Title: A generalized neural tangent kernel for surrogate gradient learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Tangent Kernel

Surrogate Gradient Descent

Binary Neural Networks

Infinite Width
Abstract: State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation.

The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide a generalization of the NTK, which we call the surrogate gradient NTK, that enables the analysis of SGL. First, we study a naive extension of the NTK to activation functions with jumps, demonstrating that gradient descent for such activation functions is also ill-posed in the infinite-width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems on the NTK with mathematical rigor. Further, we illustrate our findings with numerical experiments. Finally, we numerically compare SGL in networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.

A generalized neural tangent kernel for surrogate gradient learning

Abstract

State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation. The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide a generalization of the NTK, which we call the surrogate gradient NTK, that enables the analysis of SGL. First, we study a naive extension of the NTK to activation functions with jumps, demonstrating that gradient descent for such activation functions is also ill-posed in the infinite-width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems on the NTK with mathematical rigor. Further, we illustrate our findings with numerical experiments. Finally, we numerically compare SGL in networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.

1 Introduction

Artificial neural networks (ANNs) originate from the biologically inspired perceptron [Rosenblatt, 1958]. While the perceptron has a binary output that is faithful to the all-or-none behavior of spiking neurons in the nervous system, most activation functions considered nowadays for ANNs are smooth (like the logistic function) or at least semi-differentiable (like the ReLU function). Differentiable network functions enable the learning of network weights with methods that leverage the gradient of the network function with respect to the network weights like backpropagation [Rumelhart et al.,

1986]. These methods are very successful [LeCun et al., 2015], but cannot be used without welldefined gradients.

This is a problem when considering more biologically plausible ANNs, which are typically used in computational neuroscience to understand how the networks of spiking neurons in our nervous system work. These include spiking neural networks (SNNs). Motivated by the energy efficiency of our brain, SNNs and similar networks, such as binary neural networks (BNNs), are considered in the context of neuromorphic computing [Merolla et al., 2014]. Both discrete-time SNNs and BNNs have in common that their activation functions do not have useful derivatives, which renders standard gradient-descent training impossible [Neftci et al., 2019, Taherkhani et al., 2020, Tavanaei et al., 2019, Roy et al., 2019, Pfeiffer and Pfeil, 2018]. For the scope of this paper, these activation functions can be thought of as step-like functions like the sign function, in which cases the derivative vanishes almost everywhere and is thus no longer informative about the shape of the activation function.

Surrogate gradient learning resolves this issue by providing the missing information about the activation function in the form of a surrogate derivative [Hinton, 2012, Bengio et al., 2013, Zenke and Ganguli, 2018]. As a result, the gradient-based methods for classical ANNs can be leveraged with great success [Zenke and Vogels, 2021]. However, a theoretical basis underpinning the intuition is missing and it is often unclear which surrogate derivative should be chosen for a particular network. For a review focusing on surrogate gradient learning methods, which we are most interested in, see Neftci et al. [2019]. For a comprehensive review of other learning methods for SNNs, we refer to Taherkhani et al. [2020], Tavanaei et al. [2019], and Eshraghian et al. [2021].

The neural tangent kernel (NTK) introduced by Jacot et al. [2018] allows to formulate gradient descent as a kernel method. Just as ANNs with randomly initialized weights converge under certain conditions to Gaussian processes (GPs) in the infinite-width limit [Matthews et al., 2018, Lee et al., 2018], the NTK converges at initialization and during training to a deterministic kernel in the same limit. Moreover, the NTK then describes how the network function changes under gradient descent in the infinite-width regime. This led to both a better theoretical understanding of gradient descent and practical applications to neural network training; see Section 1.2 for more details.

1.1 Contribution

We study the NTK for networks with sign function as activation function. As the NTK theory is not directly applicable due to an ill-defined derivative, we consider the NTK for a sequence of activation functions that approximates the sign function and then derive a principled way of generalizing the NTK to gain more theoretical insight into surrogate gradient learning. Our contributions are as follows:

We provide a clear definition of the infinite-width limit in Section 2.2, capturing the different notions used in the literature due to the different choices of rates at which the layer widths increase. This definition is used consistently in all mathematical statements and the respective proofs.

In Section 2.3, we demonstrate that the direct extension of the NTK for the sign activation function using an approximating sequence is not well-defined due to the divergence of the kernel's diagonal. This illustrates, from the NTK perspective, that gradient descent is ill-defined for activation functions with jumps and how this problem will be mitigated by surrogate gradient learning. Moreover, we connect this divergence to results by Radhakrishnan et al. [2023] in Theorem 2.3 and show that the direct extension of the NTK can be seen as a well-defined kernel for classification.

We define a generalized version of the NTK in Section 2.4 using so-called quasi-Jacobian matrices and prove the convergence to a deterministic, in general asymmetric, kernel in the infinite width limit at initialization in Theorem 2.4. Using the generalized NTK, we formulate surrogate gradient learning in terms of the generalized NTK for networks with differentiable activation functions. This novel NTK is named the SG-NTK and we prove its convergence to a deterministic kernel during training in Theorem 2.5.

For both the diverging direct extension of the NTK and the SG-NTK with sign activation function and arbitrary surrogate derivative, we derive exact analytical expressions in sections D.1, D.2 and E.2. In particular, we identify the terms that emerge from SGL and that prevent divergence.

In Section 3, we illustrate our findings, in particular Theorem 2.4 and Theorem 2.5, using simulations. Numerical experiments show that the distribution of networks trained with SGL shows agreement with the distribution given by the SG-NTK.

Mathematically precise versions of all statements can be found in the appendix, which is selfcontained to ensure rigor and a consistent notation throughout all theorems and proofs.

1.2 Related work

The neural tangent kernel. The convergence of randomly initialized ANNs in the infinite-width limit to Gaussian processes under appropriate scaling of the weights has first been described by Neal [1996] for a single hidden layer and has been extended to multiple hidden layers and other network architectures like CNNs [Matthews et al., 2018, Garriga-Alonso et al., 2018, Yang, 2019b, Lee et al., 2018]. The NTK was introduced by Jacot et al. [2018] with first results on its convergence at network initialization and during training. Theoretical results on the implication of the convergence of NTKs on the behaviour of trained wide ANNs were given by Lee et al. [2019], Arora et al. [2019], Allen-Zhu et al. [2019]. In Section C.2, we review central theorems on the NTK to enable a clear comparison to our theoretical results.

The NTK has been generalized to all kinds of ANN standard architectures Yang [2020] such as CNNs Arora et al. [2019] and attention layers Hron et al. [2020]. In particular, it has been generalized to RNNs Alemohammad et al. [2020], which are particularly interesting in light of SNNs due to their shared temporal dimension. Bordelon and Pehlevan [2022] derive a generalization of the NTK called effective NTK for different learning rules using an approach similar to ours. Note that all of these extensions require well-defined gradients.

The ability of overparameterized neural networks to converge during training and to generalize can be explained using the NTK [Allen-Zhu et al., 2019, Bordelon et al., 2020, Bietti and Mairal, 2019, Du et al., 2019]. The NTK has also been used in more applied areas such as neural architecture search [Chen et al., 2021] and dataset distillation [Nguyen et al., 2021].

Surrogate gradient learning. In the context of computational neuroscience, the idea of replacing the derivative of the output of a spiking neuron with a surrogate derivative was introduced by Bohte [2011]. To deal with the temporal component in SNNs or more generally RNNs, the resulting gradient is usually combined with backpropagation through time (BPTT). Prominent examples of surrogate gradient approaches include SuperSpike by Zenke and Ganguli [2018] and a number of works with different surrogate derivatives [Wu et al., 2018, 2019, Shrestha and Orchard, 2018, Bellec et al., 2018, Esser et al., 2016, Wo´ zniak et al., 2020]. In the general ANN literature, the method is better known as straight-through estimation (STE) and was introduced by Hinton [2012] and by Bengio et al. [2013] in more detail. It was successfully applied by Hubara et al. [2016] and Cai et al. [2017].

Surrogate gradient learning or STE is only heuristically motivated and it is hence desirable to derive a theoretical basis. The influence of different surrogate derivatives on the method has been analyzed through systematic numerical simulations Zenke and Vogels [2021], revealing that the particular shape has a minor impact compared to the scale. In a more theoretical work, Yin et al. [2019] analyzed the convergence of STE for a Heaviside activation function with three different surrogate derivatives and found that the descent directions of the respective surrogate gradients reduce the population loss when the clipped ReLU function is used as surrogate derivative. Gygax and Zenke [2024] examine how SGL is connected to smoothed probabilistic models [Bengio et al., 2013, Neftci et al., 2019, Jang et al., 2019] and to stochastic automatic differentiation [Arya et al., 2022]. In particular, they consider SGL for differentiable activation functions, as we do in our derivation of the SG-NTK.

2 Theoretical results

2.1 Notation and NTK parametrization

We consider multilayer perceptrons with so-called neural tangent kernel parametrization. For a network with depth L , layer width n l for 0 ≤ l ≤ L , activation function σ , weight matrices W ( l ) ∈ R n l × n l -1 , and biases b ( l ) ∈ R n l , the preactivations with NTK parametrization are given by

where σ w > 0 , σ b ≥ 0 , and we initialize W ( l ) ij , b ( l ) i iid ∼ N (0 , 1) for all i, j . The NTK parametrization differs from the standard parametrization by a rescaling factor of 1 / √ n l in layer l +1 . The network function is then given by f ( · ) = h ( L ) ( · ): R n 0 → R n L and notably the last layer is a preactivation layer. We denote the total number of weights by P . More details can be found in Section C.1.

̸

Notation (see Remark C.1 and C.2) By default, we will interpret any vector as a column vector, i.e., we identify R n with R n × 1 . This is the case even when writing x = ( x 1 , . . . , x n ) ∈ R n for handier notation. Row vectors will be indicated within calculations using the transpose operator, x ⊺ . For a function f : R n 1 → R n 2 and X = ( x 1 , . . . , x d ) ∈ R d · n 1 , we define the vector f ( X ) := ( f ( x 1 ) , · · · , f ( x d )) ∈ R d · n 2 . For n 2 = 1 , we denote the gradient of f by ∇ f ( x ) ∈ R n 1 for all x ∈ R n 1 . If n 2 > 1 , we denote the Jacobian matrix of f by Jf : R n 1 → R n 2 × n 1 . A subscript of the form J θ f ( x ) denotes Jacobian matrices with respect to a subset of variables. We write f ( n ) ∝ ∼ g ( n ) to denote that f ( n ) and g ( n ) are asymptotically proportional, i.e., f ( n ) ∼ Kg ( n ) for some constant K = 0 .

2.2 The infinite-width limit

We will consider neural networks in the limit of infinitely many hidden layer neurons, i.e., n l →∞ for all 1 ≤ l ≤ L -1 . We will see later, when paraphrasing existing results from the literature, that different ways of taking the number of hidden neurons to infinity can be found. To formalize these notions, we use the definition of a width function from Matthews et al. [2018] with slight modifications:

Definition 2.1 (Width function) . For every layer l = 0 , . . . , L and any m ∈ N , the number of neurons in that layer is given by r l ( m ) , and we call r l : N → N the width function of layer l . We say that a width function r l is strictly increasing if r l ( m ) < r l ( m +1) for all m ≥ 1 . We set

R L := { ( r l ) L -1 l =1 | r l is a strictly increasing width function for all 0 < l < L } , the set of collections of strictly increasing width functions for network depth L .

Every element of R L provides a way to take the widths of the hidden layers to infinity by setting n l = r l ( m ) for any 1 ≤ l < L and considering m →∞ . The notions of infinite-width limits used in the literature will now correspond to classes R ⊆ R L for which the respective limiting statements hold. This is captured in the following definition.

Definition 2.2 (Types of infinite-width limits) . Consider a statement S of the form 'Let an ANN have depth L and network layer widths defined by n 0 , n L , and n l := r l ( m ) for 1 ≤ l < L and some ( r l ) L -1 l =1 ∈ R L . Then, for fixed n 0 and any n L , the statement P holds as m →∞ .' We also write the statement S as S ( r ) .

(i) We say that such a statement S holds strongly, if S ( r ) holds for any r ∈ R L . This can be interpreted as requiring that the statement holds as min 1 ≤ l<L ( n l ) →∞ . We will also write ' P holds as n 1 , . . . , n L -1 →∞ strongly'.

(ii) We say that such a statement S holds for ( n l ) 1 ≤ l ≤ L -1 ∝ ∼ n , if S holds for all r ∈ R L with r l ( m ) ∝ ∼ m for all 1 ≤ l < L . This means that S ( r ) holds for all r ∈ R L such that r p ( m ) /r q ( m ) → α p,q ∈ (0 , ∞ ) as m →∞ . We will also write ' P holds as ( n l ) 1 ≤ l<L ∝ ∼ n '.

(iii) We say that such a statement S holds weakly, if S holds for at least one r ∈ R L . We will also write ' P holds as n 1 , . . . , n L -1 →∞ weakly'. This type of infinite-width limit is tightly connected to the sequential infinite-width limit.

Remark 2.1 (Connection between weak and sequential infinite-width limits) . In the sequential infinite-width limit, meaning that n 1 →∞ , . . . , n L -1 →∞ sequentially, the layer widths are not strictly finite. This is opposed to applications, where layer widths may be large but finite. Hence, the infinite-width limits using width functions as explained above are more meaningful in practice. For a sequential limit lim n 1 →∞ lim n 2 →∞ f ( n 1 , n 2 ) = f ∗ , we can find functions n 1 ( m ) and n 2 ( m ) such that lim m →∞ f ( n 1 ( m ) , n 2 ( m )) = f ∗ . However, the rate at which n 1 ( m ) , n 2 ( m ) diverge as m →∞ cannot generally be controlled. Since the weak infinite-width limit allows for arbitrary rates, any sequential infinite-width limit can hence be turned into a weak infinite-width limit as defined in Definition 2.2 (iii).

We use Definition 2.2 to paraphrase the convergence of ANNs to GPs in the infinite-width limit:

Theorem 2.1 (Theorem 4 from Matthews et al. [2018]) . Any network function f of depth L defined as in Section 2.1 with continuous activation function σ that satisfies the linear envelope property, i.e., there exist c, m ≥ 0 with | σ ( u ) | ≤ c + m | u | for all u ∈ R , converges in distribution as n 1 , . . . , n L -1 →∞ strongly to a multidimensional Gaussian process ( X j ) n L j =1 for any fixed countable input set ( x i ) ∞ i =1 . It holds X j iid ∼ N (0 , Σ ( L ) ) where the covariance function Σ ( L ) is recursively given by

We also write Σ ( L ) = Σ σ . The theorem states that the distribution of the network function, which is given by its randomly initialized weights, approaches the distribution of independent GPs as the hidden layer widths increase. Hence, a large-width network will approximately be a realization of the respective GPs. Note that this result can be generalized to non-continuous activation functions without well-defined derivatives that fulfil the linear envelope property, like σ ( z ) = sign ( z ) . We provide a rigorous proof of this kind of generalization for the case n 1 , . . . , n L -1 →∞ weakly in Theorem E.3. Σ σ remains well-defined in this case since the expectation in Equation (1) does. When approximating the sign function with scaled error function, i.e., σ ( z ) = erf m ( z ) := erf( m · z ) , it holds that lim m →∞ Σ erf m = Σ sign due to the dominated convergence theorem.

2.3 Direct extension of the neural tangent kernel in the infinite-width limit

We consider a dataset D = ( X , Y ) with X = ( x 1 , . . . , x d ) ∈ R d · n 0 and Y = ( y 1 , . . . , y d ) ∈ R d · n L . To solve the regression problem, i.e., to find weights θ ∈ R P such that f ( x i ; θ ) = y i for all i = 1 , . . . , d , we apply gradient descent in continuous time, also know as gradient flow, with learning rate η and loss function L : R d · n L × R d · n L → R . This means that, using the chain rule, the learning rule can then be written as

To derive the NTK, we observe that the network function then evolves according to

where we implicitly defined the empirical neural tangent kernel as ˆ Θ( x, x ′ ) := J θ f ( x ; θ ) J θ f ( x ′ ; θ ) ⊺ , which depends on the current weights θ = θ t . This means that the learning dynamics of the network functions during training are given by a kernel whose entries are the scalar products between the gradients of the network's output neuron activity, ˆ Θ i j ( x, x ′ ) = ⟨∇ θ f i ( x ; θ ) , ∇ θ f j ( x ′ ; θ ) ⟩ . The key result on the NTK is that the empirical NTK converges in the infinite-width limit to a constant kernel, the analytic NTK, at initialization, θ = θ 0 , and during training, θ = θ t :

Theorem 2.2 (Theorem 1 from Jacot et al. [2018] for general σ w > 0 ) . For any network function of depth L defined as in Section 2.1 with Lipschitz continuous activation function σ , ˆ Θ =: ˆ Θ ( L ) converges in probability to a constant kernel Θ ( L ) ⊗ I n L as n 1 , . . . , n L -1 →∞ weakly. This means that for all x, x ′ ∈ R n 0 and 1 ≤ i, j ≤ n L , it holds ˆ Θ ( L ) i j ( x, x ′ ) - → δ ij Θ ( L ) ( x, x ′ ) in probability, where δ ij denotes the Kronecker delta. We call Θ ( L ) the analytic neural tangent kernel of the network, which is recursively given by

where Σ ( l ) are defined as in Theorem 2.1 and we define

We also write Θ ( L ) = Θ σ . If θ t are the weights during gradient flow learning at time t ≥ 0 as before, Theorem 2.2 shows that ˆ Θ ( L ) t → Θ ( L ) ⊗ I n L for t = 0 in the infinite-width limit. This reveals

that the gradients of the output neurons, ∇ θ f i ( x ; θ ) , are mutually orthogonal. Under additional assumptions this convergence also holds for the whole training duration, t > 0 , see Theorem 2 of Jacot et al. [2018] for the case n 1 , . . . , n L -1 →∞ weakly, and Chapter G of Lee et al. [2019] for ( n l ) 1 ≤ l<L ∝ ∼ n . Hence, the kernel that describes the learning dynamics stays constant in the infinite-width limit. This implies that the distribution of the network function during training also converges to GPs [Lee et al., 2019, Theorem 2.2]. Then, any output neuron after training has mean m ( x ) = Θ ( L ) ( x, X )Θ ( L ) ( X , X ) -1 Y under the assumption of a mean squared error (MSE) loss, see Section C.2.1. The expression for the mean is equivalent to kernel regression with the NTK.

The above results show that gradient flow for networks with randomly initialized weights is characterized by the analytic NTK Θ ( L ) in the infinite-width limit. Since we are interested in gradient flow for networks with activation functions inadequate for gradient descent training, we want to know whether the analytic NTK can be extended to such activation functions. We see that the derivative of the activation function does not have to be defined point-wise for Equation (4). In particular, activation functions with distributional derivatives, e.g., d d z sign ( z ) = 2 δ ( z ) can be taken into consideration, where δ denotes the Dirac delta distribution. If we again approximate the sign function with scaled error functions erf m , m ∈ N , it holds

̸

in case the right-hand side exists. A rigorous analysis of this limit can be found in Section D.1. Heuristically, a simple observation suffices: if x = y , we have a one-dimensional integral over two delta distributions, which yields infinity. On the other hand, if x = y and Σ sign is non-degenerate, a two-dimensional integral over two delta distributions yields a finite value. We derive analytic expressions for lim m →∞ Θ erf m =: Θ sign in Lemma D.3. We call this kernel singular, because Θ sign ( x, x ) = ∞ and Θ sign ( x, y ) ∈ R if x = y . By the same reasoning, this divergence occurs for any activation function with jumps. Note that for the mean given by kernel regression, this implies m ( x i ) = y i and m ( x ) = 0 if x is not a training point. Intuitively, this is because the network is initialized with zero mean and different input points are uncorrelated under the singular kernel, so only the training points are learned. A limit kernel with this property also arises if the activation function is fixed but the depth of the network goes to infinity as was shown by Radhakrishnan et al. [2023]. We adopt the ideas of their proof to show that the sign of m is still useful for classification:

̸

Theorem 2.3 (Inspired by Lemma 5 of Radhakrishnan et al. [2023]; see Theorem D.4) . Let σ 2 b > 0 or let all x i ∈ R n 0 be pairwise non-parallel. Let L ≥ 2 and x i ∈ S n 0 -1 R for all i = 1 , . . . , d , where S n 0 -1 R ⊆ R n 0 is the sphere of radius R . Assuming that Θ ( L ) ∞ ( x, X ) Y ̸ = 0 for almost all x ∈ S n 0 -1 R , it holds

The estimator Θ ( L ) ∞ ( x, X ) Y = ∑ n i =1 Θ ( L ) ∞ ( x, x i ) y i has the form of a so-called Nadaraya-Watson estimator and is well-defined for singular kernels such as Θ ( L ) . If we assume a classification problem in the sense that Y ⊆ {1 , 1 } n , it holds sign ( Θ ( L ) ∞ ( x i , X ) Y ) = Θ ( L ) ∞ ( x i , X ) Y at training point x i .

2.4 Generalization of the neural tangent kernel and application to surrogate gradient learning

The above singularity of the limit kernel can be avoided by considering surrogate gradient learning instead of gradient descent. First, we introduce a generalization of the NTK that later leads to the surrogate gradient NTK.

By definition and originally due to the chain rule, the empirical NTK consists of two Jacobian matrices of the network function with respect to the weights. The Jacobian matrix can be thought of as a recursive formula, J θ f ( x ; θ ) = G ( x, θ ; σ, ˙ σ ) , which is given by the input x and the architecture of the network, including the activation function σ and its derivative ˙ σ . This formula can be modified to define a quasi-Jacobian matrix, J σ 1 , ˜ σ 1 ( x ; θ ) := G ( x, θ ; σ 1 , ˜ σ 1 ) , where ˜ σ 1 does not have to be the derivative of σ 1 . Analogous to the definition of the empirical NTK we define the empirical generalized NTK to be ˆ I ( x, x ′ ) := J σ 1 , ˜ σ 1 ( x ; θ ) J σ 2 , ˜ σ 2 ( x ′ ; θ ) ⊺ , where σ 1 , ˜ σ 1 , σ 2 , ˜ σ 2 are specified in the following theorem.

Theorem 2.4 (Generalized version of Theorem 1 by Jacot et al. [2018]; see Theorem E.4) . For activation functions σ 1 , σ 2 and so-called surrogate derivatives ˜ σ 1 , ˜ σ 2 such that σ 1 , σ 2 , ˜ σ 1 , and ˜ σ 2 satisfy the linear envelope property and are continuous except for finitely many jump points, denote the empirical generalized neural tangent kernel

as before. Then, for any x, x ′ ∈ R n 0 and 1 ≤ i, j ≤ n L , it holds ˆ I ( L ) ij ( x, x ′ ) P - → δ ij I ( L ) ( x, x ′ ) , as n 1 , . . . , n L -1 →∞ weakly. We call I ( L ) the analytic generalized neural tangent kernel, which is recursively given by

̸

Σ 1 and Σ 2 denote the covariance functions of the Gaussian processes that arise from network functions f 1 , f 2 with activation functions σ 1 , σ 2 , respectively, in the infinite-width limit. The covariance between these two Gaussian processes is denoted by Σ 1 , 2 . This covariance function is asymmetric in the sense that Cov[ f 1 ( x 1 ) , f 2 ( x 2 )] = Cov[ f 1 ( x 2 ) , f 2 ( x 1 )] in general.

We show in Theorem 2.4 that the generalized empirical NTK tends to a generalized analytic NTK at initialization in an infinite-width limit. Now, the SGL rule can be written using the generalized NTK, similar to Equation (3):

Here, we chose σ 1 = σ 2 = σ , ˜ σ 1 = ˙ σ and ˜ σ 2 = ˜ σ in the previous definition of the generalized NTK. This we call the surrogate gradient NTK (SG-NTK) with activation function σ and surrogate derivative ˜ σ . Compared to the classical NTK, one of the true Jacobian matrices is replaced by the quasi-Jacobian matrix, since the learning rule is SGL instead of gradient flow. Note that we assume that the derivative of the activation function, ˙ σ , exists. Theorem 2.4 shows convergence at time t = 0 . We extend this convergence to t > 0 in the following theorem:

Theorem 2.5 (Based on Theorem G.2 from Lee et al. [2019]; see Theorem E.5) . Let σ, ˙ σ, ˜ σ as before, all Lipschitz continuous, and ˙ σ, ˜ σ bounded. Let f t be a network function with depth L initialized as in Section 2.1 and trained with MSE loss and SGL with surrogate derivative ˜ σ . Assume that the generalized NTK converges in probability to the analytic generalized NTK of Theorem 2.4, ˆ I ( L ) → I ( L ) ⊗ I n L , as ( n l ) L -1 l =1 ∝ ∼ n . Furthermore, assume that the smallest and largest eigenvalue of the symmetrization of I ( L ) ( X , X ) , S ( L ) := ( I ( L ) ( X , X ) + I ( L ) ( X , X ) ⊺ ) / 2 , are given by 0 < λ min ≤ λ max < ∞ and that the learning rate is given by η > 0 . Then, for any δ > 0 there exist R > 0 , N ∈ N and K > 1 such that for every n ≥ N , the following holds with probability at least 1 -δ over random initialization:

We also write I ( L ) = I σ, ˜ σ or simply I σ . The explicit analytic expression of the SG-NTK is derived in Section E.2 for activation function σ = erf m , m ∈ N and surrogate derivative ˜ σ = erf as well as general surrogate derivatives. I erf m , ˜ σ converges as m →∞ , because compared to Equation (5) we obtain E [ ˙ erf m ( g ( x )) ˜ σ ( g ( y ))] → E [2 δ ( g ( x )) ˜ σ ( g ( y ))] and the delta distribution yields a finite value. In Section E.2, we show this rigorously and derive the analytic expressions. Hence, we define I sign , ˜ σ := lim m →∞ I erf m , ˜ σ .

For any m ∈ N we can consider the SGL dynamics given by Equation (7) in the infinite-width limit to obtain

With MSE error, this equation is solved by a GP with mean m ( x ) = I erf m ( x, X ) I erf m ( X , X ) -1 Y for t →∞ and it is natural to assume that the networks trained with SGL converge in distribution to this GP in the infinite-width limit, analogous to the standard NTK case [Lee et al., 2019, Theorem 2.2]. However, the empirical counterpart to I sign does not exist as we cannot formulate an empirical SG-NTK for the sign activation function due to the missing Jacobian matrix, compare Equation (6). We suggest that even in this case the network trained with SGL will converge in distribution to the GP given by I sign, since SGL with activation function erf m will approach SGL with activation function sign as m →∞ and the GP given by I erf m will approach the GP given by I sign as m →∞ . Numerical experiments in Section 3 indicate that this is indeed the case. We conclude that, remarkably, the SG-NTK can be defined for network functions without well-defined gradients and is informative about their learning dynamics under SGL.

3 Numerical experiments

We numerically illustrate the divergence of the analytic NTK, Θ erf m , shown in Section 2.3 and the convergence of the SG-NTK in the infinite-width limit, ˆ I ( L ) → I ( L ) , at initialization and during training shown in Section 2.4. Simultaneously, we visualize the convergence of the analytic SG-NTK, I erf m → I sign. We consider a regression problem on the unit sphere S 1 = { x ∈ R 2 : ∥ x ∥ = 1 } with |X| = 15 training points, which is shown in Figure B.1, and train 10 fully connected feedforward networks with two hidden layers, and activation function erf m for t = 10000 time steps and with MSE loss. The NTK only depends on the dot product [Radhakrishnan et al., 2023] and thus the angle between its two arguments, ∆ α = ∢ ( x, x ′ ) . Hence, we plot the NTKs as functions of this angle, where ∆ α = 0 corresponds to x = x ′ .

In Figure 1, the empirical and analytic NTKs for the networks described above and trained with gradient descent are plotted for m ∈ { 2 , 5 , 20 } and hidden layer widths n ∈ { 10 , 100 , 500 , 1000 } . In addition, the analytic NTK for m →∞ is plotted. Note that the steep slope of erf m for m = 20 results in erf m being very close to the sign function. For any m , we observe that the empirical NTKs converge to the analytic NTK both at initialization and after training as NTK theory states. Figure B.3 illustrates this further by displaying the mean squared errors between the empirical NTKs and the respective analytic NTK. The convergence slows down for larger m . Further, the plots confirm that the analytic NTKs diverge as m →∞ if and only if ∆ α = 0 . To show this more clearly, we scaled the y-axis with the inverse hyperbolic sine (asinh), which is approximately linear for small absolute values and logarithmic for large absolute values.

10 2

10 1

10 0

10 0

10 1

10 2

m = 2

n = 10

n = 100

n = 500

n = 1000

10 2

10 1

10 0

10 0

10 1

10 2

m = 5

0

m

0

10 2

10 1

10 0

10 0

10 1

10 2

m = 20

0

0

0

Analytic NTK

Emp. NTK at t=0

Emp. NTK at t=1e4

For Figure 2, we use the same setup as before, but train the networks using SGL with surrogate derivative ˜ σ = ˙ erf , and compare the empirical and analytic SG-NTKs instead of NTKs. We observe that the empirical SG-NTKs converge to the analytic SG-NTK as n →∞ both at initialization and after training in accordance with Theorem 2.4 and Theorem 2.5. Figure B.4 illustrates this further by displaying the mean squared errors between empirical SG-NTKs and respective analytic SG-NTK.

Moreover, we observe that the analytic SG-NTKs indeed converge to a finite limit as m →∞ , as shown in Section 2.4.

5

0

5

m = 2

n = 10

n = 100

n = 500

n = 1000

5

0

5

m = 5

0

m

0

5

0

5

m = 20

0

0

0

Analytic SG-NTK

Emp. SG-NTK at t=0

Emp. SG-NTK at t=1e4

Finally, we consider SGL for networks with the same architecture and training objective as before, but with sign activation function, which can be seen as the case m →∞ of the setups above. We examine whether the distribution of network functions trained with SGL agrees with the distribution of the GP given by the limit kernel I sign. Specifically, we compare 500 networks trained with SGL for t = 30000 time steps, which represent the distribution of the network function after training, to the mean and confidence band of the GP. The mean of the GP is given by kernel regression using the SG-NTK, m ( x ) = I sign ( x, X ) I sign ( X , X ) -1 Y , and the confidence band is given by m ( x ) ± 2 σ GP ( x ) , where σ GP ( x ) is the standard deviation at x . We observe in Figure 3a that the mean of the trained networks is close to the GP's mean for network width n = 500 and that most networks lie within the confidence band. The mean of the networks differs from the kernel regression using the kernel Σ sign . Figure 3b shows that this agreement between SGL and the SG-NTK already exists for a network width of n = 100 , demonstrating that the SG-NTK predicts the SGL dynamics of networks with moderate width. Note that the variance in the networks' output and the confidence band can be reduced (see Arora et al. [2019] and Section A).

0

2

1

0

1

2

3

(a)

0

0.5

0.0

0.5

1.0

1.5

2.0

n=100

n=500

(b)

4 Conclusion

Gradient descent training is not applicable to networks with sign activation function. In the present study, we have first shown that this is even true for the infinite-width limit in the sense that the NTK diverges to a singular kernel. We found that this singular kernel still has interesting properties and allows for classification, but it is unusable for regression.

We then studied SGL, which is applicable to networks with sign activation function. We defined a generalized version of the NTK that can be applied to SGL and derived a novel SG-NTK. We proved that the convergence of the NTK in the infinite-width limit extends to the SG-NTK, both at initialization and during training. Strikingly, we were able to derive an SG-NTK for the sign activation function, I sign, by approximating the sign function with error functions. We suggest that this SG-NTK predicts the learning dynamics of SGL, and support this claim with heuristic arguments and numerical simulations.

A limitation of our work is that due to the considered NTK framework, our results are naturally only applicable to sufficiently wide networks with random initialization. Further, we only prove the convergence of the SG-NTK during training for activation functions with well-defined derivatives. More rigorous analysis should be carried out on how the connection between SGL and the SG-NTK carries over to activation functions with jumps, as shown by our simulations.

Our derivation of the SG-NTK opens a novel path towards addressing the many unanswered questions regarding the training of binary networks, in particular regarding the class of functions that SGL learns for wide networks and how that class differs for different activation functions and surrogate derivatives.

Acknowledgments and Disclosure of Funding

Wethank Andreas Eberle for helpful discussions. We thank the German Federal Ministry of Education and Research (BMBF) for support via the Bernstein Network (Bernstein Award 2014, 01GQ1710).

References

Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. arXiv preprint arXiv:2006.10246 , 2020.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems , 32, 2019.

Gaurav Arya, Moritz Schauer, Frank Schäfer, and Christopher Rackauckas. Automatic differentiation of programs with discrete randomness. Advances in Neural Information Processing Systems , 35: 10435-10447, 2022.

Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in Neural Information Processing Systems , 31, 2018.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems , 32, 2019.

Patrick Billingsley. Convergence of Probability Measures . John Wiley & Sons, 2nd edition, 1999.

Sander M Bohte. Error-backpropagation in networks of fractionally predictive spiking neurons. In International Conference on Artificial Neural Networks , pages 60-68. Springer, 2011.

Blake Bordelon and Cengiz Pehlevan. The influence of learning rule on representation dynamics in wide neural networks. In The Eleventh International Conference on Learning Representations , 2022.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

Daniele Bracale, Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Large-width functional asymptotics for deep Gaussian neural networks. arXiv preprint arXiv:2102.10307 , 2021.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave Gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5918-5926, 2017.

Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 , 2021.

Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in Neural Information Processing Systems , 22, 2009.

Giuseppe Da Prato. An Introduction to Infinite-Dimensional Analysis . Springer Science & Business Media, 2006.

Luc Devroye, Laszlo Györfi, and Adam Krzy˙ zak. The Hilbert kernel regression estimate. Journal of Multivariate Analysis , 65(2):209-227, 1998.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 16751685. PMLR, 2019.

Jason K Eshraghian, Max Ward, Emre Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2109.12894 , 2021.

Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, and Dharmendra S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences , 27:201604850, 2016.

Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow Gaussian processes. arXiv preprint arXiv:1808.05587 , 2018.

Gene H. Golub and Charles F. Van Loan. Matrix Computations . John Hopkins University Press, London, 3rd edition, 1996.

Julia Gygax and Friedemann Zenke. Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks. arXiv preprint arXiv:2404.14964 , 2024.

Insu Han, Amir Zandieh, Jaehoon Lee, Roman Novak, Lechao Xiao, and Amin Karbasi. Fast neural kernel embeddings for general activations. In Advances in Neural Information Processing Systems , 2022. URL https://github.com/google/neural-tangents .

Herbert Heyer. Structural Aspects in the Theory of Probability , volume 8. World Scientific, 2009.

Geoffrey E Hinton. Coursera video lectures: Neural networks for machine learning. https: //www.cs.toronto.edu/~hinton/coursera_lectures.html , 2012. Online; accessed 25 July 2023.

Jørgen Hoffmann-Jørgensen and Gilles Pisier. The law of large numbers and the central limit theorem in Banach spaces. The Annals of Probability , pages 587-599, 1976.

Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning , 2020. URL https://github.com/google/neural-tangents .

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in Neural Information Processing Systems , 29, 2016.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems , 31, 2018.

Hyeryung Jang, Osvaldo Simeone, Brian Gardner, and Andre Gruning. An introduction to probabilistic spiking neural networks: Probabilistic models, learning rules, and applications. IEEE Signal Processing Magazine , 36(6):64-77, 2019.

Olav Kallenberg. Foundations of Modern Probability , volume 3. Springer, 2021.

Annika Lang and Christoph Schwab. Isotropic Gaussian random fields on the sphere: regularity, fast simulation and stochastic partial differential equations. The Annals of Applied Probability , 25(6): 3047-3094, 2015. ISSN 10505164.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: Isoperimetry and Processes , volume 23. Springer Science & Business Media, 1991.

Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as Gaussian processes. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=B1EA-M-0Z .

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems , 32, 2019.

Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems , 33: 15954-15964, 2020.

Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.11271 , 2018.

Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spikingneuron integrated circuit with a scalable communication network and interface. Science , 345 (6197):668-673, 2014.

Radford M. Neal. Bayesian Learning for Neural Networks . Springer New York, NY, 1st edition, 1996.

Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine , 36(6):51-63, 2019.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021.

Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in Python. In International Conference on Learning Representations , 2020. URL https://github.com/ google/neural-tangents .

Roman Novak, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Fast finite width neural tangent kernel. In International Conference on Machine Learning , 2022. URL https://github.com/ google/neural-tangents .

Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: Opportunities and challenges. Frontiers in neuroscience , 12:774, 2018. ISSN 1662-4548. doi: 10.3389/fnins.2018.00774.

Constantine Pozrikidis. An Introduction to Grids, Graphs, and Networks . Oxford University Press, 2014.

Yu V Prokhorov. Convergence of random processes and limit theorems in probability theory. Theory of Probability & Its Applications , 1(2):157-214, 1956.

Yuming Qin. Integral and Discrete Inequalities and Their Applications. Volume I: Linear Inequalities . Springer, 2016.

Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Wide and deep neural networks achieve consistency for classification. Proceedings of the National Academy of Sciences , 120(14): e2208779120, 2023.

Frank Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review , 65(6):386, 1958.

Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature , 575(7784):607-617, Nov 2019. ISSN 1476-4687. doi: 10.1038/s41586-019-1677-2. URL https://doi.org/10.1038/s41586-019-1677-2 .

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature , 323(6088):533-536, 1986.

Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in Neural Information Processing Systems , 31, 2018.

Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz, and Jaehoon Lee. On the infinite width limit of neural networks with a standard parameterization, 2020. URL https://github.com/ google/neural-tangents .

Aboozar Taherkhani, Ammar Belatreche, Yuhua Li, Georgina Cosma, Liam P Maguire, and T Martin McGinnity. A review of learning in biologically plausible spiking neural networks. Neural Networks , 122:253-272, 2020. doi: 10.1016/j.neunet.2019.09.036.

Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural Networks , 111:47-63, 2019.

Aad W. van der Vaart. Asymptotic Statistics . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998. doi: 10.1017/CBO9780511802256.

Christopher Williams. Computing with infinite networks. Advances in Neural Information Processing Systems , 9, 1996.

David Williams. Probability with Martingales . Cambridge University Press, 1991.

Stanisław Wo´ zniak, Angeliki Pantazi, Thomas Bohnstingl, and Evangelos Eleftheriou. Deep learning incorporating biologically inspired neural dynamics and in-memory computing. Nature Machine Intelligence , 2(6):325-336, 2020.

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience , 12:331, 2018.

Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence , 33 (01):1311-1318, 2019.

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760 , 2019a.

Greg Yang. Wide feedforward or recurrent neural networks of any architecture are Gaussian processes. Advances in Neural Information Processing Systems , 32, 2019b.

Greg Yang. Tensor programs II: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548 , 2020.

Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662 , 2019.

A. Zalesski˘ ı, V . V . Sazonov, and V . V . Ulyanov. A precise estimate of the rate of convergence in the central limit theorem in Hilbert space. Sbornik: Mathematics , 68(2):453-482, 1991.

Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation , 30(6):1514-1541, 2018.

Friedemann Zenke and Tim P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation , 33(4):899-925, 2021.

A Additional remarks on the numerical experiments

Weight initialization and implementation. All networks are initialized with σ w = 1 , σ b = 0 . 1 For the implementation of the NTK and SG-NTK we use the JAX package [Bradbury et al., 2018] and Neural Tangents package [Novak et al., 2020, 2022, Han et al., 2022, Sohl-Dickstein et al., 2020, Hron et al., 2020] with modifications. Computations were done using an Intel Core i7-1355U CPU and 16 GB RAM. The simulations for Figure 1 and Figure 2 took two hours each. The simulations for Figure 3 took 12 hours.

Variance reduction trick. The variance in the outputs of the networks trained with gradient descent or SGL and the confidence band given the NTK or SG-NTK respectively can be reduced by multiplying the weights of the last layer with a constant κ < 1 at initialization [Arora et al., 2019]. This is explained in detail at the end of Section C.2.1. We can see from Figure B.2 that the agreement between the distribution of the trained networks and the distribution given by the SG-NTK still holds; however, network width and training time have to be increased.

B Additional figures

0

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

0

2

1

0

1

2

3

10 1

10 2

10 3

10

2

10 0

10 2

10 4

m = 2

10 1

10 2

10 3

m = 5

10 1

10 2

10 3

m = 20

hidden layer widths

n

MSE at t=0

MSE at t=1e4

10 1

10 2

10 3

10

3

10

2

10

1

10 0

10 1

m = 2

10 1

10 2

10 3

m = 5

10 1

10 2

10 3

m = 20

hidden layer widths

n

MSE at t=0

MSE at t=1e4

C Neural tangent kernel theory

C.1 Introduction of the neural tangent kernel

We begin by defining an artificial neural network with a parameterization suitable for considering the limit of infinitely many hidden neurons. This parameterization is called neural tangent kernel parameterization and differs from standard parameterization of multilayer perceptrons by a rescaling factor of 1 / √ n l in layer l +1 , where n l is the width of layer l . We follow the slightly more general definition in [Lee et al., 2019, Equation (1)]. A discussion of this kind of parameterization can be found in [Jacot et al., 2018, Remark 1].

Definition C.1 (Artificial neural network with NTK parameterization) . Let L be the depth of the network, n k for k = 0 , . . . , L the widths of the layers, and σ : R → R an activation function. We draw network weight matrices W ( l ) ∈ R n l × n l -1 and biases b ( l ) ∈ R n l for l = 1 , . . . , L from a probability distribution such that W ( l ) ij , b ( l ) i iid ∼ N (0 , 1) . For the parameters, we denote by θ ( l ) = ( W ( l ) , b ( l ) ) the parameters of layer l and by θ (1: l ) = ( θ (1) , θ (2) , . . . , θ ( l ) ) the parameters of layers 1 up to and including l . Given some σ w > 0 and σ b ≥ 0 we then define for all x ∈ R n 0

Therefore, h ( l ) ( · ; θ (1: l ) ) is a map from R n 0 to R n l and we use the short-hand notation h ( l ) ( x ; θ ) = h ( l ) ( x ; θ (1: l ) ) . Finally, we define our network function

With this definition, the total number of parameters, P = | θ | , is

Given such a network function with NTK parameterization, we consider a dataset D = ( X , Y ) with X = ( x 1 , . . . , x d ) ∈ R d · n 0 and Y = ( y 1 , . . . , y d ) ∈ R d · n L . First, we want to solve the regression problem, i.e., find parameters θ ′ such that f ( x i ; θ ′ ) = y i for all i = 1 , . . . , d . Later, we will also consider the classification problem, i.e., we assume y i ∈ {-1 , 1 } and want to solve sign ( f ( x i ; θ ′ )) = y i for all i = 1 , . . . , d . Tackling both cases from the regression perspective, we define the so-called loss functional and loss function. The following two definitions are similarly formulated by Jacot et al. [2018].

Definition C.2 (Loss functional and loss function) . Let F = { g | g : R n 0 → R n L } and L : F → R a so-called loss functional. In addition, let L be convex, i.e., for all λ ∈ [0 , 1] and g 1 , g 2 ∈ F it holds

We will assume that L can be written as

so that the so-called loss function L ( · ; Y ): R d · n L → R is differentiable.

Remark C.1 (Function evaluation at sets and vector notation) . We want to detail the notation used in Definition C.2. For a function f : R n 0 → R n L and X = ( x 1 , . . . , x d ) ∈ R d · n 0 we define the vector

By default, we will interpret any vector as a column vector, i.e., we identify R n with R n × 1 . This is the case even when writing x = ( x 1 , . . . , x n ) ∈ R n for handier notation. Row vectors will be indicated within calculations using the transpose operator, x ⊺ .

Let us first consider regular gradient descent learning in continuous time, also known as gradient flow. For this, we assume that our network function is differentiable with respect to its parameters.

Remark C.2 (Gradient and Jacobian matrix notation) . Let f : R n → R m . For m = 1 , we denote the gradient by ∇ f ( x ) ∈ R n for all x ∈ R n . If m > 1 we denote the Jacobian matrix of f by Jf : R n → R m × n . Therefore, Jf ( x ) is a m × n matrix for all x ∈ R n . We do not always want to consider the gradient or Jacobian matrix with respect to all variables. We indicate this with subscripts as follows. Let f : R n × R P → R m and g x ( θ ) = f ( x ; θ ) for fixed x ∈ R n . Then, we write ∇ θ f ( x ; θ ) := ∇ g x ( θ ) and J θ f ( x ; θ ) := Jg x ( θ ) . In particular, for a map f : R P → R , the gradient with respect to the j -th variable, 1 ≤ j ≤ P , is a scalar and denoted by ∂ j f ( θ ) := ∇ θ j f ( θ ) . This is called the partial derivative of f with respect to the j -th variable.

With this notation, we consider the gradient flow method with learning rate η > 0 and recall the derivation of the neural tangent kernel. We move the weights in the opposite direction of the gradient of the loss function with respect to the parameters of the network evaluated at the training points:

using the chain rule for the second equality and with A ⊺ denoting the transpose of a matrix A . Again using the chain rule, this then implies for any x ∈ R n 0

We therefore define the neural tangent kernel as follows:

Definition C.3 (Neural tangent kernel) . Let f be a network function of depth L as in Definition C.1 with parameters θ , not necessarily drawn randomly. Then, we define the neural tangent kernel (NTK) as:

Therefore, it holds for all x, y ∈ R n 0 and 1 ≤ i, j ≤ n L

Notice that the NTK depends on the parameters of the networks. It is therefore initialized randomly and varies over the course of the training. With notation f t ( x ) = f ( x ; θ t ) and ˆ Θ ( L ) t = ˆ Θ ( L ) for parameters θ t at training time t we can now rewrite Equations (S9) and (S10) as follows:

We are hence able to express the change of the network function during training in a kernel fashion. Later, we will consider this change of the network function in the infinite-width limit, i.e., n 1 , . . . , n L -1 →∞ .

Before doing so, we will generalize the NTK definition in order to apply the NTK to surrogate gradient learning later. In particular, we will break the symmetry of the above definition and generalize the Jacobian matrices to quasi-Jacobian matrices by replacing the derivatives of the activation function by surrogate derivatives. Let us write the recursive formula of the Jacobian matrix of the network function given by the chain rule as J θ f ( x ; θ ) = G ( σ ; ˙ σ ; x ; θ ) , where ˙ σ is the derivative of the activation function σ . Then, surrogate gradient learning replaces J θ f ( x ; θ ) with G ( σ ; ˜ σ ; x ; θ ) for the surrogate derivative ˜ σ of the activation function σ . We call this the quasi-Jacobian matrix:

Definition C.4 (Quasi-Jacobian matrices for neural networks) . Let L be the depth of the network, n k for k = 0 , . . . , L the width of the layers, σ : R → R the activation function, and ˜ σ : R → R the so-called surrogate derivative of the activation function. Let f be the network function, h ( l ) , l = 1 , . . . , L -1 , the intermediate layers as in Definition C.1 and θ the network parameters. We then define the quasi-Jacobian matrix J ( L ) of f at point x recursively as follows:

Remark C.3 (Notations for the quasi-Jacobian) . With the above definition of the quasi-Jacobian matrix of the network function f with activation function σ and surrogate derivative ˜ σ we write

For a data set X instead of a single point x , we concatenate the matrices row-wise as before, namely J ( L ) ( X ; θ ) ∈ R d · n L ×| θ | .

Definition C.5 (The generalized neural tangent kernel) . Let σ 1 , σ 2 be activation functions and ˜ σ 1 , ˜ σ 2 the surrogate derivatives respectively. Given a network depth L and parameters θ we define the generalized neural tangent kernel as:

Remark C.4. The generalized neural tangent kernel agrees with the neural tangent kernel in the case where σ = σ 1 = σ 2 and ˙ σ = ˜ σ 1 = ˜ σ 2 .

C.2 Notation for the infinite-width limit and review of key theorems for the NTK

In this section we will formulate all important theorems on the NTK that we will need for our later analysis using the introduced notation. Furthermore, we will discuss and remark their proofs, in particular in view of the generalizations that will be proved in Section E.1.

Convergence of networks to Gaussian processes in the infinite-width limit. We will consider neural networks in the limit of infinitely many hidden layer neurons. The fact that such networks converge to Gaussian processes was first mentioned by Neal [1996]. We follow and present the formulations of Matthews et al. [2018] for the general mathematical statement. First, we formalize the limit of infinitely many hidden neurons.

Definition C.6 (Width function, as in Definition 3 of Matthews et al. [2018] with modifications) . For every layer l = 0 , . . . , L and any m ∈ N , the number of neurons at that layer is given by r l ( m ) , and we call r l : N → N the width function of layer l . We say that a width function r l is strictly increasing if r l ( m ) < r l ( m +1) for all m ≥ 1 . We set

the set of collections of strictly increasing width functions for network depth L .

Every element of R L provides a way to take the widths of the hidden layers to infinity by setting n 0 and n L to some constant, setting n l = r l ( m ) for any 1 ≤ l < L and considering m →∞ . To formally define for which ways of taking the widths of hidden layers to infinity a statement holds, we can now state the set R ⊆ R L such that the statement holds for widths given by any r ∈ R as m →∞ . Clearly, the claim 'The statement holds for all r ∈ R 1 .' is stronger than 'The statement holds for all r ∈ R 2 .' if R 2 ⊆ R 1 . On the basis of these considerations, we define three types of infinite-width limits using the previous definition in order to structure the different types of limits in this thesis as well as in the literature.

Definition C.7 (Types of infinite-width limits) . Consider a statement S of the form 'Let an ANN have depth L and network layer widths defined by n 0 , n L and n l := r l ( m ) for 1 ≤ l < L and some ( r l ) L -1 l =1 ∈ R L . Then, for fixed n 0 and any n L , the statement P holds as m →∞ .' We also write the statement S as S ( r ) .

(i) We say that such a statement S holds strongly, if S ( r ) holds for any r ∈ R L . This can be interpreted as requiring that the statement holds as min 1 ≤ l<L ( n l ) →∞ . We will also write ' P holds as n 1 , . . . , n L -1 →∞ strongly'.

(ii) We say that such a statement S holds for ( n l ) 1 ≤ l ≤ L -1 ∝ ∼ n , if S holds for all r ∈ R L with r l ( m ) ∝ ∼ m for all 1 ≤ l < L . In other words S ( r ) holds for all r ∈ R L such that r p ( m ) /r q ( m ) → α p,q ∈ (0 , ∞ ) as m →∞ . We will also write ' P holds as ( n l ) 1 ≤ l<L ∝ ∼ n '.

It then holds

(iii) We say that such a statement S holds weakly, if S holds for at least one r ∈ R L . This can be read as requiring that the statement holds as n 1 →∞ , . . . , n L -1 →∞ sequentially. We will also write ' P holds as n 1 , . . . , n L -1 →∞ weakly'.

Theorem C.1 (Theorem 4 from Matthews et al. [2018]) . Any network function f of depth L defined as in Definition C.1 with continuous activation function σ that satisfies the linear envelope property, i.e., there exist c, m ≥ 0 with

converges in distribution as n 1 , . . . , n L -1 →∞ strongly to a multidimensional Gaussian process ( X j ) n L j =1 for any fixed countable input set ( x i ) ∞ i =1 . It holds X j iid ∼ N (0 , Σ ( L ) ) where the covariance function Σ ( L ) is recursively given by

Remark C.5. First, a proof of the above theorem can be found in the paper of Matthews et al. [2018]. While it takes a lot of effort to show that the statement holds strongly in the sense of Definition C.7, the weak version of the statement can be proved via induction. This has been done by Jacot et al. [2018] and we will later adapt their proof to show a generalized version, Theorem E.3.

Second, in the context of analyzing the network behavior, we are interested in the finite-dimensional distributions first of all, since neural networks are trained and tested on a finite number of data points. From the convergence of the marginal distributions, we can infer the convergence to an stochastic process via the Kolmogorov extension theorem. However, this assumes the product σ -algebra, which is why Theorem C.1 assumes a fixed countable input set. Matthews et al. [2018] have discussed these formal restrictions in more detail (Chapter 2.2). If one does not want to be restricted to a countable index set, one could, for example, consider the condition by Prokhorov [1956, Theorem 2.1]. A similar approach was taken by Bracale et al. [2021], which applied the Kolmogorov-Chentsov criterion [Kallenberg, 2021, Theorem 4.23].

Finally, note that the theorem assumes continuity of the activation function. In the proof of Matthews et al. [2018] this is only used in order to apply the continuous mapping theorem. However, it is sufficient for the limiting process to attain possible points of discontinuity with probability zero for the continuous mapping theorem to be applicable. The theorem is thus also valid for activation functions that are continuous except at finitely many jump points, such as step-like activation functions.

Convergence of the NTK at initialization in the infinite-width limit. Jacot et al. [2018] showed that the previously defined empirical NTK converges to a deterministic limit, which we will call the analytic NTK.

Theorem C.2 (Theorem 1 from Jacot et al. [2018], slightly generalized) . For any network function of depth L defined as in Definition C.1 with Lipschitz continuous activation function σ , the empirical neural tangent kernel ˆ Θ ( L ) converges in probability to a constant kernel Θ ( L ) ⊗ I n L as n 1 , . . . , n L -1 →∞ weakly. For all x, x ′ ∈ R n 0 and 1 ≤ i, j ≤ n L , it holds

We call Θ ( L ) the analytic neural tangent kernel of the network, which is recursively given by

where Σ ( l ) are defined as in Theorem C.1 and we define

Compared to Theorem 1 of Jacot et al. [2018], the statement is slightly generalized in the sense that it allows for arbitrary σ w > 0 . The arguments in the proof work the same way.

which we also write as

Remark C.6 (Versions of Theorem C.2 in the literature) . A proof of this theorem for ( n l ) 1 ≤ l<L ∝ ∼ n is given by Yang [2019a] and his proof is also referenced by Lee et al. [2019]. However, the proof is given in terms of so-called tensor programs and therefore harder to follow. For the ReLU activation function, a proof for n 1 , . . . , n L -1 →∞ strongly is provided by Arora et al. [2019, Theorem 3.1]. We will later prove a version of this theorem for the generalized NTK.

Convergence of the NTK during training in the infinite-width limit. Not only does the NTK converge to a constant kernel in the infinite-width limit, even the kernel during training, ˆ Θ ( L ) t , converges to this constant kernel. This was also discovered by Jacot et al. [2018].

Theorem C.3 (Theorem 2 by Jacot et al. [2018]) . Assume any network function of depth L defined as in Definition C.1 with Lipschitz continuous activation function σ , twice differentiable with bounded second derivative, and trained with gradient flow as in Equation S10. Let T > 0 such that

where X ∈ O p (1) denotes that X is stochastically bounded. Then, as n 1 , . . . , n L -1 →∞ weakly, the empirical NTK ˆ Θ ( L ) t converges in probability to the analytic NTK Θ ( L ) ⊗ I n L in probability uniformly for t ∈ [0 , T ] . We therefore write

Remark C.7 (Versions of Theorem C.3 in the literature) . The proof of Jacot et al. [2018] relies heavily on a function space perspective. Since this formulation tends to lack mathematical rigor, we will rely on the proof of the theorem for the case ( n l ) 1 ≤ l<L ∝ ∼ n given by Lee et al. [2019, Chapter G]. In particular, the first inequality of (S51) in Theorem G.2 of Lee et al. [2019] implies the condition ( * ). Furthermore, a different approach to proving the above statement for the case ( n l ) 1 ≤ l<L ∝ ∼ n using the Hessian matrix of the network function was taken by Liu et al. [2020, Proposition 2.3, Theorem 3.2]. A partial proof of a version of this theorem for the generalized NTK will be given later. Only an auxiliary lemma remains to be proved.

C.2.1 Gradient flow in the infinite-width limit

Given the results of the previous section, we can formulate an infinite width version of Equation (S10) by replacing the empirical with the analytic NTK. This allows us to analyze the learning dynamics of networks in the infinite-width limit, which yields connections to kernel methods and reproducing kernel Hilbert spaces. We then discuss how far the resulting functions and solutions in the infinite-width limit deviate from the finite width networks. This is essential to evaluate to what extend the results in the infinite-width limit can inform us about the behavior of gradient flow in the finite-width networks. First, we state the infinite-width version of Equation (S11) using Theorem C.3,

where in the last line we interpret ∇ f t ( X ) L ( f t ( X ); Y ) ∈ R d × n L as a matrix of size d × n L with entries [ ∇L f t ( X ) ( f t ( X ); Y ) ] i j = 1 /d · ∂ j ℓ ( f t ( x i ); y i ) . Recall that ∂ j ℓ ( f t ( x i ); y i ) is the partial derivative of ℓ ( · ; y i ) with respect to its j -th entry, i.e., with respect to f t,j ( x i ) . Note that the last line is a row vector, which we can identify as a column vector. The fact that the NTK is now time-independent and non-random has two interesting implications:

Equation (S16) is now an differential equation that can be solved explicitly or numerically for certain loss functions.

According to Equation (S15), the time derivative of f t ( x ) can now be expressed elementwise as a linear combination of functions of the type Θ ( L ) ( · , ˜ x ): R n 0 → R . For an arbitrary symmetric and positive definite kernel k ( · , · ) , the completion of the linear span of functions of this type is called the reproducing kernel Hilbert space (RKHS) of k . Assuming that the solution of Equation (S15) is an element of the RKHS of Θ ( L ) , one can ask what the space looks like.

The ODE of Equation (S16) has already been considered by Jacot et al. [2018, Chapter 5] and Lee et al. [2019, Chapter 2.2], and we will follow the observations made there. To do this, we will assume the mean squared error (MSE) loss,

implying ∇ f t ( X ) L ( f t ( X ); Y ) = f t ( X ) -Y , where f t ( X ) and Y are again interpreted as matrices of dimension d × n L . This gives us the following ODE

Now, for simplicity, we denote Θ( x, y ) := Θ ( L ) ( x, y ) and Θ := Θ( X , X ) . Furthermore, we consider an arbitrary set of test points X T . The solution of the ODE is then given by

Recall that by Theorem C.1, the components f 0 ,j are independent and identically distributed Gaussian processes with mean zero and covariance function Σ := Σ ( L ) . Hence, γ t has mean zero and the mean of f t is given by µ t . By looking at the components of f t ,

we can conclude that they are independent and identically distributed as well. One can show that the components are indeed Gaussian processes again with mean µ t . Using γ t we can also compute the covariance matrix for our arbitrary set of test points X T ,

Assuming that Θ is positive definite immediately leads to pointwise convergence of the mean and covariance functions. This implies that the gradient flow solution for networks in the infinite-width limit converges to a Gaussian process as t →∞ with mean function µ ∞ and covariance function Γ ∞ given below. This follows from the weak convergence of the finite-dimensional marginal distributions by Lévy's convergence theorem [Williams, 1991, Section 18.1]. Again, the discussions of Remark C.5 applies. We have

Lee et al. [2019] state that a network trained with gradient flow will indeed converge in distribution to this Gaussian process as the width goes to infinity:

Theorem C.4 (Theorem 2.2 from Lee et al. [2019]) . Let the learning rate η < η critical for

with a network function f t as in Theorem C.3 with hidden layer widths n 1 = · · · = n L -1 = n and restricted to x ∈ R n 0 with ∥ x ∥ 2 ≤ 1 . If λ min (Θ ( L ) ( X , X )) > 0 , then the components of f t converge in distribution to independent, identically distributed Gaussian processes N ( µ t , Γ t ) as n →∞ for all t ∈ [0 , ∞ ) ∪ {∞} .

Hence, the result of training a finite-width network with gradient flow for an infinite amount of time will be arbitrarily close in distribution to a Gaussian process with mean function µ ∞ and covariance function Γ ∞ , if the width is sufficiently large. Note that by Equations (S17) and (S18) the variance at the training points X is zero and the mean at the training points is exactly Y .

Since we will focus on the mean, we will first sketch a trick introduced in Chapter 3 of Arora et al. [2019] to make the variance term arbitrarily small. If f 0 had mean variance, this would consequently also be the case for all f t and for the solution f ∞ := lim t →∞ f t . This can be achieved by multiplying f 0 by a small constant κ > 0 and considering the network function g 0 = κf 0 instead. It then holds

and thus we have Θ g = κ 2 Θ f . In the infinite-width limit, the derivative of g t is then given by

which implies as before

Note that the term in the second last line corresponds to the non-random mean of f trained with learning rate ηκ 2 , and that the term in the last line is random, but can be made arbitrarily small using κ . We can think of this as a trade-off between learning rate and variance. This justifies why we can focus on the mean in the next section.

To sum up, we are interested in network functions in the infinite-width limit that are trained over time according to

where we change from a row vector to a column vector in the last equation. The mean of such network functions after infinite training time is given by

D The NTK for sign activation function

The first observation to make in our attempt to apply the neural tangent kernel to networks with the sign function as activation function is that the sign function has a zero derivative almost everywhere. Thus, the derivative of the network function with respect to the network weights is zero for all weights that are not part of the last layer. The case where the weights θ (1: L -1) are frozen after initialization and only θ ( L ) is trained has already been discussed by Lee et al. [2019, Chapter 2.3.1 and Chapter D]. For a network in the infinite-width limit, this approach is equivalent to applying Gaussian process regression , i.e., knowing that f ∼ N ( 0 , Σ ( L ) ) for infinite width, one considers f | f ( X ) = Y . This can be seen by realizing that Θ ( L ) = Σ ( L ) if ˙ σ = 0 almost everywhere and applying Theorem C.4.

While this is an interesting observation, and the strategy of optimizing only the last layer can also be transferred to finite width networks, we would prefer to train the whole network and not identify the derivative of the sign function with zero, since this discards all information about the jump discontinuities in our networks. An obvious alternative would be to use the distributional derivative of the sign function, which is given by 2 δ 0 , where δ 0 denotes the delta distribution. We will see that ˙ Σ ( L ) still exists when the distributional derivative is substituted into its formula. Alternatively, we can obtain the same expression by approximating the sign function with scaled error functions,

and considering the limit m →∞ .

D.1 The NTK for error activation function

Due to the previous considerations, we begin by deriving the analytic NTK for the error function. Following the notation of Lee et al. [2019], we need to find analytic expressions for the terms

Note that by a change of variables we can alternatively consider the terms

For Σ ′ = ( x · x x · y x · y y · y ) , T (Σ ′ ) and ˙ T (Σ ′ ) are given in Chapter C of the supplementary material of Lee et al. [2019]. However, we cannot assume that Σ ′ always has this form. While ˙ T can be easily calculated, T is harder to deal with and a reference to Williams [1996, Chapter 3.1] is used. There, the main idea of the proof, how to evaluate a more general expression, is given without further details. We will derive analytic expressions for both terms explicitly.

We start by evaluating ˙ T . Note that

Lemma D.1. Given U ∼ N (0 , Σ) with invertible covariance matrix Σ ∈ R d × d and x, y ∈ R d , it holds

In particular, given ( X,Y ) ∼ N (0 , Σ) with invertible covariance matrix Σ ∈ R 2 × 2 or with X = Y and singular covariance matrix Σ ∈ R 2 × 2 , it holds

where | A | denotes the determinant of a matrix A.

Proof. It holds for U ∼ N (0 , Σ) with covariance matrix Σ ∈ R d × d and x, y ∈ R d :

using a change of variable Σ 1 / 2 u = v for Equation ( ⋆ ) and using basic properties of the determinant. We can evaluate the determinant in the last line by applying the Sylvester's determinant theorem [Pozrikidis, 2014, (B.1.16)], i.e., | I n + AB ⊺ | = | I m + B ⊺ A | for any matrices A,B ∈ R n × m . We then define A = B = ( √ 2Σ 1 / 2 x, √ 2Σ 1 / 2 y ) ∈ R d × 2 , which yields

This directly implies Equation (S21). Furthermore, Equation (S22) follows with Σ ∈ R 2 × 2 and x = ( 1 0 ) , y = ( 0 1 ) or with x = y = ( 1 0 ) and an arbitrary invertible covariance matrix Σ ′ such that Σ ′ 11 = Σ 11 .

Corollary D.1. Given ( X,Y ) ∼ N (0 , Σ) with invertible covariance matrix Σ , it holds

If ( X,Y ) ∼ N (0 , Σ) with X = Y and singular covariance matrix Σ ∈ R 2 × 2 , it holds

Proof.

Remark D.1. As mentioned at the beginning of this chapter, we can get the same result by considering the distributional derivative of the sign function, 2 δ 0 . It holds for ( X,Y ) ∼ N (0 , Σ) with invertible covariance matrix Σ as before

In the case of X = Y , the integral is no longer well-defined.

Next, we consider T by solving a more general problem, which was formulated in slightly less general form by Williams [1996, Chapter 3.1].

Lemma D.2. Given U ∼ N (0 , Σ) with invertible covariance matrix Σ ∈ R d × d and x, y ∈ R d , it holds

In particular, given ( X,Y ) ∼ N (0 , Σ) with invertible covariance matrix Σ = ( Σ 1 Σ 3 Σ 3 Σ 2 ) ∈ R 2 × 2 or with X = Y and singular covariance matrix Σ = ( Σ 1 Σ 3 Σ 3 Σ 2 ) ∈ R 2 × 2 , Σ 1 = Σ 2 = Σ 3 , it holds

Proof. We follow the proof idea given by Williams [1996, Chapter 3.1], that is, we define V ( λ ) , differentiate the expectation, and integrate by parts. We can then see that d d λ V ( λ ) = (1 -γ 2 ) -1 / 2 d γ d λ , which gives the desired arcsin . So, we define

and then differentiate with respect to λ on both sides

using a change of variables u = Σ 1 / 2 v in Equation ( ⋆ ) and using partial integration and Gauss' divergence theorem in the last equation. In addition, we used that

and the partial integration rule for scalar functions. To be precise, for differentiable scalar functions f and g it holds

which then implies the partial integration rule. Furthermore, the left-hand side vanishes in our case due to Gauss' divergence theorem:

where S d -1 ( R ) is the surface area of the sphere in R d with radius R . Continuing with our previous calculations, we see that

We evaluate the expression outside of the integral in the last line by applying the Sherman-MorrisonWoodbury formula. For A ∈ R d × d and w 1 , w 2 ∈ R d it holds [Golub and Van Loan, 1996, (2.4.1)]

For C = I d and w = w 1 = w 2 = √ 2 λ Σ 1 / 2 x this yields

With this we see that

Inserting this into Equation (S25), we obtain

As in the proof of Lemma D.1, we evaluate this using Sylvester's determinant theorem. With the same notation as before, we can define A = B = ( √ 2 λ Σ 1 / 2 x, √ 2Σ 1 / 2 y ) ∈ R d × 2 . This yields

If we insert this this again, we have so far shown

We now define and claim that

We can find a solution to the claimed equation by finding a function ˜ V that satisfies

and by setting V ( λ ) := ˜ V ( γ ( λ )) . This follows from the chain rule. ˜ V is thus simply given by ˜ V ( γ ( λ )) = 2 π arcsin( γ ( λ )) . In particular, this yields

It is now left to show Equation (S27) using Equation (S26). First, see that

Second, it holds

With the results of both calculations, we get

which yields Equation (S27) and concludes the proof. The special case Σ ∈ R 2 × 2 with invertible covariance matrix or X = Y and singular covariance matrix follows as in the proof of Lemma D.1.

Corollary D.2. If ( X,Y ) ∼ N (0 , Σ) with invertible covariance matrix Σ = ( Σ 1 Σ 3 Σ 3 Σ 2 ) ∈ R 2 × 2 or with X = Y and singular covariance matrix Σ = ( Σ 1 Σ 3 Σ 3 Σ 2 ) ∈ R 2 × 2 , Σ 1 = Σ 2 = Σ 3 , it holds

Proof. It holds

We can now use Corollary D.1 and Corollary D.2 to evaluate Σ ( L ) , ˙ Σ ( L ) and Θ ( L ) for activation function erf m , which we denote by Σ ( L ) m , ˙ Σ ( L ) m , and Θ ( L ) m respectively. We then are interested in the limit m →∞ . First recall that

by Theorem C.2. Therefore, Σ (1) m is independent of m , and it holds for any x, y ∈ R n 0

̸

Assuming that the limit lim m →∞ Σ ( L ) m ( x, y ) =: Σ ( L ) ∞ ( x, y ) exists and that Σ ( L ) ∞ ( x, x ) = 0 , Σ ( L ) ∞ ( y, y ) = 0 , we can define

Hence, it follows via induction and from the continuity of the arcsin function that lim m →∞ Σ ( L ) m ( x, y ) =: Σ ( L ) ∞ ( x, y ) is well defined. We discuss the resulting kernel Σ ( L ) ∞ in Remark D.3. For ˙ Σ ( L +1) m , L ≥ 1 , it holds

As we will see later, the limit

̸

exists for x = y apart from a few exceptions. In the case x = y , we obtain

̸

where ∼ denotes asymptotic equality. Therefore, for x = y , the limit lim

m

→∞

Θ

(

L

m

(

x, y

) =: Θ

(

L

∞

(

x, y

)

exists. However, due to Equation (S32), the NTK diverges for x = y as m → ∞ . We will call a kernel with this property a singular kernel. We also say that a kernel with this property is singular along the diagonal .

Remark D.2 (Distributional neural tangent kernel) . An alternative conceivable approach to obtain the same kernel Θ ( L ) ∞ would have been to consider the distributional Jacobian matrix of the network function with step-like activation function. Whether or not a distributional NTK can be formulated is a question for further research. Here, we only want to point out that the corresponding formulas for the recursive definition of the analytical distributional NTK would then naturally read as follows,

where

Note that Equation (S34) can be derived from Remark D.1, and that Equation (S33) is also easy to show.

)

)

̸

Wenow want to explore the implications of our findings in this section for f NTK , which we introduced at the end of Section C.2. Equation (S20) yields

̸

Note that Θ m ( X , X ) := Θ ( L ) m ( X , X ) is invertible for sufficiently large m , as the matrix will be dominated by the diagonal for large m . Using this and the fact Θ m ( x, X ) m →∞ - - - - → Θ ∞ ( x, X ) if x = x i for all i = 1 , . . . , d , we obtain for such x :

However, if x = x i ∈ X and denoting the i -th basic vector by e i ∈ R d we get:

Therefore, f m NTK converges pointwise to a function that is zero almost everywhere, but interpolates exactly at the data points. The singular kernel Θ ( L ) ∞ , which we can see as the analytic NTK for the sign function, is for that reason not suitable for regression. We will deal with this problem in the following section.

D.2 From singular kernel to Nadaraya-Watson estimator

Radhakrishnan et al. [2023] considered the limit of infinite depth, lim L →∞ Θ ( L ) , and a singular kernel emerged similar to the previous section. It was shown that the resulting estimator using this singular kernel behaves like a Nadaraya-Watson estimator when classification tasks are considered instead of regression tasks. We will adapt the ideas of Radhakrishnan et al. [2023] to show similar results. To consider a classification task, we assume that Y ∈ {1 , 1 } d . The network is then trained with data points ( X , Y ) as before, but we apply the sign function at the end to obtain the final classifier. As Radhakrishnan et al. [2023], we assume that we are operating in the infinite-width limit after infinite training. So, we are interested in

We will see that the resulting classifier is equal to

The form of this classifier is similar to a Nadaraya-Watson estimator. To be precise, for a singular kernel k and data points ( X , Y ) , the Nadaraya-Watson estimator is defined as

̸

Since k ( x, x i ) →∞ as x → x i , it holds that c ( x ) → y i as x → x i . Thus, the continuous extension of c to R n 0 interpolates the data points.

We now begin to further evaluate Σ ( L ) ∞ and ˙ Σ ( L ) ∞ to analyze Equation (S35):

̸

Regarding the assumptions made for Equation (S29), note that that Σ ( L ) ∞ ( x, x ) = 0 if L ≥ 2 and Σ (1) ∞ ( x, x ) = 0 if x = 0 or σ 2 b > 0 .

̸

̸

After Equation (S31), we mentioned exceptions to the existence of ˙ Σ ( L ) ∞ ( x, y ) for x = y that were postponed at that time. For ˙ Σ (2) m with x = y we can see that

̸

assuming that either σ 2 b > 0 or that x, y are not parallel, i.e., |⟨ x, y ⟩| ̸ = ∥ x ∥∥ y ∥ . The term 1 2 m 2 A m is given by

For L ≥ 3 the expression simplifies:

with the term 1 2 m 2 B m given by

From Equation (S39) and Equation (S41) we see that

Although we did not use dual activation functions to denote the NTK as [Jacot et al., 2018, Section A.4], we still find that the property (ˆ σ ) ′ = ̂ ( σ ′ ) applies, i.e., the derivative of the dual activation function is the dual of the derivative of the activation function. Here ˆ σ denotes the dual activation function of σ .

In the case x = y we get

We summarize the above calculations in the following lemma:

̸

̸

Lemma D.3. For m,L ∈ N let Σ ( L ) m and ˙ Σ ( L ) m be as in Theorem C.2 for activation function erf m . It then holds for any x = y with x, y = 0 :

̸

and, assuming that x, y are not parallel or that σ 2 b > 0 ,

Remark D.3 (Addendum to Remark C.5) . In Remark C.5 we discussed the topology of the space of the Gaussian processes to which ANNs with continuous activation functions converge in the infinite-width limit. The product σ -algebra restricts us to a countable input set, so it is not possible to check for properties such as continuity or even differentiability. While Theorem C.4 is stated only for continuous activation functions with linear envelope property, we will see in Theorem E.3 that the convergence also holds in the (weak) infinite-width limit even for step-like activation functions. For the sign function as activation function, the covariance function of this Gaussian process is then given by Σ ( L ) ∞ . Since we know explicitly what this covariance function looks like, we can examine the sample-continuity of the process. Note that to get the full picture, one would still have to show functional convergence of the network to this process, as Bracale et al. [2021] have done.

Σ ( L ) ∞ is isotropic when restricted to a sphere, as will be discussed in the next section. In the case of isotropic covariance functions, k ( x, y ) = k ( ∥ x -y ∥ ) , the simplest way to show sample-continuity using the Kolmogorov-Chentsov criterion is to show Lipschitz continuity of k at zero. This can be seen from the proof of Lemma 4.3 by Lang and Schwab [2015]. In our case, the covariance function is basically given by a composition of arcsin functions. Lipschitz continuity of k at zero is therefore equivalent to Lipschitz continuity of the arcsin function at 1, which does not hold. In conclusion, it is not possible to show sample-continuity of the Gaussian process given by Σ ( L ) ∞ in the established way using the Kolmogorov-Chentsov criterion.

Clearly, the kernel can be rewritten in terms of the arccos function. Cho and Saul [2009] analyzed arc-cosine kernels in the context of deep learning in detail.

̸

Corollary D.3. For m,L ∈ N let Θ ( L ) m as in Theorem C.2 for activation function erf m . If x = y and either x, y not parallel or σ 2 b > 0 , then the limit

exists. Furthermore, it holds, asymptotically as m →∞ ,

Proof. The first statement directly follows from Lemma D.3 and the definition of Θ ( L ) m . The recursive definition can be resolved to the following formula:

With we get from Lemma D.3 that ˙ Σ (2) m ( x, x ) ∼ K ( ∥ x ∥ 2 ) · m and ˙ Σ ( L ) m ( x, x ) ∼ K ( n 0 ) · m for L ≥ 3 . This implies ∏ L -1 l =1 ˙ Σ ( l +1) m ( x, x ) ∼ K ( ∥ x ∥ 2 ) K ( n 0 ) L -2 · m L -1 and ∏ L -1 l = k ˙ Σ ( l +1) m ( x, x ) ∼ K ( n 0 ) L -k · m L -k for any k ≥ 2 . For L ≥ 2 , we get that

Theorem D.4 (Inspired by Lemma 5 of Radhakrishnan et al. [2023]) . Let σ 2 b > 0 or let all x i ∈ R n 0 be pairwise non-parallel. Let L ≥ 2 and x i ∈ S n 0 -1 R for all i = 1 , . . . , d , where S n 0 -1 R ⊆ R n 0 is the sphere of radius R . Then, with

and assuming that Θ ( L ) ∞ ( x, X ) Y ̸ = 0 for almost all x ∈ S n 0 -1 R , it holds

Proof. First note that almost all x ∈ S n 0 -1 R are not parallel to any x i ∈ X . We denote Θ ( L ) m ( X , X ) =: Θ m , m ∈ N ∪ {∞} , for convenience. Let a m > 0 be a positive constant that we will choose later. It then holds for almost all x ∈ S n 0 -1 R

We now show that A m and B m go to zero as m →∞ for a suitable choice of a m . First, note that

since ∥Y∥ 2 < ∞ and by Corollary D.3 it holds that Θ ( L ) m ( x, x i ) → Θ ( L ) ∞ ( x, x i ) for all x i ∈ X as m →∞ . Second, we have

̸

Again by Corollary D.3 it holds that ∥ Θ ( L ) m ( x, X ) ∥ 2 →∥ Θ ( L ) ∞ ( x, X ) ∥ 2 as m →∞ . Furthermore, for all 1 ≤ i ≤ d we have Θ ( L ) m ( x i , x i ) ∼ C ( R ) · m L -1 for some constant C ( R ) which depends on R = ∥ x i ∥ . Since it holds Θ ( L ) m ( x j , x i ) → Θ ( L ) ∞ ( x j , x i ) as m → ∞ for any i = j , we choose a m = C ( R ) · m L -1 and conclude

̸

This implies a -1 m Θ m → I d as m →∞ . In particular, { I d } ∪ { a -1 m Θ m | m ∈ N } is a compact set with respect to ∥ · ∥ 2 . Since D ↦→ D -1 is a continuous function, { I d } ∪ { a m Θ -1 m | m ∈ N } is bounded. We get

and thus

Recall that Θ ( L ) ∞ ( x, X ) Y ̸ = 0 , which now concludes the proof:

Remark D.4. One can generalize the above theorem by dropping the restriction to S n 0 -1 R . By Lemma D.3 it holds Θ ( L ) m ( x i , x i ) ∼ √ σ 2 w n 0 ∥ x i ∥ 2 + σ 2 b · C · m L -1 . For some constant C independent of L and ∥ x i ∥ . If we now define a m := C · m L -1 , we get

where diag { v } of a vector v is a square matrix with diagonal v . As in the proof above, and using the continuity of the inverse map D ↦→ D -1 , this implies

if we define

In conclusion, dropping the restriction leads to a similar form for our classifier, but with a modified kernel.

Theorem D.4 shows that the classifier

has an easily interpretable form that is close to the Nadaraya-Watson estimator with singular kernel Θ ( L ) ∞ . This is despite the fact that the pointwise limit of f m NTK is trivial, i.e., regression is no longer possible

D.3 Checking for Bayes optimality

In addition to the ideas used in the above section, Radhakrishnan et al. [2023] proved that the resulting estimator is Bayes optimal under certain conditions. This property is also referred to as consistency. Let the probability distribution P data on our data space be given by a random variable ( X,Y ) ∈ R n 0 ×{-1 , 1 } and let

be the Bayes classifier with respect to this distribution. Denote X d = ( x 1 , . . . , x d ) and Y d = ( y 1 , . . . , y d ) for ( x i , y i ) drawn independently from P data . We then define Bayes optimality as follows:

Definition D.1 (Bayes optimality) . Let c d ( · ) = c d ( · ; X d , Y ) be classifiers for d ∈ N and estimators of c ( · ) . We then say that ( c d ) d ∈ N is Bayes optimal, if it is a consistent estimator of c , i.e., for all ε > 0 and X -almost all x

First, we summarize the results of Radhakrishnan et al. [2023] and consider Θ ( L ) . If the singular limiting kernel for L →∞ behaves like a singular kernel of the form k ( x, y ) = ∥ x -y ∥ -α , then the classifier for L →∞ will be of the form

assuming ∑ d i =1 1 / ∥ x -x i ∥ α > 0 . This classifier satisfies Bayes optimality for α = n 0 by Devroye et al. [1998]. Radhakrishnan et al. [2023] generalized the results of Devroye et al. [1998], expressed α in terms of the activation function and its derivative, and chose them in such a way as to achieve α = n 0 . Going back to our setup, we want to see if we can write

for some constant α and for some function R : R + → R bounded away from zero as ∥ x -y ∥→ 0 . We start by proving that Θ ( L ) m ( x, y ) = G ( ∥ x -y ∥ ) for some function G . Recall that we have to restrict ourselves to a sphere by Theorem D.4. For simplicity, we restrict ourselves to the unit sphere S n 0 -1 . Then, it holds ⟨ x, x ⟩ = 1 for all x ∈ S n 0 -1 , which gives us

In the following, we will substitute z = ⟨ x, y ⟩ . Taking ∥ x -y ∥→ 0 is then equivalent to taking z → 1 . We can conclude from Lemma D.3 that we can indeed write Σ ( L ) ∞ ( x, y ) = Σ ( L ) ∞ ( z ) , ˙ Σ ( L ) ∞ ( x, y ) = ˙ Σ ( L ) ∞ ( z ) , and hence Θ ( L ) ∞ ( x, y ) = Θ ( L ) ∞ ( z ) for all L ≥ 1 . Note that Σ ( L ) ∞ (1) = σ 2 w + σ 2 b for L ≥ 2 and Σ ( L ) ∞ (1) = σ 2 w /n 0 + σ 2 b . Next, we consider ˙ Σ (2) ∞ ( z ) for z → 1 using Lemma D.3

For L ≥ 3 , we can observe that

We will now prove a lemma that allows us to analyze the behavior of ˙ Σ ( L ) ∞ ( z ) as z → 1 .

Lemma D.5. It holds:

Proof. Since the numerator and the denominator both go to zero as z → 1 , we apply l'Hôpital's rule and differentiate both. This yields

Thus,

concluding the proof.

For L = 3 it now holds

where we used Lemma D.5 at ( ⋆ ) . For L ≥ 4 it holds

again using Lemma D.5 at ( ⋆ ) . This implies for arbitrary L ≥ 2

for some constant K ( L ) depending on L . Recall the formula for the NTK, that was used in the proof of Corollary D.3

for some constants K ( L ) , K ′ ( L ) . It is therefore to possible find a function R that is bounded away from zero near ∥ x -y ∥ → 0 , such that

However, we have that α (1) = 1 and α ( L ) ↑ 2 as L →∞ . So it is only possible to choose L such that α ( L ) = n 0 , if n 0 = 1 . But this is a trivial case, since we have restricted ourselves to the unit sphere. In conclusion, we cannot prove that c ( L ) is a Bayes optimal classifier for any choice of L . Chapter 4 of Devroye et al. [1998] suggests that, in fact, the estimator will not be universally consistent.

E The NTK for surrogate gradient learning

In this chapter we explore surrogate gradient learning , introduced in Section 1, by connecting it to the NTK. Recall that the standard gradient flow dynamics of the parameters are given by

As mentioned several times before, the Jacobian matrix J θ will vanish for all parameters except those in the last layer if we consider the sign activation function due to its zero derivative. The idea of surrogate gradient learning is to circumvent the zero derivative of the sign function by replacing the derivative with a surrogate derivative [Neftci et al., 2019]. We can replace the derivative in two main ways:

The activation function in the full network can be replaced by a differentiable surrogate activation function. In particular, we obtain a non-vanishing surrogate derivative of the activation function and thus non-vanishing network gradients. Let g denote the network with surrogate activation function. Therefore, we can train the weights according to

or consider the loss with respect to f and train according to

Instead of replacing the activation function, we can only replace the derivative of the activation function ˙ σ with a surrogate derivative ˜ σ in Equation (S8). Let J σ, ˜ σ be the quasiJacobian matrix as in Definition C.4 with activation function σ and surrogate derivative ˜ σ . Then the training is given by

with ˆ I ( L ) as in Definition C.5 for σ 1 = σ , ˜ σ 1 = ˙ σ , σ 2 = σ and ˜ σ 2 = ˜ σ . For Equations (S48) and (S49) we assume that ˙ σ exists and is non-vanishing, but we deliberately train with a surrogate gradient. For example, we can again consider erf m as the activation function. Its derivative explodes at zero and vanishes everywhere else as m →∞ . The hope is that lim n 1 ,...n L -1 →∞ ˆ I ( L ) m = I ( L ) m exists and lim m →∞ I ( L ) m is not a singular kernel as before.

In this chapter we will deal with the second approach, because J σ, ˜ σ ( x ; θ t ) =: G ( σ ; ˜ σ ; x ; θ t ) is closer to J θ f ( x ; θ t ) = G ( σ ; ˙ σ ; x ; θ t ) than J θ g ( x ; θ t ) = G ( η ; ˜ σ ; x ; θ t ) as a formula if J θ f ( x ; θ t ) exists and is non-vanishing. Here, η denotes the surrogate activation function with derivative ˜ σ . To do this, we provide asymmetric generalizations of Theorem C.1, Theorem C.2, and Theorem C.3. These generalizations would, in principle, even allow us to compare the two approaches.

E.1 Asymmetric generalization of the neural tangent kernel

For this section we adopt the so-called linear envelope property from [Matthews et al., 2018, Definition 1] to ensure that all expectations exist in the following theorems:

First, we consider networks under the weak infinite-width limit, and are thus interested in taking the number of hidden neurons to infinity sequentially. This is done inductively while using the central limit theorem and the weak law of large numbers. In order to do this rigorously, we state and prove two lemmata.

The first lemma is stated in terms of Gaussian measures on Hilbert spaces. An introduction to Gaussian measures on Hilbert spaces can be found in Chapter 1 of Da Prato [2006]. A rigorous derivation and definition of convergence in distribution on arbitrary metric spaces is given by Heyer [2009, Chapter 1.2]. This includes a definition of weak convergence in terms of continuous and bounded functions (Remark 1.2.5 (b)), a version of the Portemanteau theorem (Theorem 1.2.7), and the fact that convergence in probability implies convergence in distribution (Application 1.2.15).

Lemma E.1. Let H i m and Z i , i, m ∈ N , be random variables with values in a separable Hilbert space H . Furthermore, let Z i be independent and identically distributed with finite mean and covariance operator V . If ( H m 1 , . . . , H m k ) D - → ( Z 1 , . . . , Z k ) as m →∞ for all k ∈ N , then there exists k : N → N such that k ( m ) →∞ monotonically as m →∞ and

for an H -valued Gaussian random variable Z with mean and covariance operator like Z 1 . Similarly, there exists k ′ : N → N such that k ′ ( m ) →∞ monotonically as m →∞ and

Proof. Since H is separable and complete, convergence in distribution and convergence with respect to the Prokhorov metric d (also known as the Lévy-Prokhorov metric) are equivalent [Billingsley, 1999, Theorem 6.8]. By the central limit theorem for separable Hilbert spaces [Zalesski˘ ı et al., 1991], this implies

In addition, the assumption together with the continuous mapping theorem gives that

In particular, for any k ∈ N , there exists some m k ∈ N such that

We now want to choose k ( m ) as large as possible for any m , but small enough to ensure Inequality (S54), i.e., m ≥ m k ( m ) . So we define

First note that { k | m ≥ m k } ̸ = ∅ , if m ≥ m 1 . The map k : N → N is therefore well-defined, as we consider m →∞ . Similarly, we can find a m ≥ m k for any given k . This yields

By definition of k ( m ) , it holds m ≥ m k ( m ) for all m ∈ N and thus

Together with Equation (S53) this yields the claim, (S51):

For the second claim, (S52), one can follow the same procedure but use the law of large numbers for Banach spaces instead of the central limit theorem. Suitable results are given by Ledoux and Talagrand [1991, Corollary 7.10] and Hoffmann-Jørgensen and Pisier [1976, Theorem 2.1]. Note that even the strong law of large numbers holds, but the weak law is sufficient.

In the second lemma, some properties about convergence in distribution and convergence in probability are stated.

Lemma E.2 (Theorem 2.7 from van der Vaart [1998], modified and (iv) added) . Let X n , X and Y n be random vectors. Then

Proof. (i) - (iii). The proofs are given by van der Vaart [1998].

(iv) . Let f be a bounded and continuous function. Then,

where we used the independence of W and ( X n ) n ∈ N in Equation ( ⋆ ) and the boundedness of f for the interchange of limit and integration. This proves convergence in distribution.

Remark E.1. The above theorem can be generalized to metric spaces. One can easily check that the proofs in [van der Vaart, 1998, Theorem 2.7] also work for metric spaces using the Portemanteau theorem provided by Heyer [2009, Theorem 1.2.7]. However, it is necessary to derive some more equivalent characterizations of convergence in distribution, which are given and used by van der Vaart [1998] but are missing in the work of Heyer [2009].

Theorem E.3 (Generalized version of Proposition 1 by Jacot et al. [2018]) . For activation functions σ 1 and σ 2 with property (S50), which are continuous except for finitely many jump points, let f 1 ( · ; θ ) and f 2 ( · ; θ ) be network functions with hidden layers h ( l ) 1 ( · ; θ ) , h ( l ) 2 ( · ; θ ) , for 1 ≤ l ≤ L , respectively

as in Definition C.1 and with shared weights θ . Then ( f 1 ( · ; θ ) , f 2 ( · ; θ )) converges in distribution to a multidimensional Gaussian process ( X ( L ) j , Y ( L ) j ) j =1 ,...,n L as ( n l ) 1 ≤ l ≤ L -1 → ∞ weakly for any fixed countable input set ( z i ) ∞ i =1 . The Gaussian process is defined by X ( L ) j iid ∼ N ( 0 , Σ ( L ) 1 ) , Y ( L ) j iid ∼ N ( 0 , Σ ( L ) 2 ) , where we have for x, x ′ ∈ R n 0

̸

Furthermore, X ( L ) i and Y ( L ) j are independent if i = j and

Proof. We write h ( l ) ( x ) = h ( l ) ( x ; θ ) . We prove the theorem by induction, as in the proof of Proposition 1 of Jacot et al. [2018], but expand on the technical details.

L = 1 . By definition, we have

This implies that h (1) k 1 ,i ( x ; θ ) and h (1) k 2 ,j ( x ′ ; θ ) , the i -th and j -th component of h (1) k 1 ( x ; θ ) and h (1) k 2 ( x ′ ; θ ) respectively, are independent for any k 1 , k 2 ∈ { 1 , 2 } , x, x ′ ∈ R n 0 and i = j . To prove that ( h (1) 1 ( · ; θ ) , h (1) 2 ( · ; θ )) is a Gaussian process, it is thus sufficient to show that vectors of the form ( h (1) 1 ,i ( x 1 ) ... h (1) 1 ,i ( x n ) h (1) 2 ,i ( x 1 ) ... h (1) 2 ,i ( x n ) ) ⊺ have a multivariate Gaussian distribution for any x 1 , . . . , x n ∈ ( z i ) ∞ i =1 . It holds

̸

and therefore

with ✶ d a column vector of ones with length d . Now since it holds

✶ ✶ Therefore, the vector has a multivariate Gaussian distribution with the covariances required for Equation (S56) and the first case of Equations (S58).

L → L + 1 . We assume that the convergence holds for depth L . This means that there exists some r ∈ R L such that, for given constant width n 0 , any width n L , and widths n l = r l ( m ) , 1 ≤ l < L , it holds

.

To be precise, there is no such r in the case L = 1 → L +1 . However, this only makes the proof simpler and one can still follow the same steps as for L ≥ 2 .

By the continuous mapping theorem [Billingsley, 1999, Theorem 2.7] it holds

The theorem is applicable despite the finitely many jump points, since ( X ( L ) , Y ( L ) ) assumes the values of the jump points with zero probability.

We now need to find an increasing width function r L : N → N such that, if we additionally set n L = r L ( m ) , it holds for any fixed n L +1

.

Note that by Remark C.5 we consider the product σ -algebra. Therefore, to show convergence in distribution for the whole process, it is sufficient to show convergence in distribution for the marginal distributions. By definition, we have for k ∈ { 1 , 2 } that

The marginal vector for points x 1 , . . . , x n as before can thus be written as

With the same arguments as before and using the continuous mapping theorem in combination with Lemma E.2 (iv) and the independence of W ( L +1) , it holds

̸

Since ( X i , Y i ) and ( X j , Y j ) are independent for i = j , the conditions of Lemma E.1 are satisfied. Now, there exists k : N → N such that k ( m ) → ∞ monotonically as m → ∞ and, when setting n L := r L ( m ) := k ( m ) , it holds

for a Gaussian random variable ( R ( L +1) x 1 . . . , R ( L +1) x n , S ( L +1) x 1 , . . . , S ( L +1) x n ) ∈ R 2 · n L +1 · d with

̸

Before considering the covariances, we want to comment on r L . First, note that this sequence may not initially be strictly increasing, but this can be circumvented by considering a strictly increasing subsequence. Second, r L could theoretically depend on X . However, this can be resolved by not evaluating the pair ( h ( L ) 1 , h ( L ) 2 ) at certain data points, but doing the same calculation as above for ( h ( L ) 1 , h ( L ) 2 ) . The starting point for this is Equation (S59). To apply Lemma E.1, note additionally that ( h ( L ) 1 , h ( L ) 2 ) ∈ ⊗ ∞ i =1 R 2 · n L , which is a separable Hilbert space because we are considering a countable input set. Above, we worked with the marginal distribution because this makes the following calculation of the covariances easier. Finally, the choice of r L should be independent of n L +1 . This follows from the independence of W ( L +1) ji and W ( L +1) j ′ i for j = j ′ .

To verify Equation ( ⋆ ) , we check the covariances of the random vector. They are given by

Similarly, we get that

and together this implies using the independence of biases b ( L +1) and weight matrices W ( L +1)

We therefore proved Equation (S57). For the second case of (S58), we see that

with, by induction hypothesis,

This finished the proof, since it now holds

Theorem E.4 (Generalized version of Theorem 1 by Jacot et al. [2018]) . For activation functions σ 1 , σ 2 and so-called surrogate derivatives ˜ σ 1 , ˜ σ 2 such that σ 1 , σ 2 , ˜ σ 1 , and ˜ σ 2 are continuous except for finitely many jump points with property (S50), let f 1 ( · ; θ ) and f 2 ( · ; θ ) be network functions with hidden layers h ( l ) 1 ( · ; θ ) , h ( l ) 2 ( · ; θ ) , 1 ≤ l ≤ L , respectively as in Definition C.1 with shared weights θ . Denote the empirical generalized neural tangent kernel

as in Definition C.5. Then, for any x, x ′ ∈ R n 0 and 1 ≤ i, j ≤ n L , it holds

as n 1 , . . . , n L -1 →∞ weakly. We call I ( L ) the analytic generalized neural tangent kernel, which is recursively given by

with Σ ( L ) 1 , 2 as in Theorem E.3 and

Proof. We prove the theorem by induction over L , as in the proof of Theorem 1 by Jacot et al. [2018]. We denote J ( L ) ,k ( z ) = J ( L ) ,σ k , ˜ σ k ( z ; θ ) for k ∈ { 1 , 2 } , z ∈ R n 0 .

L = 1 . For 1 ≤ i, j ≤ n 1 it holds

This proves Equation (S60).

L → L + 1 . Now we assume that the statement is true for L and need to prove it for L +1 . Instead of considering an explicit r ∈ R L as in the proof of Theorem E.3 and taking m →∞ , we will just write ' n 1 , . . . , n L -1 →∞ weakly'.

In the induction step, we would like to use Theorem E.3. However, it is not obvious that this is possible. In the setting of Definition C.7, let S 1 and S 2 be two statements that hold weakly. In our setting, these are the induction hypothesis and Theorem E.3 for depth L . Then there exist s, t ∈ R L such that S 1 ( s ) and S 2 ( t ) are true. It is not clear that there exists some r ∈ R L such that S 1 ( r ) and S 2 ( r ) are true. It would be natural to define r by r l ( m ) := max { s l ( m ) , t l ( m ) } for all 1 ≤ l < L and m ∈ N , but it is still unclear that this implies S 1 ( r ) and S 2 ( r ) . Instead, one can consider the combined statement ' S 1 and S 2 ' as a new statement S . In our case, for any depth L , we would need to find a r ∈ R L such that the statements in Theorem E.3 and Theorem E.4 are both true, which can be done. Since we used the first part of Lemma E.1 to prove Theorem E.3 and will use the second part of Lemma E.1 for this proof, we would have to define r L ( m ) := min { k ( m ) , k ′ ( m ) } . For simplicity, we will assume that the r ∈ R L we get by the induction hypothesis also gives us the convergence statement of Theorem E.3.

Using the definition of the generalized NTK and the quasi-Jacobian matrices, we obtain for 1 ≤ i, j ≤ n L +1

We want to apply the second part of Lemma E.1. We will consider the terms (S62) and (S63) separately. First note that

as n 1 , . . . , n L -1 →∞ weakly by Theorem E.3 and the continuous mapping theorem with

Again, we used that the values of the jump points are assumed with zero probability. Thus, again by the continuous mapping theorem and by the second part of Lemma E.1, it holds

Here, as in the proof of Theorem E.3, n L → ∞ is given by n L := r L ( m ) := k ′ ( m ) , which is in turn is given by Lemma E.1. Note also that the limit is a constant, which implies convergence in probability according to Lemma E.2 (ii). In conclusion, we obtain

For term (S63), we can apply the induction hypothesis to obtain

Also, we can again use the convergence given by (S64), but with σ 1 , σ 2 replaced by ˜ σ 1 , ˜ σ 2 respectively. This gives using Lemma E.2 (iv)

Together with (S66) and Lemma E.2 (iii) this implies

For the summands in (S63) we therefore have by the continuous mapping theorem

These terms are independent for different 1 ≤ m ≤ n L . Therefore, the second part of Lemma E.1 can be applied as before. This yields

Together with (S65) this yields Equation (S61) and concludes the proof.

The two theorems above are complemented by the convergence of the generalized NTK in the infinitewidth limit during training, which we will prove below. The theorem is a generalization of Theorem G.2 from Lee et al. [2019]. There, the convergence of the NTK at initialization as ( n l ) L -1 l =1 ∝ ∼ n is assumed. More precisely, they refer to a result of Yang [2019a]. In consequence, we have to assume the same for the next theorem. The proof of Yang [2019a] could also be generalizable to our case. Alternatively, one could also try to generalize the statement of Jacot et al. [2018] on stability during training in the weak infinite-width limit.

Theorem E.5 (Based on Theorem G.2 from Lee et al. [2019] ) . Let σ be a Lipschitz continuous and differentiable activation function. Let the derivative of the activation function ˙ σ and a socalled surrogate derivative ˜ σ be Lipschitz continuous and bounded. Let f t ( · ; θ ) be the corresponding network function with depth L initialized as in Definition C.1 and trained with MSE loss and surrogate gradient learning, i.e., according to Equation (S49) with surrogate derivative ˜ σ . The hidden layers are denoted by h ( l ) l ( · ; θ ) , for 1 ≤ l < L , as in Definition C.1. Assume that the generalized NTK converges in probability to the analytic generalized NTK of Theorem E.4 as ( n l ) L -1 l =1 ∝ ∼ n ,

Furthermore, assume that the smallest and largest eigenvalue of the symmetrization of I ( L ) ( X , X ) ,

are given by 0 < λ min ≤ λ max < ∞ and that the learning rate is given by η > 0 . Then, for any δ > 0 there exist R > 0 , N ∈ N and K > 1 such that for every n ≥ N , the following holds with probability at least 1 -δ over random initialization

where ∥ · ∥ F denotes the Frobenius norm.

Proof. We follow the proofs in Chapter G of Lee et al. [2019]. The analogous statement is given by Theorem G.2 of Lee et al. [2019]. Since we are considering the infinite-width limit ( n l ) L -1 l =1 ∝ ∼ n , we will assume that the width of the hidden layers are given by n ∈ N . The more general case can easily be proved using the same steps. Recall that

The loss function is the MSE loss by assumption. This yields

The change of weights over time is hence given by

By Lemma E.6 below, for any δ 1 > 0 there exists a K > 0 such that for any C > 0 it holds with probability at least 1 -δ 1

Together, this implies

for all θ t ∈ B ( θ 0 , C ) with probability at least 1 -δ 1 . Inequality ( ⋆ ) is a consequence of the Cauchy-Schwarz inequality.

To estimate the term ∥ g ( θ t ) ∥ 2 in Inequality (S67), we use Grönwall's inequality in the differential form, e.g., Lemma 1.1.1 from Qin [2016]). First, note that as a consequence of the proof of Lemma E.6, for all δ 2 > 0 there exist R 0 > 0 and n 0 ∈ N such that with probability at least 1 -δ 2 it holds for all n ≥ n 0

We now set C := 3 KR 0 /λ min . Next, observe that

̸

with S ( L ) t := 1 2 ( ˆ I ( L ) t ( X , X ) + ˆ I ( L ) t ( X , X ) ⊺ ) . To show ( ⋆⋆ ) , we will prove that for any 0 = z ∈ R n L · n

̸

First, note that by assumption for all z = 0

Since the generalized NTK converges at initialization in probability, we can assume that with probability at least 1 -δ 3 for any δ 3 > 0 that

for any n ≥ n 1 ∈ N . For the symmetrizations, this yields

Furthermore, for θ t ∈ B ( θ 0 , C ) it holds that

with probability at least 1 -δ 1 using Lemma E.6 as before. With the same calculations as for Inequality (S71), this implies for the symmetrizations that

for all n ≥ N := max { m 0 , m 1 , ( λ min 18( K ′ ) 2 KR 0 ) 2 } . Now Inequalities (S70), (S71) and (S73) give us

and thus imply Inequality (S70). To summarize, it holds with probability at least 1 -δ 2 -δ 3 for any n ≥ N and θ t ∈ B ( θ 0 , C ) that

Grönwall's inequality now implies

We can now return to Inequality (S67) to obtain with probability at least 1 -δ 1 -δ 2 -δ 3

Integrating the inequality on both sides yields for all θ t ∈ B ( θ 0 , C )

Let t 1 := inf { t : ∥ θ t -θ 0 ∥ 2 < C } . Now, if t 1 < ∞ , it holds

This is a contradiction, so we conclude that t 1 = ∞ . In particular, Inequality (S74) holds for all t > 0 . We can now repeat the calculations for Inequality (S72) with the Frobenius norm to finally obtain

Therefore, if we choose δ 1 = δ 2 = δ 3 = 1 3 δ , the desired inequality holds for any n ≥ N with probability at least 1 -δ .

Lemma E.6 (Based on Lemma 1 and Lemma 2 from Lee et al. [2019]) . In the setting of Theorem E.5, for any δ > 0 there exists a K > 0 such that for any C > 0 , with probability at least 1 -δ it holds

for all θ, ˜ θ ∈ B ( θ 0 , C ) and ˆ σ ∈ { ˙ σ, ˜ σ } . Due to the equivalence of matrix norms, the same inequalities hold for the norm ∥ · ∥ 2 and some constant K ′ > 0 .

Proof. For ˆ σ = ˙ σ and a different but equivalent parameterization of the network parameters [Lee et al., 2019, Chapter F], the proof can be found in Section G.2 of Lee et al. [2019]. The surrogate derivative ˜ σ is bounded and Lipschitz continuous. Also, J ( L ) ,σ, ˜ σ ( X ; θ ) and J ( L ) ,σ, ˙ σ ( X ; θ ) = J θ f ( X ; θ ) share the same recursive formula. Therefore, the proof, which builds on the so-called Gaussian conditioning technique [Yang, 2019a, Section E.1], should also hold for the surrogate derivative.

Remark E.3 (The analytic generalized NTK for surrogate gradient learning) . Since in Theorem E.5 we consider only one activation function σ instead of two different ones σ 1 , σ 2 as in Theorem E.3 and E.4, it holds

The analytic generalized NTK in the setting of Theorem E.4 is thus given by

We therefore still obtain an asymmetric kernel due to the contribution of ˜ Σ ( L ) 1 , 2 .

Remark E.4 (Positive definiteness of the generalized NTK) . In Theorem E.5 we require that the matrix I ( L ) ( X , X ) be positive definite. Equivalently, its symmetrization,

should be positive definite. For applications this can be checked numerically. More generally, it would be interesting to know whether the symmetric kernel given by

is positive definite. One could try to approach this question inductively. However, this leads to problems in the induction step. We want to present a different ansatz, which reduces the question to a question about ˜ Σ ( L ) 1 , 2 . We use the closed form of the analytic NTK and the symmetry of Σ ( l ) for all l ∈ N to see that

This defines a symmetric positive definite kernel if Σ ( L ) is positive definite (see Jacot et al. [2018, Section A.4] for comparison) and the symmetrized kernels,

are positive semi-definite for all k = 1 , . . . , L -1 .

E.2 The analytic NTK for surrogate gradient learning with sign activation function

We would like to proceed as in Section C.2.1 and replace the empirical generalized NTK in Equation (S49), the equation defining surrogate gradient learning, with the analytic generalized NTK obtained by Theorem E.5. Since the activation functions considered in the above section are Lipschitz continuous with bounded and Lipschitz continuous derivative, we will again approximate the sign function and its distributional derivative with the error function as in Section D. The results will not depend on the approximation of the weak derivative of the sign function. This approach can only lead to a useful result if the resulting kernel is not singular. We will check this in the following.

We choose activation function erf m ( z ) = erf( m · z ) , m ∈ N , with surrogate derivative ˜ σ ( z ) . We will also consider the special case ˜ σ ( z ) = ˙ erf( z ) . As discussed in Remark E.3 and with the final results of Section D.1, this immediately yields

̸

Here, we have to assume that σ 2 b > 0 or x, y = 0 to ensure that Σ (1) ∞ ( x, x ) , Σ (1) ∞ ( y, y ) = 0 . This has already been discussed after Equation (S39).

E.2.1 The derivative of the error function as surrogate derivative

Next, we want to calculate ˜ Σ ( L ) 1 , 2; ∞ ( x, y ) := lim m →∞ ˜ Σ ( L ) 1 , 2; m ( x, y ) . We will first consider the case ˜ σ = ˙ erf , for which we can use the already established tools. In particular, we will discuss the differences to the results of Section D.

It holds where

again using Remark E.3 to simplify the covariance matrix. The notation Σ ( L -1) 1 , 2; m , which comes from Theorem E.3 in combination with the scaling variable m of the activation function, should not be confused with Σ ( L -1) m ; x,y , which is a shorthand notation for the Gram matrix Σ ( L -1) m ( { x, y } , { x, y } ) .

̸

We denote e 1 = ( 1 0 ) , e 2 = ( 0 1 ) and U = ( Z m 1 Z m 2 ) . This yields

using Lemma D.1 in Equation ( ⋆ ) . For the penultimate equality we used Equation (S36) and Equation (S37). Compared to Equation (S41),

̸

which in fact holds for both L = 2 and L ≥ 3 , an additional term appeared in Equation (S76). It always holds σ 2 w + σ 2 b > 0 and it holds σ 2 w ∥ x ∥ 2 /n 0 + σ 2 b > 0 if σ 2 b > 0 or x = 0 . As discussed earlier, we always assume that x = 0 is satisfied. It follows that this asymmetric NTK is not singular, since

̸

Note that this is reminiscent of the constant factor depending on x in the asymptotics of ˙ Σ ( L ) m ( x, x ) as m →∞ , given by (S42) and (S43),

According to Remark E.3 it holds

Since Σ ( L ) m ( x, y ) and ˜ Σ ( L ) 1 , 2; m are continuous functions of the entries of the matrix Σ ( L -1) m ; x,y , it follows by induction using Equations (S75) and (S76) that the limit of the analytic NTK is well defined for m →∞ . We can write

Thus, by approximating the sign function with error functions, we found that the analytic NTK for surrogate gradient learning with sign function and error function as surrogate derivative is welldefined and non-singular as a kernel on R n 0 × R n 0 . Furthermore, comparing this NTK with the

NTK we derived in Section D, the term Σ ( L ) ∞ does not change. This is a direct consequence of not replacing the activation function with a surrogate activation function. In this sense, we inherit more properties with this approach than by replacing not only the derivative but the entire activation function including its derivative with a surrogate. Comparing Equation (S41) with Equation (S76), we see that ˜ Σ ( L ) 1 , 2; ∞ ( x, y ) can be obtained from ˙ Σ ( L ) ∞ ( x, y ) by adding a regularizing term depending on x , Σ ( L -1) ∞ ( x, x ) / 2 .

E.2.2 General surrogate derivative

Now we turn to the general case with a general surrogate derivative ˜ σ . Similar to before, for m ∈ N ∪ {∞} we denote

With u = ( z 1 z 2 ) and for invertible Σ = Σ ( L -1) ∞ ; x,y , it holds for L ≥ 2

where Equation ( ⋆ ) seems natural, but requires further reasoning. We will prove the above rigorously in Lemma E.7. For now, we assume that the equality holds. Since ˜ σ is bounded and continuous, this yields

This agrees with the fact that

where we again assumed an equality very similar to Equation ( ⋆ ) . It is easy to check that Equations (S76) and (S77) can be recovered by inserting ˜ σ = ˙ erf into the derived formulas.

We now prove the missing part of Equation (S78).

Lemma E.7. Let ˜ σ be a bounded and continuous function and (( Z m 1 , Z m 2 )) m ∈ N a sequence of random variables, ( Z m 1 , Z m 2 ) ∼ N (0 , Σ m ) . If the covariance matrices are invertible and converge to an invertible matrix, Σ m → Σ ∞ ∈ R 2 × 2 as m →∞ , then it holds

̸

If Z m 1 = Z m 2 for all m ∈ N so that the covariance matrices are given by Σ m = ( Σ m 1 Σ m 1 Σ m 1 Σ m 1 ) , and if Σ m 1 → Σ ∞ 1 = 0 as m →∞ , then it holds

Proof. We begin with the case of invertible covariance matrices. Again denoting u = ( z 1 z 2 ) , it holds by assumption

for

The determinant of B m is given by

This now yields

The continuity of matrix inversion implies ( B m ) -1 → ( B ∞ ) -1 as m → ∞ . The characteristic functions of finite-dimensional Gaussian random variables are fully defined by their means and covariance matrices. Thus, the convergence of the covariance matrices implies convergence of the characteristic functions, which in turn implies convergence in distribution. Since ˜ σ is continuous and bounded and the determinant is continuous, we obtain

This proves Equation (S80). In the case Z m 1 = Z m 2 , we have

again using the boundedness and continuity of ˜ σ in the last line. This proves Equation (S81).

With Σ m = Σ ( L -1) m ; x,y the lemma yields

In conclusion, we found that the analytic NTK for surrogate gradient learning with sign function is well-defined and non-singular for any bounded and Lipschitz continuous surrogate derivative. As in the special case of the error function, the term ˜ Σ ( L ) 1 , 2; ∞ ( x, y ) can be expressed in terms of Σ ( L -1) ∞ ( x, x ) and the determinant ∣ ∣ Σ ( L -1) ∞ ; x,y ∣ ∣ .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As stated in the abstract, we analyze the NTK for activation functions with jumps and generalize the NTK to surrogate gradient learning. We focus on the sign activation function in detail, which is also stated in the abstract. The derived theorems, however, are valid for a general class of activation functions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the theorems are discussed throughout the paper and a general discussion of limitations can be found in Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, all definitions, theoretical results, and proofs are given in great detail in the appendix and have only been slightly shortened for the paper itself. Only for Lemma E.6, which is used to prove Theorem 2.5, the proof is not given in full detail. Most proofs rely on similar results for the classical NTK and these results are properly discussed and referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All information needed to reproduce the numerical experiments is given in Section 3 and Section A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the code is provided in the supplementary material with instructions to reproduce the experiments and figures.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Since only plain gradient descent and gradient descent with surrogate gradients (surrogate gradient learning) is used in the paper, no further additional specifications apart from the ones made in Section 3 and Section A are necessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, confidence bands are used for Figure 3 to demonstrate the agreement between two distributions. Figure 1 and Figure 2 are complemented by Figure B.3 and Figure B.4 respectively, showing the mean squared errors.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources used for the numerical simulation are described in Section A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics and the paper conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is solely theoretical and does not have conceivable societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not contain models. The data is only created to illustrate and examine the theoretical results and has no risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The used packages have been properly referenced in Section 3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper addresses the challenge of applying gradient-based training methods to neural networks with non-differentiable activation functions, such as binary and spiking neural networks. These networks use surrogate derivatives to enable gradient descent, but this approach lacks theoretical foundation. The authors propose a generalization of the NTK, called the surrogate gradient NTK (SG-NTK), to provide a rigorous theoretical basis for SGL.

STRENGTHS:
1. Novel extension of NTK to non-differentiable activation functions using surrogate gradients using the results in [1,2,3,4].
2. Strong empirical validation through numerical experiments.

[1] Gaussian process behavior in wide deep neural networks
[2] Neural tangent kernel: Convergence and generalization in neural networks.
[3] Wide and deep neural networks achieve consistency for classification
[4] Wide neural networks of any depth evolve as linear models under gradient descent

WEAKNESSES:
1. The complexity of the theoretical framework may limit accessibility for practitioners
2. Additional experiments on real datasets (MNIST, CIFAR10, ImageNet) could further validate the approach.

QUESTIONS:
What is the application of the results? The activation function, the ReLU function is commonly used and I believe it can be solved in previous works. Are there any commonly used activation functions that can solved in your paper but not in previous works?

LIMITATIONS:
The results in this paper are only applicable to sufficiently wide networks with random initialization.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers an extension of neural tangent kernel methods for the analysis of training of neural networks with non-differentiable activations with surrogate gradient learning.  The basic approach is the define a generalized NTK (the SG-NTK) based on the quasi-Jacobian matrix (that is, the Jacobian constructed using the surrogate gradients) rather than the (ill-defined) Jacobian based on the actual gradients of the activations.  This construct is shown to be deterministic (distributionally determined) in the infinite width limit.  For the sign activation (with erfm acting as surrogate gradient) both the NTK and SG-NTK are derived.  Finally some experimental results are given that show that the distribution of networks trained with SG-NTK matches reality.\n\nSTRENGTHS:\n- The paper is clearly presented.\n- Motivation, aims etc are clear and compelling.\n- The paper appears to fills an important gap in existing literature (admittedly I am relatively new to this area of research so there may be predecessors I am unaware of).\n- The derivations given seem correct to the best of my understanding, though I did skim some of the proofs in the rather long appendix.\n- The experimental results would appear to confirm that the SG-NTK matches with reality.\n\nWEAKNESSES:\n- it would perhaps be more useful to present figure 1 with a log y scale as the linear scale used, combined with the divergence at $\\Delta \\alpha = 0$, tends to flatten all features of the NTK, masking the differences between the NTKs away from the point of divergence.\n- also regarding figure 1 is it feasible to plot the limiting case $m \\to \\infty$ with the point of divergence elided?\n\nQUESTIONS:\nSee previous sections.\n\nLIMITATIONS:\nThe authors have addressed limitations adequately and there appear to be no obvious negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explore the neural tangent kernel (NTK) with regards to surrogate gradient learning for non-differentiable activation functions. The authors show that the standard neural tangent kernel is not equipped to deal with such activation functions and causes the kernel function to become singular. They provide a derivation of a surrogate derivative based NTK that can handle such activations thus further generalizing the NTK for a larger subset of networks.\n\nSTRENGTHS:\nThis paper provides a brand new derivation of the NTK for non-smooth ANN activations and thus is very original. The overall paper is of high quality and clearly developed. The results of this paper are of high significance and allow for new avenues for application of NTK analysis.\n\nWEAKNESSES:\nWhile the paper is well developed overall, there are a few spots that can benefit from additional clarity regarding notation:\n\nLine 144 (Definition 2.1) - $r_l(m)$ refers to the number of neurons for a particular layer however $m$ is not directly described. My understanding is that it is referring to the output size dimension but this is not immediately clear.\n\nLine 215 - I am assuming that $\\delta(z)$ refers to the Dirac delta function (delta distribution). \n\nLine 244 - Here we have $\\delta_{ij}$ which I assume is *not* the Diract delta and instead a constant for a given kernel matrix entry.\n\nLine 289 - Since we are talking about divergence, notationally it would be better to use $\\Theta_{\\text{erf}_m} \\not\\rightarrow \\Theta\\_{\\text{sign}}$ or specify the formal definition of divergence *or* avoid it all together.\n\nIn regard to the experiments line 303 mentions that the NTK diverge as $m \\rightarrow \\infty$, however this does not seem to be clear from the plots especially since $m=20$ does not seem to be sufficient to show the divergence occurring. In addition, since we are discussing divergence, I believe the paper could benefit by including the average error values between the analytic and empirical kernels since graphs tend to do a poor job of illustrating this.\n\nQUESTIONS:\nAre $\\delta(z)$ and $\\delta_{ij}$ the same?\n\nLIMITATIONS:\nEverything is adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper adapts the neural tangent kernel framework to surrogate gradient learning (and so to learning in spiking neural networks).

STRENGTHS:
- The paper generalizes NTK to (some algorithms for) spiking neural networks, which is an important scenario for neuroscience and neuromorphic computing 
- Like the original NTK results, analyzing learning dynamics with this approach seems easy (i.e. it works like kernel regression, Eq. 7)
- The paper is well-written and self-contained

Overall, I think this paper is important for theoretical analysis of spiking neural networks, and contains novel results. I didn't closely follow the derivations in the appendix, but I'm familiar with the NTK literature/proofs, and the approach of this paper seems reasonable to me (+ it follows the original derivations to some extent).

WEAKNESSES:
The main weakness is the NTK approach itself: NTK is known to be a poor approximation to learning in deep networks in many cases, which limits what kind of conclusions we can make with NTK-driven theoretical analysis. However, this weakness is not specific to this paper.

QUESTIONS:
- I don’t see how definition 2.2 iii can work for strictly increasing width functions. Sequential limits mean some of the widths stay fixed, right? 
- Fig.1 should have the black line at the back I think, otherwise it’s not clear if the peaks always overlap

LIMITATIONS:
The authors have addressed the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12