A generalized neural tangent kernel for surrogate gradient learning
Authors: Luke Eilers
Raoul-Martin Memmesheimer
Sven Goedeke
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Neural Tangent Kernel
Surrogate Gradient Descent
Binary Neural Networks
Infinite Width
Primary area: neuroscience_and_cognitive_science
TL;DR: We derive a generalized neural tangent kernel that describes surrogate gradient learning.

Abstract: State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation.

The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide a generalization of the NTK, which we call the surrogate gradient NTK, that enables the analysis of SGL. First, we study a naive extension of the NTK to activation functions with jumps, demonstrating that gradient descent for such activation functions is also ill-posed in the infinite-width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems on the NTK with mathematical rigor. Further, we illustrate our findings with numerical experiments. Finally, we numerically compare SGL in networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.

A generalized neural tangent kernel for surrogate gradient learning
Luke Filers"
Department of Physiology. University of Bemn, Switzerland Institute for Applied Mathematics, University of Bonn, Germany luke. eilers@unibe.ch ch
Raoul-Martin Memmesheimer
Institute of Genetics, University of Bonn, Germany
menmmehheimerduni-oone
Sven Goedeke
Bernstein Center Freiburg, University of Freiburg, Germany Institute of Genetics, University of Bonn, Germany
sven goeen.oodeee@bc...o wni-freibbrg
Abstract
State-of-the-art neural network training methods depend on the gradient the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's 's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation.
The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide generalization of the NTK, which we call the surrogate gradient NTK. that enables the analysis of SGL. First, we study naive extension of the NTK to activation functions with jumps. demonstrating that gradient descent for such activation functions is also ill ill-posed in the infinite-widhe width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems the NTK with mathematical rigor. Further, illustrate findings with numerical experiments. Finally, we numerically compare SGL networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.
Introduction
Artificial neural networks (ANNs) originate from the biologically inspired perceptron [Roseblaat 1958]. While the perceptron has a binary output that is faithful to the all-or-none behavior of spiking neurons in the nervous system, most activation functions considered nowadays for ANNs smooth (like the logistic function) or at least semi dimi-iifferentiable (like the ReLU function). Differentiable network functions enable the learning of network weights with methods that leverage the gradient of the network function with respect to the network weights like backpropagation [Rumelhart et al.)
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1986]. These methods are very successful LeCun et al. 2015]. but cannot be used without well- defined gradients.
This is a problem when considering more biologically plausible ANNs, which are typically used in computational neuroscience understand how the networks of spiking neurons in our nervous system work. These include spiking neural networks (SNNs). Motivated by the energy efficiency of our brain, SNNs and similar networks, such as binary neural networks (BNNs), are considered in the context of neuromorphic computing (Merolla et al. 2014). Both discrete-time SNNs and BNNs have in common that their activation functions do not have useful derivatives, which renders standard gradient-descent training impossible [Neftei et al., 2019 Taherkhani et al. 2020 Tavanaei et al. 2019 Roy al.] 2019 Pieiffer and [ For the scope of this paper, these activation functions can be thought of as step-like functions like the sign function, in which cases the derivative vanishes almost everywhere and is thus no longer informative about the shape of the activation function.
Surrogate gradient learning resolves this issue by providing the missing information about the activation function in the form of surrogate derivative [Hinton] 2012 Bengio et al. 2013 Zenke and Ganguli 2018). As a result, the gradient-based methods for classical ANNS can be leveraged with great success Zenke and Vogels 202T]. However, a theoretical basis underpinning the intuition is missing and it is often unclear which surrogate derivative should be chosen for a particular network. For a review focusing on surrogate gradient learning methods, which we are most interested in, see Neftei et al. [2019]. For comprehensive review of other learning methods for SNNs. we refer to Taherkhani et al. 2020). Tavanaci et al. [2019), and Eshraghian et al. [20T]]
The neural tangent kernel (NTK) introduced by Jacot et al. (0088] allows to formulate gradient descent a kernel method. Just as ANNS with randomly initia lzzd weights converge under certain conditions to Gaussian processes (GPs) in the infinite-width limit [Matthews et al.) 2018 Lee et al. 2018). the NTK converges at initialization and during training to deterministic kernel in the same limit. Moreover, the NTK then describes how the network function changes under gradient descent in the infinite-width regime. This led both better theoretical understanding of gradient descent and practical applications to neural network training; see Section 1.2 for more details.
1.1 Contribution
We study the NTK for networks with sign function as activation function. As the NTK theory is not directly applicable due to an ill-defined derivative, we consider the NTK for a sequence of activation functions that approximates the sign function and then derive a principled way of generalizing the NTK to gain more theoretical insight into surrogate gradient learning. Our contributions are as follows:
We provide a clear definition of the infinite-width limit in Section 2.2 capturing the different notions used in the literature due to the different choices of rates at which the layer widths increase. This definition is used consistently in all mathematical statements and the respective proofs.
In Section 2.3 we demonstrate that the direct extension of the NTK for the sign activation function using an approximating sequence is not well-defined due to the divergence of the kernel's s diagonal. This illustrates, from the NTK perspective, that gradient descent is ill-defined for activation functions with jumps and how this problem will be mitigated by surrogate gradient learning. Moreover, we connect this divergence to results by Radhakrishnan et al. [2023) in Theorem 2.3 and show that the direct extension of the NTK can be seen as a dell-nd kernel for classification.
We define generalized version of the NTK in Section 2.4 using so-called quasi-Jacobian matrices and prove the convergence to a deterministic, in general asymmetric, kernel in the infinite width limit at initialization in Theorem 2.4 Using the generalized NTK, we formulate surrogate gradient learning in terms of the generalized NTK for networks with differentiable activation functions. This novel NTK is named the SG-NTK and we prove its convergence deterministic kernel during training in Theorem 2.5
For both the diverging direct extension of the NTK and the SG-NTK with sign activation function and arbitrary surrogate derivative, we derive exact analytical expressions in sections D.I D.2 and
In particular, we identify the terms that emerge from SGL and that prevent divergence.
In Section 3 we illustrate our findings, in particular Theorem 2.4 and Theorem 2.5] using simu- lations. Numerical experiments show that the distribution of networks trained with SGL. shows agreement with the distribution given by the SG-NTK.

---

Mathematically precise versions of all statements can be found in the appendix, which is self- contained to ensure rigor and consistent notation throughout all theorems and proofs.
1.2 Related work
The neural tangent kernel. The convergence of randomly initialized ANNS in the infinite-width limit to Gaussian processes under appropriate scaling of the weights has first been described by Neal [996] for a single hidden layer and has been extended to multiple hidden layers and other
network architectures like CNNs [Matthews al., 2018 Garriga-Alona et al., 2018 Yang 2019b
Lee et al. 2018). The NTK was introduced by Jacot et al. [0118] with first results on its convergence
network initialization and during training. Theoretical results on the implication of the convergence
of NTKs on the behaviour of trained wide ANNs were given by Lee et al. (2199] Arora et al. 2019).
Allen-Zhu et al. 2019) In Section C.2
we review central theorems on the NTK to enable clear
comparison to our theoretical results.
The NTK has been generalized all kinds of ANN standard architectures Yang 2020 such as CNNs
Arora et al. [2019] and attention layers] Hron et al. [2020). In particular, it has been generalized to
al. [2020), which are particularly interesting in light of SNNs due their
shared temporal dimension. Bordelon and Pehlevan [2022 derive a generalization of the NTK called
effective NTK for different learning rules using an approach similar ours. Note that all of these
extensions require well-defined gradients.
The ability of overparameterizze neural networks converge during training and generalize be explained using the NTK [Allen-Zhu et al. [019 Bordelon et al. [2200 Bietti and Mairal [019 Du ett al.) 2019]. The NTK has also been used more applied areas such neural architecture search [Chen et al. 2021 and dataset distillation [gguyen et al. 2021]
Surrogate gradient learning. In the context of computational neuroscience, the idea of replacing the derivative of the output of a spiking neuron with a surrogate derivative was introduced by [Bohte| [2 To deal with the temporal component in SNNs more generally RNNs, the resulting gradient is usually combined with backpropagation through time (BPTT). Prominent examples of surrogate gradient approaches include SuperSpike by Zenke and Ganguli (018 and number of works with different surrogate derivatives [Wu et al. 201 8 1000sshhrestha and Orchard 2018 Bellec et al. 2018
Esser et al. 2016 Wožniak et al. 2020 In the general ANN literature, the method is better known as straight-through estimation (STE) and was introduced by Hinton 2012] and by Bengioo et al. [2013
in more detail. It was successfully applied by |uubara et al. [2016] and Cai et al. | [2017]]
Cni
Surrogate gradient learning or STE is only heuristically motivated and it is hence desirable to derive theoretical basis. The influence of different surrogate derivatives the method has been analyzed through systematic numerical simulations Zenke and Vogels (002]) revealing that the particular shape has a minor impact compared to the scale. In more theoretical work, Yin al. (2019) analyzed the convergence of STE for a Heaviside activation function with three different surrogate derivatives and found that the descent directions of the respective surrogate gradients reduce the population loss when the clipped ReLU function is used as surrogate derivative. Gygax and Zenke [2024) examine how SGL is connected smoothed probabilistic models (Bengio et al.) 2013 Neftei al., 2019 Jang et al.) 2019) and to stochastic automatic differentiation [Arya et al. 2022 In particular, they consider SGL for differentiable activation functions, as we do in our derivation of the SG-NTK.
2 Theoretical results
2.1 Notation and NTK parametrization
We consider multilayer perceptrons with so-called neural tangent kernel parametrization. For a network with depth L,, layer width ny for 0 S 1 S L, activation function a, weight matrices ( E -= and biases ((0) E "I, the preactivations with NTK parametrization are given by
1 (x) = Tn (1) x + or b(1)
/mo
(+1) (x) = ym
ow
w((+1) (1))(z) (x)) + 06 (((1)) for l = I,..., L -1,
3

---

where ow > 0, 2 0, and we initialize W 66 lid (O 1) for all i.j. The NTK parametrization differs from the standard parametrization by rescaling factor of 1/ynij in layer 1 + 1. The network function is then given by f(-) = h (.): R' - * and notably the last layer is a preactivation layer. We denote the total number of weights by P. More details can be found in Section C.1
By default, we will interpret any vector as a column vector, Le., we identify R" with R"x! This is the case even when writing (æ1 (#1....., Fn.) E R" for handier notation. Row vectors will be indicated within calculations using the transpose operator, I.. For function I: I + R and X
Notation (see Remark C.1 and C.2
........,..)) E Rd define the vector f(x) = (f(x1),... f(za)) E [ddm, For 722 1, denote the gradient of f by Vf(x) I E 18Th If m2 > 1, we denote the Jacobian matrix of f by Jf: I + R" xgn A subscript of the form fx) denotes Jacobian matrices with respect to subset of variables. We write f(n) g(n)
R for all
to denote that f() and g(n) are asymptotically proportional, i.e., f(n) ~ (g(n) for some constant K + O.
2.2 The infinite-width limit
We will consider neural networks the limit of infinitely many hidden layer neurons, i.e., fy + oo for all S ( S L 1. We will see later, when paraphrasing existing results from the literature, that different ways of taking the number of hidden neurons to infinity can be found. To formalize these notions, we use the definition of width function from Matthews et al. [2018) with slight modifications:
Definition 2.1 (Width function). For every layer O.... in that layer is given by r(mm), and we call r1: N- - N. the width function of layer I. We say that width function Ti is strictly increasing if r((m) ri(m + 1) for all m 1. We set
., L and any m N, the number of neurons
RL R== = (m))a rr is a strictly increasing width function for all < <I<
the set of collections of strictly increasing width functions for network depth L..
Every element of R., provides way to take the widths of the hidden layers to infinity by setting ny = (m) for any S < and considering m -> co. The notions of infinite-width limits used in the literature will now correspond to classes R C Ry for which the respective limiting statements hold. This is captured in the following definition.
Definition 2.2 (Types of infinite width limits). Consider a statement of the form "Let an ANN have depth L and network layer widths defined by no. nh, and == ri (m)) 1 < L and some (ri)==1 L-1 E RR. Then, fixed no and any n., the statement P holds as m + cc." " We also write the statement S as S(r).
(i) We say that such statement S holds strongly, if S(r) holds for any E Rr, This can be interpreted as requiring that the statement holds as mini (iinggre((m)) + co. We will also write "P holds n1,..., RLL-1 + oc strongly".
(ii) We say that such a statement S holds for (m)) X n. if S holds for all r E Rr with ri(m) & m for all S l < L. This means that S(r) holds for all r E Rr, such that co. We will also write "P holds (mI) (u)is X n."
Ty (m(m//r((m)) + ap.g E (0, oo) as
(ii) We say that such statement S holds weakly, if S holds for at least one r E R1. We will also write "P holds as connected to the sequential indinite-width limit.
+ 00 weakly". This type of wifth limit is tightly
Remark 2.1 (Connection between weak and sequential infinite-width limits). In the sequential infinite-width limit, meaning that n4 + "o,,,.,, + 30 sequentially, the layer widths are not strictly finite. This is opposed to applications, where layer widths may be large but finite. Hence, the sffnite-width limits using width functions as explained above are more meaningful in practice. For sequential limit lim +0 limng->o f(11, m22) f we can find functions nı nz(m) and (2(m)) such that litnm-soo [mn (m.) (2(m)) = f However, the rate at which nu (1()) 12(()) diverge as - x cannot generally be controlled. Since the weak infinite-widhh limit allows for arbitrary rates, any sequential infinite-wind width limit can hence be turned into a weak infinitewiddth limit as defined in Definition 2.2 (iii)

---

We use Definition 2.2 to paraphrase the convergence of ANNs to GPs in the infinite- widhh limit:
Theorem 2.1 (Theorem from Matthews et al. 2018). Any network function f of depth defined as in Section 2.7 with continuous activation function that satisfies the linear envelope property. i.e. there exist c, m 2 0 with [o{u)] & с + m|u] for all u E R. converges in distribution as
mi...., n
strongly to multidimensional Gaussian process (X = for any fixed count- able imput set (1=) It holds X3 id NOO ())) where the covariance function EXL) recursively given by
o (x,
(x, x²) + of (X =
oz. (I)
We also write E(L) E. The theorem states that the distribution of the network function, which is given by its randomly initialized weights, approaches the distribution of independent GPs as the hidden layer widths increase. Hence, large-width network will approximately be realization of the respective GPs. Note that this result can be generalized to non-continuous activation functions without well-defined derivatives that fulfil the linear envelope property, like (=)) sign(z). We provide a rigorous proof of this kind of generalization for the case 11,.... _L_- + oo weakly in Theorem E.3 Mg remains well-defined in this case since the expectation in Equation (1 does. When
approximating the sign function with scaled error function, i.c., (z) erfm (z) = erf(m 2), it holds that limm Derfm
Sggn due the dominated convergence theorem.
2.3 Direct extension of the neural tangent kernel in the infinitewwidh limit
We consider a dataset D = (X. )) with X = (z1...., Fa) E Rd and y = (y1...... Ya) E Rdd.. To solve the regression problem, i.e., find weights 0 E RR. such that fx;;;) y/ for all i= 1, d, we apply gradient descent in continuous time, also know as gradient flow, with learning rate 1j and loss function L: TRd- Rdazz -> R. This means that, using the chain rule, the learning rule can then be written as
=
V (((x))))
(2)
To derive the NTK, we observe that the network function then evolves according to
d a. (1; 0.) Jos(2; 0t) 0t [e] V V1xx = -n X ((x,0.) ((((2;0));
(3)
di
where we implicitly defined the empirical neural tangent kernel as 0(z,x') [e] which depends on the current weights functions during training are given by a kernel whose entries are the scalar products between the gradients of the network's output neuron activity, θ., x²) (Vof. 0), Vefs (ef ()): The key result on the NTK is that the empirical NTK converges in the infinite-width limit to a constant kernel, the analytic NTK, initialization, 0 = Oo and during training. 0 = 0,:
0,. This means that the learning dynamics of the network
Theorem 2.2 (Theorem from Jacot et al. (2018) for general ow > 0). For any network function of depth L defined as in Section 2.1 with Lipschitz continuous activation function a, ê =: b()) converges in probability to a constant kernel (1.)) @ In,, as fi,..., - oo weakly. This means that for all æ, E n and 1 S i.j S nL, it holds 6/7) (x,z²) + &, (x, x²) ) in probability, where diy denotes the Kronecker delta. We call (LL) the analytic neural tangent kernel of the network. which is recursively given by
((xxx') = (
(x,xx)) = (x) + ((.-1 (11) (I,x'))
where ()))) are defined as in Theorem
2.1 and we define
(1)) (xx, x') = -tt E E (ā(g(x)) (()())))))))((())))))) +
We also write (L) = θ,, If fg are the weights during gradient flow learning at time t 2 0 as before, Theorem 2.2 shows that (,()) (lL) 2 Ing for t 0 in the infinite-width limit. This reveals
5

---

that the gradients of the output neurons, Vafi(a; f), are mutually orthogonal. Under additional assumptions this convergence also holds for the whole training duration, t 0, see Theorem
oacot et al. [0188] for the case ..,,...: -1--11 - oo weakly, and Chapter G of of et al. [2019] for (m)) X n. Hence, the kernel that describes the learning dynamics stays constant in the infinite-width limit. This implies that the distribution of the network function during training also
converges to GPs [Lee et al. 2019. Theorem 2.2]. Then, any output neuron after training has mean
m(x) ()))(( (x. ()) (X, X) y under the assumption of mean squared error (MSE) loss, see Section C.2.1 The expression for the mean is equivalent to kernel regression with the NTK.
The above results show that gradient flow for networks with randomly initialized weights is character- ized by the analytic NTK ((4) in the infinite-width limit. Since we are interested in gradient flow for networks with activation functions inadequate for gradient descent training, we want to know whether the analytic NTK can be extended to such activation functions. We see that the derivative of the activation function does not have to be defined point-wise for Equation (4). In particular, activation d functions with distributional derivatives, e.g., de sign(2) = 28(z) can be taken into consideration, where 6 denotes the Dirac delta distribution. If we again approximate the sign function with scaled error functions erfm,. m. m. E N, it holds
(g(x)) eif eifm ml((())()))))))))))))))))) m=+00 (0
( (25(g(z)) 28((()))
(5)
in case the right-hand side exists. A rigorous analysis of this limit can be found in Section D.I Heuristically, simple observation suffices: if = y, we have one-dimensional integral over two delta distributions, which yields infinity. On the other hand, if x o and Esign is non-degenerate,
two-dimensional integral over two delta distributions yields finite value. We derive analytic expressions for m- Oeef... =: Osign in Lemma D.3 We call this kernel singular, because Osiga (I, x²) oo and O Oign (I, y) E R if x + y. By the same reasoning, this divergence occurs for any activation function with jumps. Note that for the mean given by kernel regression, this implies m(x;) = 1/s and m(x) 0 if is not training point. Intuitively, this is because the network is initialized with zero mean and different input points are uncorrelated under the singular kernel, so only the training points are learned. A limit kernel with this property also arises if the activation function is fixed but the depth of the network goes to infinity as was shown by RRabakrishhan et al.) 2023 We adopt the ideas of their proof to show that the sign of ma still useful for classification:
Theorem 2.3 (Inspired by Lemma 5 of Radhakrishhan et al. [2033]; see Theorem D.4j. Let o > 0 or let all 2; E L be pairwise non- non-paralle pon-paalllel. Let L 2 2 and x, E S for all i I,.... d. where Sno-1 C R is the sphere of radius R. Assuming that O (ar, X) # 0. almost all x E Sno-1 it holds
lim sign (2,) (2, 1)) = sign (1,,))) a.e. on S m +0c
22)
(L) The estimator ( (x, x)2 = Σm=
(xx, ( has the form of a so-called Nadaraya-Watso estimator and is well-defined for singular kernels such as (()), If we assume a classification problem in the sense that y S {-1, 13mh, it holds sign( ( (I., @66 (x;, XY at training point z;-
2.4 Generalization of the neural tangent kernel and application to surrogate gradient learning
The above singularity of the limit kernel can be avoided by considering surrogate gradient learning instead of gradient descent. First, we introduce a generalization of the NTK that later leads to the surrogate gradient NTK.
By definition and originally due to the chain rule, the empirical NTK consists of two Jacobian matrices of the network function with respect the weights. The Jacobian matrix can be thought of as recursive formula, f fz; 0) = G(z, 6; a, d), which is given by the input and the architecture of the network, including the activation function o and its derivative o. This formula can be modified to define a quasi-Jacobian matrix, J @0 (z; 0) == G(r, 01 ,1), where ōn does not have to be the derivative of σ1. Analogous to the definition of the empirical NTK we define the empirical generalized NTK be [(x,x')= = J (z; 8) 2a (z': (a2;))) B)T, where Ti 3i 02. 02 are specified in the following theorem.

---

Theorem 2,4 (Generalized version of Theorem 1 by Jacot et al. [2018); see Theorem E.4|. For activation functions ds. 0g and so-called surrogate derivatives 0T, 02 such that 01, 02 TI. and og satisfy the linear envelope property and are continuous except for finitely many jump points, denote the empirical generalized neural tangent kernel
[1 x') J(L),on. (x:0) (1 (d::)T for xF ER.
as before. Then, for any z, E To and 1 S i.j S nL, it holds f!l)
i9 (x,x'') Sx deg (g (x,x²) x'), as B1....,, 11-1 + Do weakly. We call ((L) the analytic generalized neural tangent kernel, which is recursively given by
1 (z, 2 (1, 1 1t (z, (z,z')
11. (z, x²) + _ (1-11)(x, (I,
11, (2.x2)) for L 22 with (x, x²) + o where
21, = o B 02 (22) of for L ニ and E1,2(1,
no
Blo (2) (22)) and (Z1, 22) ~ N (o, (0,))) (m,a)) (a,a") 2 (z
5(2) 1,2
Σn and Σg denote the covariance functions of the Gaussian processes that arise from network functions f1 with activation functions 01 d2 respectively, in the infinite-width limit. The covariance between these two Gaussian processes is denoted by 21,2. This covariance function asymmetric in the sense that Cov (ov[ff(=1) [2(2)]] + Cov (z2), f2(1))] in general.
We show in Theorem 2.4 that the generalized empirical NTK tends to generalized analytic NTK at initialization in an infinite wndth limit. Now, the SGL rule can be written using the generalized NTK, similar to Equation (3):
d di f(z,0t) =(o,f))=Jof((z Jof 01) _ 0 --= (x; ( v1xxx))
(6)
= i x) ( Here, we chose 01 02 = 0. NTK. This we call the surrogate gradient NTK (SG-NTK) with activation function a and surrogate derivative o. Compared to the classical NTK, one of the true Jacobian matrices is replaced by the quasi-Jacobian matrix, since the learning rule is SGL instead of gradient flow. Note that we assume that the derivative of the activation function, exists. Theorem 2.4 shows convergence at time 0. We extend this convergence to / >> O in the following theorem:
(7)
à and 82 = ā in the previous definition of the generalized
Let o, ,, ā as before, all Lipschitz continuous, and o, ā bounded. Let Ja be network function with depth L initialized as in Section 2. Rand trained with MSE loss and SGL with surrogate derivative ō. Assume that the generalized NTK converges in probability to the analytic generalized NTK of Theorem Furthermore, assume that the smallest and largest eigenvalue of the symmetrization of (( X). S(L) == (I(L) + [ (x) /2, are given by 0 < Amin & Xmax < oo and that the learning rate is given by 1) >> 0. Then, for any 8 > 0 there
Theorem 2.5 (Based on Theorem G.2 from Lee et al. (2019]; see Theorem E.5
2.4 I(L) + +I(L) & Logs (nu)t=) L-1
/TT/22
exist 0, N. E N and K >> 1 such that for every N, the following holds with probability at least 1 - 8 over random initialization:
6K
sup k0)0,00)
- (x,
<
where
||
|| denotes the Frobenius norm.
Amin
We also write (L) Io,a or simply I.. The explicit analytic expression of the SG-NTK is derived in Section E.2 for activation function erfm, E N and surrogate derivative ē = erf well
as general surrogate derivatives. Leetm. converges as m + co, because compared to Equation (5) we obtain Ejerf, ((())) [((()) + [z))(g(z)) (g(V)) and the delta distribution yields a finite
value. In Section E.2 we show this rigorously and derive the analytic expressions. Hence, we define
I Isgnna limm .r.
For any E N we can consider the SGL dynamics given by Equation 7 in the infinite-widne width limit to obtain
d
Ieet f2,ft) -n Teetmn (z, X3) V 5(x1)) (((xx))
dt
7

---

With MSE error, this equation is solved by a GP with mean m(x) = Terl (z, Ter for + oo and is natural to assume that the networks trained with SGL converge distribution to this GP in the infinite-widde width limit, analogous to the standard NTK case [Lee et al. 2019 Theorem
2.2]. However, the empirical counterpart to sign does not exist as we cannot formulate an empirical SG-NTK for the sign activation function due to the missing Jacobian matrix, compare Equation (6. We suggest that even in this case the network trained with SGL will converge in distribution to the GP given by Lign. since SGL with activation function erfm will approach SGL with activation function sign as m + oo and the GP given by lrf,.. will approach the GP given by Lign as m + oo. Numerical experiments in Section inndictee that this is indeed the case. We conclude that, remarkably, the SG-NTK can defined for network functions without well-defined gradients and is informative about their learning dynamics under SGL.
3 Numerical experiments
We numerically illustrate the divergence of the analytic NTK, Gerf., shown in Section
2.3 and the
convergence of the SG-NTK in the infinite-width limit, j(L) - ((t) at initialization and during training shown in Section 2.4 Simultaneously, we visualize the convergence of the analytic SG-NTK, Lerfm + sign- We consider a regression problem on the unit sphere S {o E R || 1} with | = 15 training points, which is shown in Figure B.1 and train 10 fully connected feedforward networks with two hidden layers, and activation function erf,, for t = 10000 time steps and with MSE loss. The NTK only depends
the dot product [Radhakrishnan et al. 2023] and thus the angle between its two arguments, Aa <(x, x'). Hence, we plot the NTKs as functions of this angle, where Aa 0 corresponds to x = ='.
In Figure D the empirical and analytic NTKs for the networks described above and trained with gradient descent are plotted for In addition, the analytic NTK for m + oo is plotted. Note that the steep slope of erfm for m = 20 results in erfm being very close to the sign function. For any m, observe that the empirical NTKs converge to the analytic NTK both at initialization and after training as NTK theory states. Figure B.3 illustrates this further by displaying the mean squared errors between the empirical NTKs and the respective analytic NTK. The convergence slows down for larger m. Further, the plots confirm that the analytic NTKs diverge m - do if and only if Aa 0. To show this more clearly, scaled the y-axis with the inverse hyperbolic sine (asinh), which approximately linear for small absolute values and logarithmic for large absolute values.
{2, 5, 20) and hidden layer widths E {10, 100, 500, 1000).
102
101
10
102
101
-10-
102
Analytic NTK Emp. NTK Emp. NTK at t=1e4
-10
-10a
-7
-0
77
-7
4a
Figure 1: We plot empirical and analytic NTKs of 10 networks for different hidden layer widths n and activation functions erfm- The kernels are plotted at initialization and after gradient descent training with t 1e4 time steps, learning rate 77 0.1, and MSE error. The y y-axis is asinh-scaled.
For Figure 2 we use the same setup as before, but train the networks using SGL with surrogate derivative = erf, and compare the empirical and analytic SG-NTKs instead of NTKs. We observe that the empirical SG-NTKs converge to the analytic SG-NTK as > oo both at initialization and after training in accordance with Theorem 2.4 and Theorem 2.5 Figure B.4 illustrates this further by displaying the mean squared errors between empirical SG-NTKs and respective analytic SG-NTK.
8

---

Moreover, we observe that the analytic SG-NTKs indeed converge to a finite limit m - 00, as shown in Section 2.4
100
100
5.00
1000
5
-5
Analyti SG-NTK Emp. Sipp.GG-MT SG-NTK at t=1ee
0
-5
-1
27 Aa
-2
-1
Figure 2: We plot empirical and analytic SG-NTKs of ten networks for different hidden layer widths and activation functions erfm. The kernels plotted at initialization and after surrogate gradient learning with 1e4 time steps, learning rate 1 0.1, MSE error, and surrogate derivative ā erf.
Finally, we consider SGL for networks with the same architecture and training objective before, but with sign activation function, which can be seen as the case m - oo of the setups above. We examine whether the distribution of network functions trained with SGL agrees with the distribution of the GP given by the limit kernel Lsign- Specifically, we compare 500 networks trained with SGL for t 30000 time steps, which represent the distribution of the network function after training, the mean and confidence band of the GP. The mean of the GP is given by kernel regression using the SG-NTK, m(x) sign (3., Iign (X, y and the confidence band is given by m(x) + (0)G(() where OGP (z) is the standard deviation at x. We observe in Figure 3a that the mean of the trained networks is close the GP's mean for network width confidence band. The mean of the networks differs from the kernel regression using the kernel Esign- Figure 3b shows that this agreement between SGL and the SG-NTK already exists for network width of 100, demonstrating that the SG-NTK predicts the SGL dynamics of networks with moderate width. Note that the variance the networks' output and the confidence band can be reduced (see Arora et al. 20199] and Section
500 and that networks lie within the
n=20
2.0
n=500
1.5
0.5
-2
-0.5
-1
(a)
(b)
Figure 3: Comparison of SGL learning in networks with different hidden layer widths with SG-NTK predictions. (a) 500 networks (blue) with sign activation function and hidden layer widths n 500 trained with SGL. using the surrogate derivative = erf for f = 3e4 time steps plotted together with their mean (cyan), the SG-NTK-GP's mean (black) and confidence band (grey), and the Esign kemel regression (dashed). Training points are indicated with crosses. (b) The mean ensembles 500 networks is plotted as (a) for different hidden layer widths.

---

4 Conclusion
Gradient descent training is not applicable to networks with sign activation function. In the present study, we have first shown that this is even true for the infinite-width limit in the sense that the NTK diverges singular kernel. We found that this singular kernel still has interesting properties and allows for classification, but it is unusable for regression.
We then studied SGL, which is applicable networks with sign activation function. We defined a generalized version of the NTK that can be applied to SGL and derived a novel SG-NTK. We proved that the convergence of the NTK in the infinite-width limit extends the SG-NTK, both at initialization and during training. Strikingly, we were able to derive an SG-NTK for the sign activation function, sign= by approximating the sign function with error functions. We suggest that this SG-NTK predicts the learning dynamics of SGL, and support this claim with heuristic arguments and numerical simulations.
limitation of our work is that due to the considered NTK framework, our results are naturally only applicable to sufficiently wide networks with random initialization. Further, we only prove the convergence of the SG-NTK during training for activation functions with well-defined derivatives. More rigorous analysis should be carried out on how the connection between SGL and the SG-NTK carries over to activation functions with jumps, as shown by our simulations.
Our derivation of the SG-NTK opens novel path towards addressing the many unanswered questions regarding the training of binary networks, in particular regarding the class of functions that SGL learns for wide networks and how that class differs for different activation functions and surrogate derivatives.
Acknowledgments and Disclosure of Funding
We thank Andreas Eberle for helpful discussions. We thank the German Federal Ministry of Education and Research (BMBF) for support via the Bernstein Network (Bernstein Award 2014, O1GQ1710).
References
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. arXiv preprint arXiv:2006. J0246, 2020.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterizaatonn In International Conference on Machine Learning, pages 242-252 PMLR. 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
Gaurav Arya, Moritz Schauer, Frank Schäfer, and Christopher Rackauckas. Automatic differentiation of programs with discrete randomness. Advances in Neural Information Processing Systems, 35: 10435-10447, 2022.
Guillaume Bellec, Darjan Salaj, Anand Subramoncy, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-leant in networks of spiking neurons. Advances in Neural Information Processing Systems, 31, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 308.v:1308.343:: 2013.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
Patrick Billingsley. Convergence of Probability Measures. John Wiley & Sons, 2nd edition, 1999.
Sander M Bohte. Erron-backproppggation in networks of fractionally predictive spiking neurons. In International Conference Artificial Neural Networks, pages 60-68. Springer, 2011.
10

---

Blake Bordelon and Cengiz Pehlevan. The influence of learning rule on representation dynamics in wide neural networks. In The Eleventh International Conference on Learning Representations. 2022.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.
Daniele Bracale, Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Large-width functional asymptotics for deep Gaussian neural networks. arXiv preprint arXiv:2102.1 10307, 2021.
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milmn and
Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github. com/ google/jax
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave Gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5918-5926, 2017.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. arXiv preprint arXiv:2102./15:: 2021.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in Neural Informa- tion Processing Systems, 22, 2009.
Giuseppe Da Prato. An Introduction to Infiiit-Dimmensional Analysis. Springer Science & Business Media, 2006.
Luc Devroye, Laszlo Györfi, and Adam Krzyžak. The Hilbert kernel regression estimate. Journal of Multivariate Analysis, 65(2):209 227, 1998.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks, In International conference on machine learning, pages 1675- 1685. PMLR, 2019.
Jason K Eshraghian, Max Ward, Emre Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mo- hammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2 109. / 2894, 2021.
Steven K. Esser, Paul A. Merolla, John Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Armon Amir, Brian Taba, Myron D. Flickner, and Dharmendra Modha. Convolutional networks for fast, energy- energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 27:201 27:201604850, 2016
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks shallow Gaussian processes. arXiv preprint arXiv:1808.0 05588, 2018.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. John Hopkins University Press, London, 3rd edition, 1996.
Julia Gygax and Friedemann Zenke. Elucidating the theoretical underpinnings of surrogate gradient learning in spiking neural networks. arXiv preprint 4rXiv:2404. 2024.
Insu Han, Amir Zandieh, Jaehoon Lee, Roman Novak, Lechao Xiao, and Amin Karbasi. Fast neural kernel embeddings for general activations. In Advances in Neural Information Processing Systems, 2022. URL httpt:ttttttthtttte com/ google/ wwgoogl//eeuralllooer http//euullllannraaaa
Herbert Heyer. Structural Aspects in the Theory of Probability, volume 8. World Scientific, 2009.
Geoffrey E Hinton. Coursera video lectures: Neural networks for machine learning. https
7/ //www.ce cs. torontro www.tooooooooooooioooo wwww"hinton/ wwwrrsrerreectteeeeaaaea html 2012. Online; accessed 25
July 2023,

---

Jørgen Hoffmann-Jorgeen Jörgensennnn and Gilles Pisier. The law of large numbers and the central limit theorem in Banach spaces. The Annals of Probability, pages 587-599, 1976.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, 2020. URL
atttp:://gituue www.coo//ooooo http///ww...ooooogg/////////e//e toongg/-eoorrrt-taettttt
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Hyeryung Jang, Osvaldo Simeone, Brian Gardner, and Andre Gruning. An introduction to probabilis- tic spiking neural networks: Probabilistic models, learning rules, and applications. IEEE Signal Processing Magazine, 36(6):64-77, 2019.
Olav Kallenberg. Foundations of Modern Probability, volume 3. Springer, 2021.
Annika Lang and Christoph Schwab. Isotropic Gaussian random fields on the sphere: regularity, fast simulation and stochastic partial differential equations. The Annals of Applied Probability, 25(6): 3047-3094, 2015, ISSN 10505164.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: Isoperimetry and Processes, volume 23, Springer Science & Business Media, 1991.
Jaehoon Lee, Jascha Sohl-Dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks Gaussian processes. In International Conference on Learning Representations, 2018. URt.tttts://///// htttss://opeereiieeee nett/forumidd--EEA--- htstss//ppeeerrrreeeeeee//oootta
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems. 32, 2019.
Chaoyue Liu, Libin Zhu, and Misha Belkin, On the linearity of large non-linear models: when and why the tangent kernel constant. Advances in Neural Information Processing Systems, 33: 15954-15964, 2020.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: 1804.1121,, 2018.
Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, al. A million spiking- neuron integrated circuit with a scalable communication network and interface. Science, 345 (6197):668 673, 2014.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer New York, NY, 1 edition, 1996.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-basee based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-8))) 2019.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5198,
2021.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in Python. In International Conference on Learning Representations, 2020. URL htt.sstss: ///githubb com/ google/nee maalel/neural.. tangentss
12

---

Roman Novak, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Fast finite width neural tangent kernel. In International Conference on Machine Leariing 2022. URL httpp:///gtthutt com/
google/neural. -alaaltttee tangents
Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: Opportunities and challenges. Frontiers in neuroscience, 12:774, 2018. ISSN 1662-4548. doi: 0..3389fninn.2018..000
Constantine Pozrikidis. An Introduction to Grids, Graphs, and Nenworks. Oxford University Press, 2014.
Yu V Prokhorov. Convergence random processes and limit theorems in probability theory. Theory of Probability & Its Applications, 1(2):157-214, 1956.
Yuming Qin. Integral and Discrete Inequalities and Their Applications. Volume I: Linear Inequalities. Springer, 2016.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Wide and deep neural networks achieve consistency for classification. Proceedings of the National Academy of Sciences, 120(14)) c2208779120, 2023.
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386, 1958.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards based machine intelligence with neuromorphic computing. Nature, 5757777))77777 Nov 2019, ISSN 1476-4687. doi: 10.1008/411586-011--6777. URL httpt: org/0/ 01118//11566001----111
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by
mack-propagating errors. Nature, 323(6088):533-536 1986.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in Neural Information Processing Systems, 31, 2018.
Jascha Sohl-Dickstein, Roman Novak, Samuel S. Schoenholz, and Jachoon Lee. On the infinite width limit of neural networks with standard parameterization, 2020. URL https://github. com/
google/neurall tangents
Aboozar Taherkhani, Ammar Belatreche, Yuhua Li, Georgina Cosma, Liam P Maguire, and T Martin McGinnity. A review of learning biologically plausible spiking neural networks. Neural Networks, 122:253-272 2020. doi: 10. 10.10066).neunet... www.eett.1.....000.....
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural Networks, 111:476 2019.
Aad W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998. doi: 10.1017/CB09780511 11002256.
Christopher Williams. Computing with infinite networks. Advances in Neural Information Processing Systems, 9, 1996.
David Williams. Probability with Martingales. Cambridge University Press, 1991.
Stanislaw Wožniak, Angeliki Pantazi, Thomas Bohnstingl, and Evangelos Eleftheriou. Deep learning incorporating biologically inspired neural dynamics and in-memory computing. Nature Machine Intelligence, 2(6):325-336 2020.
Yujie Wu, Lei Deng. Guoqi Li. Jun Zhu, and Luping Shi. training nigh-peefforance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
backpropagation for
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01):1311-1318, 2019.
13

---

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 1rXiv:192..04999 942.04760,, 2019a.
Greg Yang. Wide feedforward recurrent neural networks of any architecture are Gaussian processes. Advances in Neural Information Processing Systems, 32, 2019b.
Greg Yang. Tensor programs Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14544 2020.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing serraighttthruug estimator in training activation quantized neural nets. arXiv preprint AXXiv:9003.0666 2019.
B. A. 7alesskii, V. Sazonov, and V. Ulyanov. A precise estimate of the rate of convergence in the central limit theorem in Hilbert space. Shormik: Mathematics, 68(2):453-44 482, 1991.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation, 30()):514-1541, 2018.
Friedemann Zenke and Tim P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):899-925, 2021.
14

---

A Additional remarks on the numerical experiments
Weight initialization and implementation. All networks are initialized with ow 1, oz 0.1
For the implementation of the NTK and SG-NTK use the JAX package [Bradbury et 2018
and Neural Tangents package [Novak et al., 2020 2022 Han et al. 2022 Sohh-Diccstein et al. 2020
Hron et al. 2020] with modifications. Computations were done using Intel Core 17-1355U CPU
and 16 GB RAM. The simulations for Figure and Figure took two hours each. The simulations
for Figure 3 took 12 hours.
Variance reduction trick. The variance in the outputs of the networks trained with gradient descent or SGL and the confidence band given the NTK or SG-NTK respectively can be reduced by multiplying the weights of the last layer with constant K < 1 at initialization [Arora et al. (019)]. This is explained in detail at the end of Section C.2.1 We can see from Figure th2 the agreement between the distribution of the trained networks and the distribution given by the SG-NTK still holds; however, network width and training time have to be increased.
B Additional figures
2.0
-0.5
-1.0
Figure B.1: Target function and training points for the numerical experiments, f(x,y) 4xy²
0. + 1.2y² 0.8x2y.
3
2
1
-1
Figure B.2: Network functions of 100 networks with sign activation function and hidden layer widths y 500 trained with SGL using the surrogate derivative erf for / 2e4 time steps plotted together with the SG-NTK-GP's mean and confidence band. The parameters of the last layer have been multiplied with к 0.2 at initialization.
15

---

10
10
10
MSE at t=0
10-2
at t=1e4
10a
102
10a
10a
102
10a
10a
102
10a
hidden layer widths n
Figure B.3: Mean squared errors between empirical NTKs and analytic NTKs in Figure 1 for all 10 networks (thin lines) and averaged over the networks (thick lines).
m
20
10a
10
10
10
MSE at t=0
10
at t=1e4
101
102
10a
10a
102
10
10A
102
10
hidden layer widths n
Figure B.4: Mean squared errors between empirical SG-NTKs and analytic SG-NTKs in Figure for for all 10 networks (thin lines) and averaged over the networks (thick lines).
C Neural tangent kernel theory
C.1 Introduction of the neural tangent kernel
We begin by defining an artificial neural network with a parameterization suitable for considering the limit of infinitely many hidden neurons. This parameterization is called neural tangent kernel parameterization and differs from standard parameterization of multilayer perceptrons by rescaling factor of 1/vTi in layer / + 1, where is the width of layer I. We follow the slightly more general definition in [Lee et al. 2019 Equation (1)). A discussion of this kind of parameterization can be
found in (Jacot et al. 2018 Remark 1].
Definition C.1 (Artificial neural network with NTK parameterization) Let T, be the depth of the network, n& for O,... , the widths of the layers, and a: IR. - an activation function. draw network weight matrices W()) E Roixn- and biases (()) Th for 1 1....... L from a probability
N 1). For the parameters, we denote by g1)) (W 6.0)) the parameters of layer I and by g(1:
distribution such that w!!)) b ünd
(0()) g(()) ..... f the parameters of layers up and including l. Given some ow > 0 and σ 2 we then define for all x E Ro
h(t) (z; (1;001))
Te (1) 6(1)
Vno
((+1) (I;0(1:(11)) Vni
Ow
w(+1) (1 (2; t0 for / I,..., L
16

---

Therefore, (0:00: 0) is a map from R to R" and we use the short-hand notation (2; (x; Finally, we define our network function
{( h (::6): TR + R
With this definition, the total number of parameters, P= 1 is
P
Σ mm(mi1+ 11.
Given such a network function with NTK parameterization, we consider a dataset D_ = (x'3) with X' (Z1....,II) E R. and y (y1..... 3d) Rd First, we want to solve the regression problem, i.e., find parameters 0' such that fax; = ys for all i = 1, ..... d. Later, we will also consider the classification problem, i.e., we assume y/ (-1,1} and want to solve sign([(r;) 0¹)) = y/ for all i = ......d d. Tackling both cases from the regression perspective, we define the so-called loss functional and loss function. The following two definitions are similarly formulated by Jacot et al. [2018).
Definition C.2 (Loss functional and loss function). Let == Lg g: 1 + 1 and L: F - R so-called loss functional. In addition, let & be convex, i.e., all X E [o, 1] and 9 92 E F holds
2(g + ()-2) S (R(or) + (1 A)2()
We will assume that 2 can be written as
1 S d Σ ((f(x;)): =: (((()))))
so that the so-called loss function C(::)); Rd + R is differentiable.
Remark C.1 (Function evaluation at sets and vector notation). We want to detail the notation used in Definition C.2 For a function f: I + T and X (T)) id define the vector
s(x) = (J(r), " ((( E Rd
By default, we will interpret any vector as a column vector, i.e., we identify R" with R=x1, This is the case even when writing x (21...., Inn) IK"" for handier notation. Row vectors will be indicated within calculations using the transpose operator, xT.
Let us first consider regular gradient descent learning continuous time, also known as gradient flow. For this, we assume that our network function is differentiable with respect to its parameters.
Remark C.2 (Gradient and Jacobian matrix notation). Let f: IR + TRm For m 1, we denote the gradient by fx)) E R. for all x E R If > we denote the Jacobian matrix of f by f I + Rmxxn Therefore, fz)z is m matrix for all z E R". We do not always want to consider the gradient or Jacobian matrix with respect to all variables. We indicate this with subscripts follows. Let f: Rn x .R + I and 9u (0) = f(x; 0) for fixed x E R2, Then, write Vef 0) = (6) and f 0) = (f). In particular, for a map f R. - R. the gradient with respect to the j j-th variable, j P. is scalar and denoted by 0,f(t) f(). This is called the partial derivative of f with respect to the j-th variable.
With this notation, we consider the gradient flow method with learning rate 1j > and recall the derivation of the neural tangent kernel. We move the weights in the opposite direction of the gradient of the loss function with respect to the parameters of the network evaluated at the training points:
dt 0, V
(S8)
1 d =-n d
Jef 0z) ((f
17

---

using the chain rule for the second equality and with AT denoting the transpose of a matrix A. Again using the chain rule, this then implies for any z E R
d di [ 0.) [as 0₂)
d _ 0z
(S9)
di
d
7 Σ
(S10)
We therefore define the neural tangent kernel as follows:
Definition C.3 (Neural tangent kernel), Let f be network function of depth L as in Definition C. with parameters 0, not necessarity drawn randomly. Then, we define the neural tangent kernel (NTK)
as:
(
R x R + R
(I,y)
Jaf((;
Therefore. it holds for all z. Ro and 1 i S n
p Σ (w) p=1
(z,
Notice that the NTK depends the parameters of the networks. is therefore initialized randomly and varies over the course of the training. With notation (x) f(x; 0.) and 0.(²) 6 for
parameters 0, at training time / we can now rewrite Equations (S9) and (STO) as follows:
d = -n
dt -
(I,
XVV ((x
(S11)
1 =ー Σ
6 ( (z, x4
V (
We are hence able express the change of the network function during training in kernel fash- ion. Later, we will consider this change of the network function in the infinite-width limit, i.e.,
11....., -1 + 0o.
Before doing so, we will generalize the NTK definition in order to apply the NTK to surrogate gradient learning later. In particular, we will break the symmetry of the above definition and generalize the Jacobian matrices to quasi-Jacobian matrices by replacing the derivatives of the activation function by surrogate derivatives. Let us write the recursive formula of the Jacobian matrix of the network function given by the chain rule as [a; f) G( o x;; f), where à is the derivative of the activation function σ. Then, surrogate gradient learning replaces Jo ff (a: 8) with G(o; ō; 8) for the surrogate derivative ē of the activation function a.. We call this the quasi-Jacobins matrix:
Definition C.4 JausiaJacobiaan matrices for neural networks). Let L be the depth of the network, ny for k = 0, ...... the width of the layers, σ: R R the activation function, and ō: R. + R the so-called surrogate derivative of the activation function. Let f be the network function, h(), [.IIadd and 0 the network parameters. then define the Jacohan matrix J(L) of I at point x recursively as follows:
1 = 1,..., I, 1, the intermediate layers as in Definition C.I
ifO w((1)
j()) (z; (2:8(1)
E RR" x
with
((1) 3k0 (2:0(1)=
=10
/no"
(S12)
Sbs ob
ifp b(4
j(0 ((:))
E TRIIi|0
for 2 515 To with
dzi vii-i (n:-)( 0))
if f0, w:O (1)
k,, (x;8(2:1))
_!
dzi Tb
f00, (1)
Ow W--
EEE wi(1) & (n
(z: 8)) 5/ (z; (1;0(11))
ifO, E (
(S13)
18

---

Remark C.3 (Notations for the Juasi-Jacobian). With the above definition of the quasi-Jacobian matrix of the network function f with activation functiooo a and surrogate derivative we write
J(L), (x; = =j(() (x; (12))
It then holds
j(L) (x; = (a]
For data set X instead of a single point x, concatenate the matrices row-wise before, namely J E Rd X|0
Definition C,5 (The generalized neural tangent kernel), Let 0T, 02 be activation functions and ō 02 the surrogate derivatives respectively. Given network depth I, and parameters 0 we define the generalized neural langent kernel as:
1(7); R x R + 1 X
(S14)
(v.v) + j(L), (x:0) o( O),
Remark C.4. The generalized neural tangent kernel agrees with the neural tangent kernel in the case where 01 02 and ā 01 02
C.2 Notation for the infinite-width limit and review of key theorems for the NTK
In this section we will formulate all important theorems on the NTK that will need for our later analysis using the introduced notation. Furthermore, we will discuss and remark their proofs, in particular in view of the generalizations that will be proved Section E.T
Convergence of networks to Gaussian processes in the infinite-width limit. We will consider neural networks in the limit of infinitely many hidden layer neurons. The fact that such networks converge to Gaussian processes was first mentioned by Neal (1996]. We follow and present the formulations of Matthews et [2018] for the general mathematical statement. First, we formalize the limit of infinitely many hidden neurons.
Definition C.6 (Width function, as in Definition Matthew et al. [2018] with modifications). For every layer 1 O...., L and any E N, the number of neurons that layer is given by ri(m). and we call ry: N- + the width function of layer We say that a width function ri is strictly increasing f
r(m) < rr(m 1) for all ' 1. We set
1)
R₁, = (rr)t=]) Im is a strictly increasing width function for all < < L}.
the set of collections of strictly increasing width functions for network depth T..
Every element of Rr provides way to take the widths of the hidden layers to infinity by setting no and nt, to some constant, setting ny = rr(m) for any 1 < < T. and considering m - co. To formally define for which ways of taking the widths of hidden layers to infinity a statement holds, now state the set R C R2 such that the statement holds for widths given by any m + co. Clearly, the claim "The statement holds for all r E RR." is stronger than "The statement holds for all r E Rg." if R2 C R, On the basis of these considerations, we define three types of infinite-widee limits using the previous definition in order to structure the different types of limits in this thesis well as in the literature.
R as
Definition C.7 (Types of infinite-wide limits). Consider statement S of the form "Let ANN have depth and network layer widths defined by na, ny and n4 == ri (m) for S L and some (ri)t= E RR, Then, for fixed ng and any ni, the statement P holds as m + co. " We also write the statement S as S(r).
(i) We say that such a statement holds strongly, if S()) holds for any r E Ry, This can be interpreted as requiring that the statement holds as minggier(m)) + co. We will also write "p holds as Tı. , + op strongly".
TiL-1
(ii) We say that such a statement S holds for (MI) X n,, S holds for all r Rr with re(m) X mffor all 1 < 1 < L. In other words S(r) holds for all E Rr, such that ry fm(m)/r/mm) - ap.g E (0, 00) as m - sa. We will also write "P holds as (ni) & n"
19

---

(iii) We say that such a statement S holds weakty. if S holds for at least one r E RL. This can be read as requiring that the statement holds as n4 + . - 30 sequentially, We will also write "'P holds ..,...:: .RI.11 - Do weakly".
functinfunnciin f of depth L defined Theorem C.1 (Theorem 4 from Mathhews et al. (2018), ). Any network function as in Definition C.7 with continuous activation function that satisfies the linear envelope property, Le., there exist C, m 2 0 with
(a(u)) <c+1 + m/ul Vu IR,
converges in distribution as , - - oo strongly to a multidimensional Gaussian process (X:)3=1 for any fixed countable input set (1: (2=)) It holds X3 id NO, E(4) where the covariance function ( is recursively given by
(x,,x' = 0e (xr, x²) + ob.
n10
(x, o R R v)(o(g())) (g((')))] + of
Remark C.5. First, proof of the above theorem can be found in the paper of Matthews al. (2078]. While it takes a lot of effort to show that the statement holds strongly in the sense of Definition C.7 the weak version of the statement can be proved via induction. This has been done by Jacot al. (2018) and we will later adapt their proof to show a generalized version, Theorem E.3
Second, in the context of analyzing the nefwork behavior, we are interested in the fmite-dimensiooal distributions first of all. since neural networks are trained and tested on a finite number of data points. From the convergence of the marginal distributions, can infer the convergence to an stochastic process via the Kolmogorov extension theorem. However, this assumes the product o o-algebra, which is why Theorem C.I assumes fixed countable input set. Matthews et al. (2018) have discussed these formal restrictions in more detail (Chapter 2.2). If one does not want to be restricted to a countable index set, could, for example, consider the condition by prokhorov [1956] Theorem 1. A similar approach was laken by Bracale et al. 2021), I. which applied the Kolmogorov- Cheentssvvvv criterion (Kallenberg 2021 Theorem 4.23).
Finally, note that the theorem assumes continuity of the activation function. the proof of Mathhess et al. [2018] this is only used in order to apply the continuous mapping theorem. However, it is sufficient for the limiting process to attain possible points of discontinuity with probability zero for the continuous mapping theorem to be applicable. The theorem is thus also valid for activation functions that are continuous except at finitely many jump points, such as step-like activation functions.
Convergence of the NTK at initialization in the infinite-widthe limit. Jacot et al. [2018) showed that the previously defined empirical NTK converges to a deterministic limit, which we will call the analytic NTK.
Theorem C.2 (Theorem from Jacot et al. [2018) slightly generalized), For any network func- tion of depth L defined as in Definition C.1 with Lipschitz continuous activation function o, the empirical neural tangent kernel (T)) converges in probability to a constant kernel (I.)) @ Ing. as nL-1 + Do weakly. For all x E =n and 1 < i.j S n,, it holds
_ dig ( (x,
P
(1.
which we also write as
S(L) P (()) & Inn.
We call (L) the analytic neural tangent kernel of the network, which is recursively given by
(14))) -5(() (x, x²) ((1-1)) (1,)),
where 5 are defined as in Theorem C.1 and we define
501) x') o E B S0) (( ((g(t'))
Compared to Theorem Jacot et al. [2018) the statement slightly generalized in the sense that allows for arbitrary ow > 0. The arguments in the proof work the same way.
20

---

Remark C.6 (Versions of Theorem C.2 in the literature). A proof of this theorem for (mi) 8 given by Yang (2019a) and his proof is also referenced by Lee al.) (2019). However, the proof given in terms of so-called tensor programs and therefore harder to follow. For the ReLU activation function, proof for 01,..., 11111 - oo strongly is provided by Arora et al. (2019 Theorem 3.
We will later prove a version of this theorem for the generalized NTK.
Convergence of the NTK during training in the infinite-width limit. Not only does the NTK converge to a constant kernel in the infinite-width limit, even the kernel during training, 6(() converges to this constant kernel. This was also discovered by Jacot et al. (2018).
Theorem C.3 (Theorem by Jacot et al. (2018). Assume any network function of depth L. defined as in Definition C.7 with Lipschitz continuous activation function a, twice differentiable with bounded second derivative, and trained with gradient flow as Equation STO Let T > O such that
J
dt dt=/ :t=f =f"" ((0
dt Op(1),
where X E Op(1) denotes that X is stochastically bounded. Then, as P1...,,LL-1 weakly, the empirical NTK 6 converges in probability to the analytic NTK @(L) @ Ing in probability uniformly for & [0, T.. We therefore write
P 6 0( @ Inc.
the literature). The proof of Jacot et al. 2018 relies heavily on function space perspective. Since this formulation tends to lack mathematical rigor, we will rely on the proof of the theorem for the case (ft) & n. given by Lee et al. 22019 Chapter G]. In particular, the first inequality of (S51) in Theorem G.2 of Lee et al. [2079] implies the condition (+). Furthermore, a different approach to proving the above statement for the case (mt) X n using the Hessian matrix of the network function was taken by Liu et al. (2020) Proposition 2.3, Theorem 3.2]. A partial proof of a version of this theorem for the generalized NTK will be given later. Only an auxiliary lemma remains be proved.
Remark C.7 (Versions of Theorem
C.2.1 Gradient flow in the infinite-width limit
Given the results of the previous section, we can formulate an infinite width version of Equation (S1O) by replacing the empirical with the analytic NTK. This allows us to analyze the learning dynamics of networks in the infinite- width limit, which yields connections to kernel methods and reproducing kernel Hilbert spaces. We then discuss how far the resulting functions and solutions in the infinite-wide witth limit deviate from the finite width networks. This is essential to evaluate to what extend the results in the infinite-widd limit can inform us about the behavior of gradient flow the finite-width networks. First, we state the infinite-widnh version of Equation (S11) using Theorem
di [((x) = -n (o 8 1md) (x) V ((((()))))
1
Σ
( (x, In, V((f((
= =- d
i=1
d
= - Σ ( (x, .1) (((S(())))))
(S15)
=n (z, x)= [((f((())))) ... v((f((ra):
= -704 (z. X) V (((x)
(S16)
where in the last line we interpret V (xx (X)):)) E Rdxms as a matrix of size d n. with
1/d (f.()((z)); v.). Recall that ()(()(())) is the partial derivative of E(:: with respect to its j-th entry, i.e., with respect to (1;) Note that the last line row vector, which we can identify as column vector. The fact that the NTK now time- iime-independent and non-random has two interesting implications:
entries (V
21

---

Equation (S16) is now an differential equation that can be solved explicitly or numerically for certain loss functions.
According to Equation (S15). the time derivative of (2:) can now be expressed element- wise lincar combination of functions of the type (( (-)))
Z): I + R. For arbitrary symmetric and positive definite kernel k(', ,, -), the completion the linear span of functions
of this type is called the reproducing kernel Hilbert space (RKHS) of k. Assuming that the
solution of Equation (S15) is element of the RKHS of ((4), one can ask what the space looks like.
The ODE of Equation (S16) has already been considered by Jacot et al. 20188 Chapter 5] and Lee al. [2019] Chapter 2.2]. and we will follow the observations made there. To do this, we will assume the mean squared error (MSE) loss
1 - 2
()
y. where f((x') and y are again interpreted as matrices of dimension d x nr. This gives us the following ODE
implying V
d dt f((x) - ( (x, X) (fi(Z) 3/)
Now, for simplicity, we denote Ozx. == (1l)(,, v) and θ = (XX, Furthermore, we consider an arbitrary set of test points Xr. The solution the ODE is then given by
fe((() = p((x)) + (e(rr for Mx(xT) = (( (1s-c y and
(xrrr fo(())) O(X,, x)O- e fo(
Recall that by Theorem C.1 the components fu,, are independent and identically distributed Gaussian processes with mean zero and covariance funetion = (L), Hence, "tt has mean zero and the mean of ft is given by f4. By looking at the components of ft.
(rr( fo., (Xr) X(x,,x)6 (Ia - ela-e-mey ( (fr - )1y)
we can conclude that they are independent and identically distributed as well. One can show that the components are indeed Gaussian processes again with mean Me. Using % can also compute the covariance matrix for our arbitrary set of test points Xr.
= (ff,, I (r = E = (o, (XT))
E [fo,)()r) (Ia- - - -
E [)( e fo. (2)
+ +E(0(E7 ( (2. (X) (Ia- o -0
-(( - X(xr; (I4 -e ) 0 0(X,
X00 (Ia - e E(X,dT) + + x)e (Ia e I(X, X) (La- θ --
Assuming that is positive definite immediately leads pointwise convergence of the mean and covariance functions. This implies that the gradient flow solution for networks in the infinite-width limit converges to Gaussian process as - oc with mean function yooo and covariance function To given below. This follows from the weak convergence of the fnite-dimensional marginal distributions by Lévy's convergence theorem [Williams 1991. Section 18.1]. Again, the discussions of Remark applies. We have
Mox (Xr) = 0(Xr X)0 and Tho (26, E(Kr, Xx) E(Z,, X0,,, -,
(S17)
O X)0 X')00-00--- '''' Xr) -1
(S18)
Lee et al. 2019 state that network trained with gradient flow will indeed converge distribution to this Gaussian process the width goes infinity:
22

---

Theorem C.4 (Theorem 2.2 from
2019)). Let the learning rate Meriticll for Lee et al.
Meritical 2mm (E (1) (X, X)) Amax ( (X,
with network function ft as in Theorem C.3 with hidden layer widths nı =... RL-1 = and restricted to E R with ||=|| S. 1. If A (O( (X,X)) > O, then the components of ft converge in distribution to independent, identically distributed Gaussian processes N(H,, as + so for
all t E co) U foo}.
Hence, the result of training a wiith network with gradient flow for an infinite amount of time will be arbitrarily close in distribution to Gaussian process with mean function Hoo and covariance function r so. if the width sufficiently large. Note that by Equations (S17) and (S18) the variance at the training points X' is zero and the mean at the training points is exactly )
Since we will focus on the mean, we will first sketch trick introduced in Chapter 3 of et al. [2019) to make the variance term arbitrarily small. If fo had mean variance, this would consequently also be the case for all fr and for the solution foe lim ft. This can be achieved by multiplying fo by a small constant K > 0 and considering the network function Do kfo instead. It then holds
(g(r, y) Jogo(:; (x; Jogo (y; 0)T fols Jo 0)T =<= (z,
and thus we have θ x² O In the infthite-width limit, the derivative of ge is then given by
d
dt 01(x) = nn 0, x3) (0) -)
= = (x) y),
which implies before
(x( -u(x)) (2,,, (1o-e-00,))) (9o(2)) -3)
= =0,(,,
(12 -e----m" 01xx,10)
(Ia- - qhs ()(2))))) ) fo(2)).
Note that the term in the second last line corresponds the non-random mean of f trained with learning rate mpr?, and that the term in the last line is random, but can be made arbitrarily small using K. We can think of this as a trade-off between learning rate and variance. This justifies why we can focus on the mean in the next section.
To sum up, we are interested in network functions in the infinite-width limit that are trained over time according to
d
di (n (A(x)) - - = Σ 0 xi) (-n (fe(z;) y:)),
(S19)
where we change from a row vector to a column vector in the last equation. The mean of such network functions after infinite training time is given by
(NTK(x) == O(z, ()0((, = Moo (x:).
(S20)
D The NTK for sign activation function
The first observation to make in our attempt to apply the neural tangent kernel to networks with the sign function activation function is that the sign function has zero derivative almost everywhere. Thus, the derivative of the network function with respect to the network weights is zero for all weights that are not part of the last layer. The case where the weights ((:It-1) are frozen after initialization and only (LL) trained has already been discussed by Lee et al. [2019] Chapter 2.3.1 and Chapter D]. For network in the infinite-width limit, this approach is equivalent to applying Gaussian process
regression, i.e., knowing that f ~ N (0, E(4) for infinite width, one considers f f(X)) = y. This
can be seen by realizing that (4) (L) if o almost everywhere and applying Theorem C.4
23

---

While this is an interesting observation, and the strategy of optimizing only the last layer can also be transferred to finite width networks, we would prefer train the whole network and not identify the derivative of the sign function with zero, since this discards all information about the jump discontinuities in our networks. An obvious alternative would be to use the distributional derivative of the sign function, which is given by 2 28g. where dg denotes the delta distribution. We will see that ()) still exists when the distributional derivative is substituted into its formula. Alternatively, we can obtain the same expression by approximating the sign function with scaled error functions,
2 V Jo
m-
erf (2) erf (m 2)
dt,
and considering the limit m + co.
D.1 The NTK for error activation function
Due to the previous considerations, we begin by deriving the analytic NTK for the error function. Following the notation of Lee et al. [019]. we need find analytic expressions for the terms
(Z) = -xx. cx
and
I' (E) (Z) = (ox eif, (Y)).
Note that by a change of variables we can alternatively consider the terms
T(m². Σ) ((x)))>w//m==>>)/((((((>
= Tm (E) and
T(m² Σ) = E(X Bxx
= m Tm (E).
For E' 2- (.)) T(E') and ((E') are given in Chapter C of the supplementary material of Lee al. 2019]. However, we cannot assume that E' always has this form. While T can be easily calculated, T is harder to deal with and a reference to Williams [1996] Chapter 3.1] is used. There, the main idea of the proof, how to evaluate more general expression, given without further details. We will derive analytic expressions for both terms explicitly.
We start by evaluating T. Note that
2m -m??² VT
erf(2)= _ and
erf (2)
d=
V
dz
Lemma D.1. Given U ~ (00, with invertible covariance matrix E E Raxd and x, y E R4, it holds
Bleif(UT<) eif((Ty)] [l)i((((aaee(((UT))))== = ((1 20++ )
4
-1/2
(S21)
In particular, given (X,Y) ~ NO0, Σ) with invertible covariance matrix E E 2x2 or with X = Y and singular covariance matrix Σ 222 it holds
4 T(E) = E(ei((X) ei)))()))))))))))))))
-l12 21-11
(S22)
π
where A denotes the determinant of a matrix A.
Proof. It holds for U (O Σ) with covariance matrix Σ BRdx and x, y E Rd:
(e =
e ((TT)))
2 c (uTT)) e -1L du
Ra [
4
1
uT + 2/ 1/2 + 1/ /2 do
π JRd (2m)d/2
exp
2
1/2
r.. (I+ 25
TI12
1/2 1/2 yy yy T5//)
112 1/2
=
Ia
+ 2Σ 1/2 1zz T 1/2 22 1/2 y 1/2
24

---

using a change of variable E¹ for Equation (+) and using basic properties of the determinant.
We can evaluate the determinant the last line by applying the Sylvester's determinant theorem
[Pozrikidis] 2014, (B.1.16) i.e., In, + ABT Im. + 1T A| for any matrices A, B E R=x". We
then define A = B (v221/2x, 2 /2) E IRdx2 which yields
y)
Ia + 25 12 Er Σ 2
yy Σ
LL + ABTI
(
11+2x
2x ><x 2xy
2x
+ BTA|
1 + 2y Ey
This directly implies Equation (S21
Furthermore, Equation (S22) follows with E E R and
(0) (2) or with z ()) and arbitrary invertible covariance matrix E' such that
En = E11.
Corollary D.1. Given (X NO, Σ) with invertible covariance matrix E, it holds
Tm Ejerfm (X) erf (Y)
2
Σ (2m
1/2 m0000
2 Σ 1/2
(S23)
(XX,Y) ~ ~1 with X = and singular covariance matrix D E g2x2, it holds
I (2))
120000
Proof.
D.I 4m
2m2 (2mm122/(m²)) E)) -1/2
(2) = m²² T(m? -) Lomma
11 + 2m 221211111
2 Σ - 15+12/(2m) 12/m(mm))
1/2 m-000
(
if / invertible, -1/2
π
singular.
Remark D.1. mentioned at the beginning of this chapter, we can get the same result by considering the distributional derivative of the sign fanction, 28.. It holds for (X,Y) ~ (0, 2) with invertible covariance matrix E as before
/
1
2 | 1 200(22) - dz
(l(o(X)) 6o(Y))
JK2 2 20g(21)200(2)
π
In the of X Y, the integral is no longer well-defined.
Next, we consider T by solving a more general problem, which was formulated in slightly less general form by Williams [1996] Chapter 3.1].
Lemma D.2. Given U ~ (0, E) with invertible covariance matrix E E Rdxd and x, y E R&, it holds
2 2x
V == ==e((([(UTT> V= vU) = arcsin
2I vI+2y
In particular, given (X,Y) ~ (0, E) with invertible covariance matrix D ( Ea
E gxx2 or
Eg
with X = and singular covariance matrix Σ
(2) Fa Ez
2x22 Σ₁ Eg Ez, it holds
2 2
[lerf(X)
aresin
VI+2> v 1
Proof. We follow the proof idea given by Williams 1996. Chapter 3.1], that is, we define V(A), differentiate the expectation, and integrate by parts. We can then see that d. which gives the desired aresin. So, define
TX V(A) = (1 (1- 1/2 dy
dx
((A) = EAerf(A- UUx) (r)(r))( S
erf(A - (rf((A-Ta)errt (T))er((uT))) -uu du,
(2/1//211/1/2 121//
25

---

and then differentiate with respect to Ả on both sides
((A)
/ /
L
VT e
X²
xuT
erf(uTy) y) - du
yp
(2
20/2
121//
T
Jmu (2Th) 20/2 (2
2u
erl(u y) E - du
|2/1//2
V
1/1
Σ 1/2
1 (-20 (la 2A -== 1) du crf(uTE)/23) exp (-1
erf(
VI
d/2 (2/) 1//1/2
2/
(1a + 2x
(T(22)d/22
1 - (1=2222)((x v) du
xx erf(vT21/2g)) y) (ld + 22221/2 /2
exp
2r1E1/2
(la + 1/2 xx 1-
(T((2/)d/2
1 TT (-207
La 2/251//2221100 / u) V cr)( do,
x exp
(S24)
using change of variables u / in Equation (+) and using partial integration and Gauss' divergence theorem in the last equation. In addition, we used that
V E 1 (1
(Ia + and the partial integration rule for scalar functions. To be precise, for differentiable scalar functions f and g it holds
V(S g) = Vg VS. which then implies the partial integration rule. Furthermore, the left-hand side vanishes our case due to Gauss' divergence theorem:
(erf(uTE1/2)) e /2 20))) du
V.
Rd
erf(o (1 de
lim
R
(d-1(R)
e -o (1,t223'1/" du
< lim
R-+00
5á-1
lim S c R Amin 2m
0,
where (R) is the surface area of the sphere in Rd with radius R. Continuing with previous calculations, we see that
1/2
S24
Vi(2m)dd/2 1/22 (L.
22
xr 1-111
x
e 1UT (1+2222'"
1112
(12) /
du
J2d
(Ia 1 22 122
e
de.
(2m d/2
Rd
(225)
We evaluate the expression outside of the integral in the last line by applying the Sherman- Sherman-Morrison
Woodbury formula. For A E ddxd and W1, W2 gd it holds (oolu and Van Loan 1996
(2.4.1)]
C F.wwwwe.. WWCCC
(C + (C+ C-1
1 + w C-1wi
26

---

For C' = pa and w = wi = w2 = /2AE//2
/2x this yields
wwoT 2/2
(Ia + 22
(Ia + uw la
1 wTw
1 222x1Ex
With this we see that
2X2E1/200T
x x /2 (Ia + + xx x
= =I/1//2 12
Id
E1
1 2>2xTEx
Eyy (1
2A²xT Ex
ITEy
Ey 1 212x22 Ex
1 2x2
1 2x
Inserting this into Equation S25 we obtain
π 2x2xrT Ex Jg. (2x)a/2 E = 11/3)00 de
2 2x Ey
$25
T.y
-1 1/2 22 xx + / yy
π 1 + 2x²xTEr
2 2x Tx
1/2
Ia + 2
1 11/2 + /2 yy yy
π + 222
As in the proof of Lemma D.I, we evaluate this using Sylvester's 's determinant theorem. With the same notation as before, we can define A = B = (V2A5 /2 E RRdx2 This yields
La + 2/221/1/221 xx 11/2 + // yy / 1/2 = Ia + ABT = + ATA
(1
22 2yx
2xr
= +1
4X (ZT
1 2yT
If we insert this this again, we have so far shown
d
V(
n # 222 ((1 21 2x
2 Ey
2y Eny - -1/2
4x7 1/2
(S26)
We now define
2Ax
7(A) v(1 2X2x 122 2yyT
and claim that
2 (1 (A)9)
dd (()) = d
d
yd v()).
(S27)
We can find a solution to the claimed equation by finding function V' that satisfies
d v(((2)) π (1))(()))-11)))
dy(A)
and by setting V(A) = (A)))) This follows from the chain rule. V is thus simply given by
((x(A)) = 2 ar)))i(>(A)).. In particular, this yields
2 aresin
2x Ey
V = V(1 -()))(((()))))
v(1 + 2r Er + 2y Ey)
It is now left to show Equation S27) using Equation (22)). First, that
(2 1y)²
-1/2
(1 (1- 1122 1/2
(1 2Ax Exe 22 En)
(1 + Txx) + 2y TEy)
1/2
2ArTEx 2y En) 422
27

---

Second, it holds
A(1 + (11+)\2=Ex)-112 = dp + 2x +1~2+2x
-x) 1/2 = (x- + 2x T Ex) 3/22
yd
= =1+ 21+222x Ex) 3/2
(V1+ + 212xTEx) 3
With the results of both calculations, we get
2 (1 (1- -1/2 1/2 d
2 - (1-7(1)3) 112 1/2
2x
AA (2)
A(1 + )1+
vI + 2yT2 Yd
π
VI + 2A 2y
2x ZxTDy
1
((1 + 2xx Exe Eny 4x2 (xt ZyTEy
(VI+ 212rT22)
3
2x
526
V(A),
(11 + 2xx EyyT 4X 4
2x²xTEr
Yp
which yields Equation S27) and concludes the proof. The special case E 2 with invertible
covariance matrix or X and singular covariance matrix follows as in the proof of Lemma
D.I
Corollary D.2. (XX)) ~ 2) with invertible covariance matrix Σ =
E1 Eg
2x or Eg E2
with X = Y and singular covariance matrix E
RR
2x2
ΣS = Σg = Eg. holds
Eg E2
Σg
Σ3 VI
Tm 2) = arcsin
m=+00
aresin
(S28)
1
Σ2
2m2
2mm
Proof. It holds
2m 2m? Lig
Tm (2) T ))
Lemma
aresin
π
v1 + ~ /1 + 2m
2
m-+20 2
Lg
aresin
aresin
VEivE2>
m
1 2m
+ Σn
2m + Eg
π
We can now use Corollary D.1 and Corollary D.2 to evaluate Σ(L), (2L) and (L) for activation function erfm. which denote by x(L) s2(2) and (L)) respectively. We then are interested in the limit m - 2o. First recall that
(1) (x, x') = (z, x¹) + of
and
n1
(x, = - B (on. +of,
by Theorem C.2 Therefore, x(1) is independent of m, and it holds for any x, y E me
o²
(0) (z. y) = (x. v) + o0 = Σ (x, v).
no
28

---

Assuming that the limit oom Em (x, (x, = I'6i (I, exists and that D6e (2, * 0,
>66) (y.y) + O, we can define
(x y) σ E
mm
(x,))
ferlm (X) erf (V) of = - Tm ((ndda)) + oB
D3 20 w
{(L)) (I,
Cor,
aresin
oB
1 + I(()) (I,,2) V 2 +-mi (v.
2m2
m- 20 aresin
E0e, (x, 11
+ tof
rob (y) v)
D0 (x,, x)
=: 2 E((+))
(z, y).
(S29)
Hence, it follows via induction and from the continuity of the arcsin function that limm >/ (x. y) = >i (a, y) is well defined. We discuss the resulting kernel m in Re-
mark D.3 For 2m +1(+1) L 2.2 1. it holds
( (z, 3) σ² E l (r mi
eerfm (X) ex)m o Fm (I(A))
D 20
1/2
20(L) m
+
12
2m
20
((I))(1, x)) +
(x)
1
) 2m
1
E/h 1)
-1/2
2m²
(S30)
As we will see later, the limit
lim {-(1) ( 3) = Σ(( (x. y)
(S31)
m-000
exists for x + y apart from few exceptions. In the case x = y. we obtain
20
1 2
-1/2
S(L+11 (z, x))
x(4) m
(x,
+
E(L) (I, 2)²
π
2m
20
-1/2
20 2
-1/2
(2())
(x, + 4m
m
x(L]
(I, x) +
π
m
4m
20
m
(L) (x,
1/2 - m=+00 oC,
(S32)
π
where denotes asymptotic equality. Therefore, for x # y. the limit
lim O (2, v) 0 ! (v.
m=++00
exists. However, due to Equation (S32) the NTK diverges for = y as m + co. We will call
kernel with this property a singular kernel. We also say that a kernel with this property singular along the diagonal.
Remark D.2 (Distributional neural tangent kernel). An alternative conceivable approuch to obtain the same kernel ( would have been to consider the distributional Jacobian matrix of the network function with step-like activation function. Whether or not distributional NTK can be formulated is a question for further research. Here, we only want to point out that the corresponding formulas for the recursive definition of the analytical distributional NTK would then naturally read as follows,
(x. x²)= = =1() (x,a²)
((L) (z, x2) -1(ا (z, x ((L-))(2,z))) (1,(), for L 2 2,
where
x(Lb (x, x²) 2² E -' (s))n(((x)) sign( of for L 2 2,
(S33)
$111 (x,xx')) 2 E ( - (2))((())) 200()())) for L 2.
(S34)
Note that Equation (S4) can be derived from Remark D.I and that Equation S33 is also easy to show.
29

---

We now want to explore the implications of our findings in this section for fNIK, which we introduced the end of Section C.2 Equation (S20) yields
lim NNK (2) lim 2 (() m0-200
Nik (z)
m 00
Note that Om (X, x) = (2, is invertible for sufficiently large m, as the matrix will be dominated by the diagonal for large m. Using this and the fact (m, 2) , O (2, X) if de for all ,,.,,,, d, we obtain for such x::
(r) = (m, (2
m=
However, I 24 X and denoting the i-th basic vector by e Rd we get:
(a;) Om (zt, o (X' () = e yi Therefore, [Nix converges pointwise to function that is zero almost everywhere, but interpolates exactly at the data points. The singular kernel o%, which we can see as the analytic NTK for the sign function, is for that reason not suitable for regression. We will deal with this problem in the following section.
D.2 From singular kernel to Nadaraya- Hedaraya-Waason estimator
Radhakrishnan et al. [2023] considered the limit of infinite depth, lim @(L), and a singular
kernel emerged similar to the previous section. It was shown that the resulting estimator using this singular kernel behaves like a Nadaraya- Watson estimator when classification tasks are considered Wattroo
instead of regression tasks. We will adapt the ideas of Radhakrishhan et al. [2023] to show similar results. To consider a classification task, we assume that E {-, The network is then trained
with data points (X before. but apply the sign function at the end to obtain the final classifier. As Radhakrishnan et al. 2023]. , we assume that we are operating in the infinite-width limit after infinite training. So, we interested in
lim sign (0m(z; x)) ()))
(S35)
We will see that the resulting classifier is equal to
d
sign (0 (2,, x)D) sign
O (x,, Σ
The form of this classifier is similar Nadaraya- Watson estimator. To be precise, for singular kernel i and data points (X, )), the Nadaraya- Watson estimator is defined as
cxx) = Σ (x,
for all x + xy 1 siisa
Ei,,k0 xx)
Since k(x, + x0 as x - Fy, holds that c(x) + y as x + Fz. Thus, the continuous extension of c to Fe0 interpolates the data points.
We now begin to further evaluate (L) and ())) analyze Equation S35
) (x,.)
(5., y) ob.
(S36)
n4
(L) (Ir, x))
$29
20
a1) + of 0² + +o3
for all
I, 2 2,
(S37)
d² no
2x(2) (I, y)
$29
20 u
aresin
(z. v)
+ +00 and
(S38)
c ||ا|| + of
| 2 +
ا(ااع (z, 4)
21(+1) (I, (1,++)((,,))
S29
20
arcsin
+ 0
π
b(6i) (z. x²) v -اااا (y. v)
837
20 s aresin
الاي (x, y) 2² + o
+ t for all L> 2.
(S39)
30

---

Regarding the assumptions made for Equation S29 note that that Σ (I, + 0f I. 2 2 and
> (x, + if + 0 or of > O.
After Equation (S31 we mentioned exceptions to the existence of E(6 (I, v) for # that were postponed at that time. For ) with æ * y we can see that
(2)) m (xr y)
930
20
2
|x:1 + of +
|/yl
+ +o + 2m? ) -
o (x. y) + of
2
-1/2
2m?
nu
20 2
0
- 1122
2
[url|/|yyll 1 (ar, (2,,))²)
+
(1|=112 + |1,/12 - -2(0r, v)) + 2m?
Am
π
n1
-1/2
m=0000
20
(or, w))
oi
| - yl
= => (cr, y),
(S40)
no
no
assuming that either of > 0 or that x, y not parallel, i.e., (,r, w) + ||0
The term - Am is
given by
Am
((or
||||| |1r|1 +
c no
+o²)
- +
ش
|1y| + oo
+
1
m00 0.
2m
2m2
no
2m2
For L 2 3 the expression simplifies:
$(L (x, y)
830
20
((
2m² ) (x((~"1(.1))
1
(g,
1
1-((-1) (I,
(x,, (1,,))2)
-1/2
2m2²
π
20
(5D
(x,, Σ((-11) (g,,) (x, v)))
1
Bm
-1/2
π
2m2
20
m-+00
(x((--1) (I,, r) M([--1) 1) (v. v) D1b-1) (x.
(z. w)²)
937
20
(1o + o -1(6--6610 I(L- (x, ((),))))))))))))) -1/2 = (x, (z.
(S41)
π
with the term 2m Bm given by
Bm
1) 1) (xx, x) + 2o(h111) 8) +
1
O.
2m
2m
2mn²
From Equation
$39 and Equation
S41 we see that
d
E(1) (Ix = -5(0)) (xr, v).
I 11 (z, v)
Although we did not use dual activation functions to denote the NTK as (Jacot et al. 2018 Section A.4], we still find that the property (ô)' (a') applies, i.e., the derivative of the dual activation function is the dual of the derivative of the activation function. Here & denotes the dual activation function of o.
In the case x= = y we get
-1/2
(2) (I, x)
in
832
202
m
o²
| + of
and using (S37
(S42)
110
( (z,x)
in
S32
20 m
m
(o"
+ oī)) 112
for L 2 3.
(S43)
We summarize the above calculations in the following lemma:
31

---

Lemma D.3. For m. L E N let ) and D(LL) be as in Theorem C.2 then holds for any x + y with z, y + 0:
for activation function erf It
2 (z. = lim 20 (x, y)
536
Iu (x, v) + to. 2
m-000
710
E (x.y) v)
lim x22 (x. y)
538
20
aresin
Lu 12g
(a,, v)
+ oz
+ o
m 00
02 no
|
ob
au no
|1y12 + o6
(x.)) v)
lim 20th (I. v)
539
20
arcsin
(5,
+ +ot for all
L 2 3.
m-000
o + +03
2 (x. x)= = lim x(D) (I,
9:37
+ o
for all
L 1 2,
m0
and, assuming that 2, y are not parallel or that of >
|| y112) -
20
(1, = lim
112) (I. v)
(|,b|||||a|
(x.,)))
m-+00
π
no
20
E (z, v),))) -
( (z,
E
y. y))
π
20 m S41
-
()(L) (x, = lim )
(r,y)
(oth + o - x1(-1) (xx,
for all L2 3,
m-000
π
20 2
2
-
$02 (x)
$42
w
|
of
m
π
ngg
510)) (1,,0,,))) (x) x)
543
20 w m (o + o8)
1
for L 2 3.
Remark D.3 (Addendum to Remark C.5 In Remark C.5 we discussed the topology of the space of the Gaussian processes to which ANNs with continuous activation functions converge in the infinite-width limit. The product a-algebra restricts us to a countable input set, so il is not possible to check for properties such as continuity or even differentiabiliry. While Theorem C.4 is stated only for continuous activation functions with linear envelope property, we will see in Theorem E.3 that the convergence also holds in the (weak) infinite-width limit even for step-like activation functions. For the sign function as activation function, the covariance function of this Gaussian process is then given by E Since we know explicitly what this covariance function looks like, we can examine the sample-continuity of the process. Note that to get the full picture, one would still have to show functional convergence of the network this process, as Bracale et al. [2027] have done.
E!D) isotropic when restricted to sphere, as will be discussed in the next section. In the case of isotropic covariance functions, k(z, y) = k[[[æ= yl), the simplest way to show comple-contiuuity using the Baannogoro--hentttee criterion is to show Lipschitz continuity of at zero. This can be seen from the proof of Lemma 4.3 by Lang and Schwab) (chwab)(2033) In our case, the covariance function basically given by a composition of arcsin functions. Lipschitz continuity of at zero is therefore equivalent to Lipschitz continuity of the aresin function at 1, which does not hold. In conclusion, it is not possible to show sample-continne of the Gaussian process given by x!e in the established way using the Kolmogorov-Chene Chentsov criterion.
Clearly, the kernel can be rewritten in terms of the arccos function. Cho and Saul 2009 analyzed arc-cosine kernels in the context of deep learning in detail.
Corollary D.3. For m. L E N let (L) in Theorem C.2 for activation funciiooo erfm If x 4 y and either x, y not parallel or of > O, then the limit
(L) (z.
lim o0 (I,y) m mo
exists. Furthermore, it holds, asymptotically + oo,
((L) m (I,x) ~
20 _ + oτ²
| 2
2202 π
(o E + o?)
-1
L-2 mb-1 for L 2z 2.
π n0
32

---

Proof. The first statement directly follows from Lemma definition can be resolved to the following formula:
D.3
and the definition of Om (2)
The recursive
I.
I.
(1) m (I, v)
=
(k)
(a,
y)
s m
(z, v).
k=1
1=
With
20
- -== + ob
-
K(z) =
2
n40
we get from Lemma D.3 that 5:(2)
(z, 2) K (1<+13²) .m and Eim (2,x) x) (m for L 3. This
implies II
L-1 (+++ (xx x) ~ [[| K 4-2 m b-1 m and M TI--111 +((+1) (x. ~ K(no)
|=1 2-m
m lonk for any k 2 2. For L 2 2, we get that
Mz (2, K(no) m 1 (2,x) 2 (z, y) K (1|=112 K K(no) (n=))=-2 mm--1 Σ
k=2
20
-
-
||=1|
+
0
|1:|| 2 + of
K(no) 1
110
π
n0
20 2
20 2
-5
1-1
| 2 +
ob
(o
+ o3)
m 1-2
77u
Theorem D.4 (Inspired by Lemma 5 of Radhakrishhan et al. 2023 ) Let of > 0 or ler all x; E R
be puirwise non-parallel. Let L 2 and x, E Sn Sm for all i 1..... d, where Sn Spa-1 C R the sphere of radius R. Then, with
(1) = lim sign (2)
X)0 (X,
and assuming that @ (x. XYy + for almost all x E Sh Sho-1, it holds
L x sign (0% (1, (0w)(,, x))) a.e. on Sno-1
Proof. First note that almost all x E S are not parallel any x, X. We denote @ Om m E N U {oo}, for convenience. Let > 0 be a positive constant that we will choose later. It then holds for almost all E S
(X,
sign ( (z. x)0,'y) sign (am (00 (z, x)))
sign
(2, - (1, x)> +
(a,
(z. x)) (r, (x, -
=:Am
B,
+ +
We now show that Am and Bm go to zero as m - xc for a suitable choice of an- First, note that
Bml < ( (x, e (z,
(or, x3|| ||'||2 m- 0,
since ||1|| < oc and by Corollary D.3 it holds that 2 (x, 2Tx) - O (xx mi)) for all xa E X as
m - cc. Second, we have
|Am| < ((!) (z, X')
o x))
o (z,
17'1 - 10% (z, 23)| amm Lallz 17'll (S44)
33

---

Again by Corollary D.3 it holds that 110m (2t 22 + 10% (cc, 2)12 as + xc. Furthermore, for all S v we have 0 (xi, 4i) ~ ((R) m --1 for some constant ((R) which depends
(L)
on R ||, Since it holds Om (zJ, 0 24) as + for any i o J. choose CR)) -mf-1 and conclude
[1 if
1o if
This implies am m. + la as m + oc. In particular, Uam m with respect to bounded. We get
m E NJ is a compact set {am @ m E N) is
1ll. Since D + D-1 continuous function, {Ia]
am 9m -dlll2 - am - 12 La am 2 1mml2 m00 0,
and thus
A
| (z, (x) x) aam 0 La la 1131|l m-+00, 0.
Recall that @ (r, xY o O, which now concludes the proof:
lim sign (x, ()) = lim sign (Am + Bm + + (x, (x, m-000
m=+00
sign 0 (r,
Remark D.4. One can generalize the above theorem by dropping the restriction to SR Sma-1 By Lemma D.3 it holds Onl) (I;, ri)
|1l|2 + of C. mm--1, For some constant C independent of L
and ||=:|]- we now define a... = C m
m
1-11, we wet
am 1 m (2, 100000 diag
(VoilaeiP/n0
=D,
isisa
where diag(u} of vector a square matrix with diagonal As in the proof above, and using the continuity of the inverse map D + D-1, this implies
lim sign 000
(0 m
(x, (),x)0 >)) sign (e,
sign (x, (x,)
(x,,
if we define
(x, = (ou + o,) 1/22 (o2/lal12//w + a.) 11/2 ( v)-
1/2
In conclusion, dropping the restriction leads similar form for our classifier, but with modified kernel.
Theorem
D.4 shows that the classifier
lim sign
lim sign (f)I
has an easily interpretable form that is close to the Tadaraya-Watson estimator with singular kernel ( This is despite the fact that the pointwise limit of fKTK is trivial, i.e., regression is no longer possible
D.3 Checking for Bayes optimality
In addition the ideas used in the above section, Radhakrishnan al. (023 proved that the resulting estimator is Bayes optimal under certain conditions. This property is also referred to as consistency. Let the probability distribution P our data space be given random variable
(X,Y) E no x 1- and let
c(x)
arg max IP = 5 X- =x)
5E(-1.1)
be the Bayes classifier with respect to this distribution. Denote Xa (xI, vra) and V4
(y)...., ..,,.),)))) yd) for (xi,, y:) drawn independently from Pa Paata- We then define Bayes optimality as follows:
34

---

Definition D.1 (Bayes optimality). Let cal ) = cal : Ka.Y) be classifiers for d E N and estimators of ce ). We then say that (Ca)aen is Bayes optimal, if it is a consistent estimator of c, i.e., for all & > 0 and X X-almost all x.
lim Pllca(2) c(x) > =0
d
First, we summarize the results of Radhakrishnan et al. (2023) and consider ((²). If the singular limiting kernel for - oc behaves like singular kernel of the form k(x,y) = |2 ee-yl "" yyll"* then the classifier for L + oo will be of the form
E /o/ || x=ll"
sign
1 1/ |2 2::1
assuming 1/ xz 0. This classifier satisfies Bayes optimality for a no by Devroye
et all. 1998). Radhakrishnan et al. [0233 generalized the results of Devroye et al. [1998]. expressed
terms of the activation function and its derivative, and chose them in such a way as to achieve
Rg. Going back to our setup, we want to see if we can write
lim ( v) R( x vl) | y/
for some constant a and for some function R: R. - R. bounded away from zero as || - y O. We start by proving that 0 (z, v) = G[I yll) for some function G. Recall that we have to restrict ourselves to a sphere by Theorem D.4 For simplicity, we restrict ourselves to the unit sphere S Then, it holds (z, z) = 1 for all r E S which gives us
| y| 2 (xr u y) = (x) 2(x, + (v.v) (x1-(x,,)) v))-
In the following, we will substitute = = (x, v). Taking ||x- - -1 0 then equivalent to taking
2 + 1. We can conclude from Lemma D.3 that we can indeed write Σ(6) (ar, y) = 5 (z). 5-(1] (x, v) 5:60( and hence @55 (2r, v) @ for all L 2 1. Note that Eo (1) = +of
for L 2 2 and D' (1) = / + of. Next, we consider 5:(2) (z) for + 1 using Lemma D.3
20
o1 (1
L
-
(12) (z)
2² +
2(1 - -2)
π
ng
nlu
20 2
20 2 w
(1 - 21 + 2) +
(1 (1=2)
=
ng
n20
20
20
20 (1 2)
20
200
102
+ oo ))
(1
- 2) +
(1 2)-
n
ng
70
no
110
20
1/
o0
2
(1 =)-b b
(S45)
π
no
For L 2 3, we can observe that
2
(o w + 2 --((6-1( (x)²) -
(x)
π
2x
((02) + to
- - ==)) (ot + +o + xo0=))
π
20 w
2
(2 + of)) 1 (00 + + - x(6-1)²)) -
111
(S46)
We will now prove a lemma that allows us to analyze the behavior of >i6 (z) as +
Lemma D.5. It holds:
(1 - 1/2
π
lim
1 2 arcsin (z)
2v2
35

---

Proof. Since the numerator and the denominator both go to zero as 2 - 1, we apply I' I'Hôpital's 's rule and differentiate both. This yields
dz - 2)² =
and
d
2 1 --
arcsin (2)
--2
2 (1 21-
d=
Thus.
(1 2)1/2
2 zv +2
π
π
lim
lim 2-+1
2-1 2 aresin (z)
2 V1 -
2V
concluding the proof.
For L. =3 it now holds
20 (2 (o 10 +
of)) - (lo2 + ooā - -2 (z) -
5 (z)
π
20 w
2
(2 (o² m
+ +o?))
v 1
(
2
aresin
o 02zz/no + o / nno + 08
π
π
(+) 20
(2 (0% + +oā))
π
o z2 + o
π
V
22
Lte //o + oo
20 w
(2 (a w
+ +0))) - ~
π
0 /o/noo (1-2) =))
π
V
222
0 0a//no nn + +o²
-1
2 w
(2/2 (o + +o?)) -
(1 2)
0 /2//no no + o
where we used Lemma D.5 at (+). For L 2 4 it holds
20 10 (2 (0" + ) - (o- + o,?) - 10 -))) -
(z)
S46
L-123
20 w (2 (0" + +o+))
-1
1 -
2
aresin
20 2) (z)
π
oh o
20 10
(2 (o w
+ of))
- v0
π
1
I66 (1 (2)
π
2v2
2 + of
20
(o oō) (o + +0² - 12 (s)) -
(2 (o + of))
1 V
π
22/2
S46
20
(2 (o" + +)))
0 1
(o + +ož) /5(6-11
π
2/2
v 1)(1-11 1) (z)/2,
again using Lemma D.5 at (+). This implies for arbitrary L 2
D6 (z) ~ K(L) (1 (1- (1--)-11/1 -1
for some constant K(L) depending on L. Recall the formula for the NTK, that was used in the proof of Corollary D.3
L T-
L.
T-1
(z) =
(x) ~
Σ
(1))
(z)
II
K(I + 1) - (1 - 2) 1/22
00
1=1
k=1
1=
xXll (1)
K(I 1): (1 - 1/22- = K(L) (1 (1-111
[=1
K'L)) || - vll 12-1/2/-3) = (Z) 1| yl -a())
36

---

for some constants (L),, (L). is therefore to possible find function R that is bounded away from zero near || 15--y yll 0, such that
R(I vll) | y [L
@ (z. (L)
However, we have that (<1)) = and (LL)) + as I. co. So it is only possible to choose such that (T) No. if no = 1. But this is trivial case, since we have restricted ourselves the unit sphere. In conclusion,
cannot prove that e(L) is Bayes optimal classifier for any choice of L. Chapter 4 of Devroye et al. [1998] suggests that, in fact, the estimator will not be universally consistent.
E
The NTK for surrogate gradient learning
In this chapter we explore surrogate gradient learning, introduced in Section I by connecting it to the NTK. Recall that the standard gradient flow dynamics of the parameters given by
0:
S8
As mentioned several times before, the Jacobian matrix Jø will vanish for all parameters except those in the last layer if we consider the sign activation function due to its zero derivative. The idea of surrogate gradient learning is to circumvent the zero derivative of the sign function by replacing the derivative with a surrogate derivative (Neftci et al.) 2019). We can replace the derivative two main ways:
The activation function in the full network can be replaced by a differentiable surrogate activation function. In particular, obtain non-vanishing surrogate derivative of the activation function and thus non-vanishing network gradients. Let denote the network with surrogate activation function. Therefore, can train the weights according to dt
0, = - Jog(X 0.) o(x,,, L(x 0:);
consider the loss with respect and train according to
d 0, Jog(x 0.) [((x))
(S47)
dt
Instead of replacing the activation function, we can only replace the derivative of the activation function ā with surrogate derivative o in Equation (S8). Let Jo,a be the quasi- Jacobian matrix as in Definition C.4 with activation function o and surrogate derivative Then the training is given by
d
nn Jo. (N) ((x,,)
dt
d
fr; 0₂)
(S48)
dt
-o (X 8.) (2. (
0,):
- X: 0 1xx _ [(I(
(S49)
with f(L) as in Definition C.5 for 01 = a, σ = o 02 = a and 3. = 5. For Equations S48) and (S49) we assume that ē exists and is vanishing, but we deliberately train with a surrogate gradient. For example, we can again consider erf, as the activation function. Its derivative explodes at zero and vanishes everywhere clse m + The hope is that
limtyn... 1 ((L) (( exists and lim (1()) not a singular kernel as before.
In this chapter we will deal with the second approach, because Ja (x; 02) = G(a; o; x:; f2) is closer to Jo](x; 0,) = G(a; Ø;; 0.) than Jag(2; 0.) = G(n; ō; x;; 0,) as formula if Jo [=; 0,) exists and is non-vanishing. Here, denotes the surrogate activation function with derivative ā. To do this, we provide asymmetric generalizations of Theorem C.T Theorem C.2 and Theorem
These
generalizations would, in principle, even allow to compare the two approaches.
37

---

E.1 Asymmetric generalization of the neural tangent kernel
For this section we adopt the so-called linear envelope property from [Mathhww al. 2018 Definition 1] to ensure that all expectations exist in the following theorems:
m. c2.c20 O VUER: [o(u) S c m//u
(S50)
First, we consider networks under the weak infinite-widd limit, and are thus interested in taking the number of hidden neurons to infinity sequentially. This done inductively while using the central limit theorem and the weak law of large numbers. In order to do this rigorously, we state and prove two lemmata.
The first lemma is stated in terms of Gaussian measures on Hilbert spaces, An introduction to Gaussian measures on Hilbert spaces can be found in Chapter 1 of Da Prato [2006]. A rigorous derivation and definition of convergence in distribution on arbitrary metric spaces is given by Heyer 2009 Chapter 1.2]. This includes definition of weak convergence in terms of continuous and bounded functions (Remark I 1.2.5 (b)), version of the Portemanteau theorem (Theorem 1.2.7), and the fact that convergence probability implies convergence in distribution (Application 1.2.15).
Lemma E.1. Let H and Z, i, E N, be random variables with values in separable Hilbert space H. Furthermore, let Z; be independent and identically distributed with finite mean and
covariance operator V. If (H ..... H 2, (Zi....; Zk) as m - 0o for all l E N, then there exists k: N + M such that k(m) - 50 monotonically as m - 2o and
k(m)
Z,
(S51)
(k(m)
M-
for H-valued Gaussian random variable Z with mean and covariance operator like Z₁. Similarly, there exists W;; N + N such that k' (mm)) + co monotonically as m - oc and
(m)
H
E F
(S52)
W m
ML-
Proof. Since H is separable and complete, convergence in distribution and convergence with respect to the Prokhorov metric (also known as the Lévy-Prokhoru metric) are equivalent (Billingsley
1999 Theorem 6.8]. By the central limit theorem for separable Hilbert spaces (Zalesskii et al.) 1991
this implies
h = -0.
lim d k
Zi, 2
(S53)
1=1
In addition, the assumption together with the continuous mapping theorem gives that
k
lim
Zi =
for all e N.
m 200
Vk
Vk
f=1
In particular, for any N, there exists some my
N such that
(
k
k
1
H.
Z
for all 2 my.
(S54)
Vk
Vk
k
We now want to choose k(m) as large as possible for any m, but small enough to ensure Inequality S54 i.e., m 2 mk(m)- So we define
k(m) sup(k m 2 ma)-
First note that {1: m 2 m * Ø, if 2 m1. The map k: + NN is therefore well-defined, as we consider cc. Similarly,, find 2 my for any given k. This yields
lim k(m) = lim sup(k m 2 m2} = co.
m=+00 m m0
38

---

By definition of k(m), it holds m 2 me(m) for all m E N and thus
k(m)
k(m)
H
Z,
for all m E N.
(S55)
Vk(m)
k(m)
Together with Equation
(S53) this yields the claim, (S51
k(m)
d
Σ H
Z
(k((m)
k(m))
k(m)
k(m)
d
H
Zi
+
1
Σ
Zzg
km)))
k(m)
[k(m)
k(m)
+ d
Zig Z
m--00
0.
k(() Vk(m)
For the second claim, S52 one can follow the same procedure but use the law large numbers for Banach spaces instead of the central limit theorem. Suitable results given by Ledoux and Talagrand 1991] Corollary 7.10] and Hoffmmnn--grgensseee and Pisier 1976. Theorem 2.1 Note that the strong law of large numbers holds, but the weak law is sufficient.
In the second lemma, some properties about convergence in distribution and convergence in probabil- ity are stated.
Lemma E.2 (Theorem 2.7 from van der Vaart 1998 modified and (iv) added). Let Xny X and Yn be random vectors. Then
(i) Xn P X implies Xn 2, X: (ii) X,
D P for a constant e if and only if Xn, - c;
(iii) Xn, 2a X and Yn 29 c.for constant then (Xn, Yn) 2 (X,c);
(iv) X,, 2, X and W is random vector independent of (Xn)nEN then (Xn, W) 2, (X,
Proof. (i) (iii). The proofs are given by van der Vaart 1998
(iv). Let f be a bounded and continuous function. Then,
/ [I W) W] dP())
lim E[ W)] lim O=
n0000
E|J(X,, W(w)) dP(w) = f lim [[S(X,, W(w)) dP(w)
lim n
n-
B [[ dP(w)= EE [[(( W) (w) dl B(J(X,W),,
where we used the independence of W and (X,))ee in Equation (+) and the boundedness of f for the interchange limit and integration. This proves convergence distribution.
The above theorem be generalized to metric spaces. One easily check that the proofs in Avan der Vaari 1998 Theorem 2.7] also work for metric spaces using the Portemanteau theorem provided by Heyer [2009 Theorem 1.2.7). However, it is necessary derive more equivalent characterizations of convergence in distribution, which are given and used by yaan der Vaart 1998] but are missing the work of Heyer (2009).
Remark E.1.
Theorem E.3 (Generalized version of Proposition by Jacot et al. 2018)) D. For activation finctions o and 02 with property $50 which are continuous except finitely many jump points, let f :)) and fa( () be network functions with hidden layers to (-;0), b (:00), for < L L. respectively
39

---

as in Definition C.1 C.1.ndd and with shared weights 8. Then (f((=;8)ff((;;3)) converges in distribution
to a multidimensional Gaussian process (X) .Y3 ()(l)) )5=1,....,, as (mi)isisl-1 + oo weakly for
any fixed countable imput set (2x) The Gaussian process is defined by x²") idd N (1.
((()) ldd N (0, (..))))). where we have for x, E Ro ~
0 (x,x') + oh
(2,,'' Σ2 (x,x')
(S56)
n1
Ell (2x, = سz g~ g~~ (o. fo (g(((()))
on (g((0)) +o for L L (1.2). (S57)
Furthermore, x(() and y are independent if # and
x Y,
( (x')
L (2, = (x,x') 222) ()))) (x,,')
for 1, for L 2 2,
(S58)
oh Elo (2)) + +0
where (21. Z2) N (1.
( (2,z)
21.2 21f--) - (o',)))
Proof. We write (x) h00 x; t). We prove the theorem by induction, as the proof of Proposition 1 Jafot al. [0188, but expand on the technical details.
L By definition, we have
h (x) (x)
w x + ozb(1) vno
(1) This implies that h 0) and b (z'; :o, 0), the i-th and j-th component of fig (z; 0) and Rig (2'; respectively, are independent for any h1, h2 E {1,2), x' We and * j. To prove that (n()):: ();0), 122 ( (-:0)) is a Gaussian process, it is thus sufficient to show that vectors of the form (m)) ... (aw) (an) (())))))) have a multivariate Gaussian distribution for any Ti:...;2n Zn (2) It holds
(1)
(x::0)
((('T ((1)
M = Tw w!!! + tabb(") (a /T/o ob)
n00
and therefore
(W((!)T) bi
(m) (x.)
2
n2 (xm)
OwntT/n oolo) owrt oolo)
with la column vector of ones with length d. Now since
((JT
~ ~O, Ino+1),
(())
it holds
(uu ( (wuXT/ng od3d)) (ff.)())) ... aifl (a.)) (m) ... (a.)) N (0, (0. (T/g o. XT mgad) )
Therefore, the vector has multivariate Gaussian distribution with the covariances required for Equation (S56) and the first case of Equations (S58).
L 1++++ 1. We assume that the convergence holds for depth L. This means that there exists some r E Rr such that, for given constant width ni. any width n,,, and widths nu = rr(m). , 1 < L, holds
(n12(),
D
(x)0 xx
m-+00
j=1....
To be precise, there is no such r in the case ==1 + L + 1. However, this only makes the proof simpler and one can still follow the same steps for L 2.
40

---

By the continuous mapping theorem [Billingsley 1999.
Theorem 2.7] it holds
(a1 (n/2()) o2 (h2 (+)
D
[x (x o2 )x)
01
(S59)
m--20
j=
The theorem is applicable despite the finitely many jump points, since (X(4), V(L) assumes the values of the jump points with zero probability.
We now need to find an increasing width function Fz + N such that, if we additionally set an (m),, it holds for any fixed nL+1
12 ())
D. (x.1) x -=
Note that by Remark C.S we consider the product o-algebra. Therefore, to show convergence in distribution for the whole process, is sufficient show convergence in distribution for the marginal distributions. By definition, we have for ho {1,2} that
(I++1)
w((+)) on (1 obbl4-1)
oh (h (z) (1+1) + tzbb(4)
(
h&
VLL
VTL
The marginal vector for points 21.... In before can thus be written as
h1
((+1)(2))) (x1)
a (1 (xI))
1i (((-)) (xn) hi n (x1)
σ1 (h (L) (I:z) Tnnsn
b(++1)
Ow
w
VUL
02 (122 (x1) I1
(L)
ob
b(a+1)
(xn))
02 (h2 (In)) ) In
B(++1)
With the same arguments as before and using the continuous mapping theorem in combination with Lemma E.2 (iv) and the independence of (((+1), it holds
σ (1 (xx)) (x:) (na4)
n2
0i (x (x1))
a1 (1 ( (L) 02 (2 TTet)s
(x! (In) Iuan 02 (v!L)) (ar Inc-
w(L+1)
D m-+00
w(L+1)
: 02 (2 (II)) Ime
22 (12 (In)) Inz
(L)
1=1 Since (X,, Yi) and (X Yj) are independent for i A j. the conditions of Lemma E. are satisfied.
Now, there exists &: + such that k(m) + oo monotonically as m - oo and, when setting nz = (m) = k(m), it holds
11 ( (x1)
R
(1+1))
B((-1)
X ((L1))
h (x)t))
((++1)
D au
w
hn (((+))( (x1)
s(L+1)
+ ob
B((-1)
v(L+))
1=0000
SKl
((++))(xm))
((++1)
B(b-1)
y(L+i)
liz
Sez
for a Gaussian random variable (Re (1()+1
S Rel (L+1) s((+1)
s(l+1) E m22 with
R
(L+1)
(x (L)
Inz)
: p((+1)
on (x (x(() (In))
W((+1)
s((+1)
(z (2:))
S s(L+1)
oz (x (Ix) ( Inzes
41

---

Before considering the covariances, we want to comment on rL. First, note that this sequence may not initially be strictly increasing, but this can be circumvented by considering a strictly increasing subsequence. Second, rr. could theoretically depend on X. However, this can be resolved by not evaluating the pair (114), ( at certain data points, but doing the same calculation as above for (h1") (.) The starting point for this is Equation (S59). To apply Lemma E. note additionally
that (high) (. E 0i R2-se which is a separable Hilbert space because we are considering a countable input set. Above, we worked with the marginal distribution because this makes the following calculation of the covariances easier. Finally, the choice of r. should be independent of
AL+1- This follows from the independence of (((++1) and w/ for j # j'.
To verify Equation (+), we check the covariances of the random vector. They are given by
[ov[n(4tt)) | [ R R foi (x(1 (az))
w.(+!") (w!(+!)) T on (x))(a))) (1.) x() (xz) or (x))
Cov
-E 01 (a:)) E (w(lt (w.(4t))²
=E (x (1) (xi)) In (x( (x)) = E foi (xl (ai)) os (x))) InDi
Similarly, we get that
Cov s(L+1), s(++1)] E 02 (xi)) 02 (1)0())))) Ieets
si
and together this implies using the independence of biases ([++1) and weight matrices W(L+1)
Cov om R + to
(1+1)
oa R + onb]att]]
[oi =Ōa (o E oi (x1² on (x)) + oi))
Cov x (x(tt2) (2i)), (1z))
Cov (wsst + aobh (1+
owsltt" + +oobf/tli]]
=ōæi (0 E Jo (r(4)(z)) 02 (y( (x:)) + oτ) Cov [ (x:)), r-(L+1) (15)
(xx)),
We therefore proved Equation
S57 For the second case of (S58). we see that
Cov
pl(l+1),
(((++)) E fo (x(*) w:(ctt)
(w((t))² T 2o (x)
-E 01 x (L) (x:)) E (i(l(tt
(w:(ttt)* 1 xx)) (z²)) (z,) do (x140) (zx)
=E on (x(x (x:))
(r (v))) TTzi1:
with, by induction hypothesis,
(Z.), N (o.
Σ (1r,
((L) (Ii, (z;,
(2,,
ΣS (zg,
This finished the proof, since now holds
Cov Beatt + (osb(tt)) , -awstht + =oof(t11)
=ōxe (22 os (o,
(x)) 02 (x)) + +o?)
Cov [x (x((t)))(()))))))))))))))))))( (zx), Y (((t)(())))
Remark E.2. In the preceding proof, we checked the marginal distributions of arbitrary size in order to give complete proof of the convergence to Gaussian process. However, the covariances can be derived by considering only a pair of data points (æ1, which drastically simplifies the notation. Also, we only need the distributions of pairs for the next theorems.
42

---

Theorem E.4 (Generalized version of Theorem 1 by Jacot et al. 2018 ). For activation functions di 0g and so-called surrogate derivatives o 02 such that 0 22 TT and og are continuous except for finitely many jump points with property (550), ler fi(::)) :) and f2( be network functions with
hidden layers (.; h22 (-:8), 1 L, respectively as in Definition C.I with shared weights
0. Denote the empirical generulized neural langent kernel
(x,)')) J(L),1.. (x; 0) J(L),02,2 (x':;0)) for xr, IR.
as in Definition C.5
Then, for any x, x' E 10 and 1 S i,j S nu, it holds
(x,x') Es (x,
as m1..... R do weakly. We call ((4) the analytic generalized neural tangent kernel, which is recursively given by
10 = (x,
(S60)
(x,x') 21 (xt x') (x,x') (12 x') for L 2 2,
(S61)
with 21 as in Theorem
E.3 and
11 (x,xx x') = d (1)) for 2,
where (Z1. N N((0,
(2,0) 3 (( (m,x²) 10 (2,x') (o(,x))
Proof. We prove the theorem by induction over L, as in the proof of Theorem 1 by Jacot et al. (2018) We denote J(4). (z) = J(L)) (z; (z;0) for & E (1,2), 2 E Ro
== For s i Σ it holds
(x,x') ب (x)
Σ
y (x) (x))
Tu
an Ōk:
Tw
8Rcob oxy
110
v1o
Kkknn 181Sng
IskSn
2
o2 dig
dij
haدa + ofoi
x') + oß = -1
no
no
Iskgng
This proves Equation (660).
L- 1. 1. Now we assume that the statement is true for L and need to prove it for L + 1. Instead of considering an explicit r E Rr as in the proof of Theorem E.3 and taking m - cc, we will just write Nt....., + weakly"".
In the induction step, we would like to Theorem possible. In the setting of Definition C.7 let S and Sy be two statements that hold weakly. In our setting, these are the induction hypothesis and Theorem E.3 for depth L. Then there exist s, E Rr such that Sj (s) and (t) are true. It is not clear that there exists some r E Rr such that Si(r) and S2(r) are true. It would be natural to define r by Ti rm)) == (m))
However, it is not obvious that this is
for all 1 < L and m E N, but it is still unclear that this implies S (r) and S2(r). Instead, one can consider the combined statement "S, and Sy" " as new statement to find a E R2 such that the statements in Theorem E.3 and Theorem E.4 are both true, which can be done. Since we used the first part of Lemma E. to prove Theorem E.3 and will use the second part of Lemma [o. this proof, we would have to define rz. (m)) == min(k(m)) (mm)). For simplicity, we will assume that the r E R.,, we get by the induction hypothesis also gives us the convergence statement of Theorem
In our case, for any depth L, we would need
3
43

---

Using the definition of the generalized NTK and the quasi-Jacobian matrices, we obtain for 1 - i S 11+1
1i g
(x,x)) (((tt11i(())
(x)
E
Il (x) 1/02((0) (x')
Ōki
Ow
[111 (L) (x)
ow ōus
(n2 (2')
+
Oxgeb ōz
VIL
VnL
IEKSn211 Isisnn
15k5n111
+
(x)) (x))
pmndi di (x)
Σ
w
82 (hg (2'))
8'EgCL:
L)
-bj
01
11..1 (x))
02 [1/ (x')
+ o
(S62)
nLL
n
w
wllt!) (1
σ1
02 (+')
Σ
m (o) (x')
(S63)
jor
TTT
We want to apply the second part of Lemma separately. First note that
E.1 We will consider the terms S62 and
S63
(a1 (x)) 02
(1 (d'))
2x (ar (x)" (z) 02 (v,
(S64)
as 1...... PL-1 + oo weakly by Theorem E.3 and the continuous mapping theorem with
( (2,
M =12())))))
(x). idd N
0,
1 (I,
I(L) 2,2 2 (x',²')
Again, used that the values of the jump points assumed with zero probability. Thus, again by the continuous mapping theorem and by the second part of Lemma E.I holds
Z...
T1 (1) 02 (12 (x))
oB
E 01 (x) 02 (x)
of.
n
weakly
7h is given by nn rm()m) = ((m), which is in turn is given by Lemma E. Note also that the limit is a constant, which implies convergence in probability according to Lemma E.2
Here, as in the proof of Theorem
E.3
(ii). In conclusion, obtain
nL
8ig
10
σ
(1 (x)) 02 (117 (2') + +og
8ig 112 (x,
(S65)
nL
1=1
For term S63
we can apply the induction hypothesis obtain
Σ
(1 (2). (v) [1.) (x,y)
8 8m (L) (z, 8).
(S66)
ni... weakly
e ergan:: 4)
Also, we can again use the convergence given by (S64). but with 0i, 02 replaced by ãi, 22 respectively. This gives using Lemma E.2 (iv)
ف6 (1i ()) 51
(12 ()) w((--))
D ār (x(()). o (r(c.). w((t)).
ai..... weakly
Together with (S66) and Lemma
E.2 (iii) this implies
σ1 [1 (1) (.) or (n) w((--)), ((L) (I,
(ō (x( . (y(l)). w((--1), Smm (y(l-1))mmm1(1 (z,
22)
RL. weakly
44

---

For the summands in (S63) we therefore have by the continuous mapping theorem
W
w
(D) (x)) 82 (12 (d))
Σ
fmm (x) (a')
(
mm
8'EaG: m
D P1
W (itt fm
(x)) 82 (x))) Smr (()
jor
weakly
wittt W -w
(x)) 82 (x'))
(2,
These terms are independent for different m nu. Therefore, the second part of Lemma E. can be applied as before. This yields
W m.
w((t!) 0y (M 82 (x')
Σ
J (2) (x')
Jir
D
σ E [ (x, x')
W (x) (x (x))
Bi weakly
1,1
=T(4) (2,
8ig on
E Øn (x((()) 02
(x9) = 8zg (() (z, - 212
Together with (S65) this yields Equation (S6] and concludes the proof.
The two theorems above are complemented by the convergence of the generalized NTK in the infinite- width limit during training, which will prove below. The theorem is a generalization of Theorem G.2 from Lee et al. [2019] There, the convergence of the NTK initialization (nt) X is assumed. More precisely, they refer to a result of Yang [2019a]. In consequence, we have to assume the same for the next theorem. The proof of Yang [019a] could also be generalizable to case. Alternatielly, one could also try to generalize the statement Jacot et al. 2018) on stability during training in the weak infinite-wice limit.
Theorem E.S (Based on Theorem G.2 from Lee et al. 2019 ). Let a be a Lipschitz. continuous and differentiable activation function. Let the derivative of the activation function à and a so- called surrogate derivative ō be Lipschitz continuous and bounded. Let fc(-;0) be the corresponding network function with depth L initialized in Definition C.1 and trained with MSE loss and surrogate gradient learning, i.e., according to Equation (S49) with surrogate derivative ō. The hidden layers are denoted by h (-:0), for & 1 < I,, as in Definition C.I Assume that the generalized NTK converges in probability the analytic generalized NTK of Theorem E.4 as (ni) Ø n,
(JUW)a (J(h)N))= = f(()
P P_ ((4) & Ing.
n-400
Furthermore, assume that the smallest and largest eigenvalue of the symmetrization of ((L) (X, X).
S(L) = (1 + ( (x,
are given by < Amin S Amax < OC and that the learning rate is given by 1) > O. Then, for any 8 > there exist R > 0, N IN and K > 1 such that for every n N, the following holds with probability at least 1 - 8 over random initialization
6K R
(000-1(1)))00 (x,
sup (E(0,00)
5
Xmnn
where ||
|| denotes the Frobenius norm.
Proof. We follow the proofs in Chapter G of Lee et al. 2019]. The analogous statement is given by Theorem G.2 of et al. [2019]. Since we are considering the infinite-width limit (n4)/=1 L-1 2 n, we will assume that the width of the hidden layers are given by N. The more general case can easily be proved using the same steps. Recall that
di ((r -1 f0. (x, x; (ea,,))0
S49
45

---

The loss function is the MSE loss by assumption. This yields
V ((xx,))
V 2 ( - -211 = -y=g(0)
The change of weights over time is hence given by
d Pt
S48
7) (1.),00
(x::0)T g(t))
dt
By Lemma E.6 below, for any 81 > 0 there exists K > 0 such that for any C' > 0 it holds with probability least 1 51
J (x;0,) |, S K for all 0, E B (0o, (),
Together, this implies
10 dt
d
0oll2
(+)
d
02
= 17 111000000000000000000 s(00)|
di
2
s < j(L)ao (x; (x2:0.] 1o(00))l2 K (1((.))l..
(S67)
for all 0g E B (fo. C) with probability at least - 51- Inequality (+) is a consequence of the Cauchy-Schwarz inequality.
To estimate the term (0g) in Inequality S67D, we use Grönwall's inequality in the differential form, e.g., Lemma 1.1.1 from Qin (2016)). First, note that consequence of the proof of Lemma
E.6 for all 82 > 0 there exist Ry > 0 and no E such that with probability at least 62 it holds for all n 2 no
[g(0)]11 S Ro.
(S68)
We now set C' == 3o/ Ro/ Amin- Next, observe that
d
(0t)
dt [xX;; 0,)
$49
7 J( (X: J(L)aa (x) g(,))
de
di [ls(0.)112 d
d
= (9(0)) at 0(0))
((.)) = =2 dt
2m (0(0.), J(L),,, (x;0)) J(L)) (x); (2; s(0,))
2n (9(0)), 11 (X, x) g(0,)) = -22 (g(0), ((* 9(0.)) (**)
S -217 ((()). 3 Amiu (g(())
2 Menin (0)()
(S69)
3
with s((0) = (f.0) x) + 11 (x,x))) To show (**), we will prove that for any 0 #
(s
1
Avoin 2 2 O.
(S70)
3
First, note that by assumption for all z 2#0
2, (5()) - )min)
1)= 2) 20)20
Since the generalized NTK converges at initialization in probability, we can assume that with probability at least 1 - 53 for any 83 > that
(xxx S
2 3
Amins
for any 2 ni E N. For the symmetrizations, this yields
| 1 |
1
(x,
- (x,,
||,
2
2
2
= -116 10 x) 5 Amin-
(S71)
3
46

---

Furthermore, for 0, E B (fu. C) it holds that
1 (X, (x xt) - (1 (2,
J( (X: 00) J(,), 00) -11)) J(L),0 (2: (x);0,) (2:
1 (x; - (2:
(1;00));011) 01)
| (2:00)
|
2
[ (2: 01)
|
_10 (2; 00)
J((i)a (x: 2
)
100 00012 5 6(K')2KR
(S72)
vn
Vn
with probability at least 51 using Lemma E.6 as before. With the same calculations as for Inequality (S71. this implies for the symmetrizations that
|(())
5/ |1 s 10 (x, - 11 1X (x,
6(K')
1
Amin:
(S73)
Vn
3
21 )
for all n 2 N = max
mo, m1,
Amin A IICKT)<KK
Now Inequalities (S70). (S71 and (S73 give
us
2, ((1)) = 2, (5(2) + 56( -((L) + +((() -((4)
S(L. 2) + (2, (((() - -s 2) + (+2 2, (s - - 2)
Amin 1|=1|² - (2,, (((() - -s(!)) z) - (z, (s - 5b( 2
1 Amia I
Amin || 2
3 Amin |1/1
3 Amin || 2, 3
and thus imply Inequality (S70). To summarize, it holds with probability at least N and 0g B (fo, C') that
82 - -3 for any
S68
RØ and dt (l)(0))1
d
S69
- nmmi [lg(0:)11
2
Grönwall's inequality now implies
(l.)0,) S e
- 3 llo(0))122 Se
- innmiia RR.
We can now return to Inequality (S67) to obtain with probability at least 1 - 81 82 - 83
S67
dt 100 0 2
7Kl19(())12 S 7 e Endmint
Integrating the inequality on both sides yields for all O₂ E B(oo, C)
3K (1-e-3nnnee c-tndmint)
3K Ro C. Amin
|| Ball2 S
(S74)
Amin
Let 1 == inf {t: ||0g- - 0ol12 < C}. Now, if t < co, it holds
C C= 00, 00l12 C.
ttt,
This is a contradiction, so conclude that t = In particular, Inequality (S74) holds for all t > O. We can now repeat the calculations for Inequality (S72) with the Frobenius norm to finally obtain
2k²
6K Ro
(x,x) (x,,))
10, 0oll2
Vn
Amin
Therefore, if we choose 81 = 82 = 83 = 35, the desired inequality holds for any n 2 N with probability at least 1 - d.
47

---

Lemma E.6 (Based on Lemma and Lemma 2 from Lee et al. (2019). In the setting of Theorem E.5
for any 8 > 0 there exists K such that for any C > with probability at least it holds
K
1 (x: - J(L),a, (2: (2:0)
(2:;
0
and
vn
(2:0) (2:0000120011 - K,
for all 0,0 B (fo, C) and o E (ō,8}. Due the equivalence of matrix norms, the same inequalities hold for the norm 11 and some constant K' > O.
Proof. For 0 = ē and a different but equivalent parameterization of the network parameters [ee et al. 2019 Chapter FJ. the proof can be found in Section G.2 of Lee et al. (219). The surrogate derivative
is bounded and Lipschitz continuous, Also, ((4), (X 0) and J (4,00 (x: 0) Ja (X'; ; 0) share
the same recursive formula. Therefore, the proof, which builds on the so-called Gaussian conditioning
technique Yang 2 Section E. should also hold for the surrogate derivative.
Remark E.3 (The analytic generalized NTK for surrogate gradient learning). Since in Theorem E.S we consider only one activation function a instead of two different ones 02 as in Theorem and E.4 it holds
E.3
D(L) -
(L))
=
The analytic generalized NTK in the setting of T'Theorem F.4 is thus given by
7 20) and
T(L+1)= -)(+1) + T(T) S((+1)
$1.2
We therefore still obtain asymmetric kernel due to the contribution of {(L)
Remark E.4 (Positive definiteness of the generalized NTK). In Theorem E.5 we require that the matrix (X, X) ) be positive definite. Equivalently, its symmetrization,
S(L) = 1 21 (x)
should be positive definite. For applications this can be checked numerically. More generally, would be interesting to know whether the symmetric kernel given by
(x, x²) + 7( (2',
is positive definite, One could try to approach this question inductively. However, this leads to problems in the induction step. We want to present a different ansatz, which reduces the question question about 21 We use the closed form of the analytic NTK and the symmetry of E(!) for all I to see that
1 (110 (2) (10
5th
s(h)) (2,2') 2
1
2
(k) (x, x')
((+1)
12
(x,
+
Σ
(k) (x),x)
(k)
II
2 (ot,
b=k
I=
(l (
- (2,
II
(1(+1) 21.
(ad .x) .0)
1=k
This defines symmetric positive definite kernel if (L) is positive definite (see Jacot et al. 2018 Section A.4] for comparison) and the symmetrized kernels.
L--1
L-1
)
1
212
(4+ (z.
(o.22))
+
II 1
sa (al
(dl))))))))))
2
1
positive semi-definite for all k L.... -
48

---

E.2
The analytic NTK for surrogate gradient learning with sign activation function
We would like to proceed as in Section C.2.1 and replace the empirical generalized NTK in Equation (S49) the equation defining surrogate gradient learning, with the analytic generalized NTK obtained by Theorem E.5 Since the activation functions considered in the above section are Lipschitz continuous with bounded and Lipschitz continuous derivative, we will again approximate the sign function and its distributional derivative with the error function as in Section D The results will not depend on the approximation of the weak derivative of the sign function. This approach can only lead to useful result if the resulting kernel not singular. We will check this in the following.
We choose activation function erfm (z) = erf(m z), E N, with surrogate derivative (().. We will also consider the special case ((2) erf(2). As discussed in Remark E.3 and with the final results of Section D.I this immediately yields
E0 y)
lim m0000
E0 (x,
(xr, v) o and
ng
529 20
I (x. y)
((++1) (x, = lim (x++1) (x, 1)
aresin
to..
m00
VIW (xx, xz V/ 1(66 (v.1)
(S75)
Here, we have to assume that of > 0 or r. # 0 to ensure that > (I, x). >!l (v, 8) + 0. This has already been discussed after Equation (S39).
E.2.1 The derivative of the error function surrogate derivative
Next, calculate (LL) 21,2.00 (2r, y) = lim 1,,2m (2r, y)- We will first consider the
((L)
ō = erf, for which we can use the already established tools. In particular, we will discuss the differences to the results of Section
holds
(D)) 1.2.m (x.v) o2 Eferrfm (2("
cif( cif("") for L 2,
where
(Z 2 . (0,
21.2m ( (maa) s((-")) (v.v)
- (0,
11 (2iw) xll-1)) (0 )) N (0. (0.-)))))
Ilc (2,))) EXL-1) (m..)
again using Remark E.3 simplify the covariance matrix. The notation Σ 1 which comes from
(L-1)
Theorem E.3 in combination with the scaling variable m of the activation function, should not be confused with Lingny which is shorthand notation for the Gram matrix s-(--1) (4x, y), (2.
LL-)
49

---

We denote e1 = =6)). e2 = (9) and U ((2))
7
22 ). This yields
(:(2) 1,2mm (2, y) = o BBlerf ( erf( [2""] = o E meef ((m - UU) erf (eJu)
2) -
(+) 40 m
1 (1+m²² m² 2 - (1+ 20 201 - (2m of - e22
20 m
4m ((1
+ 2m
2m/ 2013
(1 (1 (1,
v)) 4m D m)))
20 w
1
1//--1 (2,
1
(1.
"x -
+
π
2m2
2
m-+00
20 m
1 +
1
E (x,,1) -
π
20 mm
-
((-1)
+
(16-1)
2 21)
2 00
π
-
2
-
1)
+
+ 2
for L = 2:
2no
2
x(b-1) 1)
o +
+
for L 2 3.
(L) 1,0000
(2r, y),
(S76)
using Lemma D.I in Equation (+). For the penultimate equality we used Equation S36 and Equation SS37
Compared to Equation S4I..
20
20 (1--1) 3
() (Ir, y)
m 10-000
100 (v. 1/6 (x,
=x v)²)
π
π
S41
which in fact holds for both L 2 and L 3. an additional term appeared in Equation (76)) It always holds of + of > and it holds o | ///g + of > 0 if of > or x o O. As discussed earlier, we always assume that f 0 is satisfied. It follows that this asymmetric NTK is not singular, since
(TL) 1,2:00
20 (x,
I --
(2 (2.
-
+
2
π
2
20
0
1 (x,
F
20 v2
(1,,))
- E RR
(S77)
π 2
00
Note that this is reminiscent of the constant factor depending on æ in the asymptotics of 5:! (a, as m - 00, given by S42 and (S43).
20 m (x,
(I,
22
According to Remark E,3 it holds
1 (x, v) E{L) (z. v) + I 1(f-11(2, v) E1, (I. v).
Since E(h 111'(x,))) y) and mm are continuous functions of the entries of the matrix Dmaar it follows by induction using Equations (S75) and (S76) that the limit of the analytic NTK is well defined for m xc. We can write
(C)
(L--1)
(2) lim (LL (z,
m=
Thus, by approximating the sign function with error functions, we found that the analytic NTK for surrogate gradient learning with sign function and error function as surrogate derivative is well- defined and non-singular as kernel on nh x no Furthermore, comparing this NTK with the
50

---

NTK we derived in Section D the term ΣX6 does not change. This is direct consequence of not replacing the activation function with a surrogate activation function. In this sense, we inherit more properties with this approach than by replacing not only the derivative but the entire activation function including its derivative with a surrogate. Comparing Equation (541) with Equation (S76). we see that 5)(4 1,02 (z,, v) can be obtained from Ex (Ir, v) by adding regularizing term depending on x(--1) (x, //2.
E.2.2 General surrogate derivative
Now we turn to the general case with a general surrogate derivative ã. Similar to before, for m E {oo} we denote
24 (or,o)) s42 (2..))
(ZT 4 ~ No (0,
2 (may) 22m (v.v)
= =N (0
2m I(2-1) (2,2) 2 ((
121--10 (2,8) xyl-)) )) =N (0. -/
With u - (2)) and for invertible Monjyy it holds for L
$1,2,0 (I.y) = lim
m- 1.2:m (z. v) = lim
o E
eif (Z 0 (Z
m-
0 20 (170) o (23 ) = 20
| )τ(2)) (22) e furE tu du
R2 2T
1
11
2 d
Ej
)22)
e
2
[Σ
-
By
Y (o. / ولحر 1) " ( [o(Y),,
(S78)
where Equation (+) seems natural, but requires further reasoning. We will prove the above rigorously in Lemma E.7 For now, we assume that the equality holds. Since ā is bounded and continuous, this yields
lim S(L) 21,2000 (v. y)
a x+
- 2 (x,
lim
V π
20
(o.oro.///0... 128 (x,,)) [o
+0
(x,)))
ō(0).
This agrees with the fact that
((L)
1,2:00 (I, = lim 2
erf ml (7Th) ō (ZTm))
= E
(S79)
m=
2
(1 (x, x))
a(0) = oz v (x
2
o(0),
22T
where we again assumed an equality very similar to Equation (*). It is easy to check that Equations (76 and (S77) can be recovered by inserting ē erf into the derived formulas.
We now prove the missing part of Equation (S78
Lemma E.7. Let ă be a bounded and continuous function and ((ZT)) 7 MeEN sequence of random variables, (7T 7. ~ NO, Em)) If the covariance matrices are invertible and converge to an invertible matrix, Mmm + D 2x2 as 0o, then it holds
lim E | 2TTh m=000
( (
E Y
Y (0. x- /a/ls//////) /E,, [o(Y)) (0())
(S80)
51

---

2 = for all E N. so that the covariance matrices are given by -m
1.1 21
and if
Ei
ETT + Eg + 0 as + oo, then it holds
2 V -V (>T)
lim E (
= 6(0)
(S81)
m-000
Proof. We begin with the case of invertible covariance matrices. Again denoting u = 21 ), holds by assumption
erf m (71h))
722
2m rn - x(22)
VTT C -m²z] e (x")) tM du
lta
2m
2
# /1/11)) ((Em)- (31//1)) +2uTerefu) du
mm
(22)
JR2
2T
π
y
(22)
2
tu B du.
(S82)
Ra 2T
for
Bmd
Im /
m 1.2/m m
E///mm 11,1
+
0
2
0
0
00000
(
/
Ex,
/
= = (S83)
The determinant of T is given by
Σ
Σm
1
2
2
m² mm
mm
m
mm
2
m
2 mm
mm
This now yields
S82
-
(Bm)-
(Bm)
(222) e - 3uTB"" du
Ra 2T
2
&
(Yr (V,,,)))-w(0,,10m)) [o(
The continuity of matrix inversion implies (Bm) (Bm)) (Bx) as m + oc. The characteristic functions of finite-dimensional Gaussian random variables are fully defined by their means and covariance matrices. Thus, the convergence of the covariance matrices implies convergence of the characteristic functions, which in turn implies convergence in distribution. Since is continuous and bounded and the determinant is continuous, we obtain
2
1
E (rTm ym --)00
VI
m=+00
000
B
3 E
(y (p 1) [o((3²))
V
S83
2
1
F
22
Y [o(Y)
2
-
(2111)
B
BY (0,10> / [o(Y)
π
52

---

This proves Equation (80).
In the case 2 Z2", we have
E erf
(
0 (-2)
By-N((0ET)) cerl (erl (V)
1
1
2
m² .ā(y) E 1 क dy
JR 22T V
Vi
2
c ビ//" (1/ET +2m ) dy (1)
JR 2TT
V
VT
m
m
1 (1/E1 + 2m 2mm)2.0000) -tv dy
VT V EN
VI/EI
2m
JR 2
1
(0(8))
TT V1/m² + 25
12-00
(0) -V (ΣT) ) =3))).
VT 2
again using the boundedness and continuity of in the last line. This proves Equation (S81
With E = I(L-1) the lemma yields
S(L) 1,2000 (I,, v) lim 2 (x, lim o E eirf (2T²) o mgn)
2 o V (2(6-))),
E y b (xx..)) ((a))))) [o(Y]],
In conclusion, found that the analytic NTK for surrogate gradient learning with sign function is well-defined and non son-singular for any bounded and Lipschitz continuous surrogate derivative. As in
the special case of the error function, the term 1,2;00 (z,, y) can be expressed in terms of D& (=L--1) and the determinant >
(:())
(xr x)
53

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
As stated in the abstract, we analyze the NTK for activation functions with jumps and generalize the NTK to surrogate gradient learning. We focus the sign activation function in detail, which is also stated in the abstract. The derived theorems, however, are valid for general class of activation functions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the theorems are discussed throughout the paper and general discussion of limitations can be found in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
54

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, all definitions, theoretical results, and proofs are given in great detail in the appendix and have only been slightly shortened for the paper itself. Only for Lemma EE.6 which is used to prove Theorem 2.5 the proof is not given in full detail. Most proofs rely on similar results for the classical NTK and these results are properly discussed and referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All information needed to reproduce the numerical experiments is given in Jadd Section[A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
55

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the code is provided the supplementary material with instructions to reproduce the experiments and figures.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/ poollccgguies/// CodeSubmissionoollo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https wttp:/lccggiiee//eeeeeeee http://ggguieee/eeeeuuubiiiii www.oodeuumiiisoonnee..
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Since only plain gradient descent and gradient descent with surrogate gradients (surrogate gradient learning) is used in the paper, no further additional specifications apart from the ones made Section and Section
necessary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, confidence bands are used for Figure to demonstrate the agreement between two distributions. Figure T and Figure 2 are complemented by Figure B.3 and
Figure B. respectciv,, showing the mean squared errors.
56

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources used for the numerical simulation described in Scction
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code Ethics and the paper conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
57

---

Answer:
[NA]
Justification:
The paper is solely theoretical and does not have conceivable societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not contain models. The data only created illustrate and examine the theoretical results and has no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used packages have been properly referenced in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
58

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
59

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
60


Reviews:
[{"review_id": "WnE0NkXs67", "reviewer": "Reviewer_PShm", "summary": "The paper adapts the neural tangent kernel framework to surrogate gradient learning (and so to learning in spiking neural networks).", "questions": "- I don’t see how definition 2.2 iii can work for strictly increasing width functions. Sequential limits mean some of the widths stay fixed, right? \n- Fig.1 should have the black line at the back I think, otherwise it’s not clear if the peaks always overlap", "limitations": "The authors have addressed the limitations of the work.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper generalizes NTK to (some algorithms for) spiking neural networks, which is an important scenario for neuroscience and neuromorphic computing \n- Like the original NTK results, analyzing learning dynamics with this approach seems easy (i.e. it works like kernel regression, Eq. 7)\n- The paper is well-written and self-contained\n\nOverall, I think this paper is important for theoretical analysis of spiking neural networks, and contains novel results. I didn't closely follow the derivations in the appendix, but I'm familiar with the NTK literature/proofs, and the approach of this paper seems reasonable to me (+ it follows the original derivations to some extent).", "weaknesses": "The main weakness is the NTK approach itself: NTK is known to be a poor approximation to learning in deep networks in many cases, which limits what kind of conclusions we can make with NTK-driven theoretical analysis. However, this weakness is not specific to this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging, thorough and detailed review and for your helpful suggestions.\n\n- **Short answer:** Indeed, by parameterizing all hidden layer widths with the parameter $m$ and considering $m \\to \\infty$, we cannot cover the sequential infinite-width limit as described in [1]. This is because the parameterization requires all hidden layer widths to be finite during the entire limit procedure, which is not the case for the sequential limit. The fact that the width functions are strictly increasing is not creating this problem. However, by following the inductive proof of the sequential limit, we construct a parameterized limit using the width functions (see next paragraph). A consequence of this proof technique is that we have no control over the rates at which the hidden layer widths diverge. However, this is not a problem if one wants to prove weak convergence as defined in Definition 2.2 (iii), see also Section E.1, Lemma E.1 and E.2. This is why we wrote \"In practice, this means that the statement holds as $n_  1,\\dots,n_L \\to \\infty$ sequentially\". We agree that this wording is not very clear and we will add a more elaborate clarification.  \n**Connection between the sequential infinite-width limit and our notion of weak convergence:** The simplest form of a sequential limit takes the form $\\lim_{n_1 \\to \\infty} \\left( \\lim_{n_2 \\to \\infty} f(n_1,n_2) \\right)$. Let us assume that $\\lim_{n_2 \\to \\infty} f(n_1,n_2) = \\hat{f}(n_1)$ exists for all $n_1$ and that $\\lim_{n_1 \\to \\infty} \\hat{f}(n_1) = a$ also exists. If $f$ is continuous, one can always find a parameterization $n_1(m), n_2(m)$, such that $\\lim_{m \\to\\infty} f(n_1(m),n_2(m)) = a$. Now, the sequential limits in [1] correspond to the limit of the form $\\lim_{n_1 \\to \\infty} \\left( \\lim_{n_2 \\to \\infty} f(n_1,n_2) \\right)$. Following this analogy, we provide a way to find the parametrization $n_1(m), n_2(m)$ with Lemma E.1. This allows us to show convergence in the weak sense as defined in Definition 2.2 (iii).  \n**Why parameterised limits are preferable:** There are two reasons why we think that our proposed way of unifying different infinite-width limits is relevant, even though it excludes the sequential infinite-width limit. First, in practice, the hidden layer widths can never be set to infinity, as required by the sequential infinite width limit. In this sense, it is more meaningful in practice to consider the weak convergence we have introduced instead of the sequential limits. Second, the infinite-width limits considered by [2] and [3] are both parameterized and provide elaborate ways of dealing with the infinite-width limit. Our definition shows how they relate to each other.\n\n- We have added grid lines in all revised plots (see global response PDF) to better show the horizontal alignment of the peaks. In addition, the quasi-logarithmic scaling in Figure R2 (see global response PDF) improves the visibility of the vertical alignment of the peaks. The black line is not sufficiently visible if it is placed at the back of the plot.\n\n[1] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural tangent kernel: Convergence and generalization in neural networks.\" Advances in neural information processing systems 31 (2018).  \n[2] Lee, Jaehoon, et al. \"Wide neural networks of any depth evolve as linear models under gradient descent.\" Advances in neural information processing systems 32 (2019).  \n[3] Matthews, Alexander G. de G., et al. \"Gaussian process behaviour in wide deep neural networks.\" arXiv preprint arXiv:1804.11271 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DNU9Fc50N3", "reviewer": "Reviewer_L5ST", "summary": "This paper explore the neural tangent kernel (NTK) with regards to surrogate gradient learning for non-differentiable activation functions. The authors show that the standard neural tangent kernel is not equipped to deal with such activation functions and causes the kernel function to become singular. They provide a derivation of a surrogate derivative based NTK that can handle such activations thus further generalizing the NTK for a larger subset of networks.", "questions": "Are $\\delta(z)$ and $\\delta_{ij}$ the same?", "limitations": "Everything is adequately addressed.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper provides a brand new derivation of the NTK for non-smooth ANN activations and thus is very original. The overall paper is of high quality and clearly developed. The results of this paper are of high significance and allow for new avenues for application of NTK analysis.", "weaknesses": "While the paper is well developed overall, there are a few spots that can benefit from additional clarity regarding notation:\n\nLine 144 (Definition 2.1) - $r_l(m)$ refers to the number of neurons for a particular layer however $m$ is not directly described. My understanding is that it is referring to the output size dimension but this is not immediately clear.\n\nLine 215 - I am assuming that $\\delta(z)$ refers to the Dirac delta function (delta distribution). \n\nLine 244 - Here we have $\\delta_{ij}$ which I assume is *not* the Diract delta and instead a constant for a given kernel matrix entry.\n\nLine 289 - Since we are talking about divergence, notationally it would be better to use $\\Theta_{\\text{erf}_m} \\not\\rightarrow \\Theta\\_{\\text{sign}}$ or specify the formal definition of divergence *or* avoid it all together.\n\nIn regard to the experiments line 303 mentions that the NTK diverge as $m \\rightarrow \\infty$, however this does not seem to be clear from the plots especially since $m=20$ does not seem to be sufficient to show the divergence occurring. In addition, since we are discussing divergence, I believe the paper could benefit by including the average error values between the analytic and empirical kernels since graphs tend to do a poor job of illustrating this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and thorough review and for your helpful suggestions.\n- We apologise for any confusion caused by the use of the variable $m$ in lines 133 to 137 and will change the variable. The parameter $m$ is used in Definition 2.1 to be able to consider a limit by taking $m \\to \\infty$. It does not refer to any dimension or size of the network itself, but rather indexes the limit. This is described directly after the Definition 2.1 in lines 148 to 149: \"Every element of $\\mathcal{R}_L$ provides a way to take the widths of the hidden layers to infinity by setting $n_l = r_l(m)$ for any $1 \\leq l < L$ and considering $m \\to \\infty$\". Example: We have two hidden layers and hence layer widths $n_0, n_1, n_2, n_3$. Suppose that $n_0 = n_3 = 2$ are fixed, layer one grows quadratically, and layer two grows cubically. Then we have $r_1(m) = m^2$ and $r_2(m) = m^3$. \n- Yes, $\\delta(z)$ denotes the delta distribution. We will add this information right after the first occurrence of the delta distribution.\n- Yes, $\\delta_{ij}$ denotes the Kronecker delta, i.e., $\\delta_{ij} = 1 $ if $i=j$ and $\\delta_{ij} = 0$ otherwise. To avoid confusion with the delta distribution, we will add a clarification after the first occurrence of the Kronecker delta.\n- It is true that $\\Theta_ {\\mathrm{erf}_ m}$ converges to the singular kernel $\\Theta_\\mathrm{sign}$, $\\Theta_ {\\mathrm{erf}_ m} \\to \\Theta_\\mathrm{sign}$, because this is how $\\Theta_\\mathrm{sign}$ is defined in lines 221 to 222. This means that $\\Theta_ {\\mathrm{erf}_ m}(x,y) \\to \\infty$ as $m\\to\\infty$ for $x = y$. To be more precise, we will instead write in line 289, \"We numerically illustrate the divergence of the analytic NTK, $\\Theta_ {\\mathrm{erf}_ m}$, as $m \\to \\infty$, [...]\".\n- It is unclear in which way the plots do not sufficiently illustrate the divergence. Comparing the y-axis and the size of the peak at $\\Delta \\alpha = 0$ for $m=1,5,20$, the effect of the divergence can be seen, as the peak grows from $<5$ for $m=2$ to $>100$ for $m=20$. Figures can never fully confirm or disprove convergences or divergences and we will weaken the formulation in lines 302 to 303, \"the plots confirm that the analytic NTKs diverge\". In addition, we introduce a common quasi-logarithmic y-axis using the inverse hyperbolic sine to better illustrate the convergence for all $\\Delta\\alpha \\not= 0$ (see global response PDF, Figure R2), as suggested by Reviewer Ljsu. By comparing the analytic NTKs for finite $m$ with the singular kernel for $m \\to \\infty$ in Figure R2, we can indeed see that $m=20$ is large enough to illustrate the convergence to the singular kernel. Finally, we agree that average error values help to illustrate the convergence of the empirical kernels to the analytic kernels and provide them in Figure R3 and Figure R4 (see global response PDF).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtwVczsmzY", "reviewer": "Reviewer_Ljsu", "summary": "The paper considers an extension of neural tangent kernel methods for the analysis of training of neural networks with non-differentiable activations with surrogate gradient learning.  The basic approach is the define a generalized NTK (the SG-NTK) based on the quasi-Jacobian matrix (that is, the Jacobian constructed using the surrogate gradients) rather than the (ill-defined) Jacobian based on the actual gradients of the activations.  This construct is shown to be deterministic (distributionally determined) in the infinite width limit.  For the sign activation (with erfm acting as surrogate gradient) both the NTK and SG-NTK are derived.  Finally some experimental results are given that show that the distribution of networks trained with SG-NTK matches reality.", "questions": "See previous sections.", "limitations": "The authors have addressed limitations adequately and there appear to be no obvious negative societal impacts.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly presented.\n- Motivation, aims etc are clear and compelling.\n- The paper appears to fills an important gap in existing literature (admittedly I am relatively new to this area of research so there may be predecessors I am unaware of).\n- The derivations given seem correct to the best of my understanding, though I did skim some of the proofs in the rather long appendix.\n- The experimental results would appear to confirm that the SG-NTK matches with reality.", "weaknesses": "- it would perhaps be more useful to present figure 1 with a log y scale as the linear scale used, combined with the divergence at $\\Delta \\alpha = 0$, tends to flatten all features of the NTK, masking the differences between the NTKs away from the point of divergence.\n- also regarding figure 1 is it feasible to plot the limiting case $m \\to \\infty$ with the point of divergence elided?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging and positive review and for your helpful suggestions.\n\n- We have refrained from using logarithmic scales to keep the plot as simple as possible and to facilitate the comparison between Figure 1 and Figure 2. However, we agree that a logarithmic y-scale helps to illustrate the divergence at $\\Delta \\alpha = 0$ and the convergence at $\\Delta \\alpha \\not = 0$ in Figure 1, and provide an updated figure with a quasi-logarithmic y-axis using the inverse hyperbolic sine (see global response PDF, Figure R2).\n- Yes, we agree with this suggestion and add the analytic NTK for $m \\to \\infty$ as in Figure 2 (see global response PDF, Figure R2). Note that this singular kernel, $\\Theta_\\mathrm{sign}$, can never be fully plotted with a logarithmic y-axis, since $\\Theta_\\mathrm{sign}(\\Delta \\alpha) \\to \\infty$ as $\\Delta \\alpha \\to 0$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHq3ewunzP", "reviewer": "Reviewer_TYqA", "summary": "The paper addresses the challenge of applying gradient-based training methods to neural networks with non-differentiable activation functions, such as binary and spiking neural networks. These networks use surrogate derivatives to enable gradient descent, but this approach lacks theoretical foundation. The authors propose a generalization of the NTK, called the surrogate gradient NTK (SG-NTK), to provide a rigorous theoretical basis for SGL.", "questions": "What is the application of the results? The activation function, the ReLU function is commonly used and I believe it can be solved in previous works. Are there any commonly used activation functions that can solved in your paper but not in previous works?", "limitations": "The results in this paper are only applicable to sufficiently wide networks with random initialization.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel extension of NTK to non-differentiable activation functions using surrogate gradients using the results in [1,2,3,4].\n2. Strong empirical validation through numerical experiments.\n\n[1] Gaussian process behavior in wide deep neural networks\n[2] Neural tangent kernel: Convergence and generalization in neural networks.\n[3] Wide and deep neural networks achieve consistency for classification\n[4] Wide neural networks of any depth evolve as linear models under gradient descent", "weaknesses": "1. The complexity of the theoretical framework may limit accessibility for practitioners\n2. Additional experiments on real datasets (MNIST, CIFAR10, ImageNet) could further validate the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your helpful suggestions. We appreciate the focus on the applicability of our work. \n\n- While the mathematical details behind the NTK theory are generally complex and not particularly accessible to practitioners, the analytic NTK can be easily calculated for different data sets, network depths and activation functions, e.g., using the Neural tangents package (see citations in line 843). The analytic NTK then shows how the class of functions learned with gradient descent looks like, without actually having to learn a network. As we write in lines 22 to 23, this is possible whenever the activation function is differentiable or at least semi-differentiable. This also includes the case of the ReLU activation function. However, whenever gradient descent is not applicable, we cannot define the NTK anymore. This includes binary neural networks, time-discrete spiking neural networks and any activation function with jumps. Surrogate gradient learning is then commonly used by practitioners, see lines 102 to 120. Surrogate gradient learning introduces the surrogate derivative, which is an additional unknown (besides network depth and activation function) that changes the class of functions learned with SGL. So far, there is no theoretical tool like the SG-NTK to systematically choose the surrogate derivative and the network depth. In particular, the SG-NTK yields a prediction for the class of functions that are learned with SGL.\n- We agree that our approach would benefit from additional experiments. Moreover, as we write in lines 338 to 339, \"[a more] rigorous analysis should be carried out on how the connection between SGL and the SG-NTK carries over to activation functions with jumps, as shown by our simulations\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their thorough reviews and helpful suggestions.

We provide four additional figures in the attached PDF, where we have implemented the suggestions of Reviewer PShm and Reviewer Ljsu. The figures address the question raised by Reviewer L5ST. In Figure R1, we have added grid lines to Figure 1 to illustrate that the peaks overlap horizontally. In Figure R2, we have added grid lines and an asinh-scaling for the y-axis (approximately linear for small absolute values and logarithmic for large absolute values) to Figure 1. Moreover, we have added a plot of the singular kernel, which we obtain from the analytic NTK as $m \to \infty$. Note that this allows for a nice comparison with Figure 2, where we have plotted the non-singular kernel, which we obtain from the analytic SG-NTK as $m \to \infty$. In Figure R3 and Figure R4, we have plotted the mean squared errors between the empirical and analytic kernels for Figure 1 and Figure 2 respectively. We can see the convergence of the empirical SG-NTKs in R4, in accordance with our theoretical results.

We answer the individual reviews in the order of reviewer comments.

Meta-review: The paper does a thorough analysis of the NTK regime in in the presence of surrogate gradients. The technique seem general enough that they might be relevant to other models. Most reviewers are excited about the result and consider the paper to be worth of a spotlight.
Decision: Accept (spotlight)