Equivariant Neural Tangent Kernels
Authors: Philipp Misof
Pan Kessel
Jan E Gerken
Venue: ICML 2025 poster (2025)
Keywords: neural tangent kernel
equivariant neural networks
geometric deep learning
data augmentation
Primary area: theory->deep_learning
TL;DR: We introduce NTKs for group convolutional neural networks, establish a relation between data augmentation and GCNNs, and show that equivariant NTKs outperform non-equivariant NTKs in a number of application domains.

Abstract: Little is known about the training dynamics of equivariant neural networks, in particular how it compares to data augmented training of their non-equivariant counterparts. Recently, neural tangent kernels (NTKs) have emerged as a powerful tool to analytically study the training dynamics of wide neural networks. In this work, we take an important step towards a theoretical understanding of training dynamics of equivariant models by deriving neural tangent kernels for a broad class of equivariant architectures based on group convolutions. As a demonstration of the capabilities of our framework, we show an interesting relationship between data augmentation and group convolutional networks. Specifically, we prove that they share the same expected prediction over initializations at all training times and even off the data manifold. In this sense, they have the same training dynamics. We demonstrate in numerical experiments that this still holds approximately for finite-width ensembles. By implementing equivariant NTKs for roto-translations in the plane ($G=C_{n}\ltimes\mathbb{R}^{2}$) and 3d rotations ($G=\mathrm{SO}(3)$), we show that equivariant NTKs outperform their non-equivariant counterparts as kernel predictors for histological image classification and quantum mechanical property prediction.

Equivariant Neural Tangent Kernels
Philipp Misof
Pan Kessel 2
Jan E. Gerken
Abstract
Schüitt al., 2021; Unke et al., 2021). Other application areas include particle physics (Bogatskiy et al., 2020), cos- mology (Perraudin et al., 2019) and even fairness in large language models (Basu et al., 2023).
Little is known about the training dynamics of equivariant neural networks, in particular how it compares to data augmented training of their non-equivariant counterparts. Recently, neural tangent kernels (NTKs) have emerged as a pow- erful tool to analytically study the training dy- namics of wide neural networks. In this work, we take an important step towards a theoretical understanding of training dynamics of equivari- ant models by deriving neural tangent kernels for broad class of equivariant architectures based on group convolutions. As a demonstration of the capabilities of our framework, we show an in- teresting relationship between data augmentation and group convolutional networks. Specifically, we prove that they share the same expected pre- diction over initializations all training times and even off the data manifold. In this sense, they have the same training dynamics. We demonstrate in numerical experiments that this still holds ap- proximately for finite-widhe ensembles. By imple- menting equivariant NTKs for roto-translations in the plane (G Cn IR?) and 3d rotations (G S3))),, we show that equivariant NTKs outperform their non-equivariant counterparts kernel predictors for histological image classifica- tion and quantum mechanical property prediction.
Recently, there has been a number of works which avoid equivariant architectures but rely on data augmentation to approximately learn equivariance, most notably Al- phaFold3 (Abramson al., 2024). 'hiis has the potential ad- vantage that non-equivariant architectures may offer better training dynamics, for example favorable scaling capabili- ties. There has been a vigorous debate on this subject with some empirical works claiming superiority of equivariant ar- chitectures (Gerken et al., 2022; Brehmer et al., 2024) while others suggest the opposite (Wang et al., 2024; Abramson et al., 2024). One challenging aspect conclusively settle the matter is that there is no good theoretical understanding of how the equivariant and the purely augmentation-base training dynamics compare.
Motivated by this observation, this paper derives equivariant neural tangent kernel (NTK) theory (Jacot et al., 2018) for group convolutional architectures. The NTK provides a powerful tool to analytically study the training dynamics of neural networks in the large width limit by analyzing the behavior of the kernel, in particular its trace, eigenvalues and other properties (Geiger et al., 2020; Mok et al., 2022; Engel et al., 2024; Tsai et al., 2023). A particularly important feature of the NTK is the fact that in the infinite width limit, it becomes constant throughout training (Jacot et al., 2018). Furthermore, at infinite width, the NTK can be computed by layer-wise recursion relations. These simplifications allow for complete analytic control over the training dynamics. In particular, the network output of an arbitrary input and at an arbitrary point in training time converges to a Gaussian process over initializations whose mean- and covariance functions can be computed analytically. This result has led to number of theoretical and practical insights (Geiger et al., 2020; Jacot et al., 2020; Yang & Hu, 2021; Yang et al., 2021; Franceschi et al., 2022; Day et al., 2023) into the initialization and training of neural networks.
1. Introduction
Equivariant neural networks (Weiler et al., 2023; Gerken al., 2023) are widely used in many applications of great practical importance, for example in medical image analysis in two and three dimensions (Bekkers et al., 2018; Winkels & Cohen, 2019; Müller et al., 2021; Pang et al., 2023) and in quantum chemistry (Duval et al., 2023; Batzner et al., 2022;
'Department of Mathematical Sciences, Chalmers University of Technology and the University of Gothenburg, SE-412 96 Gothen- burg. Sweden Prrescient Design, Genentech Roche, Basel, Switzer- land. Correspondence to: Jan Gerken <gerken @ chalmers.se>.
We derive recursive relations which determine the NTK of an equivariant neural networks for the first time. In particu- lar, we study the NTK of group convolutional layers (Cohen & Welling, 2016). These layers are some sense universal.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Equivariant Neural Tangent Kernels
Specifically, they have the unique property that they arise from imposing equivariance constraint on dense fully- connected layers and are therefore the most general linear, equivariant transformations (Kondor& & Trivedi, 2018) and have been used in a wide array of applications (Chidester et al., 2019; Celledoni et al., 2021; Moyer et al., 2021).
as width grows to infinity. Furthermore, we demon- strate the superior performance of equivariant NTKs over other kernels for medical image classification and quantum mechanical property prediction.
2. Related Work
These NTK recursion allows us to clarify the relation be- tween the training dynamics of pure data augmentation and equivariant architectures in the large width limit. Specif- ically, non-equivariant architectures trained with full data augmentation converge to certain group convolutional archi- tectures in the infinite width limit. This result holds for any input, in particular off- off-manifold, and at any training time. Thus, at least in the infinite width limit and in expectation over initializations, the training dynamics of data augmen- tation is identical to the one of certain group convolutional architectures.
Neural Tangent Kernel. Gaussian processes can be viewed as Bayesian neural networks first pointed out by (Neal, 1996) and this relation extends to deep neural networks as shown in (Lee et al., 2018). Neural tangent ker- nels allow description of training dynamics, see the seminal reference (Jacot et al., 2018) and (Golikov et al., 2022) for an accessible review. In (Lee et al., 2019), NTK theory was used to show that wide neural networks trained with gra- dient descent become Gaussian processes and generalized in more rigorous and systematic manner by (Yang, 2020). NTKs can be used to derive parametrizations that allow scaling networks to large width (Yang & Hu, 2021). They can also be used to theoretically analyze GANs (Franceschi et al., 2022), PINNs (Wang et al., 2022b), backdoor at- tacks (Hayase & Oh, 2023), pruning ( (Yang & Wang, 2023) and spectral learning biases (Bordelon et al., 2020; Canatar et al., 2021). Recently, corrections infinite width limit have been studied by (Huang & Yau, 2020; Yaida, 2020; Halverson et al., 2021; Erbin et al., 2022) using techniques inspired by perturbative quantum field theory. The NTK kernel for convolutional architectures was derived in (Arora et al., 2019). Our results can be thought as a generalization thereof to general group convolutions.
NTKs have also been shown to be interesting kernel func- tions in their own right. Since they are induced by neural network architectures, they allow to transfer the intuition gained in the extensive literature on the design of neural networks to kernel machines and have shown to outperform more traditional kernel functions (Arora et al., 2019; Li et al., 2019; Lee et al., 2020). In our experiments, we show that group equivariant kemelss outperform their non-equivariant counterparts for both regression and classification as well as for discrete roto-translations and continuous rotations. In summary, our main contributions are
We derive layer-wise recursive relations for the neural tangent kernel and neural network Gaussian process kernel of group convolutional layers, the correspond- ing lifting layers, point-wise nonlinearities and group- pooling layers.
Equivariant Neural Networks. Equivariance has been an important theme of deep learning research over the last years, see (Gerken et al., 2023) for an accessible review. Equivariant deep learning is part of the larger area of ge- ometric deep learning (Bronstein al., 2017), in which more general geometric properties of the different parts of the learning problem (e.g. the data (Dombrowski et al., 2024), model (Weiler et al., 2023) and optimization proce- dure (Amari, 1998)) are studied. Herein, we focus on group convolutional layers (Cohen & Welling, 2016) which are the unique linear equivariant layers. A comprehensive summary is given in (Cohen, 2021). These architectures have found wide-spread application in computer vision (Chidester et al., 2019; Celledoni et al., 2021; Moyer et al., 2021), medical applications (Bekkers et al., 2018; Chidester et al., 2019; Pang et al., 2023) as well as natural science use cases (Nicoli et al., 2020; Liao & Smidt, 2023; Bekkers et al., 2024).
We specialize our general results to the case of roto-translations in the plane as well as the three- dimensional rotation group SO(3). We derive and implement the kernel relations for these cases, allow- ing for efficient computations. The code provided www/philiiieccccccccoccocccocooo PhilippMi
publicly at ptattps:/ggttee so sofCH/ eeueiariiiiaaateeeeee -nt.
We prove that in the infinite width limit, a standard con- volutional or fully connected network trained with full data augmentation yields the same expected network function as a corresponding group convolutional net- work trained without data augmentation. This result holds for all training times well as off manifold. We show empirically that this holds approximately at finite width.
Learned vs. Manifest Equivariance. While equivariance can be enforced on the training data via data augmenta- tion, the imposed symmetry does in general not extend to out-of nut-of-distribution data (Moskalev et al., 2023). This is in accordance with the tighter upper bound on the generaliza-
We verify experimentally that the NTKs of unite-width equivariant networks converge to our equivariant NTKs
2

---

Equivariant Neural Tangent Kernels
tion error of equivariant networks compared to purely data- augmented ones that was found in (Wang et al., 2022a). By analyzing the spectrum of the NTK in a toy problem, (Perin & Deny, 2024) found that non-equivariant neural networks are unable to generalize symmetries learned in one class via data augmentation to another, only partially augmented class. They further show how the qualitatively different NTK spectrum of CNN improves generalization for the task under consideration. In (Gerken & Kessel, 2024), the effect of data augmentation infinitely wide neural net- work ensembles was studied. The authors found that the resulting Gaussian process is equivariant at all training times and even off the data manifold. In contrast, our results do not require data augmentation but derive an NTK for mani- festly equivariant group convolution layers. This allows us to find connection between the ensemble means of data- augmented and equivariant networks, which complements the previously mentioned results focusing on individual net- works.
through the layer + using the chain rule,
(N)c+t)(() (OA
g((+1) (I, x²) = E
a0le+1)
a0le+1)
(oN(ct)((z)
(x)
(ON(CO()))
+ E
(x7((2)
o0(
ag(M)
610
(N (N
x
(2)
Note that according to the NTK's definition (1), it holds that O = 0. The recursions (2) have been computed explicitly for number of layers, e.g. fully connected (Jacot et al., 2018), nonlinear (Jacot et al., 2018), convolution (Arora et al., 2019), and graph convolution (Du et al., 2019). An efficient implementation for many layers is available the Jax-based Python package neural neural-tangent (Novak et al., 2020).
3. Backkround
For evaluating the expectation values in (2), it is convenient to introduce the neural network Gaussian process (NNGP) kemne
This section gives brief overview of NTK theory (for optional broader introduction, see Appendix A) as well as of equivariant neural networks with a particular emphasis on group convolutional neural networks (GCNNs).
K (2, x') = (A10
[M(O) (NI1))))
(3)
whose name originates in the fact that at initialization, the neural network converges in the infinite width limit to a zero-mean Gaussian process with covariance function K (x,) x²) (Neal, 1996; Lee et al., 2018). the infi- nite width limit, K is proportional the unit matrix, so we will treat it as scalar well. The NNGP can also be computed recursively layer-by-layer. For the & O, the NNGP is the covariance matrix of the input features K (()) (I, x') = x TT"
Neural Tangent Kernels. layer-wise recursive relations (Jacot et al., 2018) starting from the definition
The NTK can be computed by
(x) agie)
(DN1O(8))
)
(2,x') E
agien
(L)
Using the definition (3) of the NNGP, we can determine the structure of the NTK recursive relations from (2). For linear layers, the first expectation value will evaluate to the NNGP, while the second expectation value will be proportional to the unit matrix due to the initialization with independent normally distributed parameters. For nonlinear layers, the first expectation value vanishes and the second expectation values will depend on the derivative of the nonlinearity.
of the layer- NTK. The NTK of the full network is given by (zz,z') = (1)) x²) for a network depth L. Here, g(t) are the parameters of the layer É and we adopt the convention that expectation values
over the initialization distribution unless otherwise stated. In the limit of infinitely wide hidden layers, not only do analytic expressions exist for (1), but (0) (z,x') also stays constant during training. Hence it is often referred to as the frozen NTK (Geiger et al., 2020). As customary in the NTK literature, we treat activations and preactivations as distinct layers and refer to AT as the layer-& features with ((x)) (x(4)(z). This allows us to treat linear- and nonlinear layers equal footing. Since (1) is proportional to the unit matrix, we can treat it as a scalar.
Group Convolutions. Group convolutions (Cohen & Welling, 2016) act on feature maps f G - R where Rin denotes the number of input features to the network. In the example of image inputs, this feature map would be s z² + R where z² is the pixel grid, R is the space of RGB colors, and the feature map f is supported on [0, A] x [0, w] for imagesize h X w. Let ( Y) denote the set of square integrable functions from X to Y. The f-th neural network layer A(() I,2 Th ) + T,3 R*))
We can find a recursion relation between ((++)) and (()) by separating the E = + 1 contribution from the sum and computing the l I contributions in terms of derivatives
3

---

Equivariant Neural Tangent Kernels
maps an input feature map f G R to an output
4. Equivariant Neural Tangent Kernels
feature map, (f) G R A particular instance of such a layer is the group convolution layer which in NTK representation is given by
This section presents our recursive relations for the NTK and the NNGP for group convolutional layers. These recursions allow for efficient calculation of these kernels for arbitrary group convolutional architectures and thus provide the nec- essary tools to analytically study their training dynamics in the large width limit. Specifically, we derive recursion relations for group convolutions (4), lifting layers (6) and group pooling layers (10) by evaluating the derivatives and expectation values in (2).
1
/ dh klg (N)O))
VnelSkl Sxl
(4)
with filter x : G + m with support S, C G. Here, we integrate over the group with respect to the Haar measure. For finite groups, the integral becomes sum over group ele- ments. Due to the invariance of the Haar measure, the layers (4) are equivariant with respect to the regular representation
4.1. Equivariant NTK for Group Convolutions
(M) = (m)
h E G.
(5)
Since the domain of the feature maps in GCNNs is the symmetry group G, the layer-f NNGP and NTK kernels do not only depend on the input feature maps f and f but also the group elements g. g at which the feature maps evaluated, i.e.,
Since the input features typically have domain X C R which is not the symmetry group G, the first layer of group convolutional neural network (GCNN) is lifting layer which maps feature map with domain X equivari- antly into a feature map with domain G (Cohen& & Welling, 2016)
K(0 (f,5") E (N) (MM()) 9.9
7
(O
/ dr ())
(6)
@ 9.9 (f,f))
onl aec!)
(awvo))))))) opers
Diu Sxx
where P is a representation of G on X. We assume here and in the following that X is a homogeneous space of G, i.c that any two elements of X are connected by a group transformation.
For these kernels, we derive the following recursion relation:
Theorem 4.1 (Kernel recursions for group convolutional layers). The layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (4) are given by
As is common for other network types as well, the non- linearities in group convolutional networks are applied component- wise across the different group elements,
K Keo (S.S"= "((""((s'))
/ dh K-(t)
gh.g'h (J,
(11)
(N(+)) a ((V())))
|5 S.
for nonlinearity a. Due to this component-wise structure, the layers (7) are equivariant with respect to the regular representation (5) as well.
(e+1) " = " (f
(f.f'
1
d
(
S
Sn
(12)
By combining lifting- and group-convolution layers with nonlinearities, one can construct expressive architectures which are equivariant with respect to the regular representa- tion, i.e. which satisfy
Proof. See Appendix B.
Given G-invariant filter supports S., these recursive defi- nitions imply an invariance of the kernels in their group- indices under right-multiplication by the same group ele- ment h E G.
N (Prre = ( (U),
G.
(8)
Many practical applications necessitate an invariant network
gh.g'h (h.gh(( R(E (f,f) K(C-D (f.f)
(13)
(Proo (9) =N()), g GEG.
(9)
9.9"
θ gh (f.f')
9.d o (f,f).
(14)
Such transformation property can be achieved by append- ing a group pooling layer to a GCNN,
While the kernels of feature maps on the group carry g.g'- indices, the kernels of the input features carry x, -iiiniices
Artet = vol(G) [ dg ())
1
(10)
( 2. T = (
2 (I.') = (15)
Using these layers, a wide variety of equivariant- and invari- ant networks with respect to a general symmetry group G can be easily constructed.
Using this, we also derive the following recursion relations:
4

---

Equivariant Neural Tangent Kernels
Theorem 4.2 (Kernel recursions for the lifting layer). The layer-wise recursive relations for the NNGP and NTK of the lifting layer (6) are given by'
for the lifting layer in Theorem 4.2, have the transforma- tion property
(()) (Preg (h) Preg Preg(h 1')) = K
h (f,f" (24) 1g
(f,
1
dr Kl0)
plg)e, plo (f, (16)
(1)
9.9²
9. (Preg (h)f, Preg (h')f') = 6 -h-igg f'). (25)
[
S...
D " (f.f")
1
dx 0
paaa old porr (f.
This left-multiplication is preserved by the recursions of both the group convolutions in Theorem 4.1 and the non- lincarities in Corollary 4.4. Therefore, before any pooling layer, we have
s.
S
Sn + 9.9
(17)
where the regular representation Preg is defined in (5).
((()) 9.0 (Preg ( Preg (h) = hn iglt 10 (f.f') (26) θ e 9.9 (Preg (h) Preg (N) O ggghiggh ig' (f. f'), (27)
Proof. See Appendix B.
The group pooling layer (10) maps feature maps on G onto channel-vectors. Therefore, the kernels lose their 9, g' indices in this layer, as is reflected in the following result:
reflecting the equivariance of the network. The recursions of the group-pooling layer in Theorem 4.3 average over the group and the kernels become invariant after the group pooling layer
Theorem 4.3 (Kernel recursions for group pooling layer). The layer-wise recursive relations for the NNGP and NTK of the group pooling layer (10) are given by
k (Preg Preg( = (f.f)
(28)
(f,+)) = (vol(G))² [c dg Se ds k(0 aag k
1
(1,8)
(Preag (f) Pieg (h')f') (f, (29)
(18)
as expected from an invariant network. Note that these transformation properties of the kemels are independent for both arguments.
((++) = (v)) / dg {( dg 9. (f,
1
(19)
4.2. Roto-Toanslaan Translations in the Plane
Proof. See Appendix B.
The kernel recursions provided in the previous section are valid for general symmetry groups G. In this section, we will specialize these expressions to the case of roto- translations in the plane with rotations by (360/n)². In this case, G = Cn k IR² where G is the semidirect product of the cyelic group C. and the translation group in two dimensions R². It was shown that adding this rotational symmetry to conventional CNNs boosts performance con- siderably for important applications such as medical image analysis (Chidester et al., 2019; Bekkers et al., 2018, Pang et al., 2023). Due to the semidirect product nature of the symmetry group, the group convolutional layers can be writ- ten stack of conventional convolutions which are summed over the rotation group. Details and explicit ex- pressions for the lifting- group convolutional- and group pooling layers in this case can be found in Appendix C.I.
The final layer necessary to compute kernels of GCNNs are the nonlinearities (7). Since these act pointwise on the feature maps, the recursive relations are the same as those for nonlinearities in MLPs (Jacot et al., 2018):
Corollary 4.4 (Kernel recursions for nonlinearities). The layer-wise recursive relations for the NNGP and NTK of the nonlinear layer (7) are given by
k(0) (f K9 (f'.
) KO (g((()))
Kl0 (
(0) (1,')) 9.0
(20)
k(e=" 9.9¹
(U,8") E ((N))- um)low (21)
x(e=")( (unn) by (.
uun lo (2)
9:0
g.6
(U,") = k-(ett) (
g.g'
s.g U.
(23)
The kernel recursion of ordinary CNN-layers can be written in terms of the operator (Xiao et al., 2018)
Using these results, the NTK and NNGP can be straightfor- wardly computed for any GCNN architecture. In particular, consider the transformation of the kernels under transfor- mations of the inputs, i.e. consider K (Prag (h)f,, Prog (N'))' and (preay (f) Preg (f'))"). From the recursion relations
[As, (K))())) 1 di K(I k((+,((+)) (30)
S S.
for which efficient implementations in terms of convolu- tions are available in (Novak et al., 2020). In Appendix C.2, we present explicit expressions for the NNGP and NTK recursions of roto-translation equivariant convolutions in
In practice, the lifting layer is usually the first layer, thus 0.
5

---

Equivariant Neural Tangent Kernels
terms of A, retaining the efficiency of the non-rotation- equivariant kernel computations. We provide implementa- tions of these recursions for 4 as new layers based on the neural-tangen package.
mentation in fact converge group convolutional networks in the ensemble mean.
5.1. Data Augmentation at Infinite Width
In the infinite width limit, the training dynamics under gra- dient descent can be solved exactly (Jacot et al., 2018). This enables us explicitly study data augmentation, showing that data augmentation and kernel averaging yield the same mean predictions, as detailed in the following
4.3. Rotations in 3d
Spherical signals subject to rotations in 3d are further important use case with numerous applications in quantum chemistry (Duval et al., 2023), weather prediction (Bonev et al., 2023) and 3d shape recognition (Fuchs et al., 2020). The group convolutions for the corresponding symmetry group (3) can be computed efficiently in the Fourier do- main, in terms of coefficients in steerable basis of spherical harmonics y Wigner matrices D D respectively (Co- hen et al., 2018; Cohen & Welling, 2017). Due to the con- tinuous nature of the (3) group, comprehensive data aug- mentation is not feasible, thus making group convolutional networks the natural choice to incorporate such symmetries. In Appendix D.I, we provide summary of the necessary Fourier space relations for (eq)i-qqui networks.
Theorem 5.1. Let ir aug. and be the mean predictions after t training steps of infinite ensembles of two neural network architectures Naau and N. Let N Afaug be trained on the fully G-augmented training data of N and assume that the NTKs of the two architectures are related by
O(f.f)
1 GG
Σ BEG
@UVE Preg
(32)
Then, pi p and μ4 converge in the infinite width limit to the same function for all t for quadratic losses, up to quadratic corrections in the learning rate.
The kernel forward equations (11), (12) simplify to purely algebraic equations in terms of Fourier coefficients
Proof. See Appendix E.
KF JaR f [AR' K (I, f')D (RYD") (R",, (31)
monmon
The proof of Theorem 5.1 proceeds inductively over training steps. At initialization, both mean functions are identically zero (Neal, 1996; Lee al., 2018). The updates for the two networks can be written in terms of the NTK and shown to agree by splitting the sum over augmented training data into a sum over samples and a sum over and using the assumption (32).
and analogously for the NTK. Note that the kernels have two group indices, thus necessitating double Fourier trans- form. Detailed relations for lifting- group convolutional- and group pooling layer in the Fourier space are provided in Appendix D.2. Again, these layers are implemented in the neural- neural-tangent -tannntns package and the necessary generalized FFTs are provided by the JAX-based package s2fft (Price& & McEwen, 2024).
In fact, the same argument can be used to show that the individual networks Aaaug and N (as opposed to their en- semble averages) agree if their empirical NTKs satisfy (32) as long as the networks are identical for all inputs and equiv- ariant at initialization. This would for instance be the case if N (x) = Naagg (z) = for all x at initialization.
5. Data Augmentation Versus Group Convolutions at Infinite Width
5.2. Kernel Averaging Yields GCNN-Kernels
The recursive relations presented in the previous sections give analytical access to the training dynamics of equivari- ant neural networks. In particular, they allow for more in-depth theoretical understanding of the similarities and differences of data augmentation and manifest equivariance than previously possible.
Theorem 5.1 shows equivalence of augmented and non- augmented networks if the NTKs of both architectures are related by group-averaging. Consider the case of training an MLP on augmented data. Then, (32) prompts us to consider the group-average of its NTK find the architecture which results in the same mean predictions if trained without data augmentation. By iterating the recursive kernel-relations found in the previous section, one can in fact show that this architecture is a GCNN, as detailed in the following
It known that ensembles of independently initialized neu- ral networks trained with data augmentation yield equivari- ant mean predictions (Gerken& & Kessel, 2024; Nordenfors & Flinth, 2024). It is however unclear how these equivariant functions relate to trained manifestly equivariant networks. Using the recursive relations from Section 4. it is possible to show that non-equivariant networks trained with data aug-
Theorem 5.2. Let NFC be an MLP acting on feature maps with output in IR and architecture
PC FC(L) 000 FF o
(33)
6

---

Equivariant Neural Tangent Kernels
where FC denotes a dense MLP layer and a point-wise nonlinearity. Let N be G-invariant GCNN with archi-
related by
(X,XN(J,(²=- vol(K) f dk K (U,Prog( Prog (k)))(())()) (37)
1
tecture
AGCC-
GPool o GC -
ΦKxN (J,N(( vol(K) f ck O ON Prog (k)) (k) (38)
1
... GConv ( oa Lifting(S)),
(34)
Proof. See Appendix E.
where S% are the supports of the convolutional filters with Sk = X, the domain of the inpul feature maps, and the other S% are invariant under G. Then, the G-averages of the kernels of the MLP are given by the kernels of the GCNN,
Remark 5.4. For K = C. and N = =R, i.c. roto- translations in the plane, A becomes an ordinary CNN and KKN N is of the form discussed in Section 4.2. Accord- ing to Theorem 5.3, the kernels of the rotation-equivariant network are then given by averaging the kernels of the CNN,
KCc (((())))= vol(G) ʃag
1
(35)
K K.RRRR2 (1,8)= 1 Σ
K (J. Preg Prog (r)
(39)
(GC(f.)") vol(G) / @Pe (f. (f. (36)
rEC
C (1.8) = 1 Σ
@CNN (F.N(
(40)
rEC,
Proof. See Appendix E.
if the spatial filter shapes agree for both networks and are rotation-invarii
Together with Theorem 5.1, this theorem shows that by augmenting arbitrary deep MLP at infinite width, one obtains a specific equivariant architecture, namely GCNN with the same depth I. and an additional group-pooling layer. This result singles out group convolutional layers among other equivariant layers and mirrors the fact that group convolutions are the unique linear equivariant layers under the regular representation. Note that according to Theorem 5.1 the equivalence between augmented and equiv- ariant networks holds throughout training and even out of distribution.
Taken togenther with Theorem 5.1, this shows that training an N-invariant GCNN on K K-augmented data results in a K N N-invariant GCNN, For the special case of N a transla- tion group and K a rotation group, this means that training CNN on rotation-augmented data is equivalent to training poo-translation equivariant GCNN on unaugmented data. In Section 6, we will show that this still holds approximately for width networks and ensembles.
5.4. Distribution of Ensemble Members
Consider training two ensembles of networks with (a) data augmentation on a non -ouivarrvai architecture and (b) no data augmentation on an equivariant architecture. Then, the distributions of the individual networks in these ensembles do not agree since most of the augmented networks will not be equivariant. However, our results show that the ensemble mean of (a) converges to the ensemble mean of (b) with a specific GCNN architecture. This establishes a highly non-trivial relation between data augmentation and GCNNs.
5.3. Augmenting CNN
Consider a generalization of the roto-translation symme- try discussed in Section 4.2, namely a general semidirect product group. G = K x N with N a normal subgroup of G. For N a translation group, this covers cases such as CNNs in two and three dimensions with additional rotation or reflection symmetry (Cesa et al., 2022).
The semidirect product structure of G allows splitting of the full cquivariance, namely training a K x --invariant GCNN is equivalent to training an N N-invariant GCNN on K K-augmented data. In order to see this, we show the corre- sponding kernel averages for Theorem 5.1 to hold:
6. Experiments
In the following, we validate the theoretical results of the preceding sections experimentally for various datasets (Ci- for QM9, MNIST, and histological data), tasks (regres- sion and classification), and groups (SO(3) and C4 x R2).
Theorem 5.3. Let. Let N be the K x N-invariant GCNN with architecture (34) and K-invariani filter supports St. which for the GConv-layers decompose as St. = =K{- x N Kf. C K, N S N. Let NN be the N-invariant GCNN with architecture (34) and filter supports ......, No and St. Then, the NNGPs and NTKs of these networks are
Kernel Convergence for C4 x IR2. Figure I confirms that the Monte-Carie estimate of the NTK converges to our analytical expression as the width increases. Our MC esti- mates are obtained by replacing the expectation values in (1)

---

Equivariant Neural Tangent Kernels
R. Frr
5003
MLP
Relative Error
MAE (me
0.4
02-
0.0
103
102
20
Network Width
Training Set Sizo
Figure 3. NTK for molecular energy prediction. Molecular en- ergy MAEs of the NTK kemnel methods in the infinite width and training time limit for different training set sizes. The results are both conventional MLP and -33va-inv GCNN.
Figure 1. Convergence of the Monte-Carlo estimates of the NTK to their infinite-width limits for G = C4 x R². Plot- ted is the relative Gram matrix for networks with ReLU or an error function non- linearity. Bands show one standard deviation of the estimator.
averaged over the components of 3
solution of the NTK-dynamics under MSE loss,
C. CNN
0.7
CNN
(z) (z, ())
(41)
0.6
where X' represents the training images, scaled down to 32 x 32 pixels, y' the training labels, given by e, $ for class c, and O X)) is the Gram matrix of the NTK. We refer to Appendix F.2 for more details.
Test Accuncy
0.5
0.4
0.3
Molecular Energy Regression with SO(3). We bench- mark the NTK-predictor resulting from SO(3)- network the QM9 dataset (Ramakrishnan et al., 2014) by predicting molecular energies Up from atom configurations utilizing (41). Comparing this to the corresponding MLP kernel, we observe a considerable performance boost for the invariant kernel over a range of training set sizes, as shown in Figure 3. For preprocessing, we construct spherical sig- nals from the atom configurations as described in (Esteves et al., 2023). For each atom i of the at most 29 atoms, the environment is represented by pairwise Gaussian smearing over atoms with the same atomic number
0.2
200
800
1000
Training Set Sizo
Figure 2. NTK for image classification. Test accuracy of the aris- ing NTK kemel methods in the infinite width and infinite training time limit for different training set sizes. The results for both conventional CNN and C4 k B2 invariant GCNN are shown.
by the sample mean of 1000 initializations of the network. We considered GCNNs with one lifting- and four group- convolution layers interspersed with ReLU nonlinearites, followed by group sooupppooling layer. The convergence of the NNGP is shown in a similar plot in Appendix F.I.
2,2 bb (177)==1)
11,2,(z) = Σ
(42)
|rag|
f:2y=a
Medical Image Classification with C₁ R.?. We show that inaatio-equivariann -Tee-rrdiitsors outperform non- equivariant NTK-predistors on dataset of histological im- ages (Kather al., 2018) containing nine distinct classes of tissues. Specifically, we compare a CNN architecture with the corresponding ntation-invarian GCNN architecture, in which we replace each of the five convolutional layers with a group-convolutioonal or lifting layer, respectively and used a group-pooling layer instead of a SumP col layer. Figure? 2 shows the improved scaling behavior of the equivariant kernel with training set size upon using the infinite-time
Choosing E {2, and considering all of QM9's five atom types leads to 29 spherical per-atom signals with 5 x 2 channels each. Each of those per-atom signals are then either processed by a two layer (33))e-quivarian network with group pooling on top or by two-layer MLP. The per- atom outputs eventually summed and fed into final fully-connected layer similarly as in (Cohen et al., 201 8). The input signals are constructed at a resolution of 12 x 11 on the sphere, corresponding to a bandlimit of L = 6, which are then downsampled to a bandlimit = for the group layer. We provide further details Appendix F.3.
8

---

Equivariant Neural Tangent Kernels
100
pooling layer and are thus specialized to invariant GCNNs. An equivariant extension would require new layers beyond those presented here since in the infinite-width limit, the NTK of an MLP becomes proportional to the unit matrix in the output channels, trivializing the feature map.
Ensemble Size
0.9
Ensemble
(7equis-oggh vun.egav--oggity
Enseuible Sizp 8
0.8
Finenble Sine
0.7
Acknowledgments
J.G. and P.M. want thank Max Guillen for inspiring dis- cussions and collaborations related projects. The work of J.G. and P.M. is supported by the Wallenberg AI, Au- tonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The compu- tations were enabled by resources provided by the National Academic Infrastructure for Supercomputing Sweden (NAISS) and the Swedish National Infrastructure for Com- puting (SNIC) at C3SE partially funded by the Swedish Research Council through grant agreements 2022-06725 and 2018-05977
10
12 13
14
16
20
Epoch
Figure 4. Convergence of finite-width ensembles trained with data augmentation to ensembles of GCNNs on MNIST. L²- distance between the logits of the equivariant- and non-equivariant ensemble trained with data augmentation for different ensemble sizes on out of distribution data, For larger ensembles, the distance decreases.
Impact Statement
Data Augmentation Versus Group Convolutions at Fi- nite Width. In Section 5, we proved that networks trained with data augmentation converge in expectation to group convolutional networks at infinite width. We also verify empirically that the mean predictions agree progressively with increasing ensembles size at finite width, supporting approximate validity away from the theoretical limit. To this end, we train ensembles of CNNs and GCNNs with symme- try group C4 x 2²² as discussed in Remark 5.4 on CIFARI I0 and MNIST using the MSE-loss against smoothed one-hot hot labels as for the medical images above. For implementing the GCNNs, we used the escnn-package (Weiler & Cesa, 2019). As shown in Figure for MNIST, the outputs of both ensembles converge to the same vector for large ensemble sizes throughout training and even out of distribution. For further details on the model architectures, out of distribution data, as well as results on CIFAR10 10 and histological images, see Appendix E.4.
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T.,
Pritzel, A., Ronneberger, O., Willmore. L., Ballard, A. J.. Bambrick, J., et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Amari, S.-i. Natural Gradient Works Efficiently in Learn- ing. Neural Computation, 10(2):251-276, February 1998. ISSN 0899-7667. doi: 10. (0.11622899976698000077744
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang. R. On Exact Computation with an Infinitely Wide Neural Net. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/pproceedingsneettt cc cc/paper cc/paper_filesteeeeeeeeeeeeeeeee https///ppppr//0/////e////b/// hbt.ceeee22aaaaaaebeeeeeebeaaee df.
7. Conclusion
This paper provides recursive relations for the NNGP and NTK for group convolutional neural networks allowing us to theoretically establish an interesting equivalence between cquivariance-barn based to aataa-uugmentaiion- based training dy- namics. We also show that equivariant kernels outperform their non-equivariant counterparts as kernel machines.
A careful comparison of equivariant GCNNs and equivari- ant data augmentation, beyond the invariant case analyzed in Section 5, would be an interesting subject for further re- search. In particular, Theorem 5.1 can be straightforwardly extended to the equivariant case, as demonstrated in Ap- pendix E. However, Theorems 5.2 and 5.3 rely on the group
Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthama- rakshan, V., Varshney, K. R., Varshney, I.. R., and Das, P. Equi-Tuning: Group Equivariant Fine-Tuning of Pre- trained Models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6788-6796, June 2023. doi: 10.1 10.1609/aaai v37i6.25832 URL

---

Equivariant Neural Tangent Kernels
https://ojs. httpp//rgg//eepppp//eaa...e index.ddexxx. wwtpexx...pp.aAaaa/a article/view/25853
Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. Geometric deep learning: Going be- yond Euclidean data. IEEE Signal Processing Maga- zine, 34(4):18-44 July 2017. ISSN 1558-0792. doi: 10.1109/MSP.2117.22222 2763444.....
Batzner, S., Musaelian, A., Sun, I.., Geiger, M., Mailoa, J. P., Kornbluth, M., Molinari, N., Smidt, T. E., and Kozinsky, B. E(3)-Equivariant Graph Neural Networks for Data- Efficient and Accurate Interatomic Potentials. Nature Communications, 13(1), May 2022. ISSN 2041- 1723. doi: 10. //doi wr..orgg////o
Canatar, A., Bordelon, B., and Pehlevan, C. Spectral bias and task alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications, 12(1), May 2021. ISSN 2041-1725, doi: 10.003//41-66--01122301111 URL ht http: doi. wog..orgg10.0m 01tp:/111.000311111700000-0--110
URL https:
Bekkers, E. J., Lafarge, M. W., Veta, M., Eppenhof, K. A. J., Pluim, J. P. W., and Duits, R. Roto-Translation Covariant Convolutional Networks for Medical Image Analysis. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018, Lecture Notes in Computer Science, pp. 440-448, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00928-11 doi:
Celledoni, E., Ehrhardt, M. J., Etmann, C., Owren, B.,
Schönlicb, C.-B., and Sherry, F. Equivariant neural net- works for inverse problems. Inverse Problems, 37))): 085006, jul 2021. doi: 10.10888/1311-6440/aecoo00 URL https:/ / dx dww.doi..oo org 10. 088 6420/ ac104f.
Bekkers, E. J. Vadgama, S. P., Hesselink, R., van der Linden, P. A., and Romero, D. W. Fast, expressive se(n) equivariant networks through weight-sharing in position-oriennattonn space. In International Conference on Learning Representations, ICLR, 2024. URL https: // //openreview net/ fort//orrreee Peeeremmmmdddddiiieiii dd HLDUgGG...
Cesa, G., Lang, I.., and Weiler, M. A Program to Build H(N)-
Equivariant Steerable CNNs. In International Conference on Learning Representations, ICLR, 2022. URL https: / / //opperevii ew nwt. wwweeeeeoourrmeee furum? d
Chidester, B., Ton, T.-V., Tran, M.-T., Ma. J., and Do, M. N. Enhanced Rotation Equuvion-quuvvariantt U-Net for Nuclear Seg- mentation. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1097-1104, June 2019. doi: 10.1109/CVPR.220990000 43.
Bogatskiy, A., Anderson, B., Offermann, J., Roussi, M., Miller, D., and Kondor, R. Lorentz group equivariant neural network for particle physics. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 902-1002, PMLR, 13-18 Jul 2020. URL https: : fttt:::/pprceediing mlr. press / ogatskiyy0a.nne
Cohen, T. Equivariant Convolutional Networks. PhD thesis, University of Amsterdam, 2021.
Cohen, T. and Welling, M. Group Equivariant Convolutional Networks. Proceedings of The 33rd International Con- ference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: http///ppceeeeg...e/////// mlr prwss/v48/co hencl 6. html.
Bonev, B., Kurth, T., Hundt, C., Pathak, J., Baust, M., Kashinath, K., and Anandkumar, A. Spherical Fourier neural operators: Learning stable dynamics on the sphere. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pp. 2806-2823, PMLR, 23-29 Jul 2023. URL httpttttt/occceeeeeeeeeeee ess ess/v202 w222///boo/223..tmmm
mlr pr
Cohen, T. and Welling, M. Steerable CNNs. In Interna- tional Conference on Learning Representations, ICLR, 2017. URL https: http//pppnrrviiiee nwt.
Bordelon, B., Canatar, A., and Pehlevan, C. Spectrum de- pendent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 1024-1034. PMLR, 13-18 Jul 2020. URL https: //proeeedi ngs. 1 press/ / v1 ht/bbrdoelooooooo
Fid=rJoKY:511.
Cohen, T. S., Geiger, M., Köhler, J., and Welling, M. Spher- ical CNNs. In International Conference on Learning Representations, ICLR, 2018. URL https:/ /openre view. www..net/ee.e forum? / d.d.kdd22kRd
Brehmer, J., Behrends, S., de Haan, P., and Cohen, T. Does equivariance matter at scale? arXiv, October 2024. ISSN 2331-8422. URL https: /olsss//axxiie /arxivvvv.. org/ abs//b///// 24 10. 13179..
Day, H., Kahn, Y., and Roberts, D. A. Feature Learning and Generalization in Deep Networks with Orthogonal Weights. arXiv, October 2023. ISSN 2331-8422. URL 07765.
https:://rxxvv.or///aaa
10

---

Equivariant Neural Tangent Kernels
Dombrowski, A.-K., Gerken, J. E., Müller, K.-R., and Kessel, P. Diffeomorphic Counterfactuals With Gener- ative Models. IEEE Transactions Pattern Analysis and Machine Intelligence, 4665):3257-32744 May 2024. ISSN 1939-3539. doi: 10.1 1..100//PAM222233339999
Fuchs, F, Worrall, D., Fischer, V., and Welling, M. Se(3)- transformers: 3d coto-translation equivariant attention networks. In Advances in Neural Information Processing Systems, volume 33, pp. 1970-1981. Curran Associates, Inc., 2020. URL httpo//tiiittaeeprrreednnnngggo s oc hccc//ppaer__LLeee/ppeer//////// ( df.
Driscoll, J. and Healy, D. Computing fourier transforms and convolutions on the 2-sphere. Advances in Applied Mathematics, 15(2):202-250, 1994. ISSN 0196-8858. doi: http:///orrrr..00000/o/aaaaa..9/
e https::/www.sceeeeedireettt 1994////0aa..9411000. URL
Geiger, M., Spigler, S., Jacot, A., and Wyart, M. Disentan- gling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(1 133001 November 2020. ISSN 1742-5468, doi:
Du, S. S., Hou, K., Salakhutdinov, R. R., Poczos, B., Wang, R., and Xu, K. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httttt//pppprr eedings www.:nuuriip..o wttp:/ccppppppppppaeeeeeeeeeeeee eecings..eerrips.ccc/ppp.. httpperrrrieee/pppppp
10.
Gerken, J. E. and Kessel, P. Emergent Equivariance in Deep Ensembles. In Proceedings of the 41 International Con- ference on Machine Learning, volume 235, pp. 1 15438- 15465, PMLR, 21-27 Jul 2024. URL hww.t:tpp////// /tsttts://proot eedings mlr. wrw.pprsssp.ee...e v235/2235///// http://///.eeeea....m html.
9a5b92d-Paper.
Gerken, J. E., Carlsson, O., Linander, H., Ohlsson, F., Pe- tersson, C., and Persson, D. Equivariance versus Aug- mentation for Spherical Images. In Proceedings of the 39th International Conference on Machine Learning, vol- ume 162, pp. 7404-7421, PMLR, 17-23 Jul 2022. URL https: https://proceddngg mlr wer..pressse press/v162 erken22a. html.
Duval, A., Mathis, S. V., Joshi, C. K., Schmidt, V., Miret, S., Malliaros, D., Cohen, T., Lio, P., Bengio, Y., and Bronstein, M. Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems. arXiv, December 2023. ISSN 2331-8422. URL /ttpt::://aaxi..../ttttttt wttpa/iwwww.../////a///aaaaaa/ 12. 07511.
Engel, A., Wang, Z., Frank, N., Dumitriu, I., Choudhury, S., Sarwate, A. D., and Chiang, T. Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models. In International Conference on Learning Representations, ICLR, 2024. URL https: //openreviee net
Gerken, J. E., Aronsson, J., Carlsson, O., Linander, H., Ohlsson, F., Petersson, C., and Persson, D. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, June 2023. ISSN 1573-7462. doi: 10.
d waaddd-KKKKKa yKksu38Bpkt
Golikov, E., Pokonechnyy, E., and Korviakov, Neural Tangent Kernel: Survey. arXiv, August 2022. ISSN 2331-8422. URL ut /tattps///axiiv. org/abs/22 08 13614.
Erbin, H., Lahoche, V., and Samary, D. O. Nonperturba- tive renormalization for the neural network- Network-QFT corre- spondence. Machine Learning: Science and Technol- ogy, 3(1):015077, March 2022. ISSN 2632-2153. doi: 10.10888/632-215534cc16
Halverson, J., Maiti, A., and Stoner, K. Neural Networks and Quantum Field Theory. Machine Learning: Science and Technology. 2(3):035002, September 2021. ISSN 2632-2153. doi: 10.1088//632-21553abbe
Esteves, C., Slotine, J.-J., and Makadia, A. Scaling spherical CNNs. In Proceedings of the 40th International Confer- ence on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 9396 011199911.1111 PMLR, 23-29 Jul 2023. URL https: http:///pooeeeeeeeeeeeee ml r. r.press/ /ttees/v2220 v222 esteves22..aa.e html.
Hayase, and Oh, Few-shot Backdoor Attacks via Neural Tangent Kernels. In International Conference on Learning Representations, ICLR, 2023. URL https: / //openreview. net forumm/oorrmmmm 701.dd=-
Franceschi, J.-Y., Bézenac, E. D., Ayed, I., Chen, M., Lam- prier, S., and Gallinari, P. A Neural Tangent Kernel Perspective of GANs. In Proceedings of the 39th In- ternational Conference on Machine Learning, volume 162, pp. 6660-69 6704. PMLR, 17-23 Jul 2022. URL https: httpp/oceedngggs www..mm.s.....eeeeeeeeeeee.eeess /v162/E ranceschi 122.. html.
Huang, J. and Yau, H.-T. Dynamics of Deep Neural Net- works and Neural Tangent Hierarchy. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119, pp. 4542 PML. 13-18 Jul 2020. URL https: : htt:://///eeeingeeeeee mlr. prwpspssssse.......... /vl 19 /9/huang201, html.

---

Equivariant Neural Tangent Kernels
Jacot, A., Gabriel, F., and Hongler, C. Neural Tangent Kernel: Convergence and Generalization in Neural Net- works. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL paper bb8a6ec6b91d224 paapraaaa...222222aaaaaerrreer
Liao, Y. and Smidt, T. E. Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs. In Inter- national Conference Learning Representations, ICLR, /e www.review.net. forw
2023. URL https: ? Rid=KwmP FARgOTD.
htttp:://proceeddnngg meurt #iLls//papee/20088EELe///S/
Mok, J., Na, B., Kim, J.-H., Han, D., and Yoon, S. De- mystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training? In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11851-11860, June 2022. doi: 10.1109/CVPR52688.222
Jacot, A., Gabriel, F., Ged, F., and Hongler, C. Order and Chaos: NTK views on DNN Normalization, Checker- board and Boundary Artifacts, arXiv, June 2020. ISSN 2331-8422. URL https: / htt///axvvvvv..///////eee/e/eevl wrgg 05715.
22.01156.
Kather, J. N., Halama, N., and Marx, A. 100,000 histologi- cal images of human colorectal cancer and healthy tissue. Zenodo, April 2018. URL https: / www.o.oooog...... 281/zenn zen/zenodo.. 1214456.
Moskalev, A., Sepliarskaia, A., Bekkers, E. J., and Smeul- ders, A. W. On genuine invariance learning without weight-tying. Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learn- ing (TAG-ML), volume 221 of Proceedings of Machine Learning Research, pp. 218-221 PMLR, 28 Jul 2023. URL hww.hhttsss//tttt /tettps://rrcceedd w....g.....es....e...ee. p.psss/v2 ht/mmskkalev23a..ttt..
Kondor, R. and Trivedi, S. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In Proceedings of the 35th International Conference on Machine Learning, vol- ume 80, pp. 2747-2755. PMLR, 10-15 Jul 2018. URL mml.g.s..r......... www...aw....e.....m............e vv00/k/v////k/
Moyer, D., Abaci Turk, E., Grant, P. E., Wells, W. M., and Golland, P. Equivariant Filters for Efficient Tracking in 3D Imaging. In Medical Image Computing and Com- puter Assisted Intervention MICCAI 2021, pp. 193-202, Cham, 2021. Springer International Publishing. ISBN 978 978-3-030-87202-1. doi: 10.1007/978-3-000-87200 1.1 9.
ador18a.html.
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Penning- ton, J., and Sohl-Dickstein, J. Deep Neural Networks as Gaussian Processes. In International Conference Learning Representations, ICLR, 2018. URL https:
Müller, P., Golkov, V., Tomassini, V., and Cremers, D. Rotation-EQquivariat Deep Learning for Diffusion MRI. arXiv, February 2021. ISSN 2331-8422. URL https: //arxiv. org/ htw.s///22..002 06942.
wwwp:eeewwwiwwwweoeeeeeee //openreview. nwt. foreem//oummmmme BeterrummidBBBEAAMM 1et.://E
Lee, J., Xiao, I.,, Schoenholz, S., Bahri, Y., Novak, R., Sohl- Dickstein, J., and Pennington, J. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/proceedingg..m nww.pisssiings..e.eese........ cpppper _f wftlee//ppprr/000////eelaaaaaaa. 8d#b1c3871c84528Bdd
Neal, R. M. Bayesian Learning for Neural Networks. Springer Science & Business Media, 1996. ISBN 978-1- 4612-0745-0.
Nicoli, K. A.. Nakajima, S., Strodthoff, N., Samek, W., Müller, K.-R., and Kessel, P. Asymptotically unbiased estimation of physical observables with neural samplers. Physical Review E, 101(2)002334, 2020.
Lee, J., Schoenholz, S., Penningtonn, J., Adlam, B., Xiao, L.,
Novak, R., and Sohl-Dickstein, J. Finite versus infinite neural networks: An empirical study. In Advances in Neural Information Processing Systems, volume 33, pp. 15156-15172. Curran Associates, Inc., 2020. URL ht / igs. neur ct/ ctt.ppsr_cppppp__.. LLae/pppper/222//111ee//ea 0773f8d0ca22eall Pal.r
Nordenfors, O. and Flinth, A. Ensembles provably learn equivariance through data augmentation, arXiv, October 2024. ISSN 2-31- URL https::///artt g abs 2410 01452.
Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl- Dickstein, J., and Schoenholz, S. S. Neural Tangents: Fast and Easy Infinite Neural Networks in Python. In International Conference on Learning Representations, ICLR, 2020. URL https: /openreview www.iiew.eeee..e. / orum?i 6r
Li, Z., Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdinov, R., and Arora, S. Enhanced Convolutional Neural Tangent Kernels. arXiv, November 2019. ISSN 2331-8422. URL https: //t.xvvv..v/v/////v///e///iiavvv wwwg/bs/////eeoooeo....oeeeeeeee
abs / 911 00809.
12

---

Equivariant Neural Tangent Kernels
Pang, S., Du, A., Orgun, M. A., Wang, Y., Sheng, Q Z.,, Wang, S., Huang, X., and Yu, Z. Beyond CNNs: Ex- ploiting Further Inherent Symmetries in Medical Image Segmentation. IEEE Transactions on Cybernetics, 53 (11):6776-6787, November Novembee 2023. ISSN 2168-2275. doi: 10.1109/TCYB.2022..33 195447
Wang, S., Yu, X., and Perdikaris, P. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110766 January 2022b. ISSN 0021-9991. doi: 10. 10.1016ffjjpp.2221.100 10768
Wang. Y. Elhag. A. A., Jaitly, N., Susskind, M., and Bautista, M. Á. Swallowing the Bitter Pill: Simplified Scalable Conformer Generation. In Proceedings of the s International Conference on Machine Learning, vol- ume 235, pp. 50400-50418, PMLR, 21 21-27 Jul 2024, URL https:: http//peeeeeennsceeeeoeeepo nggs m1 http:g/..........r..e...... ptwss//235/ ang24g. hogg2q.html. html.
Perin, A. and Deny, S. On the ability of deep networks to learn symmetries from data: A neural kernel theory. arXiv, December 2024, ISSN 2331-8422. URL https: ///axxiv... org. abs/ 211.2
Perraudin, N., Defferrard, M., Kaeprzak, T., and Sgier, R. Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological appli- cations. Astronomy and Computing, 27:130-146 April 2019. ISSN 2217-1337. doi: (0.1016/jasccom2211.. 004.
DeepSphere:
Weiler, M. and Cesa, G. General EQu)--qqi Steer- able CNNs. In Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019. URL ht
cc/p Sper_files/pappeee / htw.a/000000f411000-----eeeeee
Price, M. A. and McEwen, D. Differentiable and accel- erated spherical harmonic and wigner transforms. Jour- nal of Computational Physics, 510:113109, 2024. doi: 1..004//..pp.2222111130
Weiler, M.. Hamprecht, F. A., and Storath, M. Learning Steerable Filters for Rotation Equivariant CNNs. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018.
Ramakrishnan, R., Dral, P. O., Rupp, M., and von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Weiler, M., Forré, P., Verlinde, E., and Welling, M. Equivari- ant and Coordinate Independent Comvolutional Nenworks. 2023. URL htttts:://auriieeeeeeeeeee 00t/0//////t////aarrrrreeeeeeeee
Schütt, K., Unke, O., and Gastegger, M. Equivariant Mes- sage Passing for the Prediction of Tensorial Properties and Molecular Spectra. In Proceedings of the 38th In- ternational Conference on Machine Learning, volume 139, pp. 9377-9388. PMLR, 18 24 Jul 2021. URL https: http///pooceee...cceeooo ml press/v139/s chutt2la.htm.
i htt/cnnooooo//eennoeoonnoo h/t/////boo//ggguaaraaaaaaaaaola e IndependenttCNS pdf.
Winkels, M. and Cohen, T. S. Pulmonary nodule detection in CT scans with equivariant CNNs. Medical Image Analysis, 55:15-26, July 2019. ISSN 1361-8415. doi: 00.0066f..edia.2119...00
Tsai, C.-P., Yeh, C.-K., and Ravikumar, P. Sample Based Explanations via Generalized Representers. In Advances in Neural Information Processing Systems, volume 36, pp. 23485 23498. Curran Associates, Inc., 2023. URL 234998
Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S., and Pennington, J. Dynamical Isometry and Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks. In Proceedings of the 35th International Conference Machine Learning, vol- ume 80, pp. 5393-5402. PMLR, 10-15 Jul 2018. URL ht https::/proceed:: /ttp//eeeeeengs..cc/e/ ml 1r.. paress/v80. aol 8a html.
https //tppe/edg/cecenngggs neurips.eersppp.. _ftlee//pppeer//003///lll////SS
b1452cbb900330336000----000 pdf.
Unke, O., Bogojeski, M., Gastegger, M., Geiger, M., Smidt, T., and Müller, K.R.. SE(3)-equivariant prediction of molecular wavefunctions and electronic densities. In Advances in Neural Information Processing Systems, vol- 34, pp. 14434-14447, Curran Associates, Inc., 2021. URL httpttts/ppprceediiiii
Yaida, S. Non-Gaussian processes and neural networks finite widths. In Proceedings of The First Mathematical and Scientific Machine Learning Conference, volume 107, pp. 165-192. PMLR, 20-24 Jul 2020. https: / //proceedings mlr wwwpppres/seee vitpa////ov1/yaaa22aa.... html.
aper appe__fiiee//pppr//000///111/111 8a Batbbaas9011100119900006---
Yang, G. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation. arXiv, April 2020. ISSN 2331-8422. URL https: / //arxiv.c hwt.://rgg/b//a/aeeeo/oo 1902. 04760.
Wang, R., Walters, R., and Yu, R. Data augmentation vs. equivariant networks: A theory of generalization on dy- namics forecasting. arXiv, June 2022a. ISSN 2331-8422. URL htt.t:ts///aaatt https///axxvv.o///eeeee wtw.g///b///000 09450.
13

---

Equivariant Neural Tangent Kernels
Yang, G. and Hu, E. J. Tensor programs IV: Feature learn- ing in anfinit-wwitth neural networks. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 11727-11737. PMLR, 18-24 Jul 2021. URL htt / www.peee.oggg........./p. ml r. prwssrsssvvv.. vvl 39/ 19//ang21c... html.
Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. In Advances in Neural Information Process- ing Systems, volume 34, pp. 17084-17097. Curran Asso- ciates, Inc., 2021.
Yang, H. and Wang, Z. On the Neural Tangent Kernel Anal- ysis of Randomly Pruned Neural Networks. In Proceed- ings of The 26th International Conference on Artificial Intelligence and Statistics, pp. 1513-1553, PMLR, April 2023.
14

---

Equivariant Neural Tangent Kernels
A. Basics of Neural Tangent Kernel Theory
neural network Ne I - IR Rous which trained using continuous gradient descent
d0
OC
7) 0e
(43)
dt
on a loss function L with learning rate evolves according to dv
Asraio
OC (2, (N;)))'
1
(44)
dt
where the sum runs over the training samples z; and θ E R Khout is the NTK
0x(2,2')= @N(2)
(oN(e')
(45)
ae
00
For tnith-width networks, θ depends on the initialization and the training time and is referred to as the empirical NTK. At infinite width, θ, becomes independent of the initialization (it still depends on the initialization distribution) since it approaches its expectation value over initializations
)
(x
(xN(')) 00
(ar, x²) = Ed--piai
(46)
00
It furthermore becomes constant throughout training and proportional to the unit matrix (Jacot et al., 2018) in the NTK parametrization. For this reason, we drop the t-subscript on this frozen NTK and treat it as a scalar. the following, we will always mean (46) when we refer to the NTK unless otherwise stated. The NTK parametrization of a linear layer has an additional 1/ in prefactor and uses independent standard Gaussians initialization distributions. Hence, an MLP layer is given by
( =0
( WN (
1
(47)
with nonlinearity o, weights W and bias
B. Proofs: Kernel Recursions for GCNN-Layers
In this section, provide proofs for the theorems given in Section 4.1 in the main text. Theorem 4.1 (Kemel recursions for group convolutional layers). The layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (4) are given by
k((+1) (f.f'
x((+t)(ff') [Sx Sx
dh K-O0 (F,
(11)
((++))
-K(et") "f 9.9"
(f.f')
dh d o (f
(12)
9.98
[Sp] S.
Proof. We first compute the NNGP recursion relation. For group-convolution layers, the definition (3) of the NNGP reads
((t+1) (f. = E [w(++) (((()))))))))))))))))))
(48)
1 dh dh' (e(eet)(g (t) (g'
(49)
Sx Ja
where we have again dropped the 1/ /rrppeffccco and channel dependencies since these converge to the expectation value in the infinite width limit. Next, we shift the integration variables by g and g' which leaves the Haar measure invariant by its definition
K (B+1) 9.9"
(f.
1 / dh Sx.
E (1
(50)
15

---

Equivariant Neural Tangent Kernels
Since the kernel components sampled independently from standard Gaussians initialization, only obtain contribution to the integral when h h'' and g(t+1) has support this point, i.e.
3/1 / dh B (190))11) Su
(U.)) K 9.9¹
(51)
Comparing to (3) shows that the right-hand side is just the NNGP (f, f') of the previous layer evaluated at group indices gh and g'h. h. This proves the NNGP recursion relation stated the theorem.
For the NTK recursion relation, we start by specializing the general expression (2) to group-convolution layers and adapting the functional framework used for feature maps
θ ((e-1) (J:J"
9-0¹
S(N (F)(C))
dh E
8nle+1)(h)
8xle+1)(h)
8m
E/ dh (L (N)
(f
/
+ dğ dğ'
E
(52)
G
((I(())
(h) (k)
(
R. (1))
According to the layer definition (4), the derivatives evaluate to
8(A71+1)) (()))))))))))))))))))
(
(53)
(x((+1((h)
VnrS.
8[A7(++)) (f](g)
x (g
(54)
(
VneS..
Therefore, (52) becomes
1 / dh ([10))))))))))
θ 9.9
Sx s.
S. JG dğ @ (f. E k (o
1
(55)
k((+))) 9.9" Sx S.
(1,8)
1
dh 000
110 (I,
(56)
where we have dropped the Shannel-prefactorn usual. The last line is just the NTK recursion be proven.
Theorem 4.2 (Kernel recursions for the lifting layer). The layer-wise recursive relations for the NNGP and NTK of the lifting layer (6) are given by²
dx (0) [ S. plje, ol (f, (f, (f."()))))
1
(16)
9.9"
S.S.T / dx (())
(1) Majee plg (f.f)
Sn
+ 9.9'
k-((+1)
(f,S')
(17)
where the regular representation Preg is defined in (5).
II practice, the lifting layer is usually first layer, thus 0.
16

---

Equivariant Neural Tangent Kernels
Proof. The NNGP of the lifting layer (6) is given by
K k
(f.f)=
[ [wtet1]] (f)(()) (VTE)
(57)
9:9¹
Sx dr E ( ())))())))))) E (MM
(58)
1
/ dx Bis()) (MM
(59)
5R x
dr dr [=
(60)
Sp Su
1
k( (f
(61)
Sw Js..
where we have moved the regular representation through N onto f by using equivariance, This proves the NNGP recursion-relation.
According to (2), the NTK recursion evaluates
(8(A(())) (f)(()) (F
"ff 9:0
(f.f))
= { dr E
8x((+1)(x)
(x(
Sn
(
d di'
(62)
((
(
The derivatives in this expression are given by
(()())))))))))))))
(F)((()))
(63)
(x)
VneSA
S(ATT))) (f)l())
(
(64)
(f)(()
Vne..
Plugging this back into (62) yields the desired NTK recursion relation,
2 (f.f)
1 / de E [NNNUNNAAA
S₂ S.
+ Sx / dā ( 8'DL (
(65)
/ dr ( (t.""
(f.f")
(66)
9:9
Sr S..
Theorem 4.3 (Kernel recursions for group pooling layer). The layer-wise recursive relations for the NNGP and NTK of the group pooling layer (10) are given by
(((+))(J,))) = (vol [ dg [ dg' K(0 s.
(1,5")
(18)
(((+)) (v)) J( dg [ dg' 60 99 (f.f'.
(19)
Proof. Since we integrate over the entire domain the input feature maps (f) G + R in the pooling layer (10), are the output features (f) E R a channel-vector. Therefore, the NNGP the group pooling layer given by
k(++)) () E [r(ttt (()(()))))
(67)
(vol(G)) [ dg [ ad e (vol(()) / dg S dg' (1.8).
(68)
(69)
17

---

Equivariant Neural Tangent Kernels
The NTK recursion (2) is in this
((.t)) - 1 I [dg' E (f 0 and ( (f.f') (oCCrOn
(70)
(vol(G))² [ dg J dg ad (1,5), (T,,'
(71)
which is the NTK-relation to be shown.
C. Equivariant Kernels for Roto-Transsstions in the Plane
In this appendix, we provide explicit expressions for the kernel recursions of lifting- group convolutional- and group pooling layers for the special case of roto-translations in the plane, i.c. for the symmetry group G C, x E In this case, the general expressions in Theorems 4.1, 4.2 and 4.3 simplify and can be written in terms of the A operator (30) which can be computed efficiently in terms of ordinary 2d convolutions. However, before discussing the kernel recursions, we will first establish simplifying notation for the GCNN layer-definitions.
C.1. GCNNs for G Cn K 2
Due to the semidirect-product structure of G, any element g G can be written uniquely as product of a translation and a rotation, tr with E TR? and C, 3, We can therefore write a feature map A/(5) on G as a stack of feature maps on IR2
=
(72)
Using this representation, the lifting- and group convolutional layers can be written in terms of ordinary twoddimensional convolutions as (Cohen & Welling, 2016)
Vrin Sw / dr x(p(r- -x)
1
(f)L(()
(73)
[M+1) (N+T()))))))) (f))+t)=
de' Ki- -))
121,
(74)
Vnel SA E'E,, J32
where p the fundamental representation SO( 2 given by two-dimensional rotation matrices.
Finally, for invariant problems like classification, the group pooling layer (10) is central to making the network invariant. For On x R2, it is given by
Σ fan dn (f),
[(++ Vnl supp((O((()))
(75)
rEC.,
C.2. Kernel recursions for G Cn x R
In analogy to the notation introduced in the previous section for feature maps, we write for the NNGP and NTK on Cn x
R
(e (f,
Oy=tro'= (f. (0. (f,(t))
(76)
to emphasize the dependency on the two translations ł, E R², Furthermore, we repeat here the definition of the operator (30) for convenience
[8a] / di K(t+
[A,, (K)]((t)))
(77)
Given these definitions, the recursive relations from Theorem 4.1 for group convolutions be computed efficiently using the following
'In abuse of notation, we will denote both the abstract translation group element and its representation vector in R² by the same symbol.
18

---

Equivariant Neural Tangent Kernels
Lemma C.1 (Kernel recursions of group convolutional layers for roto-translations). In the case G Cn x IR2, the layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (74) are given by
[ Pp
KK (1,')1((,))= = -E Lo (R,
(78)
FECa
(f, = (F, (f),))) + Σ [Aptrss (6 (f p(r"
(79)
FEC,
where
C. 61 = (f
(80)
(81)
Proof. In order to compute the NNGP recursion relation in the notation (76), we first need to compute the unique decomposition of a general group multiplication gh, g. h E G into rotation and a translation. This is possible since G is a semidirect product group. Starting from = tg and = 4,77 with ty. t E IR2 and rg. rh E Ca, we have
gh lT LaTA ghug Tggh.
(82)
Since R2 is a normal subgroup of G (a further property implied by the semidirect product), rglurg E R.2. Therefore,
fgh torgtarg R² and
rgh rrgh Cn.
(83)
Since the action of fgh on a vector
R2 is given by p(tgh)xx== ()r + tgg
(84)
we obtain for the NNGP recursion from Theorem 4.1,
((S'),)))) = S&
di [K (1,r,) 4. D 1').
(85)
PEC
This we will now write in terms of the A-operator (77). However, since the A-operator shifts both slots of the argument kernel by whereas the first argument in (85) is shifted by p(r- while the second argument is shifted by p(r'-1)č, we cannot write (85) directly in terms of A(K), but need to compute A at transformed argument instead and then transform back. To this end, first consider
/ dē (f. +t)
[As.,
(86)
[8a1 S.
- [ dē (F, ( E + +i)).
(87)
[al| S...
Therefore, we obtain for the RHS of the NNGP recursion
Σ (oprrss (KA
FEC
di ((1,))00.11((((,))))))))))) + +1)
(88)
Sx
p(r)S,
PEC,
di p(r)S,
+
(89)
SN
FEC,
1
di [K10 (1 (() +()))))
(90)
Sx
PEC,
S..
The last line is just (85), proving the NNGP recursion relation.
19

---

Equivariant Neural Tangent Kernels
For the NTK, start from the NTK-recursion in Theorem 4.1. The structure of the integral appearing in that recursion is the same as the one of the integral in the NNGP recursion. Therefore, the NTK recursion is given by
| (0 (1,
(J, ((), (K
Σ
ai (
tt)
(91)
SR FEC S.
The integral can be written in terms of the A-operator following the same steps as for the NNGP above.
Therefore, by first computing the kernels k(t) and ( and then applying the A-operator, it is possible to efficiently
compute the kernel-recursions in this case. Similarly, the recursive kernel-relations for the lifting layer can also be written efficiently in terms of the A-operator, as detailed in
Lemma C.2 (Kernel recursions of lifting layers for roto-translations). The layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (73) are given by
" (U, = (ktr)s.,K/9 ( 10 (f, -
(92)
(93)
where
((((,)) = (,"" (f()tt) =
(94)
(t))
(95)
Proof. According to Theorem 4.2, the NNGP recursion is given by
K (1,8))(,1)
/ f dr Sm S.
(96)
Comparing this expression to (85) shows that the sum over F as well as the r, r'indicess on (( are absent in (96) but otherwise the two expressions agree. Therefore, we can use the same argument as above to rewrite (96) in terms of the A-operator and only need to drop the r, riindices in the definition of K (6) as well as pick the = contribution in the sum. Similarly, we can show the NTK recursion relation starting form the NTK recursinn in Theorem 4.2.
Finally, in the group pooling layer, the kernels are trivialized over their and t-indices, resulting in kemel without spatial indices:
Lemma C.3 (Kernel recursions of group pooling layers for roto-translations). The layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (75) are given by
K ((e-") =
Σ
f dt ((((,,
(97)
na| supp(( EC,
f dt (6 (f.
((--))fff)) no suppp(110
Σ Er
(98)
Proof. The integral over two copies of the group in Theorem 4.3 factorize for G C, x R2 into integrals over the translations in IR2 and sums over the discrete rotations in Cn. This immediately implies the recursions in the statement of the lemma.
The expressions given in the lemmata in this section can be straightforwardly implemented and therefore allow for explicit calculations of the NTK and NNGP of realistically-sine GCNNs.
20

---

Equivariant Neural Tangent Kernels
D. Equivariant NTK in the Fourier Domain for 3d Rotations
D.I. Group Convolutions in the Fourier Domain for G SO(3)
For compact groups, is possible to define a Fourier transformation. The group convolution (4) then becomes point-wise product in the Fourier domain, For the case of SO(3), the Fourier transformation is given in terms Wigner matrices D
30
21 +
((R)
Σ
Σ
inen Dr inn (R)
(99)
8T
1=0
M-R=-
aR mn
(100)
SO(3)
where E S3(3) is rotation matrix. Note that the presented convention corresponds to the one in the s2fft package (Price& & McEwen, 2024).
The rotations act naturally on the sphere S on which the Fourier transform given in terms of spherical harmonics Y.
80
((x) Σ
f (x)
(101)
fl dr f())
(102)
Ssa
These Fourier transformations are e.g. used spherical CNNs (Cohen et al., 2018) and steerable convolutional net- works (Weiler et al., 2018), which define equivariant group convolution layers with respect to SO(3) and act on input features defined on the sphere S The change to the Fourier space is motivated by the fact that group convolutions reduce simple multiplications of the corresponding Fourier components. For SO(3), the group convolutions (4) for filter support Sx S (3) are defined as
/ as ((R-1)))1100)0)))
(V(+))))((R)) = VnelSal Js.
(103)
Using
Dt (R-1)- = D'a (R).
(104)
Di mn (R) = (-1)m-nn mn (R),
(105)
[ SO(3)
8T²
dRDi (RYD" w (R)
Ser dan 5mm''
(106)
21 +
the Fourier components (100) of the layer in (103) be written compactly
(107)
VnelSal p=-1
Note that we have assumed real-valued kernel k. Similarly, the lifting layer (6) for features on S is
[ (fI(R))
dx x((R-12)((m)
(108)
v n Sxl sa
which, in terms of the Fourier coefficients (102) becomes
ST
WO)
21 Rl
(109)
niin Sx
27 + 1
m
21

---

Equivariant Neural Tangent Kernels
Again, have assumed real-valued kernel and used the relations
(Ro)) = Σ
mn (x),
(110)
n=-1
(x) = (-1) (x))
(111)
dr ( (4) ó
(112)
D.2. Kernel Recursions for G 50(3)
As we have seen in Section D.1 SO(3) group convolutions are frequently computed the Fourier domain. In this section, we show how also the kernel recursions from Theorems 4.1 and 4.2 for computed in the Fourier domain corresponding the spherical convolutions presented in Section D. In the following will assume filters with global support S, SO(3) or S, S respectively. The reason is that equations (106) and (112) otherwise have to be replaced by expressions including the Wigner's 3j symbols. Due to the current lack of an efficient JAX-based implementation providing their computation, we decided to restrict ourselves to the more efficient case of global filters.
layers and lifting layers can be
In terms Fourier coefficients defined (31), the recursive Kernel-relations for the spherical convolution layer (103) are specified in the following
Lemma D.1 (Kernel recursions of SO(3) group-convolutions in the Fourier domain). The layer-wise recursive relations for the NNGP and NTK of the group convolutional layer (107) G = SO(3) and global filters given by
[K(+)) P
1
8q Dp,ōnn-u
-1)--P(K ff mmpp
(113)
21 +
P' m' (0(+)) (I,
(f.
+ 6n
(-p)
(114)
Proof. We identify elements of S3(3) with 3 x 3 rotation matrices R, S..... Then, the recursive relations for the group convolution layer from Theorem 4.1 are
(e+1 (fff)= = 8T22 I as K0 (f..Rs((..f))
[SO(S)
(115)
(e+1)) R. (f. () RM (( S
/ as 0
(s,
(116)
SO(3)
The Fourier coefficients of the NNGP are given by double Fourier integral of the form (31), the recursion (115) becomes
//|
[F m'
ds
K k f' (RyD",
(117)
8T2
Plugging in the Fourier expansion of the kernel (2) s (f, f`) yields
(f =
8T2 //I
1
soron1
dd
2p 2p+122/ +1
8T²
8T2
P-y==
x
Σ
(( P.D
D RS))
RSSDD (R'))
D' (RyD
(118)
g,r=
-''-"/=-
Using (106), (105) and
D. mm
(RS) = Σ
D (R)D
(119)
mp p=
22

---

Equivariant Neural Tangent Kernels
can simplify the expression
8x
(F m'n
(-1)-
8pt Onu &n Opp On w
8T
21 +
P.P' 00 q,r= pdrrr=-p"
x (K f')arr'
(120)
21 dir nn'
(121)
y=-!
Renaming the summation index yields the desired result. The computation for the NTK is analogous.
Similarly, for the lifting layer (108) for features on the sphere, the kernel recursions can be expressed in terms of the Fourier coefficients (102) according to the following
Lemma D.2 (Kernel recursions of spherical lifting layer the Fourier domain). The layer-wise recursive relations for the NNGP and NTK of the lifting layer (109) for features on S to features on 5O(3) with global filters are given by
1
8T²
(f,
Om [K( f
(122)
4π
21
21+1)
8n
(f, 8') = (f, f
mon +
(123)
4T
21
21-11
Proof. Starting from the recursive relations for the lifting layer in Theorem 4.2, the recursions in real space for G = -O((3
are
k((+) (f, 1
K(0 dr
Ra, 1e (f.f')
(124)
4T Sa
((+1) R.R
(f,
-k(e+") (f,
1 _
/ do Rz. R2 (f.f'.
(125)
4m Ssa
Expressing the Fourier coefficients the NNGP according to (31) gives
1
/ dx
// dRdR' (0) RR (s, f'yD', mm (my (R²).
(f 1'mm
(126)
4
sa
We can now plug the Fourier expansion of the kernel on S2
K (fff)=
Σ
Σ
(f, Y (x)
(127)
4t"=0 m
m -1
to obtain
(f Pi
4T [ dr ISsocnn dRdR'
1
ΣΣ
Σ
K KO P (RT)) (K'Z')
P.P P-F'=0
x D'mm (RYD" (R').
(128)
Using (106), (112), (110) and (111) one can rewrite and simplify the expression as
8T²2
8T
[K( m
(-1)²
dipōmg By Sam am Bn dpp
4m
21
27
1
p.p =0 9.
x
|
(+)
(129)
8T²
Sn | [f( (f, ( 1' mmmm
(130)
4F
21 +
which is the claimed result.
23

---

Equivariant Neural Tangent Kernels
E. Proofs: Data Augmentation Versus Group Convolutions at Infinite Width
In this section, we provide proofs for the theorems given in Section the main text.
Theorem 5.1. Let ut and Pt be the mean predictions after t training steps of infinite ensembles of two neural network architectures Araug and N. Let Naug be trained on the fully G-augmented training data of N and assume that the NTKs of the two architectures are related by
eU,
Σ gEG
gauR preg (9))
(32)
G
Then, ii th and hi converge in the infinite width limit to the same function for all t f.for quadratic losses, up to quadratic corrections in the learning rate.
Proof. For a neural network N,, we can expand the change AN in output due to one training step of gradient descent in the learning rate 7)
ONa()
AN++(() N++1() - NG(1)
-0c+1 0,)
0(1²)
(131)
00
ONG())
aN((F)) (.(NI(F)) +0(7),
(132)
Tirrain
00
00
e.(.f))
where O₂ the empirical NTK at training step 1, y1 are the training labels and L' is the derivative of the per-sample loss with respect to the output of the network. Taking the mean and the infinite width limit yields
Pernii
(P)+1(f)
77
(133)
Thrrai
since we have assumed that L' is linear in its first argument.
The network N yang on the other hand is trained using full data augmentation over G, so we can decompose the sum over training samples into a sum over the training samples in (133) and sum over G. Note that since we assume full data augmentation and a finite training set, we restrict G being finite in this section. We obtain
Rarain
A AF+1(() (J)
Preg Preg( (100 (Preg())()))) (1)
(134)
Merain
G
BEG
As mentioned in the main text, we will prove the statement inductively over training steps t. At O, the mean output of all neural networks is zero in the infinite width limit (Neal, 1996; Lee et al., 2018). 8). For the induction step, assume that aug = A/4+1 Since the ensemble mean of networks trained with data augmentation aug (f:) (.()))
14, aug = /4- Then, aug Mr+1 if Apte
exactly equivariant (Gerken & Kessel, 2024; Nordenfors Flinth, 2024), we have 44 aug (Preg(0) (g))()) = Hi by the induction assumption. Therefore,
Rerain
A A (f)
1
aaug
(135)
Merain
DEG
Using assumption (32) concludes the proof,
Ttrain
A Aptt1(1) (f) = -1 Σ
O(I,)) =AM4=1()
(136)
24

---

Equivariant Neural Tangent Kernels
As mentioned in Section 7, Theorem 5.1 be generalized equivariantly data-augmented networks trained
Rerain
U (1
(137)
i=1 9EG
where the targets ys : Roou are signals on the group.
Theorem E.1. Let Hi t and pt be the mean predictions after training steps of infinite ensembles of two neural network architectures Neux and N. Let Naog be trained on the fully equivariantly G-augmented training data of N and assume that the NTKs of the two architectures are related by
g.gg ((
Eo
g. (S
Preg Preg(()) (h)
(138)
G
hEg
Then, 14 1 and br converge in the infinite width limit the same fanction for all & for quadratic losses, up quadratic corrections in the learning rate.
Proof. This theorem is straightforward extension of Theorem 5.1, which is why we only highlight the differences. Similarly to (132), the change in the output after one training step unaugmented data is given by
Biroin
[A14+1
E Oug..g (f. (A,))CC'
+(6))
(139)
Titrain
1=1 g'EG
where L is the pointwise per-sample loss. Again, L' is the derivative with respect to the output of the network at given point. As before, we assume that L' is linear in its first argument, allowing us to simplify the expectation of the infinite-width version of (139) to
A (f)((g) =
θ 9.0/ (f. (1,.)))
(140)
Ttrain
3'EG
In a similar fashion, we derive the update of a network trained on fully equivariantly augmented data and obtain
aug (f) A
Gaug (f, Preg (() aue (Preg (1) (h)
(141)
Terain G
8' HEE
Using the equivariance property of the ensemble mean again (Gerken & Kessel, 2024), i.c. 14 aud (preg(h)))) Prog (h)p aug (fi), and shifting the summation as g' - hg', we obtain
Derain
A oug (())g))
7
Oaug g.hgy (S, Preg (0))))
(142)
Perain G
I' s'E t=1
17
Gmng (f, Preg (h) L' (0 (F("""
(143)
Perain
g'EG
1=1
hEG
where we have used that (Preg (hg)))(hg') (,," (f))(g') and analogously for Vi-
Using (138) and following the same inductive argument as in the proof of Theorem 5.1 concludes this proof.
Theorem 5.2. Let FFC be an MLP acting on feature maps with output in IR and architecture
FF FC(L) ogo... (C o FC(1)
(33)
where FC denotes a dense MLP layer and a point-wise nonlinearity. Let NGC be a G-imvariant GCNN with architecture
GC G.oolo GConv o GConv(S(- ooo.... GConv $ oa Lifting(S.))
(34)
25

---

Equivariant Neural Tangent Kernels
where 5% are the supports of the convolational filters with Sy X, the domain of the input feature maps, and the other St. invariant under G. Then, the G-averages of the kernels of the MLP given by the kernels of the GCNN,
(f vol(G)
fag - k= (feereg()) Preg(g)
1
(35)
1 (f = vol(G) fdg (f.eeee)))))
(36)
Proof. In order proof the kernel equalities, we will construct the kernels for the fully connected architecture (33) and the group convolutional architecture (34) by explicitly iterating the recursion relations.
The iteration starts with the input kernels which for the fully-connected network are
K K-PC (J.f) vol(x) far dx (x))
FC (1.8) 00
(144)
since the different points in the domain the input function take the role of different channels when the image tensor is flattened. The first layer of the FC-network fully-connected layer. These update the kernels according to (Jacot et al., 2018)
K (f, K (F,00 ( (J, (f."')
(145)
(146)
In order to write kernel transformation like this more compactly, will collect all relevant kernels layer into IR vector (f, f') according to
(f. KFCI KFOW (f', (') (F.(O(S.
(),(')
(147)
where the components (FC(O(f.f) and (FC(0 (f', f() are needed for the nonlinear layers below. In the 3FC-notation (145), (146) can be summarized by function G R + I mapping FF
(f. f') + EFC4))) defined by
ki
ki
K2
12
(148)
ka
K
θ
+
θ
Therefore, the kernels of the first fully-connected layer take the form
f(x)s()) f(x)f(z) f'(x)
vol(X) / dz
1
=FC = (f,f))
(149)
f())
In the architecture (33), fully connected layers are alternated with nonlinearities, which act according to (Jacot et al., 2018)
(f, = KPOD krCM (PC (KFOO (FOO(f)))
(150)
(151)
kra(t)())))) (f,'' = ( AFC.o/o (Fout)((fff) = -FFCC(+))) (F
AFRIos lo ( (4)oo'
(152)
(153)
on the kernels. We will denote the corresponding action the vectors by function F, R -> RA. Therefore,
fully-connected layer followed by nonlinearity can be written as
=PC(+2) (f.f) F - F0
(154)
26

---

Equivariant Neural Tangent Kernels
Hence, in this notation, the kernels of the entire FC network given by
(=. (... (Fo (SEFCC )..)
(155)
Next, we compute the kernels of the GCNN. The input kernels in this case are
K 1
GC (f.f)=
(156)
According to (34), the first layer of the network is lifting layer whose recursion relation was given in Theorem 4.2. Again, we define an -eetoot to collect all kernel components necessary for computing the kernels of the network,
koc ({,))
K Ka (f.f)
=
=GO0 (188)=
(157)
3.9
K o' (f'
@co (f,f))
In terms of EGC the kernels of the lifting layer are given by (note that the filter of the lifting layer has global support by assumption)
k G-clo
pla) (S:)
f(olg)) f(ol))) ((ool
K k-00o) plaie ol 2 (S:5)
=GOD 3.9
vol(X) / dx (f
K kGllo Dlg (F) θ @Go p(o)a. (1,
=vol(X) / dx
(158)
((
For later convenience, note here that
(((()) f(p(h))) M)x) (o) f(p(h)))f(p(g hh)z) [<))(<))
=C(L) (f,
b.g -h A.g--A
vol( () / do
(159)
/ dx
f(s)
(160)
vol(X) X
( f(z)f )x)
FCC (f, Prea(
(161)
where we shifted the integration variable in the second step and used (149).
After the lifting layer, with point-wise nonlinearty whose recursion relations given in Corollary 4.4. Since this transformation is independent for the different g, -components, can write it using the same function F, introduced above as
GC(E+1
=F. (f,
(162)
A GCNN layer transforms the NNGP and NTK according to Theorem 4.1. We can write this in terms of =G0())
9.9"
/ dhe CEEC (f.f)),
( [S Js.
(f.f')
(163)
with G as introduced in (148). The final pooling layer acts according to Theorem 4.3, which we can write as
=GO =GOl (f. (vol(G)) / dg f dg' =0 (ff)).
(164)
9.8
27

---

Equivariant Neural Tangent Kernels
With the expressions (162), (163) and (164), can write the kernels of the entire network
dh 8 o Fo Fo.
1 [ dg ( dg'
1
(vol(G))2 E (f,
a g
Fo
15% s
SS- Jst
1
dhg g (F (OO))
ho (f.f)))
(165)
sg
In order to simplify this expression, we shift h3 and absorb g ... hs into it. This will not change the integration domain of h&g since S is by assumption invariant under G. Then, the integrals over hy, h2.22..-... hss become trivial and cancel against their 1 //15 prrefactors. We are left with
( Fo (g Fo
aag (f = (vo) I dg I dg' g
1 [ s
/
dhs g (FC ho.9'o the ($. (.11))
(166)
Finally, we trivialize the g' -nteraaaa by shifting g to absorb g'. Thus, we obtain
= EG (0,)) vol(G) / dg g Fo g Fo | / dhs g (Fo(E (f, (f,f')
(167)
vol(G) [ dg Fo (g [Fo (...9 (F (Fo(E)O)) (f, Preg Preg(9)f))) (F.re()))))).... vol(G) [ dg E Preg Preeg (g)
(168)
(169)
where we used (161), trivializing the integral over h3, and then identified =FC from (155). The statement follows by taking second and fourth components of (169).
Theorem 5.3. Let NKKNN be the K & N -invvariant GCNN with architecture (34) and K K-invariatt filter supports St, which for the GConv-layers decompose as St. Kt x N K C K, N C. N. Let NN be the N-invariant GCNN with architecture (34) and filter supports NL. . N and S Then, the NNGPs and NTKs of these networks are related by
vol(K) S dk KN (f. (f..Pre(k)) 1 OKK (),')) vol(K) [ dk 2N (r.Pro
1
KKN (f.
(37)
(38)
Proof. In this proof, we will use the same notation as in the proof for Theorem 5.2 above and use results from there as well. We start by considering the kernels EK of ArKKN by specializing (165) to the case G = K x N. Due to the semidirect product structure of G, there is a unique decomposition g = kn for each E G into k E K and n E N. Since by assumption the filter supports St on also factorize over and N, we can split all G- G-integrations in (165) over N and K and obtain
EKKN (f,') (vol(K))² (vol(N))} I dk f an [ ak' / dn' [KLIN NJ.
( djr. I dmz g Fo...
Kb
/ dmy g (Fo
(f. (f,f") )..
djs
(170)
K3
N3
NN
Kg
In order to trivialize the integrals over K, as was done with the integrals over G (166), we need to rewrite the first group
28

---

Equivariant Neural Tangent Kernels
index of (3) such that all ji appear next each other. To this end, introduce several unit elements
knjrm, -. ". -immrjsmmsjmm = knjim "1 j.777ijs.a J3
msjs mg
(171)
EN
knjim.. JrJaJa (isJs) mmjjsja, J3 msj3 mg
(172)
EN
-jr... -. ja ...... .. ja) njr. Ja ... ja)
m. is m3
(173)
EN
EN
EN
We perform the same rewriting also on the second group index of (( Since a normal subgroup of G, knk N for all K, n N and Haar measure on N is invariant under shifts of the form - knk- Furthermore, the integration domains N are by assumption invariant under this transformation. Hence, we shift n, n' and the my by
n + jL jan(ju ... ja))
(174)
n' + ju ... jan' (ju js)
(175)
m + Je-2 jame jz) 2> 3.
(176)
With this (170) becomes
|na I dk f dn J ak' dn' K kg NN
/ djz / dmp I Fo...
=Kx (f,5"=
(vol(K))² (vol(N)))
K
1
(=KKN() jisiiii-ammmm -jsn jan'm. ma ((()))... (177)
djs
dmin (FolE
K
N3
Kg
No
[ dk f dn [ ak' [ dn' [Ngll J dm @ (Fo...
(vol(K))2 (vol(N))2
dmg G (Foom)snme" .my.kkk mon'mme (f. (f.f)))
dis
(178)
K&
K2
No
1
dk J dn / dn' 1
dmz ((.....
vol(K) (vol(N))3
INZ. NO
1
dmg g (Fo (KKK())) djs
(f, (()) )..
(179)
N3 K3
K3
No
Here, shifted js (kjj- (s) in the first step, trivializing the integrals over j....., Js which then cancel against their 1/IK% -prefactors. In the second step, we first trivialized the integral over 1 by shifting k - kkk and then canceled it against its 1/ por(ftr-rrffccoor.
Next, we perform another manipulation on the group indices of kkkK by first inserting suitable unit elements,
janmp ... = janja' jaM jamL-
jsmaj5 js.
(180)
EN
EN
EN
EN
and similarly for the second group index of (K)((), After shifting
J3 njs.
n'
- n'
me - 53 3 mmmja 22 3,
(181)
in (179), we obtain
kkN (f.8)= vol(K) (vo2 [ ak [ dn S an' NN JNg
/ dmg o Fo Fo...
1
djs I dmg (Fo (FKK()))))
-amm--mafskk An/ms. (f.f")) (f.P)))... :..)
(182)
N3 K
K2 Ny
29

---

Equivariant Neural Tangent Kernels
ReLU
0.8
Relative Error
0.6
04
0.2
0.0
Network Width
Figure 5. Convergence of the Monte-Carlo estimates of the NNGP to their infinite- limits for G C4 x I2? Plotted is the relative error averaged over the components of a 3 3 Gram matrix for networks with a ReLU or an error function nonlincarity. The bands correspond to + one standard deviation of the estimator.
As in the proof for Theorem 5.2, we will now write N()N()) in terms of M(), Using the shorthand m my ... my and the analogous steps (161), we find
f( f
(KN(1)) nmjjs. k nfffs (1,8)
dr
(183)
| ss
f' fookk-n''jjj n'mjja))) (o [(p(nmja))
f(plmm) x) {p((m))) (() (p(k-"n'm)et F' n'm)a) ( (((kk-n''m) m))) ((lum))) ( (o (o(kk(-nm))
dr
(184)
N(1) m in (f, Preg (k))'))
(185)
where for the second equality, we have shifted æ - plis )zz which leaves St. invariant by assumption. Plugging (185) into (182) trivializes the js-integral which cancels against its 1/ / -preffccc,,,r, yielding
=KN (f.f) = vol(K) / dk (vol(N)) / dn / dn' IN JNL
/ dmz of Fo...
dms & (FN(E M() ( momml mw.. (f, Preg (rog()())))))))
(186)
|N No
vol(K) dk EN (S. Preg (k))))
(187)
where we have identified EN by comparing to (165). The statement of the theorem follows by considering the second and fourth components of (187).
F. Further Experimental Results
In this appendix, we provide further details and results of the numerical experiments presented in Section 6.
F.I. Kernel Convergence
Figure shows the convergence of Monte-Carlo estimates of the NNGP to the analytical infinite wiitth expression derived using the theorems in Section 4.1.
30

---

Equivariant Neural Tangent Kernels
Table Architectures used for the medical image classification deserihed in Section 6. For convolutional, group-convolutional and lifting layers, the argument is the kernel size (all kernels are squared), Both pooling layers are global. The number of output neurons is finite and to correspond to the classes.
CNN
GCNN
Conv (3)
Lifting (3)
ReLU
ReLU
Conv (3)
GConv (3)
ReLU
ReLU
Conv (3)
GConv (3)
ReLU
ReLU
Conv (3)
GConv (3)
ReLU
ReLU
Conv (3)
GConv (3)
ReLU
ReLU
SumPool
GPool
Dense
Dense
ReLU
ReLU
Dense (9)
Dense (9)
F.2 Medical Image Experiments
In the infinite-width limit, the NTK becomes deterministic and time-independent under the gradient flow dynamics. In the case of MSE loss, the differential equation describing the mean output of network time t becomes a linear ODE, thus allowing for analytic expression at arbitrary time. In the limit of infinite training time, the mean is given by (41) (Jacot et al., 2018). This relation is effectively kernel method that can be used to generate prediction of the infinitely wide network.
The task consists of classifying histological images (Kather et al., 2018) containing nine classes of tissues, two of which are cancerous. The original images have resolution of 224 224 pixels each and have been down-scaled to a resolution of 32 32 pixels reduce the kernel evaluation time. Note that the size of the final kernel matrix, that needs to be inverted, is independent of the resolution because in (41) only applies for MSE loss, we constructed target vectors y [Vo,.... UUN from classes e according to / as is standard in the NTK literature (Lee et al., 2020).
group pooling or SumPool layer, respectively. Since the analytic solution
The CNN and GCNN architectures that were used are shown in Table 1. Note that the infinite- width limit refers to the number of channels, which is why we only need specify the kernel sizes. The same training and test data was used for both models with test data size of 1000 images. Both architectures have been implemented in the neural -tangents package (Novak et al., 2020).
F33. Molecular Energy Regression
We used the same kernel method resulting from the infinite-width and infinite-time limit as explained in Section F.2. Both the grid 5 well as S3(3) are equiangular Driscoll & Healy grids (Driscoll & Healy, 1994) with resolution 2L x (2L- 1)a on 52 and (2L- 1) x 2L x (2L- 1) on SO(3) (parametrized in Euler angles). L is the corresponding bandlimit defining the cutoff in the Fourier domain, i.e. only Fourier coefficients 1 < T. are considered. The input signals are sampled for L = 6.
As the labels we have used the internal energies Un of the molecules at K after substracting the atomic reference energies. The hyperparameter (3 in (42) was chosen as described in (Esteves al., 2023) according
cos(T/4) - 1)2
(188)
log(0.05)
"In the original work by (Driscoll & Healy, 1994) the grid contained actually 2T. 2L points, but have adapted our grid to the convention used in the fffftt package.
31

---

Equivariant Neural Tangent Kernels
Table Architectures used for the molecular energy regression deseribed in Section 6. 29 identical networks (encaptured by curly braces) process the inputs associated to each atom. Their outputs are then summed together. For group- group-convolutional and lifting layers, the output bandlimit is stated. The pooling layer global and single output neuron represents the predicted energy of network.
MLP
GCNN
29 per-alom networks
29 per-alom nelworks
Dense
Dense
Lifting (3)
Lifting (3) Erf
ReLU
ReLU
Erf
Dense
Dense
GConv (3)
GConv (3)
combined to molecule network
combined molecule network
FanInSum
FanInSum
Dense (1)
Dense (1)
1.0
Ensemble Size
Ensemble Size
['equiv-lon Plequiv-loggis. logiis, con-equiv-logits
Ensemble
Ensemble
0.7
0.6
0.5
0.2
6
11 Epoch
10
12
13
14
15
16
17
18
19
20
Figure 6. Convergence finite-width ensembles trained with data augmentation ensembles GCNNs on CIFARIO. Shown the L 2 -'-diitance between the logits of the equivariant ensemble and the non non-equivariant ensemble trained with data augmentation for different ensemble sizes distribution data. For larger ensembles, the distance decreases.
The precise architectures of the MLP based network and the (033)-invariant network are listed in Table The MAE loss was evaluated on a test set 100 molecules.
F.4. Data Augmentation Versus Group Convolutions at Finite Width
Figure 6 shows that large ensembles trained with data augmentation on CIFAR10 10 converge to GCNNs even out of distribution. Similarly, Figure shows the same behavior on the NCT-CRC-HE-IOOK data set histological images (Kather et al., 2018), downscaled 32 32 pixels. Samples of the out of distribution data, whose mean and variance were normalized respectively, provided in Figure 8. The architectures used for the ensemble members detailed in Table 3.
and 1,
32

---

Equivariant Neural Tangent Kernels
Ensemble Size
Ensemble Size
(equivv-Joo) souuv--logiss non non-noooonee eouuv--vgiiss))
Ensemble Size 8
0.8
Ensemble Size 16
Ensemble Size 40
0.1
6
10
15
16
17
Epoch
Figure 7. Convergence of finite-width ensembles trained with data augmentation to ensembles of GCNNs on histological images from the NCT-CRC-HE-100K data Shown the L d'-distance between logits of the equivariant ensemble and the eonve-ivvaiaant ensemble trained with data augmentation for different ensemble sizes on out of distribution data. For larger ensembles, the distance decreases. The standard deviation is estimated from 20 independent runs for each curve.
Figure 8. Examples for out of distribution data for MNIST (left) and CIFAR10 (middle) and NCT-CRC-HE-1O0K (right).
33

---

ReLU Equivariant Neural Tangent Kernels
Table 3. Architectures used for the ensemble members in the experiments described in Section 6. For convolutional, group-convolutional lifting layers, the arguments are input channels, output channels and kernel size kernels are squared), For max- max-pooling layers, the arguments are kernel size and stride. For the GCNNs, the max pooling is done only over spatial dimensions, not group dimensions. The kernel sizes were selected such that the GCNNs are exactly equivariant for the respective input sizes 28 x and 32 32.
MNIST
CIFARIO
CNN
GCNN
CNN
GCNN
Conv (1, 4, 3)
Lifting(1, 4, 3)
Conv (3, 4, 3)
Lifting (3, 4, 3)
ReLU
ReLU
ReLU
ReLU
MaxPool (2, 2)
SpatialMaxPool (2, 2)
MaxPool (2, 2)
SpatialMaxPool (2, 2)
Conv (4, 16, 4)
GConv (4, 16, 4)
Conv (4, 16, 4)
GConv 16, 4)
ReLU
ReLU
ReLU
ReLU
MaxPool (2, 2)
SpatialMaxPool (2, 2)
MaxPoot (2, 2)
SpatialMaxPool. (2, 2)
Conv (16, 32, 3)
GConv (16, 32,
Conv (16, 32, 3)
GConv (16, 32, 3)
ReLU
ReLU
ReLU
ReLU
Conv (32, 64, 3)
GConv (32, 64, 3)
Conv (32, 64, 4)
GConv (32, 64, 4)
ReLU
ReLU Conv (64, 128, 1)
ReLU
Conv (64, 128, 1)
GConv (64, 128, 1)
GConv (64, 128, 1)
ReLU Conv (128, 32, 1)
ReLU GConv (128, 32,
ReLU Conv (128, 32, 1)
ReLU GConv (128, 32, 1)
ReLU Conv (32, 10, 1)
ReLU
ReLU Conv (32, 10, 1)
ReLU GConv (32, 10, 1)
GConv (32, 10, 1)
GPool
GPool
Table 4. Architectures used for the ensemble members in the experiments described in Section F.4. For convolutional, group-convolutional lifting layers, the arguments are input channels, output channels and kernel size kernels are squared). For max max-pooling layers. the arguments are kernel size and stride. For the GCNNs, the max pooling is done only over spatial dimensions, not group dimensions. The kernel sizes were selected such the GCNNs are exactly equivariant for input size of x 32.
NCT-CRC-HE-IOOK
CNN
GCNN
Conv (3, 4, 3)
Li Lifting (3, 4, 3)
ReLU
ReLU
MaxPool (2, 2)
SpatialMaxPool (2, 2)
Conv (4, 16, 4)
GConv (4, 16, 4)
ReLU
ReLU
MaxPool (2,
SpatialMaxPoo (2, 2)
Conv (16, 32, 3)
GConv (16, 32, 3)
ReLU
ReLU
Conv (32, 64, 4)
GConv (32, 64, 4)
ReLU
ReLU
Conv (64, 128, 1)
GConv (64, 128, 1)
ReLU Conv (128, 32,
ReLU
GConv (128, 32, 1)
ReLU Conv (32, 9, 1)
ReLU GConv (32, 9, 1)
GPool
34


Reviews:
[{"review_id": "E7qpSz1Ro5", "reviewer": "Reviewer_bkX8", "summary": "In this work, the authors propose a way to understand the training dynamics of equivariant models by deriving neural tangent kernels for a broad class of equivariant architectures based on group convolutions.  For rototranslations in 2D and 3D, the authors show that equivariant NTKs outperform their non-equivariant counterparts as kernel predictors for CIFAR10, MNIST, medical image classification, and property prediction on the QM9 dataset.\n\nThe theoretical formulation consists of recursive relations for the NTK and\nthe NNGP for group convolutional layers, lifting layers, and group pooling layers. These allow efficient calculation of these kernels for arbitrary group convolutional architectures and thus provide the necessary tools to analytically study their training dynamics\nin the large width limit. \n\nExperimental claim: Networks trained with data augmentation converge to group convolutional networks at infinite width.\n\n## post rebuttal \n\nI wish to keep my score after reading the authors' rebuttal. I do not wish to update the score as the experiments still feel limitive ( although the authors do provide an additional experiment in the rebuttal)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Approximate results at finite width\n\nOur theoretical claims hold for infinitely wide networks and in the ensemble mean, i.e. for infinitely large ensembles (for comments on extensions to single networks see rebuttal to reviewer hpPR, heading “Ensembles”). In this case, they predict exact agreement between the mean prediction of the invariant GCNN (due to group pooling) and the augmented network throughout training for arbitrary inputs. In our experiments, we test the agreement of the mean predictions for finite ensembles of finite-width networks and find that the predictions align more for larger ensembles. As expected, alignment is imperfect due to finite width, thus we call these results “approximate”. We will clarify this in the revised version.\n\n## Extension to equivariant infinite width regression\n\nAn extension of our framework to equivariant tasks like quantum mechanical force field prediction is indeed an interesting point. Using the GCNN layers we analyze, it is straightforward to perform regression on targets that are signals on $\\mathrm{SO}(3)$, i.e. $f_\\text{target}: \\mathrm{SO}(3) \\to \\mathbb{R}^{n_\\text{out}}$ in an equivariant way. Here, equivariance is understood with respect to the regular representation, see eq. (5). An equivariant model that is suitable for regression on vector quantities like forces necessitates an output layer that is equivariant with respect to the defining representation of $\\mathrm{SO}(3)$. For this, the presented framework would need to be extended by additional layer types and their corresponding kernel relations. (One approach could be to take the argmax of the signal on $\\mathrm{SO}(3)$ at the output layer and apply the resulting rotation to a fixed reference vector.) Furthermore, the infinite-width limit of the non-equivariant networks needs to be taken with care in this case. We have outlined a possible way to achieve this in the reply to reviewer *kzsw* under “Extension to equivariance”. Similarly as mentioned in the discussion by Cohen et al. [ICLR 2018], an extension to e.g. steerable CNNs would also be suitable approach. We consider this an interesting further research direction and will add a discussion to the manuscript.\n\n## Complexity of the finite width experiments\n\nWe recognize the interest in experiments with more demanding tasks. Since our theorems draw a connection between data augmented and invariant GCNN ensembles, we need to choose an invariant task to test the analytic results numerically. We have now repeated the experiment on a subset of the *NCT-CRC-HE-100K* data set (https://zenodo.org/records/1214456) consisting of histological images. We have trained on down-sampled images with a resolution of `32x32` pixels and produced OOD samples in this input space. The results are shown in  https://postimg.cc/hQzcRpkf, which were obtained in the same fashion as Figure 4 and 6. We use the same architecture as for the CIFAR10 experiment apart from twice as many channels in each hidden layer and we also adapted the learning rate. Each curve is computed 20 times, we plot the means for the metric over the runs and their standard deviations. The larger ensembles reach a validation accuracy of around 80%. The results further support our theoretical claims and we will add them to the manuscript.\n\n## Citations\n\n> 1. On genuine invariance learning without weight-tying, Mokalev et al. [TAGML, ICML 23]\n> \n> 1. On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory, Perin et al.\n> 2. Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting, Wang et al. [ICMLworkshop 22]\n> 3. Fast, Expressive Equivariant Networks through Weight-Sharing in Position-Orientation Space, Bekkers et al. [ICLR 24]\n> 4. Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks, Borelon et al. [ICML 22]\n> 5. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks, Canatar et al. [Nat Comms 2021]\n\nWe thank the reviewer for suggesting a number of relevant references and will add them to a revised version of the manuscript.\n\n### Relation to the work by Perin et al. [arXiv:2412.11521]\n\nThank you for pointing out Perin et al. We will include this very recent and interesting contribution in the revised manuscript. As you point out, although they study a similar problem, they focus on a very specific toy problem for which they analyze the spectrum of the NTK in detail. The only equivariant architecture they consider are CNNs. In contrast, our framework captures the learning dynamics of arbitrary group convolutional neural networks.", "claims_and_evidence": "Yes, they are clear. The one mentioned below is unclear and if the authors could provide clarification, that would be helpful. \n\nFor the experimental claim:  ln 419.(column 2) '.. data augmentation converge to group convolutional\nnetworks at infinite width. We verify that this also holds\napproximately at finite width',  but it does not seem fully justified, as it is a much simpler experiment.", "methods_and_evaluation_criteria": "The proposed experiments, datasets, and evaluation criteria make sense for the problem and method proposed.", "theoretical_claims": "I have looked at the proof for Theorem 5.1 only. It has no issues.", "experimental_designs_or_analyses": "Yes. \n\nSee weaknesses.", "relation_to_broader_scientific_literature": "A few important citations are missed in the paper and thus, the connections to them are not made. \nThe key contributions to the paper are related to the broader literature; for example, the connection to spectral components and NTk have been made in literature. However, the GP formalism with equivariant NTK is relatively novel. Although claims in [1] are of a similar flavor as this paper, the theoretical formalism is different. \n\n1. On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory, Perin et al.", "essential_references_not_discussed": "Missed references:\n\n1. On genuine invariance learning without weight-tying, Mokalev et al. [TAGML, ICML 23]\n\n2. On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory, Perin et al.\n\n3. Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting, Wang et al. [ICMLworkshop 22]\n\n4. Fast, Expressive Equivariant Networks through Weight-Sharing in Position-Orientation Space, Bekkers et al. [ICLR 24]\n\n5. Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks, Borelon et al. [ICML 22]\n\n6. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks, Canatar et al. [Nat Comms 2021]", "other_strengths_and_weaknesses": "## Strengths\n- The paper is well motivated theoretically, and the experimental section supports the claims made in the paper. \n\n- The paper is fairly well-written.\n\n\n\n## Weaknesses\n\n- The experiments for data augmentation vs finite width of neural networks are only shown for simpler toy datasets like CIFAR10 and MNIST\n\n- Important citations and connections are missing.", "other_comments_or_suggestions": "Additional discussion/connection to missed papers in literature would be useful.", "questions_for_authors": "1.  What about comparing data augmentation and finite depth neural network work for more complex tasks like force prediction in QM9? \n\n2. Elaborate on ln 419 (column 2) 'we proved that networks trained with data augmentation converge to group convolutional\nnetworks at infinite width. We verify that this also holds approximately at finite width.' Explain what is considered approximate here.\n\n3. ln 422. (column 1) Molecular energy regression. We know that energy is a scalar and an invariant quantity for a molecule with SO(3) transformations. Compared to a vector quantity like force prediction/regression, energy is a relatively easier task (as it is a scalar). Comment on the proposed claims in the paper in the light of different tasks based on its geometric complexity. \n\nand see Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjiyEsIQwO", "reviewer": "Reviewer_hpPR", "summary": "This paper studies the training dynamics of equivariant neural networks via neural tangent kernels. The authors derive NTKs for group convolutions and nonlinearities, and also consider group convolutions for SO(3) in the Fourier domain (similar to spherical CNNs and G-steerable CNNs). The authors show that non-equivariant models with data augmentation converge to a specific GCNN architecture. Empirical results show that the NTK converges to the analytical expression for GCNNs. They also show that equivariant GCNNs outperform standard MLPs and data augmentation helps the standard MLPs converge to the same performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Ensembles\n\nThank you for this important question. Although Theorem 5.1 is formulated in terms of ensembles, the statement in fact also holds for individual models at finite width if the infinite-width NTKs in (32) are replaced by the empirical NTKs\n\n$$\n\\Theta(x,x')=\\left(\\frac{\\partial\\mathcal{N}(x)}{\\partial\\theta}\\right)^\\top \\left(\\frac{\\partial\\mathcal{N}(x')}{\\partial\\theta}\\right)\\,,\n$$\n\nas long as they are initialized in such a way that $\\mathcal{N}(x)=\\mathcal{N}^{\\text{aug}}(x)=0$ $\\forall x$ at initialization. \n\nTheorem 5.2 however only holds for infinite-width NTKs. Therefore, the combined statement of the equivalence of data augmentation and GCNNs holds also for individual models at infinite width, if $\\mathcal{N}^{\\text{FC}}(x)=\\mathcal{N}^{\\text{GC}}=0$ $\\forall x$ at initialization.\n\nWe will add a corresponding comment to the manuscript.\n\n## Sample efficiency\n\nThis is indeed an interesting line of further research for which our study lays the ground work. It is well known, see [arXiv:1912.13053], that the conditioning number of the NTK kernel is related to the convergence speed of training. One could therefore compare the conditioning number of the equivariant and standard NTKs to deduce insights into sample efficiency. This is however a task of significant technical difficulty (see Appendix B of the publication linked above which we’d need to generalize to the equivariant case) as it requires the analytical study of the NTK’s spectra as well as their corresponding phase structure which warrants follow-up work.", "claims_and_evidence": "Yes the claims are specific and clearly supported mostly with proofs and also with empirical results.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No, I did not check for correctness of any of the proofs.", "experimental_designs_or_analyses": "No", "relation_to_broader_scientific_literature": "The key contributions of this paper are deriving NTKs for group convolutions and using these results to show that a standard CNN on augmented data is equivalent to training an equivariant CNN on unaugmented data.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n- Nice derivations of the NTKs for group convolutions\n- Practical use case of the derived NTKs for comparison with data augmentation\n- Experiments seem to practically support the paper contributions/claims\nWeaknesses:\n- See questions", "other_comments_or_suggestions": "None", "questions_for_authors": "- Is the comparison with data augmentation and equivariant networks limited to ensembles only?\n- As I am not familiar with NTKs, can the NTK for equivariant NNs describe the sample complexity improvements over standard NNs (perhaps the eigenvalues)? In the comparison with data augmentation, in my opinion, the greatest benefit of equivariant NNs over data augmentation is that they can achieve better performance with fewer data samples. E.g. a C_4 equivariant network only needs to see one sample while data augmentation with a standard network needs to see all 4 samples.  I can understand why data augmentation leads to a specific GCNN, but it would seem that GCNNs would be much more sample efficient.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UW1uR23ge9", "reviewer": "Reviewer_kzsw", "summary": "This paper derives neural tangent kernels (infinite width) for group convolutional networks, and prove a equivalence between data augmentation and equivariance. They provide experimental validations for the finite-width framework. They also test group-equivariant kernels and compare them to non-equivariant ones.\n\n## update after rebuttal\n\nI thank the authors for their rebuttal. I will keep my score since In Th. 5.2 is about invariance, and not equivariance. I feel that a result on equivariance would be needed to recommend strong acceptance, since this paper is title Equivariant NTK.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Expectation and “off-manifold”\n\nWe are thankful for pointing out this confusion. The mentioned ensembles are a collection of independently initialized NNs and the average is understood over this family. For considerations about single networks, see heading “Ensembles” in the rebuttal for reviewer hpPR. The presented theorems do not make any particular assumptions over the data distribution.\n\nThe term “off-manifold” refers to the *manifold hypothesis* of the data distribution. Our presented results hold for arbitrary inputs and hence, also hold off-manifold. We will clarify this in the updated manuscript.\n\n## Average vs. max pooling\n\nIndeed our recursion relation in Theorem 4.3 only applies to average pooling. Since the expectation value and the maximum operation do not commute this is already true for MLPs and not a specific property of GCNNs.\n\nThe average pooling which appears in Theorem 5.2 on the other hand is part of our result: Training an MLP with data augmentation corresponds at infinite width to a GCNN with a group average layer.\n\n## Network depth in Theorems 5.2 and 5.3\n\nIndeed, both Theorem 5.2 and 5.3 hold for arbitrary number of layers. For example, the equivalence between the fully connected network defined in (33) and the corresponding GCNN defined in (34) holds for any number of layers $L$. We will add a clarifying remark. \n\n## Assumption of NTK-relation in Theorem 5.1\n\nTheorem 5.1 yields a condition on the NTKs to relate augmented to non-augmented networks. The group average on the right hand side of (32) is a consequence of the data augmentation. Theorems 5.2 and 5.3 then provide particular architectures that satisfy this condition.\n\n## Extension to equivariance\n\nTo extend our results to equivariant networks, one would first need to extend Theorem 5.1 to equivariant augmentation. This is indeed possible by considering a network $\\mathcal{N}$ which maps feature maps $f:X\\rightarrow\\mathbb{R}^n$ into feature maps $\\mathcal{N}(f):G\\rightarrow\\mathbb{R}^{n'}$ by optimizing $\\mathcal{N}(\\rho_{\\text{reg}}(g)f)$ against targets $\\rho_{\\text{reg}}(g)\\hat{f}$ for all $g\\in G$. The result is that the mean predictions agree if the NTKs of the two networks satisfy\n\n$$\n\\Theta^{g,g'}(f,f')=\\frac{1}{\\mathrm{vol}(G)}\\int_{G}\\mathrm{d}h\\Theta_{\\mathrm{aug}}^{hg,g'}(\\rho_{\\text{reg}}(h)f,f').\n$$\n\nIn order to find networks whose NTKs satisfy this relation, new layers need to be introduced since in the infinite-widht limit, the NTK of MLPs becomes proportional to the unit matrix in the output channels, trivializing the feature map. Here, a separation of the channels into a part which is taken to infinity and a part which is kept finite, e.g.\n\n$$\nf'_i(g)=\\sum\\_{j=1}^{n_c}\\int\\_X\\mathrm{d}x\\kappa\\_{ij}(g,x)f_j(x), \\qquad n_c \\to \\infty,\n$$\n\nwhich corresponds to a fully-connected layer at finite width, could be suitable. We haven’t repeated the calculations for this layer, but it would be an interesting extension of our results which we will mention in the manuscript.\n\n## Non-regular representations\n\nOur analysis focuses on group convolutions. These can also be used for (scalar) point clouds by defining the input feature map to be $f(x)=\\sum_i \\delta(x-x_i)f_i$ for features $f_i$ at $x_i$. Then, the GCNN layers are equivariant with respect to transformations of the positions of the input features. However, this does not cover features transforming in vector- or tensor representations. For these, the setup would need to be extended. Although we have not performed this extension, we do not see conceptual roadblocks at the moment.\n\n## Relation to Finzi et al.\n\nIn contrast to this work, we specialize to the regular representation. This has the advantage that we can use the convolution theorem to compute the kernel recursions in Fourier space. Extending our framework to arbitrary representations would be a very interesting. A starting point would be to define suitable infinte-width limits of the relevant layers, see e.g. the section “Extension to equivariance” above.\n\n## Introduction to the NTK\n\nWe acknowledge that NTKs are a topic of substantial technical difficulty. We will revise the background section to make it more accessible stressing that the additional section in the appendix is optional background material. We will also explain the term “frozen” NTK that was adapted from other references (see e.g. Mario Geiger et al. [J. Stat. Mech. 2020]). It refers to the fact that the NTK becomes time-independent during training in the infinite width limit.\n\n## Further remarks\n\n- We will add Taco Cohen’s PhD thesis to the literature review.\n- Thank you for pointing out the typos which we will fix in a revised version.\n- For full generality, we formulated a general layer-$\\ell$ lifting layer, because one may include other non-group convolutional layers before as a preprocessing step.\n- We will clarify the notation in (8). Indeed, the regular representations on both sides act on functions with different domains.", "claims_and_evidence": "The claim: “we show an interesting relationship between data augmentation and group convolutional networks. Specifically, we prove that they share the same expected prediction at all training times and even off-manifold” is a bit hard to understand.\n\nSpecifically: what do the authors mean by “expected” prediction: averaged over initializations? or over data distribution? Additionally, what is meant with “off-manifold”? In the main text, the authors refer to “ensembles” of neural networks, which I think is related. Could the authors clarify the framework and the claim there?\n\nIt seems that most results apply to equivariant neural networks with average pooling layers. How about max-pooling layers? Which of the authors’ results hold or do not hold here? If the results are restricted to average pooling, then the claims should be updated accordingly.", "methods_and_evaluation_criteria": "Yes, the datasets and benchmarks are well-suited.", "theoretical_claims": "For the theorems, not much is said about the number of layers. Does it matter? Do the equivariant and non-equivariant architectures have the same amount?\n\nIn Th. 5.1. Is the assumption of “being related with group averaging” restrictive for the equivariant neural network?\n\nIn Th. 5.2, the N^GC is G-invariant : does the theorem apply for a G-equivariant neural network? Why restricting to invariance here?", "experimental_designs_or_analyses": "I did not see any issue.", "relation_to_broader_scientific_literature": "Missing Taco Cohen’s dissertation.\n\nCould the authors discuss an extension to non-regular group-equivariant NNs (eg, steerable?): would the results generalize to this case: why or why not?\n\nHow about the equivariant architectures from Finzi et al. A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups?", "essential_references_not_discussed": "Taco Cohen’s dissertation.", "other_strengths_and_weaknesses": "The introduction to NTK could be though to follow for someone not familiar with the topic. I would also recommend *not* starting with a reference to the appendix: the main text should be self-contained without readers having to go to the appendix. Examples of clarification point: why is it called the “frozen” NTK?", "other_comments_or_suggestions": "Some typos, eg We proof -> We prove.\n\nTheorem 4.2: if it’s the lifting layer, then we should have (1) and (2) instead of (l) and (l+1)?\n\nEq (8): It cannot be the same rho acting on f or on N(f), since f and N(f) do not have the same support: f’s support is not necessarily G whereas N(f) has support G.", "questions_for_authors": "NA.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission received unanimous positive, but lukewarm reviews. Reviewer bkX8 raised the most serious issue regarding the approximate convergence between data augmentation and GCNNs in the finite width case, which was only show in simple datasets and contradicts previous published results. This does not affect the main theoretical results for the infinite width case, including the interesting connection between data augmentation and GCNNs. The reviewer recommends acceptance despite this potential issue and I agree with their recommendation.

Reviewer bkX8 also pointed out missing citations and discussions, while kzsw raised small issues with the formality, which seemed adequately addressed in the rebuttal. I urge the authors to update the writing following the reviewers suggestions.
Decision: Accept (poster)