Geometrical aspects of lattice gauge equivariant convolutional neural networks
Authors: David I. Müller
Jimmy Aronsson
Daniel Schuh
Venue: Accepted by TMLR ()

Abstract: Lattice gauge equivariant convolutional neural networks (L-CNNs) are a framework for convolutional neural networks that can be applied to non-Abelian lattice gauge theories without violating gauge symmetry. We demonstrate how L-CNNs can be equipped with global group equivariance. This allows us to extend the formulation to be equivariant not just under translations but under global lattice symmetries such as rotations and reflections. Additionally, we provide a geometric formulation of L-CNNs and show how convolutions in L-CNNs arise as a special case of gauge equivariant neural networks on SU(N) principal bundles.

Published in Transactions on Machine Learning Research (05/2024)
Geometrical aspects of lattice gauge equivariant convolu- tional neural networks
David I. Müller
dmueller@hep.Kke wtw.:e/ppppttttuin..oommm
TU Wien, Institute for Theoretical Physics, A-1040 Vienna, Austria
Jimmy Aronsson
jommyyar@ccabmerrs
Chalmers University of Technology, Department of Mathematical Sciences, SE-412 96 Cothenburg, Sweden
Daniel Schuh
schuh@hep.iit www.iiiiiennn.mmmmmmmmrm ac.at
TU Wien, Institute for Theoretical Physics, A-1040 Vienna, Austria
Reviewed OpenReview: /ttps:///ppnnevvieet net/ forum? id= 204 VHaPe
Abstract
Lattice gauge equivariant convolutional neural networks (L-CNNs) convolutional neural networks that can be applied to non-abelian lattice gauge theories without violating gauge symmetry. We demonstrate how L-CNNs can be equipped with global group equivariance. This allows us to extend the formulation to be equivariant, not just under translations but under global lattice symmetries such as rotations and reflections. Additionally, we provide geometric formulation of L-CNNs and show how convolutions in L-CNNs arise a special case of gauge equivariant neural networks on SU(N) principal bundles.
framework for
1 Introduction
In recent years, machine learning methods incorporating ideas based symmetry and geometry, often summarized under the term geometric deep learning (Bronstein et al. 2021 Gerken et al., 2023). have received much attention in both computer vision and physics. Most famously, convolutional neural networks (CNNs) (LeCun et al. 1989) have proven to be an excollent machine learning architecture for computer vision lasks such as object detection and classification. The classic examples include delermining whether an image contains particular animal (e.g, cat dog (Parkhi al. 2012)), or identifying numbers in images of hand-written digits (Deng 2012). For these tasks, it has been demonstrated that CNN architectures excel both in terms of accuracy and reduced model size, i.e. the number of model parameters. A key differentiating feature of CNNs compared to generic neural networks is that they are formulated as stacks of convolutional layers, which exhibit translational symmetry or, more accurately, translational equivariance. If translation is applied to the input of a convolutional layer, then the resulting output will be appropriately shifted as well. This equivariance property is highly useful the case of image classification, where the absolute position of a particular feature (a cat; hand-written digit) in the image not important. Translational equivariance further implies weight sharing, which reduces the number of required model parameters and training time. Consequently, CNNs provide not just more accurate but also more robust models compared to their translationally non-symmetric counterparts.
Whereas CNNs are only guaranteed to be Iranslation equivariant, the concept of equivariance in neural net- works can be extended to symmetries beyond translations, such as rotations or reflections. Group equivariant CNNs (G-CNNs) (Cohen & Welling 20166 Cohen et al.) 2019aa Aronsson 2022) and stcerable CNNs (Cohen) &r Welling 2017 Weiler 2018 Cesa al.) 2022) use convolutions on groups achieve equivariance with respect general global symmetries. Analogous to the equivariance property of traditional CNNs, group

---

Published in Transactions on Machine Learning Research (05/2024)
transformations (e.g. roto-translation) applied to the input of a group equivariant convolutional layer, lead to the same group transformation being consistently applied to the output. Group convolutional layers thus commute with group transformations. In certain applications where larger symmetries are important, these networks have been shown to further improve performance compared to networks exhibiting less symmotry (Graham el al.) 2020 Gerken et al. 2022). From physical perspective, the symmetries considered in CNNs and, more generally, in G-CNNs are analogous to global symmetrics of lattice field theories, which has led to numerous applications of CNNs in high energy physics (see [Boyda al. (2022) for a review). For example, CNNs have been applied to detect phase transitions and learn observables (Zhou et al., 2019 Blücher et al,
2020
Bachtis et al.) 2020 Bulusu et al.] 2021 Bachtis et al.; 2021] and as generative models Nicoli et al.
2021
Albergo et al. 2021a; de Haan et al. 2021 Gerdes et al. 2023
Albergo et al. 2021b in both scalar
and fermionic lattice field theories.
In addition to global symmetries, the laws of physics of the fundamental interactions are based on the notion of local symmetry, which is the foundation of gauge thcorics. Local symmetrics allow for group transforma- tions that can differ at every point in space-time. In machine learning, gauge equivariant neural networks (Cohen al. 2019b Cheng et al., 2019) (see also Gerken et al. (2023) for review) have been proposed as architectures that are well-suited for data living on curved manifolds. In high energy physics, similar methods have been applied to problems in lattice gauge theory. For example, gange symmetric machine learning models have been used as generative models (Kanwar et al. 2020 Boyda et al. 2021 Albergo et al.,
2021a Abbott et al. 2022 Bacchio et al. 2023) to avoid the problem of critical slowing down inherent to Markov Chain Monte Carlo simulations at large lattice sizes, or as machine-learned preconditioners for the
Dirac equation in lattice QCD (Lehner & Wettig, 2023a bj Knüttel et al. 2024p. Going beyond specific applications, Lattice gauge equivariant CNNs (L-CNNs) (Favoni et al.) 2022) have recently been proposed general gauge equivariant architecture for generic machine learning problems in lattice gauge theory, L-CNNs use gauge field configurations as input and can process data in a manner compatible with gauge symmetry. They consist of a set of gauge equivariant layers to build up networks as stacks of individual layers. In particular, gauge equivariant convolutional layers (L-Convs) are convolutional layers which use parallel transport to preserve gauge symmetry while combining data at different lattice sites. Because of their expressiveness, L-CNNs can be used as universal approximators of arbitrary gauge equivariant and in- variant functions. It has been demonstrated in Favoni et al. (2022) that L-CNNs can accurately learn gauge invariant observables such as Wilson loops from dalasets of gauge field configurations. Similar to CNNs, the convolutions used in L-CNNs are equivariant under lattice translations.
In this paper, we revisit L-CNNs from a geometric point of view and extend them by including a larger degree of lattice symmetry. First, we review lattice gauge theory and the original formulation of L-CNNs in Scction 2 L-CNNs were originally constructed by incorporating local symmetry into ordinary CNNs, which means that L-CNNs are equivariant under lattice translations but not under other lattice symmetries such as rotations and reflections. We remedy this in Section 3 by applying methods from G-CNNs to L- CNNs. Our main result. is a gange equivariant convolution that can be applied to tensor fields and that is oquivariant under translations, rotations, and reflections. Finally, in Section 4 we put the original L-CNNs in a broader conlext by relating them to a mathematical theory for equivariant neural norwsrks. In doing so, we demonstrate how convolutions in L-CNNs can be understood as discretizations of convolutions on SU(N) principal bundles.
2 Theoretical background
In this section, we review Yang-Mills theory and lattice gauge theory in the way these topics are usually introduced within high-energy physics, largely following the conventions of Peskin & Schroeder (1995) and Gattringer & Lang (2010). Having defined concepts such as gauge symmetry and gauge invariance, we then review aspects of L-CNNs.
2.1 Lang-Mills theory
We consider SU(N) Yang-Mills theory on Euclidean space-time M RD with D > spatial dimen- sions. We choose Cartesian coordinates n on M with p (1, D} such that the metric on M is Eu-
2

---

Published in Transactions on Machine Learning Research (05/2024)
clidean, i.c. 9av = 6pm; where ني is the Kronecker symbol. The degrees of freedom in this theory are gauge fields (x), which are su -v)a-ld vector fields on M. We further choose matrix representation of su( N); namely the fundamental representation spanned by the generators t E CNNN with E 21. N - 1}.
which are traceless Hermitian matrices usually normalized to satisfy
1 g00
Tr [
(1)
We note that the convention to use traceless Hermitian matrices to describe su(N), in contrast to anti- Hermitian matrices, is often used in high-energy physics. These two conventions are related by replacing
t" + it",, where is the imaginary unit. With a basis for both M and su(N), a gauge field can be written as the 1-form
A(x) = (x(x)d=" (x)
(2)
with components A M + R. Two different gauge fields A and A' are considered to be gange equivalent if their components can be related via a gauge transformation To,
1x(xx) (x) (xx) (x),,
(3)
where Q : M SU(N) differentiable function on space-time. Gauge fields that can be related via gauge transformations form equivalence class. Within gauge theory, (the components of) gauge fields are not considered physical, observable fields. Rather, the physical state of a system is the same for all gauge fields in any particular equivalence class. Observables within gauge theory therefore must be gauge invariant functionals A. The most prominent example of such a gauge invariant functional is the Yang-Mills action
/ d2 Tr (xx M
STA
(4)
2g²
which maps a gauge field A to a single real number S[A] E R. Here, g 0 is the Yang-Mills coupling constant, and the su(NN-valued field strength tensor is given by
(x) (x) - -x)A(()) + +i (x)),,
where [, denotes the commutator of matrices in the fundamental representation of su(N). Under gauge transformations, the field strength tensor is transformed according to
Ta
(x)
(6)
Because of the transformation behavior of the field strength tensor and the trace in the Yang- Mills action, the value of the action is invariant under gauge transformations, i.c.
STTRAA = S[A].
The invariance of the Yang-Mills action under gauge transformations is called gauge symmetry.
2.2 Lattice gauge theory
Lattice discretizations of non-abelian Yang-Mills theory with exact lattice gauge symmetry can be con- structed with the help of the link formalism of lattice gauge theory (Wilson 1974). In this formalism, the gauge fields (x) E su(N) are replaced by gauge link variables Ux., E SU(N) defined on the edges of a finite hypereubic lattice A with periodic boundary conditions. We use the fundamental representation of su(N) and SU(V) to represent gauge fields and gauge links as complex matriccs. The links Ux, connect a lattice site to its neighboring sites x μ === x + ad,, separated by the lattice spacing and the Euclidean basis vector ég. The inverse link, which connects x + to x, is written as Uxtp-- Uk
In terms of the gauge field, a gauge link is given by the path-ordered exponential
( i do dx) ds A (x(()))
Ux,, = P Pexp
(8)
3

---

Published in Transactions on Machine Learning Research (05/2024)
In the convention that is used here, the path ordering operator P shifts fields carlier in the path to the left and fields later to the right of the product. The function x(s) [0, - RD parameterizes the straight-line path connecting x to x + p.. Geometrically, the gauge links prescribe how to parallel transport along the edges of the lattice. They transform under general lattice gauge transformations Ta, Q A - SU(N) according to
ToUx, = 2xUx,alx
(9)
Gauge links are the shortest possible Wilson lines on the lattice. Longer Wilson lines are formed by multi- plying links that connect consecutive points to form an arbitrary path on the lattice. For closed paths, they are referred to as Wilson loops, and the smallest loop, which is the 1 loop, is called a plaquette and reads
Ux
- Uxtu-v
(10)
It transforms under gauge transformations as given by
TaUx, = P2xUx
(11)
The Wilson action
(Wilson 1974), which can be written in terms of plaquettes, reads
Sw [U]
2
ΣΣ
Re Tr - Ux
(12)
g²
XEA
Note that 12 is invariant under global symmetries of the lattice such translations, discrete rotations, and reflections. Its invariance under lattice gauge transformations follows from the local transformation property of the plaquettes and the trace.
In the continuum limit, for small lattice spacings a < 1, gauge links can be approximated by the matrix exponential
Ux ~ exp iaA x )4)
(13)
at the midpoint + Lu. Purthermore, in this limit, plaquettes approximate the non-abelian field strength tensor, given by Eq. (5)
Ux x exp ia (x 22+²²²²)
(x+}x+++)
(14)
and the Wilson action approximates the Yang-Mills action, introduced in Eq.
2.3 Lattice gauge equivariant convolutional neural networks
An L.-CNN is built up by individual layers +, which take as input at least one tuple (44, W). The first part of the tuple, U {Ux,hh is a set of gauge links in the fundamental representation that transform non-locally in Eq. 9 The second part, W (Wx,a,, h, is set of complex matrices Wx E C that transform locally, like plaquettes, as in Eq. 11
TaW x.a = Wx.Wa.
(15)
Here, the index a (I.....No. refers to the channel, and Non denotes the total number of channels in the layer in question. The output of the layer is, generally, again tuple (L'', W"), with possibly a different number of channels. We require every layer to be lattice gauge equivariant in the sense of
s (ToU,
(W,
(16)
where T3 denotes the application of the gauge transformation Ω to the output of the layer. Generally, one can consider layers where To * Ta. This would be the case the representation of the input tuple (4, W) is different from the one of the output tuple W'). In this work, we only consider layers that do not change the representation of SU(N) or the transformation behavior of the links i and the locally transforming matrices W in any way. Additionally, we only focus on layers that do not modify the set of gauge links U, i.e. we always require that 4' U. A network built from multiple gauge equivariant layers (Φ1, o2,...; &N
4

---

Published in Transactions on Machine Learning Research (05/2024)
through composition, n o...o D2 D1, respects lattice gauge equivariance in the sense of Eq. 16 In the following, we review some of the layers introduced in Favoni et al. [2022).
A convolutional layer usually aims to combine data from different locations with trainable weights in a trans- lationally equivariant manner. In an L-CNN, such a layer is required to respect lattice gauge equivariance as well. This fulfilled by the Lattice gange equivariant Convolutional (I-Coov) layer, which a map W) + W"), defined
W x,a
Σ b.ju,k
Vo..nn Wx Wx
(17)
with the trainable weights Va,b,u,k E C, output channel index 1 1 a S Neh.outs input channel index < b S Neh,in, lattice directions 1 u < D and distances -Nx < k < NA, where N₁ determines the ker- nel size. Note that the output channels associated with W' are a linear combination the input channels of W. In general, the number of input channels Neh,in may differ from the number of output channels Neh The matrices Ux appearing in Eq. 17 describe parallel transport starting at the point to the point k. ps. They given by
Ux
II 1=
Uxtipp = Ux Uxnuxxeppsttnnn ... -Ux((K-I)1
(18)
for positive and
k-1 II 2=0
Ux
Ux -p-p Ux -12---2--UU--1P Ux- (k-
(19)
for negalive k. Only parallel transports along straight paths are considered because the shortest, path bet ween two lattice sites is not unique otherwise. Data on lattice points that are not connected by straight paths can be combined by stacking multiple layers. A bias term can be included by adding the unit clement to the set W. A further increase in expressivily can be achieved by also adding the Hermitian conjugates of Wx, W. A general L-Conv layer thus may be written
W x,a
U2 Σ Vab.m, K W Wx+k x + Σ Wa.b W atk x voll, biakk
(20)
bip.k
with weights bakk Vabkk and a bias term Vo- For brevity, we will use the more compact form given in Eq. (17). L-Conv layers are gauge equivariant by virtue of the transformation behavior of the parallel transporters Ux From Eq. (9h it follows that
ToUx,ky = DxUx,
(21)
The matrices Ux thus allow the L-Conv layer to combine data from various lattice sites without violating gauge symmetry.
It follows that if data are combined only locally, there is no need for parallel transport to construct lattice gauge equivariant layer. This is realized by Lattice gauge equivariant Bilinear (L-Bilin) layers, which are maps (2, W), W") + (2, W"), given by
Σ Ca,bbeWxx,ble x.c b.c
(22)
x,a
The weights aa,de E C are trainable, have an output channel index 1 1 S Neh.out and two input chan- nel indices 1 < b < Nah.in and 1 S c S No chiin Analogously to the L-Conv layer, the unit element and the Hermitian conjugates can be added to W to increase the expressivity of the L-Bilin layer.
Lattice gauge equivariant Activation functions (L-Act), which are maps (U, W) + (U, W'), are the gener- alization of standard activation functions to the L-CNN. They can be applied at every lattice site and are given by
W WK.a Ux,a (WW).W.
(23)
5

---

Published in Transactions on Machine Learning Research (05/2024)
where v is any scalar-valued and gauge invariant function. One option is (x, O(Re(TTWWW. x))) with the Heaviside function O For real-valued scalars 8, this would lead to the well-known ReLU activation function, which can also be written as ReLU((s) = (ss)s.
Finally, the last important layer to consider is the Trace layer. This layer maps (u, W) H Tx which converts the lattice gauge equivariant quantities (4, W) to lattice gauge invariant quantities
Tx W) = Tr(Wx,))
(24)
A gauge invariant layer such as this is necessary if the network output is supposed to approximate a (gauge invariant) physical observable.
As an example, an L-CNN can compute the plaquettes as a pre-processing step and use them as local variables Wx,0 in the subsequent layers. With stacks of L-Conv and L-Bilin layers, it can build arbitrarily shaped loops, depending on the number of these stacks (Favoni et al. 2022). The network expressivity can be increased further by introducing L-Acts between said stacks, and if the output is a physical observable, there should be a Trace layer at the end. After the Trace layer, conventional CNN other neural network can be added without breaking lattice gauge equivariance.
3 Extending L-CNNs to general group equivariance
In the last section, we have reviewed the L-Conv operation on a hypercubic lattice A = zd_ Like a standard convolutional layer, the L-Cony layer is equivariant under (integer) translations on ZD, which, when inter- preted as a group T, can be identified with the lattice itself, T ~~ ZP. However, lattice gauge theories on hypercubic lattices typically exhibit larger isometry groups G that include discrete rotations and reflections, which we denote by the subgroup K C G. In this section, we explicitly construct L-CNN layers that are compatible with general G-symmetry.
The original approach to G-CNNs (Cohen & Welling, 2016) is based on promoting feature maps from functions on the lattice ZD, or, more generally, functions on some space M, to functions on the group G. Group cquivariant convolutions, or G-convolutions, are analogous to traditional convolutions, except that integrals (or surns in the case of discrete space M) are carried out over the whole group G. In contrast, the modern approach to G-CNNs on homogeneous spaces M uses a fiber bundle formalism (Cohen al., 2019a; Aronsson 2022) in which feature maps are modeled via fields on M, that is, via sections of associated vector bundles over M.. This approach is geometrically pleasing because it means that the inputs to and outputs from convolutional layers live directly on M. Tt also offers computational advantages since convolutional layers become integrals over M = G/K rather than integrals over the larger space G. Here, K is the subgroup of G that stabilizes an arbitrarily chosen origin in M.
However, in order to take advantage of this simplification one needs to identify feature maps f:G RN with fields on M. This imposes a constraint given by
f(gk) = p(k))
(25)
where g E G E K, and is a representation of K (see Aronsson 2022 for details). This constraint is difficult to enforce numerically and is sometimes ignored, preventing the geometric view of f as a field on M. Fortunately, ignoring the constraint effectively promotes f to a field on the group G instead, similar to the approach laid out in Cohen & Welling) (2016), and the bundle formalism still applies after changing the homogencous space from M to G. Even though ignoring the constraint makes convolutional layers more expensive to compute, it drastically simplifies their implementation in machine learning frameworks such PyTorch (Paszke et al. 2019). In case, this is because the group G of lattice symmetries is a semi-direct product G= T x K of translations x E T and rotoreflections r E G/T K. Group elements can thus be split into products maps (x) on the lattice 7D. It can be shown that G-convolutions ions can be expressed in terms of traditional zD =ocoovvoltiooss (Cohen& & Welling 2016), for which highly efficient implementations already exist.
ær. Consequently, feature maps f: + R can be viewed as "stacks" of feature
Our strategy to develop a G-equivariant framework for L-CNNs thus the following: We first review group equivariant networks without gauge symmetry by working out explicit G-convolutions for scalar fields, vector
6

---

Published in Transactions on Machine Learning Research (05/2024)
fields, and general tensor fields discretized on the lattice z", in the spirit of the original G-CNN formulation (Cohen & Welling 2016). We then show how G-convolutions can be combined with our approach to lattice gauge equivariant convolutions to obtain fully G-equivariant L-Convs. We extend our approach to bilinear layers (L-Bilin), activation layers (L-Act), trace layers, and pooling layers, which allows us to formulate fully G-equivariant L.CNNs.
3.1 Group equivariant convolutions for scalars the lattice
Convolutional layers in traditional CNNs act feature maps, e.g. functions f: - R", where is the mumber of channels. For explicitness, we consider real-valued feature maps. A real-valued convolution with n imput channels and n' output channels is given by
[t + fx(
Σ 1 YEZo
Σ
Go (y) a a (1,,,....
(26)
where 1 n RR are the kernel weights. Here explicitly boldfaced letters to denote points the lattice 7D and the symbol * to denote a -o-coovouution. The convolution operation is equivariant with respect to translations: applying translation E U, which can be identified with the point E ZD, to ( /] yields
[. (x) = 1bb f1 2)
Gab (y - -x (y)
b= yEZ
(27)
(y' xy) z)
b= y' y'D7D
= [bb (x).
The left translation L2 commutes with the convolution operation. The main idea of Cohen & Welling (0016) is to introduce convolutions that are G-equivariant, i.e. that commute with by for g E G. More specilically two types of G-convolutions (G-Convs) are introduced: the first-layer G-convolution acts on feature maps on DD and promotes them to feature maps on the group G, and the full G-convolution, which acts on feature maps on G and outputs feature maps on G. To simplify notation and without loss of generality, we set the number of channels to one.
The first-layer G-convolution acting on feature map f: 2D + +R is given by (Chhen & Welling 2016
[+b * fl(g)
Σ 1 gEG, yEZ
(28)
where 7D - are the kernel weights, and y denotes the action of the group element g- on the point y ZD. Note that denote 2 -covvolutiinns as (66), by and G-convolutions by *. Uniquely split- ting the group element g into translation x E (identified with 2D) and rotoreflection E K G/T about the origin, Ir, we have
g -'.-y-R 1
(29)
where R E DxD is matrix representation of the rotoreflection r. This split unique because G is a semi-diroct product G = T K. In addition, the translations T form normal subgroup of G:
Lag TT, Ve Er
(30)
Note that the result of the G-convolution in Eq. 28 is function [ * f] G - IR the group G. The effect of the rotoreflection r is that the feature map is convolved with the rotated kernel
Lably x) (R 2y -6yxxx x)))
(31)

---

Published in Transactions on Machine Learning Research (05/2024)
The first-layer G-Conv can therefore be written convolution over zn
1 færr = [L, fl(x),
(32)
which we refer to as the split form (see also section 7 of Cohen & Welling (2016))
The full G-convolution acts on feature maps f: - and is used after the first-layer G-convolution. is given by
(++ * fl()) = Σ 1lg (n)(()) hEG
(33)
where & + R are the kernel weights. Since we are dealing with discrete groups, we use a sum over the group elements define the convolution. Both g and h elements of the group G, and g g denotes the group product. Just as before, we would like to write this operation in terms of zd -o-convolutions (the split form) using g = zr and h = ys with 2, E T and r,8 E G/T. In order to perform this split, we need to be able to interpret functions on G as "stacks" of functions on DD_ Given f: G + IR and h = we write
f(h) = f(ys) = fz)).
(34)
where f:: + for each element 8 8 G/T. The function I is therefore equivalent a stack of func- tions (fa s G/T3. A left translation acting on f with g rr and h ys induces
( f(g
= =[((xr) yss)
(35)
= f((r 1 yrr -
= ff- (y x)))
where 2 = --- 1-'--''rr yr E T (as T is a normal subgroup of G) and r-'s G/U. In the last line we have made use of the fact that pure translations x E T can be uniquely identified with points x E 2D via the action of the translation subgroup on the origin 0. The point z associated with & is given by
0 0- ((x (r 0)) = -1 (() 0) = r (y x) = R x)),
(36)
where r 0 because rotoreflections form the stabilizer subgroup associated with the origin.
The kernel in Eq.
33 can thereby be written as
(1) = = 1, (R xx))
(37)
hence the split form of the full G-convolution is given by
b f((r) = Σ Σ 0,-,, (R- (1xy-)))f(s
NEG/TyE7D
(38)
Σ .EG/T
( fa)(x),
which is a sum of multiple z -P-covvolutions with rotated kernels L, 1, -xxx. and (38) are particularly useful for concrete implementations in machine learning frameworks. Writing the G-convolutions in terms of 2 -cconvoluutions allows us to make use of highly optimized implementations such as the Conv2D and Conu3D functions provided by PyTorch.
x). The split forms Eqs. (32
Both types of G-convolutions can be compactly written as
f(())
Σ (h)S())))) KEH
(39)
where we use H D for the first-layer G-Conv and H G for the full G-Conv. In this form, evident that the two types merely differ in the group that is being summed over (translations T 7D in the first-layer
8

---

Published in Transactions on Machine Learning Research (05/2024)
G-Conv, the full group G in the full G-Conv) and how the left translation acts the kernel db. Depending the choice of H, the left translated kernel lighh is either rotated 7 -eeenel for H = 7D or a translated kernel on the group for H = G. It is now easy to check that G-convolutions are in fact equivariant under left translations Lg. Let E G, then we have
L * fl(g) = =w f((
-E La-1gg5())000
hel
E (kb(k))
bеи
(40)
Σ Lg0(N))(
WEH
E (, MEH
= [ Lx
where we have used the substitution N/ = kh in the fourth line. For the first-layer G-Conv (H 1H=2% 2D), where E zD, n'= kh is to be interpreted as a rotated and shifted coordinate on 2D as in Eq. (9), whereas for the full G-Conv (H = G), h' kh is simply a translated group clement in G. Note that G-oquivariance can also be shown for the split forms Eqs. 32 and 38), but the proof is analogous to the one shown above.
3.2 Group equivariant convolutions for vector and tensor fields
We have explicitly shown that the G-convolutions are equivariant under general transformations g via left translations Lg. The feature maps f Z - R, which the convolutions act, transform like scalar fields under by with y rr:
= -
x)).
(41)
In order extend G-convolutions to vectors and tensors, we would expect transformations that act on the vector structure. For example, consider a vector field 7D - RD on the lattice with components of : DD + R. Acting on with general transformation g yields
(Lgu)((y) = R R'jv' (R (y ()))),
(42)
where R', are the components of the matrix representation of E G/T on R.. More generally, type (n, m) tensor field w, i.e. with vector and m co-vector components, transforms according to
Whyw in (x) = R ...R", v. whihh R....
( ... 11- jm
(43)
Based on the compact form scalar G-convolutions, Eq. 39 we now make the following guess at G-con- volutions which map tensors of type (n,, 0) to tensors of the same type:
[bb * w| f (g)) = (L in Sind (1) (h)
(44)
GEH
where H = DD for the first-layer and H = G for the full G-Conv. Here, the kernel ab is a tensor of type (n, n) and acts as a general linear transformation of the tensor components of w. For the full G-Conv, = G, left translations acting on tensor fields va type (n, m) on the group are given by
(Lett) im (h) = R'gg Rin v whtn S... (g- ()) ... (R (m-
(45)
G-convolutions for tensors of mixed type (n, m) are defined analogously.
9

---

Published in Transactions on Machine Learning Research (05/2024)
The tensor G-convolutions be cast into more compact form by introducing condensed index notation: we write the block of indices .1... as multi-indices I so that
w w w Jinnan
8 = 8 8
(46)
R = R n1 ....R
R )n
Contractions of R and R-1 can be written
R R² J( k= = R R'n (R (R- (R-
RR Jn JR 111 yhk ki ... Rt sn (R- kn
(47)
8'1 k1 ... 8' kn
= 8 k-
Using multi-indices, the lensor G-convolutions are given by
[ * w/ (g) Eleob) (N),
(48)
NEH
and left translations act on tensors according to follows from
(Tgw)' (() RC pw" 5 (i)
(49)
Equivariance of Eq.
(T w))' n w/) (k-1
Σ Rl (Lx-119)) s(h)w
bEH
Σ R¹ (R' (L. (khh) yw (h)
(50)
bEH
Σ (Lyb)' ,(h) yw (K-
NEH
= [&b * L (g).
where we have used the substitution N/ kh. We can also define G-convolutions that change the tensor type. Given multi-indices Ln = i1...... in and Jm = f J...JJm Im with n # m gencral, we define
ih 1b u/ (g) (Lg8t) Jm (h)w"m(h))
(51)
REH
with H = DD for the first-layer and Hl G for the full G-Conv. Here, the output feature map tensor of type (n, 0) while the input feature map is tensor of type (m, 0). The kernel & is of type (n, m). For example, one can use these types of G-convolutions to reduce rank 2 tensor to vector field within a G-CNN while keeping the equivariance property.
3.3 Split forms of tensor G-convolutions
As in the case of scalar G-convolutions, it is beneficial to write tensor G-convolutions Eqs. (8 in their split forms, analogous to Eqs. (32 and 38 Using h ys G with y T and G/T, write the tensor field of type (n, m)
wn ((h) s (w.)² ,
(52)
10

---

Published in Transactions on Machine Learning Research (05/2024)
where y E ZD. Analogous to Eq. 35 left translations act via
(Lgg)' (h) Rt (w,,-1)) 5 ( x))
(53)
where g g= =x,, RR', is the (n, m) matrix representation of r and R is its representation on gd, The two types of tensor G-convolutions can then be written
[1b w))(())===)))))())) (0)
E''
Rt' rvt (y-)) sw (y)
yE7D
(54)
((Tr))) 1 (xx)
x [6* ())(()) = Σ Σ Rt ((b--)))))
(x)
yEZ sEG/T
(55)
Σ sEG/T
[(L, 1a)' (xx))
where wh, denotes the lensor components of the kernels, and v/4 ; + for the first-layer and b' G R for the full G-Conv, respectively. Analogously, similar split forms can be obtained for ten- sors of mixed type and for G-convolutions that change the tensor type as in Eq. 51
3.4 Implementing group equivariance in L-Convs for scalars and tensors
Having worked out G-convolutions for general tensors, we shift our focus back to gauge equivariance. The L-Cony introduced in Eq. (17 can alternatively be written
W * #](x)( = E wby
(56)
YEZD
where write (xx) instead of Wx to denote gauge dependent feature map. We keep the mumber of input and output channels set to one for convenience. The paths y in the subscripis of the links U are kept general to further simplify notation. Since we only want to allow for straight paths, however, we set all other elements of the kernel & to zero. Note that we only need to consider how the convolution acts on the local variables W because the links are unaffected by an L-Conv.
The L-Conv operation is translationally equivariant, i.c. the gauge oquivariant convolution commutes with translations. When acting on [v= W] with a translation 2, find
[e * W](x) = [ w](x z) = Σ
+ x
W(y)
YEZ
Σ
vyy xxUxx-uyy
2
(57)
y'EzD
I L.()
where the shift by induces a translation on both W and U. We note that our notation for gauge equivariant convolutions explicitly hides the gauge links, but we want to stress that translations must also be applied to the links and that links should be viewed as part of the input. Keeping mind the transformation properties of U and W, given by Eqs. ( and (15,, is sssrtgghttorrard to check that the T-Conv equivariant under gauge transformations:
[ * (x)]
Σ bly YE7D
= [xx) (x)
(58)
Thus, the L-Conv commutes both with translations on D and with lattice gauge transformations.
Previously, we have determined the split forms G-convolutions when the group G is a semi-direct product of translations T ~ 7D and proper and improper rotations G/T. For scalar feature maps, found Eqs. 32
11

---

Published in Transactions on Machine Learning Research (05/2024)
and (38 which reduce the G-convolution to z"-convolutions with rotated kernels. Following the same ideas, we extend our lattice gauge equivariant convolutions so that they respect not only translations but larger symmetry groups G. If we consider W = + CNNN to transform as a scalar under G,
(x) = (gg 1, x) ( -)),
(59)
then we can easily extend G-convolutions to gange dependent fields W by replacing the standard Z0-con- volution with the L-Conv operation. For the first-layer G-Conv of Eq. 32 we then have
[ * (((xr) = 1 W](x)
Σ
(R (y)))
(60)
YEZD
The resulting feature map W(g)= = Ivv (g) is now a feature map on the group G and transforms under G according to
Lg (h) = W h) = (6+ * Lg
(61)
Additionally, under lattice gauge transformations we have
(g)(g) (ar) L * (zr)
(62)
[X * ( (x))
(Zr))(( (x).
i.c. the feature map W(g) W(27) (xr) transforms locally Similarly, the full G-Conv is given by
(
Wl(g)
Σ
(63)
BEG
where q + DD projects group elements g = ar down to the lattice by acting on the origin:
( 0 r.r. =xx.
(64)
More compactly, both types of G-Conv can be written as
Wl(g) = Lo
(65)
hEH
with H = DD or H = G. Similarly, we can generalize these scalar G-convolutions to tensor convolutions in the same way as the previous section. A general G-convolution for gauge dependent tensors is given by
W'()= = x = Elgo((h) J (199)) W (h)U
(66)
DEW
where W and vb are of type (n, 0) and mn, respectively, and the output feature map transforms a tensor of lype (m, 0). Note that gauge equivariant G-convolutions merely differ in the appearance of parallel transporters compared to Eq. (8)) Consequently, the proof of equivariance of Eq. (66) under left translations in G largely follows the steps performed in Eq. (50).
One aspect of G-equivariant L-Conv layers that has been missing in the discussion so far is the inclusion of bias terms. Generally, bias terms are additional terms added to the convolution,
('(()) [ ()) (l(g),
(67)
where (g) is a tensor that is independent the feature map W'()). To retain group equivariance in Eq. 667 we require b' (g) to be invariant under left translations:
(Iab) (0) =R' ,b' 2f
()).
Vk E G.
(68)
12

---

Published in Transactions on Machine Learning Research (05/2024)
Depending on the symmetry group G, the number of dimensions D, and the tensor type (n,0) of the feature map, such invariant tensors may or may not exist. In the case of scalar feature maps, namely : - CNNN (first layer) and W : G (deeper layers), bias terms simply correspond to unit matrices added to the output of the convolution:
((g) = (+b (g) =
(())) + bo
(69)
PEH
where by a trainable parameter. However, the geometric structure of these terms becomes more complicated for general tensor feature maps and depends on the symmetry group G. For example, if G consists of rotations, reflections, and translations, then there is no vector-type (i.e. tensor of type (1, 0)) bias term. For rank 2 tensors, i.c. tensors of type (2, (), bias terms correspond to Kronecker deltas 8.j. In D = 3 dimensions, rank 3 bias terms may be given by the Levi-Civita tensor Eijk assuming that reflections are not considered as symmetries. More generally for the rotation group in D = 3, higher ranks are given by products and contractions of diy and Cizk-
3.5 G-equivariant bilinear layers
Bilinear layers, which map two feature maps into one, can be generalized to respect G-oquivariance. Without including channels, G-equivariant bilinear layer for tensors reads
W' (9) = (Lا8) JK V1 (g) (g),
(70)
where V1 and V/2 are tensor feature maps of type (n1, 0) and (n22, 0), respectively. The product of V1 and V/2 is understood to be a matrix product with respect to their N matrix structure. The weight tensor & is of type (m, n1 + r2)) and the resulting tensor feature map W is of type (m, Note that the weight tensor is constant, i.e. it does not depend on the group element g. However, because of its index structure, it transforms in the usual way
(LD8)) JK = R rot J'K ( 2)" "( K,
(71)
where R is the matrix representation of the rotational part r of g = xr. It follows that the bilinear layer is equivariant under left translations:
(L&W)+ = (L JK (LAV)) (I)AV))))))))))))))
(72)
Gauge equivariance follows from the fact that when both V1 and V2 locally transforming in Eq. [11 their matrix product also transforms locally. Consequently, the resulting feature map W transforms locally as well.
The bilinear layer can be expressed as a special case of a gauge equivariant tensor convolution. Consider a tensor V of type (n1 + n2, 0) that factorizes into two tensors V/2 and V/2 with types (n1. 0) and (nz 0):
vJk= = V
773)
A general lattice gauge equivariant convolution applied to V then reads
('(g) = Engby JK (1) (1)), (
(74)
hEH
The bilinear layer is local, i.e. feature maps are all evaluated at the same element g. We therefore assume that the kernel has the particular form
(kh) = E 8k0(),,
(75)
where 5(e) = and 6(h) = 0 for h & e with the unit element e, and & is a constant tensor. The left translated kernel (h) only has a single non-zero contribution to the sum, namely when g - is the unit element e of the group, i.c. h = g. The convolution then reduces Eq. (70). because the parallel transporters reduce to unit matrices for constant paths ((g) *(g)=
U9(9)~9(9) = 1.
(76)
Bias terms may also be included in the bilinear layer. As in the case of G-oquivariant L-Conv layers, bias terms must be invariant tensors.
13

---

Published in Transactions on Machine Learning Research (05/2024)
3.6
Trace layers
It is straightforward to show that trace layers are compatible with G-equivariance. Given locally gauge transforming tensor field (g) we define the traced tensor (g) simply as
w/ (g) = Tr(WW (o)),
(77)
where the trace is taken over the CNNN matrix structure. The trace yields gauge invariant tensor, which can be shown via
Taw (g) [T(T:W"(()]
(x)
(78)
= (g)
= wg)
where we have assumed ((9)= x. The traced tensor transforms under G as tensor because the left translation Lx, Ak E G commutes with the trace operation over CNxN Note that the trace layer does not have any trainable parameters. It can be used at the end of an L-CNN to obtain gauge invariant scalars and lensors, which can then be further processed by standard group equivariant networks.
3.7 Activation functions
The purpose of an activation function to introduce non-linearity into the network. In a CNN or G-CNN with scalar input, it usually applied point-wise, ie. f'(x) = (x(()),, where f.f' G G R are the feature maps before and after applying the (scalar) activation function R R, respectively. When the input
feature map is a general tensor field of type (n, 0), we choose an ansatz similar to Eq. 23 namely
w" Cyw' = u(w)w',
(79)
with the operator Cz, which applics the activation function to the tensor. For G-equivariance hold, C has to commute with Lg
CyLyw' = LgCowt.
(80)
Keeping in mind the transformation property of w, which is given by Eq. 49
WC find
vw)(h) = (wr(h))
(81)
where w/ (h) R R'"""""" (h). Thus, the activation function has to be invariant under transformations in the group G.
The generalization of Eq. 79 to lattice gauge invariant activation functions is straightforward. We make the ansatz
w" = =)((())))))
(82)
with the local variables W and W before and after the application of the activation function, respectively, and w (g) Re(TTr The form of w(g) guarantees equivariance under lattice gauge transformations, and Eq. 81 guarantees equivariance under transformations in G.
A possible choice for an activation function is a norm non-linearity (Gerken et al. 2023
v(w(g)) = a(Iw(())))
(83)
with a norm that satisfies Eq. non-negative, choosing the Heaviside step function θ as the function a, which we have done in Section 2 to mimic the well-known ReLU activation function, would not lead to a non-linearity. Therefore, we introduue a trainable bias 2 0 and set
81 such as ||w[())|== (w Since the output of a norm is always
(([[((g)))) [[[[[(g)]| b).
(84)
In the above example, the ReLU activation function becomes active if the norm of exceeds the bias b.
14

---

Published in Transactions on Machine Learning Research (05/2024)
3.8 Pooling
Pooling layers, which are often used to reduce the domain of feature map, can also be generalized to G-CNNs. In analogy to Cohen & Welling (2016), we split pooling layers into separate pooling and sub- sampling steps. The pooling step performs a convolution-like operation on the feature map, but does not change the domain. The domain reduction happens during the subsequent subsampling step with a par- ticular stride. Typically, subsampling leads to a reduction in symmetry, depending on the stride. We first review this procedure for scalar feature maps on the group G before generalizing it to tensor fields on G and finally to the L-CNN.
As motivating example, we consider sum pooling. In traditional CNN in two dimensions, sum pooling is performed by summing up all values of a feature map in a pooling domain D C Zz, c.g. a 2 x 2 region, which is moved across the lattice. It differs from average pooling only by a constant. factor determined by the cardinality of D. Since sum pooling in a traditional CNN can be written as a special case of convolution (overy kernel coefficient in the pooling region is set to onc), it equivariant with respoct to translations. Analogously, sum pooling on a feature map f : G - R can be viewed as a special case of the full G-convolution in Eq. (33) when setting the kernel (b))) to one in the pooling domain D G and zero elsewhere:
('(9) Σ وyا ())())) hEG
Σ S(h). hEED
(85)
Here, =D= {gd: : E D} refers to the g-translated pooling region and f G RR is the feature map after the pooling step. Another well-known pooling operation is max pooling, given by
f(()) = max f(h). hEgD
(86)
Sum and max pooling can be generalized to other pooling operations by introducing an operator P that acts on a feature map f by
(())= = PL((gd)) ((d),
(87)
where P: R.V > is a function, and N is the cardinality of = (di, d2.... .., dx). The operator P g-translates the set D over the feature map, so the same pooling operation is performed everywhere on G, rendering it G-equivariant. That is, it commutes with the left translation operator La- This can be explicitly shown via
(g( (PF)(k-'))
= P(f(k
(88)
=P(Laf(gdi) (gdi),.... ..... Lu (gdx))
=(PLxf)(g.
To achieve a pooling with stride in traditional CNN, after the pooling step, the feature map is subsampled on the subgroup T, C T consisting of all translations that are multiples of elementary translations. The resulting feature map is then equivariant under I,, Analogously, for feature maps f: G + R, subsampling is performed on a subgroup H C G. This procedure is known as subgroup pooling and the resulting feature map retains equivariance only under the subgroup H.
If the pooling region D is itself subgroup of G, then gD are left cosets of D. They partition the group into disjoint, equally sized subsets. Since the left cosets are invariant under the right-action (or right translation) of D, iee. gdDD gD, Vd E D, the corresponding parts of the feature map are invariant under said action well, and we can pick one such part to be the resulting feature map without losing G-equivariance. This type of pooling is called coset pooling. For example, in a network with scalar input on ZD that is promoted to the group G = T x K by a first-layer G-convolution and further convolved with full G-convolutions, coset. pooling over K would yield a G-equivariant feature map on T ~ ZD.
In order to generalize sum pooling to tensor fields w(g) on G, we take full tensor G-convolutions, which are given by Eq. (48), with H = G as starting point and set the kernel of r(9) 8¹ 18((), where (((g is one
15

---

Published in Transactions on Machine Learning Research (05/2024)
if g D and zero everywhere else. We get
w (g) = [bb * ()) Σ ((h)))s(()))
hEG
Σ Rt rub" 5g 1 sw (h)
hEG
(89)
Σ R n.8 ( sw
hEG
Σ hEG
s w (h) = Σ (h), (h))
NEAD
which differs from sum pooling in the scalar case, Eq. 85 blance as motivation to define the general pooling operator P on tensor fields of type (n, 0), i.e. w G + V with V = (RP)", as
only by the tensor index I. We take this resem-
where the function maps P: VN - V. The pooling operation commutes with left translations (90)
wnt (g) - (Pw) = ((o(gd)).....(((x))
P(Rw(d)), Rw)(dx)) = R ((u(d)))....d((()))
(91)
which mcans that if the pooling operation commutes with the outer transformation R then it G-oquiv- ariant. This is the case for sum pooling defined in Eq. (89). Furthermore, if the norm ||-II is unaffected by the outer transformation R, then G-equivariance also holds for the max pooling operation
w w²((g) max w/ (h) w/ (g'),
(92)
hEED
where
g' = aargmax [[w(h)]] NEED
(93)
corresponds to the element g' that maximizes the norm of w in the translated pooling domain gD. The
above definitions for lensor feature maps w G + V are consistent with the corresponding definitions for scalar feature maps f G R, where V = IR and the rotation matrices R reduce to the identity operation.
To generalize sum pooling of tensor fcature maps L-CNNs, we start with the L-CNN full tensor G-con-
volution, which is given by Eq. (66), with H = G. We set the kernel vid ( 8 rو(g), where v/() is one inside the pooling domain and zero elsewhere. Steps analogous to Eq. 89 lead to
W (9) = Σ U4i W (h)U = (1)),
(94)
NEED
hEgD
where W and W are the local variables before and after the pooling step, respectively, and
W (h) (o(1) W (N)U
(95)
denotes the variable W, parallel transported from h to g. Note that the fact that only consider straight paths in the convolution restricts the shape of the pooling region D. In principle, however, it can be chosen arbitrarily as long as the paths q(h) - ((g) of the parallel transporters Uq(h)=9(u) are chosen accordingly.
A general pooling step, represented by the aforementioned pooling operator P, on local tensor variables W can be written as
= (")))=(PW)) (g) = (WW)) W (g(gd)))',
(96)
where P: V + V. It is G-equivariant if the outer transformation commutes with the pooling operation
P(RW,(d)) RW, (dx))) = - PWW (y(dx))
997)
I For simplicity, we assume that the maximum of the feature map unique. In practical applications of G-CNNs and L-CNNs, it is highly unlikely encounter cases where the maximum cannot he uniquely determined.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Similarly, we require the pooling layer to be equivariant under lattice gauge transformations
(P [()) 1 (PTRW)'())
(98)
which implies that the function P must also satisly
Wo(dn)) P(S,, ..... SW, (dv)))) = W,,ld))).,,,,,w))) ot.
(99)
We omitted the argument x = =(g) = g(ær) of Ω = [2(x) for simplicity. For example, max pooling can be defined as
w (g) max (1)( = = (')),, hEgD
(100)
with
g
argmax BegD
|| Re Re (Tr(W"())))]
(101)
where the trace leads the parallel transporters dropping out. Clearly, this form of max pooling satisfies both G-oquivariance and equivariance under gauge transformations.
As concrete example of how these pooling layers can be used, consider a network that uses local tensor variables W on 2d as input. In the first layer, the input feature maps are promoted to the group G = T x K via a first-layer lattice G-convolution. After that, the feature maps are convolved by full lattice G-convo- lutions. Coset pooling over K can then be applied to reduce the domain of the feature maps from G back to 2d while retaining both gauge and global group symmetry. For example, if we use sum pooling, each resulting feature map reads
W (g)
Σ Cw), hEgK
(102)
where the parallel transporters are trivial because the projections ((g) q(h) coincide at the same point x E DDD The feature map is invariant in the sense of W" (ar) W" for r' E K. Subsampling in K thus leads to a G-equivariant tensor (x) on D. In this way, G-equivariant L-CNNs can be used to model gauge and group equivariant functions which map input feature maps on the lattice zD to new output feature maps on z
3.9 Computational requirements of G-equivariant L-CNNs
Having defined the Geeqivvrrannt generalizations of relevant L-CNN layers, we need to comment on the computational resources required by these layers. There are two main differences to the original formulation of L-CNNs: the first concerns the domain on which feature maps are defined, and the second concerns the computational complexity of a G-convolution layer.
Regarding the first point, we recall that in standard L-CNNs, feature maps are functions on the lattice ZD (with periodic boundary conditions) consisting of Ne. NP" complex matrices in each layer, where No. is the number of channels and NP is the total number of lattice sites the hypercubic lattice. Depending on the dimensions of the lattice, our G-equivariant generalizations enlarge these domains considerably. After the first-layer G-convolution, feature maps are promoted to functions on the global symmetry group G. Since G is a semi-direct product of the translation group T ~ z and the stabilizer group K, we may write the number of group elements as GG| = |KINP. Feature maps on G may thus be represented by Na. |K] NP complex matrices, which leads an increased memory requirement by factor of |K] compared to the original L-CNN formulation. For example, if we consider rotoreflections in D = we have |K] 4 2 = 8 group elements since there are four possible rotations and two mirror operations about the two axes. For D 3 and D = 4 we have K| 48 (octahedral group) and |K| = 384 (hyperoctahedral group) respectively. These factors are large, considering the fact that the models used in the original L-CNN study [aavoni et al. 2022) easily saturated the available memory on modern GPUs. For the physically relevant case of it may be argued that the symmetry subgroup K may be smaller than the full hyperoctahedral group. In practice, one typically uses lattice sizes N N2, where N₂ is the number of cells along the time directions with N, * Ni, which reduces the symmetry. Thus, the group K should consist of rotations and
17

---

A related question is the universality of individual layers. For example, is the G-equivariant L-Conv layer the most general G-equivariant, gauge equivariant linear map with respect to the feature map W? It appears that this is not the case due to the path dependence of parallel transporters employed in the convolution. The non-universality of the linear layers can be demonstrated for the simpler case G DD. In this work, we have always assumed that the paths along which feature maps are transported must be straight lines. Clearly, Published in Transactions on Machine Learning Research (05/2024)
reflections in the spatial directions (octahedral group) and reflections along the fourth axis only, which leads to |K| = 2- 48 = 96. Nevertheless, the memory footprint of G-equivariant T-CNNs is generically much larger than of their 2 -qquivaaria counterparts.
Additionally, one has to consider the computational complexity of a G-convolution layer, i.c. the number of operations required to evaluate such convolutions. Similar to traditional CNNs, the original L--Conv layer consists of a sum over the lattice zD. We typically restrict the kernel of the L-Conv the lattice axes, such that in total D (N&x 1) + terms have to be summed over to compute the result of the convolution. G-convolutions extend this sum to over the full symmetry group G. Thus, the number of terms to consider is larger by a factor of K|
3.10 Universality of lattice gauge equivariant layers
Neural network architectures such the multilayer perceptron (MLP) (Hornik et al. 1989 equivariant MLPs (Ravanbakhshi 2020). CNNs (Zhou 2020) and G-CNNs (Sonoda el al. 2022 exhibit universality: given enough depth (number of layers) or enough width (number of channels in a layer), these networks can approximate any function in a suitable function space to arbitrary precision. One may ask to what extent these universality properties also apply to lattice gauge equivariant networks.
Favoni al. (2022) show that an L-CNN consisting of alternating convolutional and bilinear layers can be used to construct any Wilson loop, that is, any closed path of gauge links. Combining this construction of arbitrary loops with non-linear activation layers, a trace layer, and CNN as final network, the L-CNN can represent any non-linear combination of loops. With the result of Durhuus (1980), which shows that the linear span of products of traced Wilson loops is dense in the space of continuous gauge invariant functions, one can conclude that the L-CNN is a universal approximator for continuous gauge invariant functions on the lattice. Ostensibly, one can perform the same explicit construction using G-equivariant layers, which are gencralizations of the original zD -qquivariant L-CNN layers.
in order to describe arbitrary linear maps, one must consider a generalization of the usual 7 -ooqquivarinnt L- L-Conv (sce Eq. (56), which we repeat here for convenience)
Σ bly (y
(103)
YEZD
with the usual restriction of sb to only allow straight paths straight paths x + y. In principle, arbitrary paths from x to y are allowed, which leads to
I * W]'(x))
Σ Σ (.U.W( yE2 eECy
(104)
where the sum runs over the set of paths Cy-ix which start at y and end at x. The kernel &b must be promoted to a map sb= : C - IR, where is the set of all paths on the lattice. We note that translation equivariance requires that the kernel must itself be invariant under shifts, i.c if two paths e and c' are equivalent up to a translation, then it holds that vha dhe
There is an obvious practical problem with Eq. (04): the sum over paths requires (countably) infinite terms for full generality. Even if one restricts the kernel to a finite receptive field, i.c. allowing only paths that stay close to the base point x, there are still infinitely many multi-winding loops to account for, and thus infinitely many kernel coefficients. This is surprising since a standard 2 ="-convolution can be represented by a finite number of trainable parameters when restricted to finite receptive field. Clearly, lincar gauge equivariant layers such as the d -qquivvariant L-Conv are more complicated because they are only linear with respect feature maps W, but are generally non-linear functions of link variables U, which make up the
18

---

Published in Transactions on Machine Learning Research (05/2024)
parallel transporters appearing in the convolution. For practical reasons, it is sensible to restrict the set of paths, but this necessarily limits the expressivity of individual linear layers and renders them non-universal. We note that this ambiguity in the choice of paths is not unique to the L-CNN. Gauge equivariant layers on smooth manifolds (see c.g. Cohen (2021): : Weiler et al. (2023)) typically use geodesics for parallel transport,
but any other choice would be compatible with gauge equivariance. Within lattice gauge theory, a practical example along the lines of Eq. (104) can be found Lehner & Wettig (2023a), where convolutional layers
over restricted set of paths are applied to regression problem.
4 L-CNNs from a bundle theoretic viewpoint
Having laid out the details of how to extend L-CNNs to larger global symmetries in previous sections, we now focus on the mathematical foundations of gauge equivariant convolutional neural networks and how they relate to the translationally equivariant L-CNN. There is mathematical theory of equivariant neural networks that uses fiber bundles to describe symmetries and to capture geometric information in data (Cohen
et al. 2019a Aronsson 2022). This theoretical framework models data points as fields or, more generally, soctions of vector bundles associated to principal bundle that specifies relevant symmetrics. In Sections
4.3 we show that. the original L-CNN (Tavoni et al. 2022 is a discretization of a continuous model within this theory. Section | looks at how the original L- C-CN can be directly generalized to other representations,
iee beyond locally transforming matrices W(x) as input. Finally, in Section 4.5 we discuss the possibility
of placing fully group-equivariant L-CNNs into this theoretical framework.
4.1 Bundle formalism
Geometric deep learning, and equivariant neural net works in particular, uses fiber bundles because they allow nontrivial global geometries; fiber bundles generalize the (goometrically trivial) product M x F between two spaces M and F, respectively known as the base space and the characteristic fiber. We visualize this product as attaching a fiber {x] F each point x E M of the base space. A general fiber bundle is a collection of fibers
E U
(105)
XEM
where each fiber Fx ~ F in the total space E is equivalent to the characteristic fiber. There is also a projection π E + M that maps each element p E Fx C E to the point (pp) = where its fiber is attached. Although the total space E can be more complicated object than a trivial bundle M x F, it is required to look like a product D F on certain local regions D C M. A common example is the comparison between a Möbius strip and a cylinder S x [0, 1], where S is the circle. Locally, both of these objects look like segments D x 1] for D C S but they have different global geometry. The Möbius strip is thus a nontrivial fiber bundle.
Principal bundles are fiber bundles such that the characteristic fiber is a group K, often called the structure group. Since the total space consists of fibers Kx K, the structure group acts an internal degree of freedom at each point M and principal bundles are therefore used to study local gauge symmetry.
Consider the trivial principal bundle on M = RD with structure group K = SU(N),
I R:R SU(N) + RD. T (2)
(106)
which we often refer to as P RD SU(N). This bundle describes an SU(N) gauge symmetry that is incorporated into fields over RD in the following way: Let V be linear space whose clements transform according to a linear representation p of SU(N),
P(D)..
(107)
Triples (x, Q2 v) are interpreted as an clement E located at the position x E RD and expressed in the gauge Ω (U).). The transformation in Eq. 107 means that we let the identity matrix 1 E (U) represent initial gauge, then a gauge transformation 1 + Ω can be achieved by transforming v + p(L)p
19

---

Published in Transactions on Machine Learning Research (05/2024)
instead, hence the triples 2) and (x, p(Q))) are gauge equivalent. We can remove the gauge degree of freedom by defining an equivalence relation
(x, (2 ~ (x, ((2)v)
(108)
and considering the set of equivalence classes
(PxV)/ 1x, u] = -x.
= P VVXX)\/ ~ The equivalence class
(109)
is thus a gauge invariant expression for the element v E V at the position x E RD. By construction, the quotient space Ep is a so-called associated bundle
Tp E E,: RD, (xx, ul) xx
(110)
and such bundles are used extensively the mathematical theory of equivariant neural nelworks. For example, the inputs and outputs from an equivariant neural network are called data points and are defined sections of associated bundles, i.c. generalized fields RR + Ep of the form
sx) = S2 e(x)] = 1. (())u(x)).
(111)
where runs over all values of l E (U) by definition of the equivalence class. The idea behind this definition is that instead of using feature maps (which depend on the choice of gauge), we use gauge- invariant data points as input data, which map from the base space into the class of equivalent triplets. gauge invariant object and use data points to describe vector and tensor fields in a geometric (gauge and coordinate independent) way. In contrast, a feature map of locally transforming matrices ((x) is a single representative of the gange-invariant data point sw (x). Similarly, one may also consider gauge-dependent tensor fields W (x) as representatives of their respective data points.
Thus, one consider sxx at some point
Each fiber Fx of the associated bundle E.p is linear space with respect to linear combinations
a(x, Ω v| + a' [x, Ω2 r'1 ix. Ω, av + +'v',,
(112)
for scalars a, a' and v, v V. We can therefore take pointwise linear combinations as(x) + a's((x) of data points, making the set (p)) of all data points s RD + Ep into linear space.
Layers in equivariant neural network are maps
① ((pi) + [(p2),
(113)
between the spaces of data points for two possibly different representations (pt, Vi) and (pz, V2) of the structure group SU(N). Layers can be either linear or nonlinear, and we say that & is gauge equivariant if it commutes with gauge transformations
Tas(x) = Ta(x, á uxx)
(x(x), v(x) (x p(22 (x))e(x))- ((O())())
(114)
We define gauge equivariant neural networks as compositions of gauge cquivariant layers such as Eq. 113 Note that this definition makes no mention of convolutional layers or translation equivariance, allowing it to used even in the absence global symmetry. Convolutional layers are one of possibly many different types of layers. Moreover, (non-lincar) activation functions, or the composition of a lincar transformation and an activation function, are also considered layers under this definition as there is no requirement of linearity in Eq. (13).. In the following, we will investigate how the original L-CNN relates to this theory.
20

---

Published in Transactions on Machine Learning Research (05/2024)
4.2 Locally transforming variables
We claim that the locally transforming variables used in the original L-CNN are directly related to data points of the associated bundle EAd = P x CN Here, V = NNN is the linear space of complex N x N- matrices and p = Ad is the adjoint representation
Ad(O): W + Sw?t
W E NNN
(115)
In order to investigate how the data points, given by Eq. (local) gauge
are related to ((x) for this bundle, we fix
w : D.PPP
DERD,
(116)
i.e. a local section of the principal bundle P RD x SU)N). This principal bundle is trivial, and thus the gauge is given by w(x) = (x, g(x)) for unique function : D + SUNN). sf: - EAd is data point, then the gauge selects a specific representative
f(x) = (x, W(x)) E P NNN
(117)
of the equivalence class s(x) for x E D. The triple in Eq. 117 describes a matrix W(x) CNxN, placed at the position RD and expressed in the gauge g(x). As our notation indicates, we argue that W(x) is the valrix-valued, locally transforming variable used in the original L-CNN. For the purpose of verifying the transformation behavior of (xx) under gauge transformations, fix second gauge w' D' P, given by w/ w'xx) (x.1' (x)), and use it to select a representative
f' (x, (xx,
(118)
of the equivalence class s(x) E Ex for x E D'. For each x in the intersection D n D', Eqs. 1117 and (118) select possibly different representatives f(x) ~ fx(x of the same equivalence class s(x) and must therefore be related by
(x,g' w' (x.g(x) (())) (x)).
(119)
Here, [(xx) = get (x)g(x) the gauge transformation that transforms between w and w' This shows that the matrices W (x) exhibit. the correct transformation behavior
w'(x) W (x)
(120)
Channels 1...., m can be introduced by taking direct sums: The multi-channel variable
W(x) (x), (x)(x)),
(121)
transforms under Ad @ ... e Ad and represents a data point sw(x) = [x, Ω, Wx)) of the bundle
Exa = FAA @ ... 4 EAA-
(122)
Let us introduce the shorthand notations nAd = + Ad and Add = 8 Ad Ad.
4.3 Equivariant layers
The convolutional layer of Eq. 17 can be viewed discretization of continuous convoluttonn
[ * W) = =[ dy wa x)Ux (y))
(123)
with kernel components wad RD + RR that are non-zero only on the coordinate axes and
Ux-y = Pexp {
ds dx)
Au Ao(o(o))
(124)
ds
21

---

Published in Transactions on Machine Learning Research (05/2024)
is the parallel transporter along the straight line from to
As discussed in Section 2.3 this design choice is due to parallel transport being path-dependent and the non-uniqueness of shortest paths on the lattice. This convolution is a linear transformation by virtue of being an integral operator, and maps W = (W) ,...,, Wm)) to W = =W"), ,...,,," W in gauge equivariant manner:
1 * TTx])) Tole * (x)
(125)
The action of Eq. 123 on data points sw(x) [x, S2, W(x)] is therefore well-defined and independent of the choice of representative for the equivalence class. Thus, the continuous convolution defines gauge equivariant linear layer
Cony [(mAd) + P(nAd),
Sw + sW'
(126)
is a straightforward discrctization of Analogously, the original bilincar layer Eq. (2
m
W (x)
abe (x))"xx
(127)
b=1
and can be linearized using tensor products. If aw E ((mAd) and sw E [(m' Ad), then
swow E r(mAdd @ m'Ad) [mmm' Ad?).
(128)
That is, W 3n = (Wo & WT). If let be the number of output channels, the transformation in Eq. 127 defines gauge equivariant linear layer
$ Bilin [(mm' Ad?) + T(nA)), kwew + 8wn.
(129)
For each channel, trace layers (x) Tx(W"(x, Q)) compute the trace along the N x N matrix struc- ture and transform under the trivial represcntation Id. Gauge equivariance with respect to the trivial representation is equivalent to gauge invariance, so trace layers are gauge equivariant linear layers
Traace I(mAd) + I(mld),
Sw + sw"
(130)
Finally, the non-linear activation functions w* = v ("((x)) (x) transform W(x) by scaling each lo- cally transforming variable (x(x) using a non-linear and gauge invariant function v". Since these activation functions do not affect the transformation behavior of the input feature map W, they are gauge equivariant non-lincar layers
t Act : [(mAd) - I(mAd),
sw + syr
(131)
4.4 L-CNNs for data in other representations
The original L-CNN requires that (input) data take the form of N matrix variables W(x). How- ever, the connection the bundle theory for equivariant neural networks makes this requirement relatively straightforward to generalize to functions fxx taking values in linear space V, and which transform under gauge transformations
f(x) + p(2(x))f(xx)
(132)
This more general L-CNN uses the same trivial principal bundle P RD x SU(N) with the same gange links, only the data more general. This allows us to also consider matter fields as input: for example, fields in the fundamental representation C (e.g. quark fields) transform according to
(((()))((x) = (xx)((x),
(133)
where 2(x) a NNN matrix. Similarly, for fields the adjoint representation (in the sense of adjoint fermions or bosons), we have
(x()()))((x))) = ((x)
(134)
22

---

Published in Transactions on Machine Learning Research (05/2024)
with color indices a, b. E
1} and the adjoint matrix
[(x) ab 2Tr [1((())
(135)
In these cases, we have an associated bundle E,, P x, consisting of equivalence classes, Eq. 1109 and the locally transforming input data f(x) are gaugo-dependent representatives of data points sy sx(x) x 92 fx)). Allowing multiple channels,
F = ( (")..
(136)
can be accomplished by using direct sum representations up = -z==1 P. The convolution in Eq.
123 generalizes to a gauge equivariant layer & : P(mp) + l(np) given by
1 [x, (2)
Σ dy" ( x)
(137)
where wab; : RDD + IR are kernel components.
Bilinear layers and trace layers do not generalize directly to data that, unlike Wxx), are not matrix valued and must. instead be tailored to different representations. fxx is an SU(N) vector field, for instance, trace layers could be defined as
(x)
= (T()))()((()))))))))))
(138)
which acts linearly on ("(x) @ (x) and therefore defines linear layer [(m(pp & p²)) + ((mld). This trace layer is gauge invariant and can be used to define activation functions gauge equivariant non-linear layers [(mp)- + [(mp) given by
f")() = (fx())))
(139)
for each channel a 1,..., m. Here, w wx(x) = Re (II()())))
4.5 The difficulty in achieving full group equivariance
Now that we have connected the original L-CNN to the mathematical theory of equivariant neural networks, it is desirable to do the same for the fully G-equivariant T-CNN discussed in Section 3 Fully adhering to the existing theory would require us to identify suitable principal bundle that describes both the global symmetry under as well as the local gauge symmetry SU(N). The complication with this that gauge symmetry described by the principal bundle p RD (UN,,, whereas global symmetry and group equivariance is more closely related to the principal bundle
q G G.R
<(g) - - =x
(140)
with structure group K K= G/T. It not obvious whether a single principal bundle can correctly describe both symmetries simultaneously. One interesting candidate the principal bundle
q G x SU(N) - G, ((g. 2) =
(141)
It of the same general form q: 9 + g/K as Eq. 140) but with the total space G G= G x SU and the structure group KC = (UN.. The corresponding data points are sections of associated bundles E, & x, V and are given by
s(g) = 1g. Ω. f()) = lg. 1. (())f(o)
(142)
for linear representations p of (NN. Gauge equivariance works similarly as for P RD x (U).. However, this bundle describes group equivariance with respect to g = G x SUNN), not with respect to G alone. This means that SU(N) would have to represent a global symmetry in addition to the local gauge symmetry. In particular, continuous convolutions would be integrals over G x SUNN.. This is not compatible with the group equivariant L-CNN, so Eq. (141) cannot be the correct principal bundle. The question, then, is whether there is a more appropriate principal bundle or if the bundle theory for equivariant neural networks can be appropriately broadened. We leave this question for future work.
23

---

Published in Transactions on Machine Learning Research (05/2024)
5 Conclusions and outlook
from a geometrical perspective and exlended the original formulation by accounting for additional global symmetries on the lattice. The L-CNN framework introduced a set of gauge equivariant layers which can be used to build machine learn- ing models for performing computations on gauge link configurations (Uz,a). These layers, consisting of convolutions, bilinear operations, activation functions, and trace layers, are equivariant under lattice gauge transformations. This is achieved by accounting for parallel transport in the definition of gauge equivari- ant convolutions. In addition, L-CNNs, which are fundamentally based on discrete convolutions on the lattice z", are also equivariant under global translations of the input data.
In this work, we have reviewed the L-CNN framework (Favoni et al. 2022
The global symmetry group G on hypercubic lattice consists not only of translations, but also includes discrete rotations and reflections. One of the drawbacks of L-CNNs is that they only respect the translational part of the full global symmetry. To remedy this, we have revisited G-CNNs (Cohen & Welling, 2016). which use convolutional layers compatible with general global symmetry transformations. We have first reviewed how to use these G-convolutions on vector- and tensor-valued data and then combined the G-CNN approach with the L-CNN framework to obtain network architectures that are not only gauge equivariant, but also equivariant under the full global symmetry group on the lattice. There is computational drawback associated with this extension: the domain on which feature maps are defined must be enlarged to the full symmetry group G, requiring more computational memory. Similarly, it increases the computational complexity of convolutions, as they have to be carried out over feature maps on the group.
Finally, we have linked I.-CNNs to the fiber bundle thcoretic description of equivariant neural networks (Aron- sson, 2022) for the z" equiviivar case. This has allowed us to determine the associated bundles used for input data in the original T--NN formulation and also consider more general input data. for different repre- sentations of the gauge group. More generally, we have shown how L-CNNs can be understood
special discretized case of gauge equivariant neural networks on fiber bundles (Gerken et al.] 2023). Despite this, a bundle description of G-equivariant L-CNNs is still lacking. Identiying the correct principal bundle to simultancously describe both global G and local SU(N) symmetry would be a welcome extension of this work.
To conclude, we list some potential applications that could benefit from full G-equivariance. Naturally, the most interesting applications are found in four-dimensional SU(N) lattice gauge theory, which is globally symmetric under discrete translations, rotations, and mirror transformations. Lattice gauge equivariant networks have been used as a preconditioner for the Dirac operator in lattice QCD (Lehner & Wettig, 2023a b Knüttel et al.] 2024). In these works, gange equivariant convolution layers include multiple paths and are translationally equivariant but not explicitly symmetric under rotations and reflections. Recently, =P-qquivariant I.-CNNs have been successfully used to learn a classically perfect fixed-point action (Holland let al., 2024), which could help alleviate the problem of large autocorrelation times in Monte Carlo simula- tions, also known as topological freezing and critical slowing down. Hooland et al. 2024) demonstrate that even well-trained -e-equivariant L-CNNs violate rotational and mirror symmetry. Machine-learned actions with full G-symmetry would be more consistent with the physical theory and likely more accurate as well. Beyond regression problems, G-equivariant L-CNNs could find applications in generative models. Equiv- ariant normalizing (Kanwar et al. 2020 Boyda et al. 2021 Albergo et al.) 2021b) Abbott et al. 2022) or continuous (de Haan et al.; 2021 Gerdes et al.) 2023 Bacchio et al. 2023) flow models have been applied to a wide range of lattice field thcories in recent years. Flow models can generate statistically independent field configurations and, ideally, exhibit vanishing autocorrelation. In most of these models, lattice symmetries are only approximately realized in some cases, even translational symmetry is broken down to subgroup (Boyda et al. 2021h. The use of group and gauge equivariant layers would avoid these problems entirely, as the relevant physical symmetries, both global and local, are manifest in the network architecture. Clearly, more work towards an efficient. implementation of G-equivariant L-CNNs is needed.
Acknowledgments
The authors thank Andreas Ipp for many helpful discussions regarding group equivariant neural networks and comments on the manuscript. DM and DS have been supported by the Austrian Science Fund FWF
24

---

Stefan Blücher, Lukas Kades, Jan M. Pawlowski, Nils Strodthoff, and Julian M. Urban Towards novel insights in lattice field theory with explainable machine learning. Phys. Rev. D, 101(9):094507, 2020. doi: http///yyeee.......... Published in Transactions on Machine Learning Research (05/2024)
No. P32446, No. P34764 and No. P34455. DM acknowledges additional support from FWF No. P28352. JA has been supported by the Wallenberg AT, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. DM and JA thank the organizers of the Banach Center Oberwolfach Graduate Seminar "Mathematics of Deep Learning", which took place in late 2019 and sparked this collaboration.
References
Ryan Abbott et al. Gauge-equivariant flow models for sampling in lattice field theories with pseudofermions. Phys. Rev. D, 106(():074506, 2022. doi: 2.1103/PhysRevD.006..0
Michael S. Albergo, Denis Boyda, Daniel C. Hackett, Gurtej Kanwar, Kyle Cranmer, Sébastien Racanière, Danilo Jimenez Rezende, and Phiala E. Shanahan. Introduction to Normalizing Flows for Lattice Field Theory, art. arXiv:2101.08176 2021a. doi: 208555//0Xiv2110111111
Michael S. Albergo, Gurtej Kanwar, Sébastien Racanière, Danilo J. Rezende, Julian M. Urban, Denis Boyda, Kyle Cranmer, Daniel C. Hackett, and Phiala E. Shanahan. Flow-based sampling for fermionic lattice field thcories. Phys. Rev. D, 104(11):114507, 2021b. doi: Me11003/hyyReeD..... 104.114507.
Jimmy Aronsson. Homogencous vector bundles and G-equivariant convolutional neural networks. Sampling Theory, Signal Processing, and Data Analysis, 20(2):1-35 2022. doi: (L.007//43670--22-00009--99
Simone Bacchio, Pan Kessel, Stefan Schacfer, and Lorenz Vaitl. Learning Trivializing Gradient Flows for Latlice Gauge Theories. Phys. Rev. D, 107:L051504, Mar 2023. doi: 10.1003/PlysReevD..00... 10110.00
Dimitrios Bachtis, Gert Aarts, and Biagio Lucini. Mapping distinct phase transitions to a neural network. Phys. Rev. E, 102(3))053306, 2020. doi: 11103/PyyReev.100222 102.053306.
Dimitrios Bachtis, Gert Aarts, and Biagio Lucini. Quantum field-theoretic machine learning. Phys. Rev. D. 103(7):074510, 2021. doi: httg://PyyyReerD......555.151
Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S. Albergo, Kyle Crammer, Daniel C. Hackett, and Phiala F. Shanahan. Sampling using SU(N) gauge equivariant flows. Phys. Rev. D, 10377)4444 2021. doi: ht0B://PyyReeDD..3.....0000
Denis Boyda et al. Applications of Machine Learning to Lattice Quantum Field Theory. In 2022 Snoumass Summer Study, 2022.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. art. arXiv:2104.13478, 2021.
Srinath Bulusu, Matleo Favoni, Andreas lpp, David I. Müller, and Daniel Schuh. Generalization capabilities translationally equivariant neural networks. Phys. Rev. D, 104(():074504, 2021. doi: (0.1s03/PhysRevDD. 104.074504.
Gabricle Cosa, Leon Lang, and Maurice Weiler. A program to build E(N)) steerable CNNs. In International Conference on Learning Representations, 2022.
Miranda CN Cheng, Vassilis Anagiannis, Maurice Weiler, Pim de Haan, Taco Cohen, and Max Welling, Covariance in physics and convolutional neural networks. art. arXiv:1906.0245: 2019.
Taco S. Cohen. Equivoriant convolutional networks. Phd thesis, Universiteit. van Amsterdam, 2021.
Taco S. Cohen and Max Welling, Group equivariant convolutional networks. In Proceedings of The 33rd
International Conference on Machine Learning, volume 48, pp. 2990-2999. JMLR, Jun 2016.
25

---

Published in Transactions on Machine Learning Research (05/2024)
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference Learning Representations, 2017.
Taco S. Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant CNNs on homogeneous spaces. In Advances in Neural Information Processing Systems 32: Annual Conference on Newral In- formation Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 9142-9153, 2019a.
Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral CNN. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 1321-1330. JMLR, 2019b.
Pim de Haan, Corrado Rainone, Miranda C. N. Cheng, and Roberto Bondesan. Scaling up machine learning for quantum field theory with equivariant continuous flows. CoRR, abs/2110.02673, 2021.
Li Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the
Web). JEEE Signal Processing Magazine, 2966):141-12,, 2012. doi: N0.1109/MSP:2012.222
B. Durhuus. On the structure of gauge invariant classical observables in lattice gauge theories. Letters in Mathematical Physics, 4(6):515-522, Nov 1980. ISSN 1573-0530. doi: N0.1007/BF0094388
Matteo Favoni, Andreas lpp, David 1. Müller, and Daniel Schuh. Lattice Gauge Equivariant Convolutional Neural Networks. Phys. Rev. Lett., 128(3):032003, 2022. doi: 10.1103/ 10.1103/PhysRev..ce PhysRevLett.188... www.::eeettt...2220000. 032003.
Christof Gattringer and Christian B. Lang. Quantum chromodynamies on the lattice, volume 788. Springer, Berlin, 2010. ISBN 978-3-642-01849-77 -18---642-01850-3 doi: 10.1007/998-3-662--01--
Mathis Gerdes, Pim de Haan, Corrado Rainone, Roberto Bondesan, and Miranda N. Cheng. Learning lattice quantum field theories with equivariant continuous flows. SciPost Phys., 15(6):238, 2023. doi: 10.21468/ Stte//SiiPoostss.....
Jan Gerken, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Equivariance versus augmentation for spherical images. In International Conference on Machine Learning, pp. 7404 7421. PMLR, 2022.
Jan E. Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Frodrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric Deep Learning and Equivariant Neural Networks. Artíf. Intell. Rev., 56 (00:14605-14666) jun 2023. ISSN 2821- doi: 10.1007/s10462-0023 10502-7.
Simon Graham, David Epstein, and Nasir Rajpoot. Dense steerable filter CNNs for exploiting rotational symmetry in histology images. IEEE Transactions on Medical Imaging, 39(12): 4136, 2020. doi:
10.1109/TM1222003013346
Kieran Holland, Andreas lpp, David 1. Müller, and Urs Wenger. Machine learning a fixed point action for S3)3) gauge theory with a gauge equivariant convolutional neural network. 2024. doi: 44455000a/iii 2401.06481.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal ap- proximators. Neural Networks, 2(5):339 3666,,66666000 1989. ISSN 0893-6080. doi: 10.1016 ////://999000009900000
Gurtej Kanwar, Michael S. Albergo, Denis Boyda, Kyle Cranmer, Daniel C. Hackett, Sébastien Racanière, Danilo Jimenez Rezende, and Phiala E. Shanahan. Equivariant flow-based sampling for lattice gauge theory. Phys. Rev. Lett., 125(12):121601 2020. doi: (0.103//PhyseevLcth 12..tt..25...1.01.
Daniel Knüttel, Christoph Lehner, and Tilo Wettig. Gauge-equivariant multigrid neural networks. PoS, LATTICE2023:037, 2024. doi: 0.22323/1.453.000
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1 (44):441-55) 1989.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Christoph Lehner and Tilo Wettig. Gange-cquivariant neural networks preconditioners in lattice QCD.
Phys. Rev. D, 108:034503, 2023a. doi: PhyyReeD.1000000444
Christoph Lehner and Tilo Wettig. Gauge-equivariant pooling layers for preconditioners in lattice QCD. 2023b. doi: 1118550/arXii.234...000
Kim A. Nicoli, Christopher J. Anders, Tena Funcke, Tobias Hartung, Karl Jansen, Pan Kessel, Shinichi Nakajima, and Paolo Stornati. Estimation of Thermodynamic Observables Lattice Field Theories with Deep Generative Models. Phys. Rev. Lett.,
2021. doi: 10.
22.000001.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Compuler Vision and Pallern Recognition, pp. 3498 3505, 2012. doi: 10.1109/CVPR.2012 6248092.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeminn Lin, Natalia Gimelshein, Luea Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Michael E. Peskin and Daniel V. Schroeder. An Introduction to quantum field theory. Addison- Wesley, Reading, USA, 1995. ISBN 978-0-201-50397-5.
Siamak Ravanbakhsh. Universal equivariant multilayer perceptrons. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7996-8006. PMLR, 13-18 Jul 2020.
Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda. Universality of group convolutional neural networks based on ridgelet analysis on groups. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Aduances Neural Information Processing Systems, 2022.
Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. -49-858, 2018.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Equivariant and Coordinate Independent Convolutional Networks. 2023.
Kenneth G. Wilson. Confinement of quarks. Phys. Rev. D, 10.2445.
Oct 1974. doi: 10.1103PhhySeedh
Ding-Xuun Xuan Zhou. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787-7944 2020. ISSN 1063-5203. doi: www.///...cha..11..o..
Kai Zhou, Gergely Endrödi, Long-Gan Pang, and Horst Stöcker. Regressive and generative neural networks for scalar field theory. Phys. Rev. D, 100(1):011501, 2019. doi:
27


Reviews:
[{"review_id": "z68Qn1Zy7B", "reviewer": "Reviewer_dkk1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper offers a novel extension of the L-CNN framework that incorporates equivariance to action by the global symmetry group (as opposed to just translations) and presents this extension in a very general setting (including vector- and tensor-valued data). The buildup in Section 3 (Extending L-CNNs to Group Equivariance) is quite rigorous and careful; a total of nine subsections guide the reader carefully through all relevant constructions, building on group equivariant convolutions for scalars on the lattice, vector and tensor fields, later on L-Convs, as the focus shifts to gauge equivariance, and so on, up through the most general forms given at the end of Section 3.4. Generalizations of all layer types in the original L-CNNs paper (Favoni et al., 2022) are given, including bilinear layers, trace layers, activation functions, and pooling layers. \n\n2. In terms of writing, the paper is quite strong. The writing is clear and most necessarily preliminaries are carefully introduced and built upon throughout the duration of the paper.\n\n## Weaknesses\n\n1. Although the paper spends a lot of time constructing a highly generalized version of the original L-CNN framework, no applications or motivating example is mentioned. Is there actually a concrete benefit to be had in existing lattice QFT problems from these generalizations? The answer to this question is very unclear and there is no investigation as to whether each (or any, for that matter) of these steps in generalization is actually necessitated by downstream problems in lattice QFT. The original paper on L-CNNs (Favoni et al., 2022) makes some attempt at demonstrating usefulness of their approach by comparing the predictive power of L-CNNs with regular CNNs in the context of synthetic problems where the task is to predict local, gauge invariant observables. This paper does no such thing nor does it explore concrete contexts in which there would intuitively be considerable benefit over the L-CNN approach. Such an exploration feels like a missing component of the paper. The paper also talking about the increased memory and computational requirements of their method over L-CNNs: there is a multiplicative factor of $|K|$ on both (where $K$ is the stabilizer subgroup in the semi-direct product comprising $G$). However, it is unclear to what extent this $|K|$ is critical and/or prohibitive in real-world applications of this method, since, once again, it is unclear in what contexts this approach yields a significant benefit over e.g. L-CNNs. An investigation of these contexts would also help yield a better understanding of how prohibitive this factor of $|K|$ is and how generally the paper's approach can be applied in practice.\n\n## Verdict\n\nThe authors of this work extend the L-CNN framework (Favoni et al., 2022) to enable equivariance with respect to the global symmetry group and do so over very general data (tensor-valued). The paper is well-written and the construction is carefully presented. However, no time is spent considering or discussing contexts in which these generalizations are explicitly useful/necessary, over what was already given in the L-CNN paper. Certainly having a general construction is nice, but are there problems in lattice QFT that benefit considerably from this machinery? This question was left unanswered, yet I believe it is of high importance for such a paper. Although this is the case, I believe the claims made in the submission are well supported, and the findings of the paper would be interesting to some portion of TMLR's audience, specifically those working at the intersection of equivariant machine learning and high energy physics. As such, both criteria for acceptance to TMLR are satisfied; ergo, I recommend acceptance of this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This works seeks to extend the Lattice Gauge Equivariant Convolutional Neural Network (L-CNN) framework to respect not only equivariance with respect to global translations of input data, but more general action by the global symmetry group. To accomplish this, the paper revisits G-CNNs (Cohen & Welling, 2016) (which introduced convolutional layers compatible with general global symmetry transformations) and adapts this work to the L-CNN framework. The result is a network architecture that is not only gauge equivariant, but also equivariant under action by the full global symmetry group on the lattice. Generality is also lifted considerably: the paper reviews G-convolutions on vector- and tensor-valued data and extends the aforementioned framework to work in such contexts. Some additional analysis is done to show how convolution in L-CNNs arises as a special case of gauge equivariant neural networks on $\\text{SU}(N)$ principal bundles (for the $\\mathbb{Z}^D$-equivariant case).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The writing in this paper is, for the most part, very good. That being said, I have some high-level comments. First of all, Section 4, about how L-CNNs arise as a special case of gauge equivariant neural networks on $\\text{SU}(N)$ principal bundles, seems mostly like appendix material to me. Although it is a neat connection, it is not seem central to the overall methodology and is auxiliary. Section 4.5 in particular about the difficulty of achieving full group equivariance is at best speculative. It would be fine to mention that this is future work and point to a reference/stub in the appendix. I do not think the best place for this material is the main paper. Second of all, there need to be some references added for the basic high energy physics material. There was no reference added for e.g. Section 2.1 on Yang-Mills theory. A basic reference like [a] would suffice, but please choose whichever you prefer. Additionally, a lot of notation/conventions are clearly geared towards someone with a high energy physics background, less so for a math background. For example, elements of $\\mathfrak{su}(N)$ were introduced as traceless Hermitian matrices, as opposed to traceless skew-Hermitian matrices (e.g. as they are in [b]). Physicists have the convention of multiplying through by $i$ at the Lie algebra level, but this is not so for everyone. So long as this is made explicit, either by reference or in text, I have no issue. Language like \"color indices\" (page 22) is also very physics-specific; I understand it is standard in quantum chromodynamics, but in the context of this more mathematical paper, perhaps a term like \"gauge indices\" would be more appropriate.\n\nI also give a non-exhaustive list of minor corrections below:\n\n- In the abstract: \"non-Abelian\" -> \"non-abelian\" (the standard convention is to spell abelian with a lower case \"a\")\n- The sentence following equation (30): \"$[f \\star \\psi] : G \\rightarrow \\mathbb{R}$\" -> \"$[\\psi \\star f] : G \\rightarrow \\mathbb{R}$\" (following convention with equation (28))\n- In the prelude to equation (84), you likely want $b >0$ not $b \\in \\mathbb{R}$, since for $b < 0$ the \"nonlinear\" activation function would be linear\n\n### References\n\n[a] Peskin, Michael E. and Daniel V. Schroeder. “An Introduction To Quantum Field Theory.” (1995).\n\n[b] Takhtajan, Leon A.. “Quantum Mechanics for Mathematicians.” (2008).", "broader_impact_concerns": "The paper is of a rather theoretical nature; as such, I have no ethical concerns and do not believe the paper requires a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "knPGHveObI", "reviewer": "Reviewer_6LYn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Well-written\n+ Novel gauge+globally equivariant layers\n+ Nice intro to gauge theory\n\nWeaknesses:\n- No experiments\n\nFor details, see the main review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A number of recent works have looked at exploiting gauge symmetries in neural network design via gauge equivariance. This is a very promising direction of research because gauge groups can get extremely large (introducing degrees of freedom at each point in space), and thus can lead to great improvements in data efficiency. The L-CNN is a translation and SU(N)-gauge equivariant network that has been proven useful for predicting gauge invariant quantities in lattice gauge theory. The main contribution of this paper is to extend the L-CNN to also incorporate global symmetries of the lattice, using ideas from group equivariant CNNs.\n\nAdditionally, the paper contains a very well-written introduction to lattice gauge theory using physics notation (whereas most work on GDL has relied on the more abstract bundle formalism). It also explains all the relevant GDL background very well. Finally, in section 4, the contributions are placed in the bundle-theoretic framework used by various other authors in GDL. The authors are clearly experts in this area.\n\nThe action editor noted that the paper is very heavy on physics, and asked whether it is appropriate for TMLR. Personally I think it is situated at one of the frontiers of geometric deep learning, and will find interested readers in this community. There are several papers on gauge equivariant neural networks that have several hundred citations already, so although this is somewhat niche it is not micro-niche.\n\nA major gap in the paper is that it only presents the mathematics, but does not provide an implementation or experiments involving the newly proposed neural network layers. It does however provide some cursory discussion of implementation and performance issues. This is unfortunate as the interested reader will certainly be curious about the improvements in data efficiency that can be obtained relative to plain L-CNNs.\n\nAnother thing that would be nice to see is a discussion on the connection of this work to the various \"universality\" results that have been published for linear gauge equivariant layers, e.g. in [1, 2]. Granted, these are for the smooth case, but it seems similar results could be proved for lattices quite easily. One would like to know: are the proposed layers indeed the most general gauge equivariant ones?\n\nAt the end of the paper, the authors ask the question \"Identifying the correct principal bundle to simultaneously describe both global G and local SU(N) symmetry would be a welcome extension of this work\". Although I don't have the answer, I would recommend the book [3] where related issues are discussed.\n\nAll things considered, I would recommend the paper for acceptance in TMLR. Having experimental results would greatly strengthen the paper, but if the bar is \"some readers will find this interesting\", the paper meets the bar as is. It is very well written, technical sound, and has potential applications in lattice gauge theoretic calculations.\n\n[1] Weiler, Maurice and Forré, Patrick and Verlinde, Erik and Welling, Max, Equivariant and Coordinate Independent Convolutional Networks, 2023\n[2] Cohen, Taco, Equivariant Convolutional Networks, PhD Thesis, 2021\n[3] Rudolph, Gerd and Schmidt, Matthias, Differential Geometry and Mathematical Physics: Part II. Fibre Bundles, Topology and Gauge Fields, 2017", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be nice to see a demonstration of improved data efficiency. However, given how many times this has been demonstrated on various problems, it is sufficiently plausible that we would see meaningful gains that I don't consider this strictly necessary for acceptance.\n- A discussion on universality", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DbrJ5w8DSc", "reviewer": "Reviewer_UCyp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n* The constructions are theoretically principled and explained rather thoroughly.\n\nWeaknesses\n--------------\n* The material is very hard to approach. I have a background in equivariant neural networks (even touching upon specifically lattice gauge equivariance) and I found this rather hard to parse. Several factors contribute to this, perhaps the largest being the exposition in the form of pure physics (this is specifically Lattice QCD in the broader context of QFT). An exposition closer to math (e.g. dropping the reliance on \"Yangs-MIlls theory\" \"freedom of theory\" in Section 2.1) would be immensely helpful in reaching a broader audience.\n* Related to the above, but I tend to agree with the original AC comment that this paper would be better suited for a HEP journal (specifically one open to ML constructs). While I could imagine that there is **an** audience (e.g. at least the authors themselves), I struggle to see this fitting into the equivariant ML community or the physics for ML community (which would be perhaps the most relevant areas). This is supported by the fact that the paper builds off work coming from HEP journals (e.g. Favoni et al).\n* While theoretically complete, the paper has not demonstrated a practical need for these constructs (in the form of experimental/empirical evidence).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines lattice gauge equivariant CNNs and proposes two analyses. First, the authors extend L-CNNs to handle group equivariant constraints. Second, the authors show that L-CNNs (specifically the conv layers) are related to gauge equivariant neural networks (a more general construction) if we recast them using SU(N) bundles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Nothing outside of the weaknesses section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As I explained above, the general consensus is that this is well written paper that does a good job of providing clear and convincing evidence in its claims.  The biggest and only real concern is whether its advanced and physics-heavy subject is appropriate for the TMLR audience.

In the end, the majority of the reviewers and I myself agree that there is an audience for this paper at TMLR, so I am recommending acceptance.
Decision: Accept (poster)