Automatic Symmetry Discovery with Lie Algebra Convolutional Network
Year: 2022

Abstract:
Existing equivariant neural networks require prior knowledge of the symmetry group and discretization for continuous groups. We propose to work with Lie algebras (infinitesimal generators) instead of Lie groups. Our model, the Lie algebra convolutional network (L-conv) can automatically discover symmetries and does not require discretization of the group. We show that L-conv can serve as a building block to construct any group equivariant feedforward architecture. Both CNNs and Graph Convolutional Networks can be expressed as L-conv with appropriate groups. We discover direct connections between L-conv and physics: (1) group invariant loss generalizes field theory (2) Euler-Lagrange equation measures the robustness, and (3) equivariance leads to conservation laws and Noether current. These connections open up new avenues for designing more general equivariant networks and applying them to important problems in physical sciences.1

heading: 1 Introduction
text: Incorporating symmetries into a deep learning architecture can reduce sample complexity, improve generalization, while significantly decreasing the number of model parameters (Cohen et al., 2019b; Cohen & Welling, 2016b; Ravanbakhsh et al., 2017; Ravanbakhsh, 2020; Wang et al., 2020). For instance, Convolutional Neural Networks (CNN) (LeCun et al., 1989, 1998) implement translation symmetry through weight sharing. General principles for constructing symmetry-aware group equivariant neural networks were introduced in Cohen & Welling (2016b), Kondor & Trivedi (2018), and Cohen et al. (2019b).
However, most work on equivariant networks requires knowing the symmetry group a priori. A different equivariant model needs to be re-designed for each symmetry group. In practice, we may not have a good inductive bias and such knowledge of the symmetries may not be available. Constructing and selecting the equivariant network with the appropriate symmetry group becomes quite tedious. Furthermore, many existing works are limited to finite groups such as permutations Hartford et al. (2018); Ravanbakhsh et al. (2017); Zaheer et al. (2017), 90 degree rotations Cohen et al. (2018) or dihedral groups DN and E(2) Weiler & Cesa (2019).
1Code: github.com/nimadehmamy/L-conv-code
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
For a continuous group, existing approaches either discretize the group Weiler et al. (2018a,b); Cohen & Welling (2016a), or use a truncated sum over irreducible representations (irreps) Weiler & Cesa (2019); Weiler et al. (2018a) via spherical harmonics in Worrall et al. (2017) or more general Clebsch-Gordon coefficients Kondor et al. (2018); Bogatskiy et al. (2020). These approaches are prone to approximation error. Recently, Finzi et al. (2020) propose to approximates the integral over the Lie group by Monte Carlo sampling. This approach requires implementing the matrix exponential and obtaining a local neighborhood for each point. Both parametrizing Lie groups for sampling and finding irreps are computationally expensive. Finzi et al. (2021) provide a general algorithm for constructing equivariant multi-layer perceptrons (MLP), but require explicit knowledge of the group to encode its irreps, and solving a set of constraints.
We provide a novel framework for designing equivariant neural networks. We leverage the fact that Lie groups can be constructed from a set of infinitesimal generators, called Lie algebras. A Lie algebra has a finite basis, assuming the group is finite-dimensional. Working with the Lie algebra basis allows us to encode an infinite group without discretizing or summing over irreps. Additionally, all Lie algebras have the same general structure and hence can be implemented the same way. We propose Lie Algebra Convolutional Network (L-conv), a novel architecture that can automatically discover symmetries from data. Our main contributions can be summarized as follows:
• We propose the Lie algebra convolutional network (L-conv), a building block for constructing group equivariant neural networks.
• We prove that multi-layer L-conv can approximate group convolutional layers, including CNNs, and find graph convolutional networks to be a special case of L-conv.
• We can learn the Lie algebra basis in L-conv, enabling automatic symmetry discovery. • L-conv also reveals interesting connections between physics and learning: equivariant
loss generalizes important Lagrangians in field theory; robustness and equivariance can be expressed as Euler-Lagrange equations and Noether currents.
Learning symmetries from data has been studied in limited settings for commutative Lie groups as in Cohen & Welling (2014), 2D rotations and translations in Rao & Ruderman (1999), Sohl-Dickstein et al. (2010) or permutations (Anselmi et al., 2019). In the non-commutative case, GeoManCEr (Pfau et al., 2020) uses data points related by small transformations to learn non-abelian Lie groups, but it does not introduce an equivariant layer architecture. (Zhou et al., 2020) propose a general method for symmetry discovery. Yet, their weight-sharing scheme and the symmetry generators are very different from ours. Our approach use much fewer parameters and has a direct interpretation using Lie algebras (SI B.3). Benton et al. (2020) propose Augerino to learn a distribution over data augmentations. It also involves Lie algebras, but is restricted to a subgroup of 2D affine transformations and requires matrix logarithm and sampling (SI B.3). In contrast, our approach is simpler and more general. Our approach uses composition of small transformations to achieve large transformations. In this sense bears some resemblance to symnets (Gens & Domingos, 2014), but the rest of the construction is different.

heading: 2 Background
text: We review the core concepts L-conv builds upon: equivariance, group convolution and Lie algebras.
Notations. Unless explicitly stated, a in Aa is an index, not an exponent. We use the Einstein summation AaBab = ∑ aA aBab = [AB]b, where a repeated upper and lower index are summed.
Equivariance. Let S be a topological space on which a Lie group G (continuous group) acts from the left, meaning for all x ∈ S and g ∈ G, gx ∈ S. We refer to S as the base space. Let F , the “feature space”, be the vector space F = Rm. Each data point is a feature map f : S → F . The action of G on the input of f induces an action on feature maps. For “scalar” features, for u ∈ G, the transformed features u · f are given by
u · f(x) = f(u−1x). (1)
Denote the space of all functions from S to F by FS , so that f ∈ FS . Let F be a mapping to a new feature space F ′ = Rm′ , meaning F : FS → F ′S . We say F is equivariant under G if G acts on F ′
and for u ∈ G, we have u · (F (f)) = F (u · f). (2)
Group Convolution. Kondor & Trivedi (2018) showed that F is a linear equivariant map if and only if it performs a group convolution (G-conv). To define G-conv, we first lift x to elements in G (Kondor & Trivedi, 2018). Specifically, we pick an origin x0 ∈ S and replace each point x = gx0 by g. We will often drop x0 for brevity and write f(g) ≡ f(gx0). Let κ : G→ Rm
′ ⊗ Rm be a linear transformation from F to F ′. G-conv is defined as
[κ ? f ](g) = ∫ G κ(g−1v)f(v)dv = ∫ G κ(v)f(gv)dv, (3)
We denote the Haar measure on G as dv ≡ dµ(v) for brevity. Equivariance of G-conv. G-conv in equation 3 is equivariant (Kondor & Trivedi, 2018). By definition, for w ∈ G we have
[κ ? w · f ](g) = ∫ G κ(v)w · f(gv)dv = ∫ G κ(v)f(w−1gv)dv
= [κ ? f ](w−1g) = w · [κ ? f ](g) (4) Existing works on equivariance networks implement ∫ G
by discretizing the group or summing over irreps. We take a different approach and use the infinitesimal generators of the group. While a Lie group G is infinite, usually it can be generated using a small number of infinitesimal generator, comprising its “Lie algebra”. We use the Lie algebra to introduce a building block to approximate G-conv. Figure 1 visualizes a Lie group, Lie algebra and the concept we discuss below.
Lie algebra. Let G be a Lie group, which includes common continuous groups. Group elements u ∈ G infinitesimally close to the identity element I can be written as u ≈ I + iLi (note Einstein summation), where Li ∈ g with the Lie algebra g = TIG is the tangent space of G at the identity element. The Lie algebra has the property that it is closed under a Lie bracket [·, ·] : g× g→ g
[Li, Lj ] = cij kLk, (5)
which is skew-symmetric and satisfies the Jacobi identity. Here the coefficients cijk ∈ R or C are called the structure constants of the Lie algebra. For matrix representations of g, [Li, Lj ] = LiLj − LjLi is the commutator. The Li are called the infinitesimal generators of the Lie group.
Exponential map. If the manifold of G is connected 2, an exponential map exp : g → G can be defined such that g = exp[tiLi] ∈ G. For matrix groups, if G is connected and compact, the matrix
2When G has multiple connected components, these results hold for the component containing I , and generalize easily for mutli-component groups such as Zk ⊗G (Finzi et al., 2021).
exponential is such a map and it is surjective. For most other groups (except GLd(C) and nilpotent groups) it is not surjective. Nevertheless, for any connected group every g ∈ G can be written as a product g = ∏ a exp[t i aLi] (Hall, 2015). Making t i a infinitesimal steps dt
i(s) tangent to a path γ from I to g on G yields the surjective path-ordered exponential in physics, denoted as g = P exp[ ∫ γ dtiLi] (SI A, and see Time-ordering in Weinberg (1995, p143)).
Pushforward. Li ∈ TIG can be pushed forward to L̂i(g) = gLig−1 ∈ TgG to form a basis for TgG, satisfying the same Lie algebra [L̂i(g), L̂j(g)] = cijkL̂k(g). The manifold of G together with the set of all TgG attached to each g forms the tangent bundle TG, a type of fiber bundle (Lee et al., 2009). L̂i is a vector field on TG. The lift maps L̂i to an equivalent vector field on TS, which we will also denote by L̂i. Figure 1 illustrates the flow of these vector fields on TG and TS.

heading: 3 Lie Algebra Convolutional Network
text: We can use the Lie algebra basis Li ∈ g to construct the Lie group G with the exponential map. Similarly, we show that Lie algebras can also serve as building blocks to construct G-conv layers. We propose the Lie algebra convolutional network (L-conv). The key idea is to approximate the kernel κ(u) using localized kernels which can be constructed using the Lie algebra (Fig. 2). This is possible because the exponential map is a generalization of a Taylor expansion. We show that a G-conv whose kernel is concentrated near the identity can be expanded in the Lie algebra.
Let δη(u) ∈ R denote a normalized localized kernel, meaning ∫ G δη(g)dg = 1, and with support on a small neighborhood of size η centered around the identity I (i.e., δη(I + iLi)→ 0 if ‖ ‖2 > η2). We pick δη(v ) ∼ θ(η2 −‖ ‖2), for v = I + iLi ∈ TIG and δη(v) = 0 for all other v /∈ TIG (θ(·) being the Heaviside step function). Let κ0 : G→ Rm ′ ⊗ Rm be given by
[κ0] b a (u) = [ W 0 ]c a δη ( u ( I − [ i ]b c Li )) (6)
where W 0 ∈ Rm′ ⊗ Rh and i ∈ Rh ⊗ Rm are constants, and we choose ∣∣[ i]ab ∣∣ < η. Note that (I + iLi)(I − jLj) = I + [ − ]iLi +O(η2). Therefore,∫ id δη ( (I + iLi)(I − jLj) ) = i (7)
The localized kernels κ0 can be used to approximate G-conv.
Linear expansion of G-conv with localized kernel. We can expand a G-conv whose kernel is κ0(u) = W 0δη(u) in the Lie algebra of G to linear order. With v = I + iLi, we have (see SI A)
Q[f ](g) = [κ0 ? f ](g) = ∫ G dvκ0(v)f(gv) = ∫ ‖ ‖<η dv κ0(v )f(gv )
= W 0 ∫ d δη(v ) [ f(g) + igLi · d
dg f(g) +O( 2) ] = W 0 [ I + igLi · d
dg
] f(g) +O(η2) (8)
with W 0 ∈ Rm′ ⊗ Rh and i ∈ Rh ⊗ Rm, as before. Here d is the integration measure on the Lie algebra g = TIG induced by the Haar measure dv on G.
Interpreting the derivatives. In a matrix representation of G, we have gLi · dfdg = [gLi] β α df
dgβα = Tr [ [gLi] T df dg ] . This can be written in terms of partial derivatives ∂αf(x) = ∂f/∂xα as follows.
Using xρ = gρσx σ 0 , we have df(gx0) dgαβ = xβ0∂αf(x), and so
L̂if(x) ≡ gLi · df
dg = [gLi]
α βx β 0∂αf(x) = [gLix0] · ∇f(x) (9)
Hence, for each Li, the pushforward gLig−1 generates a flow on S through the vector field L̂i ≡ gLi · d/dg = [gLig−1x]α∂α (Fig. 1).
Lie algebra convolutional (L-conv) layer. Equation 8 states that for a kernel localized near the identity, the effect of the kernel can be summarized in W 0 and iL̂i. Note that we do not need to perform the integral over G explicitly anymore. Instead of working with a kernel κ0, we only need to specify W 0 and i. Hence, in general, we define the Lie algebra convolution (L-conv) as
Q[f ](x) = W 0 [ I + iL̂i ] f(x)
= W 0 [ I + i[gLix0] α∂α ] f(x) (10)
Being an expansion of G-conv, L-conv inherits the equivariance of G-conv, as we show next. Proposition 1 (Equivariance of L-conv). With assumptions above, L-conv is equivariant under G.
Proof: First, note that the components of L̂i transform as [L̂i(vx)]α = [vgLix0]α = vαβ L̂i(x) β , while the partial transforms as ∂/∂[vx]α = [v−1]γα∂γ . As a result in L̂i = [gLix0] α∂α all factors of v cancel, meaning for v ∈ G, L̂i(vx) = L̂i(x). This is because of the fact that L̂i ∈ TS is a vector field (i.e. 1-tensor) and, thus, invariant under change of basis. Plugging into equation 10, for w ∈ G
w ·Q[f ](x) = Q[f ](w−1x) = W 0 [ I + iL̂i(w −1x) ] f(w−1x)
= W 0 [ I + iL̂i(g) ] f(w−1x) = W 0 [ I + iL̂i(g) ] w · f(x) = Q[w · f ](x)(11)
which proves L-conv is equivariant.
Examples. Using equation 9 we can calculate L-conv for specific groups (details in SI A.2). For translationsG = Tn = (Rn,+), we find the generators become simple partial derivatives L̂i = ∂i (SI A.2.2), yielding f(x)+ α∂αf(x). For 2D rotations (SI A.2.1) the generator L̂ ≡ (x∂y − y∂x) = ∂θ, which is the angular momentum operator about the z-axis in quantum mechanics and field theories. For rotations with scaling, G = SO(2)×R+, we have two Li, one L̂θ = ∂θ from so(2) and a scaling with Lr = I , yielding L̂r = x∂x + y∂y = r∂r. Next, we discuss the form of L-conv on discrete data.

heading: 3.1 Approximating G-conv using L-conv
text: L-conv can be used as a basic building block to construct G-conv with more general kernels. Figure 2 sketches the argument described here (see also SI A.1). Theorem 1 (G-conv from L-convs). G-conv equation 3 can be approximated using L-conv layers.
Proof: The procedure involves two steps, as illustrated in Fig. 2: 1) approximate the kernel using localized kernels as the δη in L-conv; 2) move the kernels towards identity using multiple L-conv layers. The following lemma outline the details. Lemma 1 (Approximating the kernel). Let the kernel κ : G → F ′ ⊗ F with ∫ G ‖κ(g)‖2dg < ∞ be continuously differentiable with ‖dκ(g)/dg‖2 < ξ2, and with compact support over G0 ⊂ G. Let κk(g) = ckδη(u−1k g) be a set of N kernels with support on an η neighborhood of uk ∈ G. Then there exist ck ∈ F ′ ⊗ F and uk ∈ G such that κ̃ = ∑N k=1 κk approximates κ, meaning∫
G ‖κ(g)− κ̃(g)‖2dg < ζ2 for arbitrary small ζ ∈ R+.
Proof: See SI A.1 for details. The intuition is similar to the universal approximation theorem for neural networks (Hornik et al., 1989; Cybenko, 1989), only generalized to a group manifold instead of R. Let B0 be the set of v = I + iLi ∈ g, with ‖ ‖2 < η2. Choose a set of uk ∈ G such that the neighborhoods Bk = ukB0 ⊂ G cover the support G0 of κ. The bound ‖dκ(g)/dg‖2 < ξ2 means that on small enough neighborhoodsBk ⊂ G, for any two u, v ∈ Bk we have ‖κ(u)−κ(v)‖2 ≤ η2ξ2, where |G0| is the volume of the support of κ. Hence, for g ∈ Bk, κ(g) can be approximated with κk(g) = κ(uk)δη(u−1k g), with normalized localized kernels δη(g), and any element uk ∈ Bk. We show that the approximation error of using κ̃ = ∑ k κk to approximate κ is bounded by∫
G dg‖κ(g)− κ̃(g)‖2 < |G0|η2ξ2. Any desired error bound ζ can then be attained by choosing small
enough η for neighborhood sizes. Thus, we can approximate a large class of kernels as κ(g) ≈ ∑ k κk(g) where the local kernels κk(g) = ckδη(u −1 k g) have support only on an η neighborhood of uk ∈ G. Here ck ∈ Rm
′ ⊗ Rm are constants and δη(u) is as in equation 8. Using this, G-conv equation 3 becomes
[κ ? f ](g) = ∑ k ck ∫ dvδη(u −1 k v)f(gv) = ∑ k ck[δη ? f ](guk). (12)
The kernels κk are localized around uk, whereas in L-conv the kernel is around identity. We can compose L-conv layers to move κk from uk to identity. Lemma 2 (Moving kernels to identity). κk can be moved near identity using a multilayer L-conv.
Proof: In equation 12, write uk = v u′k, with v = I+ iLi ∈ g. Using the definition equation 10 an L-conv layer Q = I− iL̂i performs a first order Taylor expansion (SI A.1) and so Q [δη](u′−1k v) = δη(u −1 k v) + O(
2). Thus, applying one L-conv layer moves the localized kernel along v on G. Writing uk as the product of a set of small group elements uk = ∏p a=1 va, with va = I + i aLi ∈ g. Defining L-conv layers Qa = I − iaL̂i, we can write κk(g) ≈ ckQp ◦ · · · ◦Q1 ◦ δη(g) (13)
meaning κk localized around uk can be written as a p layer L-conv acting on a kernel δη(g), localized around the identity of the group. With ‖ a‖ < η, the error in uk is O(ηp+1). Thus, we conclude that any G-conv equation 3 can be approximated by multilayer L-conv. Furthermore, for compactG, using the theorem in Kondor & Trivedi (2018), we can show that any equivariant feedforward neural network can be approximated using multilayer L-conv with nonlinearities.
Equivariance of nonlinearity. Pointwise nonlinearities give equivariant maps between scalar feature maps. To see this, let σ : R → R. We extend σ : F → F by applying σ component-wise. Let f : S → F be a scalar feature map (i.e., g · f(x) = f(g−1x)). Then
g · (σ ◦ (f))(x) = σ ◦ (f)(g−1x) = σ ◦ (g · f)(x). Since the composition of equivariant maps is equivariant, given equivariant linear mappingQ : FS → F ′S (i.e. g ·Q[f ] = Q[g · f ]), the layer f 7→ σ ◦Q[f ] is equivariant. Hence we have the corollary: Corollary 1. Assume G is compact and acts on S transitively. Then any equivariant feedforward neural network (FNN) can be approximated using multilayer L-conv with point-wise nonlinearities.
Proof: A FNN is defined as σp ◦ Fp[· · · [σ1 ◦ F1[f ]](x) where Fk are linear and σk are point-wise nonlinearities. By Theorem 1 of Kondor & Trivedi (2018), any linear layer in the equivariant FNN is a G-conv, which by Theorem 1 can be approximated by multilayer L-conv. Therefore, multilayer L-conv with nonlinearity can approximate any equivariant FNN.
Finally, to our knowledge it is not known whether every equivariant function can be approximated by equivariant FNN for a Lie group G. Hence, the corollary above is not a universal approximation theorem for equivariant scalar functions in terms of L-conv. However, it does show that multilayer L-conv is equally expressive as other equivariant networks. Next, we discuss implementation details.

heading: 4 Discretized space and implementation: the tensor notation
text: In many datasets, such as images, f(x) is not given as continuous function, but rather as a discrete array, with S = {x0, . . .xd−1} containing d points. Each xµ represents a coordinate in higher dimensional space, e.g. on a 10× 10 image, x0 is (x, y) = (0, 0) point and x99 is (x, y) = (9, 9).
Feature maps and group action In the tensor notation, we encode xµ ∈ S as the canonical basis (one-hot) vectors in xµ ∈ Rd with [xµ]ν = δµν (Kronecker delta), e.g. x0 = (1, 0, . . . , 0). The features become f ∈ F = Rd ⊗ Rm, meaning d×m tensors, with f(xµ) = xTµf = fµ. Although S is discrete, the group acting on F can be continuous (e.g. image rotations). Any G ⊆ GLd(R) of the general linear group (invertible d × d matrices) acts on xµ ∈ Rd and f ∈ F . We define f(g · xµ) = xTµ gTf ,∀g ∈ G, so that for w ∈ G we have
w · f(xµ) = f(w−1 · xµ) = xTµw−1Tf = [w−1xµ]Tf (14)
Dropping the position xµ, the transformed features are matrix product w ·f = w−1Tf . We can write G-conv in this notation (SI B). Similarly, we can rewrite L-conv equation 8 in the tensor notation. Defining v = I + iLi
Q[f ](g) = W 0f ( g ( I + iLi )) = xT0 ( I + iLi )T gTfW 0T
= ( x + i[gLix0] )T fW 0T . (15)
Here, L̂i = gLix0 is exactly the matrix analogue of pushforward vector field L̂i in equation 9. The equivariance of L-conv in tensor notation is again evident from the gTf , resulting in
Q[w · f ](g) = xT0 vT gTw−1TfW 0T = Q[f ](w−1g) = w ·Q[f ](g) (16)
Tensor L-conv layer implementation The discrete space L-conv equation 15 can be rewritten using the global Lie algebra basis L̂i
Q[f ] = ( f + L̂if i ) W 0T , Q[f ]aµ = f b µ[W 0T ]ab + [L̂i] ν µf c ν [ W i ]a c
(17)
Where W i = W 0 i, W 0 ∈ Rmin ⊗Rmout and i ∈ Rmin ⊗Rmin are trainable weights. The L̂i can be either inserted as inductive bias or they can be learned to discover symmetries.
To implement L-conv, note that the formula of equation 17 is quite similar to a Graph Convolutional Network (GCN) (Kipf & Welling, 2016). For each i, the shared convolutional weights are iW 0T and the aggregation function of the GCN, a function of the graph adjacency matrix, is L̂i in L-conv. Thus, Lconv can be implemented as GCN modules for each L̂i, plus a residual connection for the fW 0T term.
Figure 3 shows the schematic of the L-conv layer. In a naive implementation, L̂i can be general d× d matrices. However, being vector fields generated by the Lie algebra, L̂i has a more constrained structure which allows them to be encoded and learned using much fewer parameters than a d× d matrix. Specifically, encoding the topology of S as a graph (see SI B.1), the incidence matrix replaces partial derivatives (Schaub et al., 2020) in equation 9 and the Li become weighting of the edges. This weighting is similar to
Gauge Equivariant Mesh (GEM) CNN (Cohen et al., 2019a). Indeed, in L-conv the lift xµ = gµx0 fixes the gauge by mapping neighbors of x0 to neighbors of xµ. Changing how the discrete S samples an underlying continuous space will change gµ and hence the gauge.
Choosing the number of Li. Beside the width of W 0 and i, the number nL of Li is a hyperparameter in L-conv. For instance, if S is a discretization of n dimensional space the symmetry group is likely G ⊂ GLn(R) n Tn, with nL ∼ O(n2). Note that nL is independent of the size d of the discretized space (e.g. number of pixels) and generally n2 d. Choosing nL larger than the true number of Li only results in an over-complete basis and shouldn’t be a problem. We conducted small controlled experiments to verify how multilayer L-conv approximates G-conv (SI C).
Learning symmetries using L-conv. Rao & Ruderman (1999) introduced a basic version of Lconv and showed that it can learn 1D translation and 2D rotation. We conducted experiments to learn large rotation angle between two images (SI C), shown in Fig. 4. Left shows the architecture for learning the rotation angles between a pair of 7× 7 random images f and R(θ)f with θ ∈ [0, π/3). Second left is the learned L ∈ SO(2) using 3 recursive layer L-conv. Middle is the L learned using L-conv with fixed small rotation angle θ = π/10 (SI C.2) and right is the exact solution R = (Y XT )(XTX)−1. While the middle L is less noisy, it does not capture weights beyond first neighbors of each pixel. (also see SI C for a discussion on symmetry discovery literature.)
L-conv can potentially replace other equivariant layers in a neural network. We conducted limited experiments for this on small image datasets (SI D). L-conv allows one to look for potential symmetries in data which may have been scrambled or harbors hidden symmetries.

heading: 5 Relation to other architectures
text: CNN. This is a special case of expressing G-conv as L-conv when the group is continuous 1D translations. The arguments here generalize trivially to higher dimensions. Rao & Ruderman (1999, sec. 4) used the Shannon-Whittaker Interpolation (Whitaker, 1915) to define continuous translation on periodic 1D arrays as f ′ρ = g(z) ν ρfν . Here g(z) ν ρ = 1 d ∑d/2 p=−d/2 cos ( 2πp d (z + ρ− ν) ) approximates the shift operator for continuous z. These g(z) form a 1D translation groupG as g(w)g(z) = g(w+z) with g(0)νρ = δ ν ρ . For any z = µ ∈ Z, gµ = g(z = µ) are circulant matrices that shift by µ as [gµ] ρ ν = δ ρ ν−µ. Thus, a 1D CNN with kernel size k can be written suing gµ as
F (f)aν = σ ( k∑ µ=0 f cν−µ[W µ]ac + b a ) = σ ( k∑ µ=0 [gµf ] c ν [W µ]ac + b a ) (18)
where W, b are the filter weights and biases. gµ can be approximated using the Lie algebra and written as multi-layer L-conv as in sec. 3.1. Using g(0)νρ ≈ δ(ρ − ν), the single Lie algebra basis [L̂]0 = ∂zg(z)|z→0, acts as L̂f(z) ≈ −∂zf(z) (because ∫ ∂zδ(z − ν)f(z) = −∂νf(ν)). Its
components are L̂νρ = L(ρ − ν) = ∑ p 2πp d2 sin ( 2πp d (ρ− ν) ) , which are also circulant due to the
(ρ− ν) dependence. Hence, [L̂f ]ρ = ∑ ν L(ρ− ν)fν = [L?f ]ν is a convolution. Rao & Ruderman (1999) already showed that this L̂ can reproduce finite discrete shifts gµ used in CNN. They used a primitive version of L-conv with gµ = (I + L̂)N . Thus, L-conv can approximate 1D CNN. This result generalizes easily to higher dimensions.
Graph Convolutional Network (GCN). Let A be the adjacency matrix of a graph. In equation 17 if L̂i = h(A), such as L̂i = D−1/2AD−1/2, we obtain a GCN (Kipf & Welling, 2016) (Dµν = δµν ∑ ρAµρ being the degree matrix). So in the special case where all neighbors of each node < µ > have the same edge weight, meaning [L̂i]νµ = [L̂i] ρ µ, ∀ν, ρ ∈< µ >, equation 8 is uniformly aggregating over neighbors and L-conv reduces to a GCN. Note that this similarity is not just superficial. In GCN h(A) = L̂ is in fact a Lie algebra basis. When L̂ = h(A), the vector field is the
flow of isotropic diffusion df/dt = h(A)f from each node to its neighbors. This vector field defines one parameter Lie group with elements g(t) = exp[h(A)t]. Hence, L-conv for flow groups with a single generator are GCN. These flow groups include Hamiltonian flows and other linear dynamical systems. The main difference between L-conv and GCN is that L-conv can assign a different weight to each neighbor of the same node, similar to GEM-CNN (Cohen et al., 2019a) with a fixed gauge set by gµ. Next, we discuss the mathematical properties of the loss functions for L-conv.

heading: 6 Group invariant loss
text: Loss functions of equivariant networks are rarely discussed. Yet, recent work by Kunin et al. (2020) showed the existence of symmetry directions in the loss landscape. To understand how the symmetry generators in L-conv manifest themselves in the loss landscape, we work out the explicit example of a mean square error (MSE) loss. Because G is the symmetry group, f and g · f should result in the same optimal parameters. Hence, the minima of the loss function need to be group invariant. One way to satisfy this is for the loss itself to be group invariant, which can be constructed by integrating over G (global pooling (Bronstein et al., 2021)). A function I = ∫ G dgF (g) is G-invariant (SI A.3).
We can also change the integration to ∫ S d
nx by change of variable dg/dx (see SI A.3 for discussion on stabilizers). MSE loss and Field Theory. The MSE is given by I = ∑ n ∫ G dg‖Q[fn](g)‖2, where fn are data samples and Q[f ] is L-conv or another G-equivariant function. In supervised learning the input is a pair fn, yn. G can also act on the labels yn. We assme that yn are either also scalar features yn : S → Rmy with a group action g · yn(x) = yn(g−1x) (e.g. fn and yn are both images), or that yn are categorical. In the latter case g · yn = yn because the only representations of a continuous G on a discrete set are constant. We can concatenate the inputs to φn ≡ [fn|yn] with a well-defined G action g · φn = [g · fn|g · yn]. The collection of combined inputs Φ = (φ1, . . . , φN )T is an (m + my) × N matrix. Using equations 8 and 9, the MSE loss with parameters W = {W 0, } becomes (SI A.3.1)
I[Φ;W ] = ∫ G dgL[Φ;W ] = ∫ G dg ∥∥∥W 0 [I + i[L̂i]α∂α]Φ(g)∥∥∥2
= ∫ S dnx∣∣∣∂x∂g ∣∣∣ [ ΦTm2Φ + ∂αΦ Thαβ∂βΦ + [L̂i] α∂α ( ΦTviΦ )] (19)
Equation 19 generalizes the free field theories in physics (Polyakov, 2018). Here ∣∣∣∂x∂g ∣∣∣ is the determinant of the Jacobian, W i = W 0 i and
m2 = W 0TW 0, hαβ(x) = iTm2 j [L̂i] α[L̂j ] β , vi = m2 i. (20)
Note that h has feature space indices via [ iTm2 j ]ab, with index symmetry h αβ ab = h βα ba . When F = R (i.e. f is a 1D scalar), hαβ becomes a a Riemannian metric for S. In general h combines a 2-tensor hab = h αβ ab ∂α∂β ∈ TS ⊗ TS with an inner product hThαβf on the feature space F .
In field theory, the motivation is to preserve spatial symmetries for the metric h. In equation 19, h transforms equivariantly as a 2-tensor v · hαβ = [v−1]αρ [v−1]βγhργ(x) for v ∈ G (SI A.3). The last term in equation 19 vanishes for many groups (SI A.3) and it is also absent in physics.
Robustness and Euler-Lagrange Equation. Equivariant neural networks are more robust. To check this, we can quantify how the network would perform for an input φ′ = φ+ δφ which adds a small random perturbation δφ to a data point φ. Robustness to such perturbation would mean that, for optimal parameters W ∗ , the loss function would not change, i.e. I[φ′;W ∗] = I[φ;W ∗], requiring I to be minimized around real data points φ.
This can be cast as a variational equation δI[φ;W ∗] = 0, which yield the familiar Euler-Lagrange (EL) equation (SI A.4). Therefore, for an equivariant network to be robust, i.e. δI[φ;W ∗]/δφ = 0, we would require the data points φ to satisfy the EL equations for optimal parameters W ∗:
Robustness to random noise⇐⇒ EL: ∂L ∂φb − ∂α ∂L ∂(∂αφb) = 0 (21)
where the partial derivative terms appear because of the L-conv layer.
Equivariance and Conservation laws. Conserved currents, via Noether’s theorem provide a way to find hidden symmetries (see also Kunin et al. (2020)). The idea is that the equivariance condition equation 2 can be written for the integrand of the loss, L[φ,W ]. If we write the equivariance equation for infinitesimal v , we obtain a vector field which is divergence free. Since G is the symmetry of the system, transforming an input φ→ w · φ by w ∈ G the integrand should change equivariantly, meaning L[w · φ] = w · L[φ]. When robustness error is minimized as in equation 21, an infinitesimal w ≈ I + ηiLi, with δφ = iL̂iφ, results in a conserved current (SI A.4)
Noether current: Jα = ∂L ∂(∂αφb) δφb − ∂L ∂xα δxα, δI[φ;W ∗] = 0 ⇒ ∂αJα = 0 (22)
The above equation shows that for equivariant networks with a given symmetry, the deviation in data along the symmetry direction (L̂i) yields a divergence free current Jα, known as Noether current. It also provides an alternative means to discover symmetry generators Li by minimizing ‖∂αJα‖. Note that this Noether current is the “stress-energy” tensor, associated with space (or space-time) variations δx (Landau, 2013) (SI A.5). We can potentially design more general equivariant networks leading to other Noether currents.

heading: 7 Conclusion and Discussions
text: We propose the Lie algebra convolutional neural network (L-conv), an infinitesimal version of Gconv. L-conv layers do not require encoding irreps or discretizing the group, and can be combined to approximate any feedforward equivariant networks on compact groups. Additionally, L-conv’s universal and simple structure allows us to discover symmetries from data. It is easy to implement, with a formula similar to GCN. We validated that L-conv can learn the correct Lie algebra basis in a synthetic experiment.
We discover several intriguing connections between L-conv and physics. Our derivation shows that equivariant neural networks based on L-conv lead to Noether’s theorem and conservation laws. Conversely, we can also optimize Noether current to discover symmetries. Furthermore, the current equivariance formulation only pertains to “spatial symmetries” (i.e. G acts on S). In physics, more general “internal symmetries” are quite common (e.g. particle physics). We can potentially design more general equivariant networks with L-conv encoding such symmetries.
Our method also shed lights on scientific machine learning, especially for physical sciences. Physicists generally use simple polynomial forms for the Lagrangian, or the loss function. These “perturbative” Lagrangian lead to divergences in quantum field theory. However, it is believed the true Lagrangian is more complicated. Hence, more expressive L-conv based models can potentially provide more advanced ansatze for solving scientific problems.
Acknowledgments and Disclosure of Funding
R. Walters is supported by a Postdoctoral Fellowship from the Roux Institute and NSF grants #2107256 and #2134178. This work was supported in part by the U. S. Army Research Office under Grant W911NF-20-1-0334, DOE ASCR 2493 and NSF Grant #2134274. N. Dehmamy and D. Wang were supported by the Air Force Office of Scientific Research under award number FA9550-19-1-0354.

Review summaries:
They propose a model for approximating invariant functions for Lie groups, the Lie algebra convolutional network (L-conv). While convolutional networks for ordinary groups require some kind of discretization, their model does not require it. Also, Lie algebra generators in our L-conv can be learned to automatically discover symmetries.

The paper proposes a method to build equivariant neural networks by defining the layers in terms of the generators of the Lie group to which the NN is to be equivariant to. At the core lies the idea that any function (or conv kernel) with local support can be expanded in the Lie algebra basis via a Taylor approximation-like approach. Then, by observing that any function can be approximated with a basis of (shifted) local functions, it can be concluded that the presented approach in theory can approximate any group convolution. In addition, the authors the investigate the option to let go of the user-imposed constraint of being equivariant to a specific group, and instead let the neural network learn the generators (of the unknown group) in addition to the learnable weights that parametrize the layers. This is an open problem and the authors make promising steps in this direction.  Finally, the authors draw connections to physics, but I must admit, this part was too much for me and I did not get it, but possibly could have by investing more time in the paper.