Automatic Symmetry Discovery with Lie Algebra Convolutional Network
Authors: Nima Dehmamy
Robin Walters
Yanchen Liu
Dashun Wang
Rose Yu
Venue: NeurIPS 2021 Poster (2021)
Keywords: symmetry discovery
equivariance
group convolutional neural network
inductive bias
group equivariant architecture
Lie group
Lie algebra
scientific machine learning
physics

Abstract: 
Existing equivariant neural networks require prior knowledge of the symmetry group and discretization for continuous groups. We propose to work with Lie algebras (infinitesimal generators) instead of Lie groups. Our model, the Lie algebra convolutional network (L-conv) can automatically discover symmetries and does not require discretization of the group. We show that L-conv can serve as a building block to construct any group equivariant feedforward architecture. Both CNNs and Graph Convolutional Networks can be expressed as L-conv with appropriate groups. We discover direct connections between L-conv and physics: (1) group invariant loss generalizes field theory (2) Euler-Lagrange equation measures the robustness, and (3) equivariance leads to conservation laws and Noether current. These connections open up new avenues for designing more general equivariant networks and applying them to important problems in physical sciences.

Automatic Symmetry Discovery with Lie Algebra Convolutional Network
Nima Dehmamy Northwestern University nimadtCbu. edu
Robin Walters Northeastern University
rwalterss@norhhastenn edu
Yanchen Liu Northeastern University liu. www.yyaaccooohhasttrn.cccm edu
Dashun Wang
Northwestern University
dashun wang0kellogg
wwwthwestern.eee saangheellgg...oobesette...
edu
Rose Yu
University of California San Diego
roseyu@ucsd. edu
Abstract
Existing equivariant neural networks require prior knowledge of the symmetry group and discretization for continuous groups. We propose to work with Lie algebras (infinitesimal generators) instead of Lie groups. Our model, the Lie algebra convolutional network (L-conv) can automatically discover symmetries and does not require discretization the group. We show that L-conv can serve as a building block to construct any group equivariant feedforward architecture. Both CNNs and Graph Convolutional Networks can be expressed as L-conv with appropriate groups. We discover direct connections between L-conv and physics: (1) group invariant loss generalizes field theory (2) Euler-Lagrange equation measures the robustness, and (3) equivariance leads to conservation laws and Noether current. These connections open up new avenues for designing more general equivariant networks and applying them to important problems in physical sciences.
Introduction
Incorporating symmetries into deep learning architecture can reduce sample complexity, improve generalization, while significantly decreasing the number of model parameters (Cohen et al., 2019h Cohen & Welling, 2016b; Ravanbakhsh et al., 2017: Ravanbakhsh, 2020; Wang et al., 2020). For instance, Convolutional Neural Networks (CNN) (LeCun et al., 1989, 1998) implement translation symmetry through weight sharing. General principles for constructing symmetry-aware group equivariant neural networks were introduced in Cohen Welling (2016b), Kondor Trivedi (2018), and Cohen et al. (2019b).
However, most work on equivariant networks requires knowing the symmetry group a priori. A different equivariant model needs to be re-designed for each symmetry group. In practice, we may not have good inductive bias and such knowledge of the symmetries may be available. Constructing and selecting the equivariant network with the appropriate symmetry group becomes quite tedious. Furthermore, many existing works are limited to finite groups such as permutations Hartford al. (2018), 8); Ravanbakhsh et al. (2017); Zaheer et al. (2017), 90 degree rotations Cohen et al. (2018), or dihedral groups D and F(2) Weiler Cesa (2019).
Code: giftullcomiinnddannngLLoooooe
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

For a continuous group, existing approaches either discretize the group Weiler et al. (2018a.b); Cohen & Welling (2016a), or use a truncated sum over irreducible representations (irreps) Weiler & Cesa (2019); Weiler et al. (201 8a) via spherical harmonics in Worrall et al. (2017) or more general Clebsch-Gordon coefficients Kondor et (2018); Bogatskiy et al. (2020). These approaches are prone to approximation error. Recently, Finzi et al. (2020) propose to approximates the integral over the Lie group by Monte Carlo sampling. This approach requires implementing the matrix exponential and obtaining local neighborhood for each point. Both parametrizing Lie groups for sampling and finding irreps are computationally expensive. Finzi et al. (2021) provide general algorithm for constructing equivariant multi-layer perceptrons (MLP), but require explicit knowledge of the group to encode its irreps, and solving a set of constraints.
We provide novel framework for designing equivariant neural networks. We leverage the fact that Lie groups can be constructed from a set of infinitesimal generators, called Lie algebras. A Lie algebra has a finite basis, assuming the group is diteesiimmsii..... Working with the Lie algebra basis allows us to encode an infinite group without discretizing or summing over irreps. Additionally, all Lie algebras have the same general structure and hence can be implemented the same way. We propose Lie Algebra Convolutional Network (L-conv), a novel architecture that can automatically discover symmetries from data. Our main contributions can be summarized as follows:
We propose the Lie algebra convolutional network (L-conv), a building block for construct- ing group equivariant neural networks.
We prove that multi-layer L-conv can approximate group convolutional layers, including CNNs, and find graph convolutional networks to be special case of L--cov.
We can learn the Lie algebra basis in L-conv, enabling automatic symmetry discovery.
L-conv also reveals interesting connections between physics and learning: equivariant loss generalizes important Lagrangians in field theory; robustness and equivariance can be expressed as Euler-Lagraage equations and Noether currents.
Learning symmetries from data has been studied in limited settings for commutative Lie groups in Cohen & Welling (2014), 2D rotations and translations in Rao & Ruderman (1999), Sohl-Dickstein al. (2010) or permutations (Anselmi et al., 2019). In the non-commutative case, GeoManCEr (Pfau et al., 2020) uses data points related by small transformations to learn non-abelian Lie groups, but it does not introduce an equivariant layer architecture. (Zhou al., 2020) propose a general method for symmetry discovery. Yet, their weight-sharing scheme and the symmetry generators are very different from ours. Our approach use much fewer parameters and has direct interpretation using Lie algebras (SI B.3). Benton et al. (2020) propose Augerino to learn a distribution over data augmentations. It also involves Lie algebras, but restricted to subgroup of 2D affine transformations and requires matrix logarithm and sampling (SI B.3). In contrast, our approach is simpler and more general. Our approach uses composition of small transformations to achieve large transformations. In this sense bears some resemblance to symnets (Gens & Domingos, 2014), but the rest of the construction is different.
2 Backkround
We review the core concepts L-conv builds upon: equivariance, group convolution and Lie algebras. Notations. Unless explicitly stated, in A is index, not exponent. We use the Einstein summation A Bab = Σ. A Bas = [AB], where a repeated upper and lower index are summed.
Equivariance. Let S be topological space on which a Lie group G (continuous group) acts from the left, meaning for all x E S and g E G, gæ E S. We refer S as the base space. Let F, the "feature space", be the vector space F IR" Each data point is a feature map ʃ S + F. The action of G on the input of f induces an action on feature maps. For "scalar" features, for u G, the transformed features f are given by
f(x)= f(u x).
Denote the space of all functions from S to F by FS, so that f E Let F be mapping to new feature space F' Rmd meaning F F + F. We say F is equivariant under G if G acts F

---

G
S
TG
L
gLg
L,x0
T,G
T
[(x)
Lie group and Lie algebra: Illustration of the group manifold of Lie group G (left). The Lie algebra g TıG is the tangent space the identity I. L, are a basis for T,G. If G is connected, Vg E G there exist paths like from I to and g can be written g = P ee I_, dt' I.). Base space Right is a schematic of the base space S as manifold. The lift æ = ggg takes a E to E G. and maps the tangent spaces S + G. Each Lie algebra basis L, E g TiG generates a vector field L, on the tangent bundle TG via the pushforward L,(g) = g 1 Via the lift, L; also generates a vector field L; = (x)x)) 0.. = [gL,zul"0.
Figure 1:
path-ordered integral
and for E G, we have
M. (F(f)) = F(u. f).
(2)
Group Convolution. Kondor & Trivedi (2018) showed that F is linear equivariant map if and only if performs group convolution (G-conv). To define G-conv, we first lift æ to elements in G pick an origin Zy E S and replace each point gzu by
(Kondor & Trivedi, 2018). Specifically, g. We will often drop za for brevity and write fg) = fgzo)) Let G + 1 x R be linear transformation from F to F'. G-conv is defined as
Is (gr)d,
[K (K()
klg (1)())))(
(3)
We denote the Haar measure G as du du(e) for brevity.
Equivariance G-conv. G-conv in equation 3 is equivariant (Kondor & Trivedi, 2018). By definition, for w G we have
[K * w- A1 = Ja k(v)w (oood [(())[(w-'gu)))
= [K [] (g) = w. [
(4)
Existing works on equivariance networks implement Sc. by discretizing the group or summing over irreps. We take a different approach and use the infinitesimal generators of the group. While Lie group G is infinite, usually it can be generated using a small number of infinitesimal generator, comprising its "Lie algebra". We use the Lie algebra to introduce building block to approximate G-conv. Figure visualizes Lie group. Lie algebra and the concept we discuss below.
Lie algebra. Let G be a Lie group, which includes common continuous groups. Group elements infinitesimally close to the identity element / be written u x 7 + Li (note Finstein summation), where I, E g with the Lie algebra = G is the tangent space of G at the identity element. The Lie algebra has the property that closed under Lie bracket [-- : g ggg- + g
L,, Ly} Cjj k LL.
(5)
which symm-symmettic and satisfies the Jacobi identity. Here the coefficients C, called the structure constants of the Lie algebra. For matrix representations of g. [L, L,] L,L,- - L,L, is the commutator. The L, are called the infinitesimal generators of the Lie group.
k E IR or C
Exponential map. If the manifold of G is connected 2 an exponential map exp g + G can be defined such that g expl(* (t L,] E G. For matrix groups, if G is connected and compact, the matrix
'When G has multiple connected components, these results hold for the component containing /, and generalize easily for mutli-component groups such Ex @ G (Finzi et al., 2021).

---

exponential is such map and it is surjective. For most other groups (except GLa(C) and nilpotent groups) it is not surjective. Nevertheless, for any connected group every g E G can be written as product = IT, exp(!" T.s] (Hall, 2015). Making i infinitesimal steps di' (s) tangent to path from I to g on G yields the surjective path-ordered exponential in physics, denoted as P Pexpls, dt' dt) (SI A, and see Time-ordering in Weinberg (1995, p143)).
Pushforward. L, E TyG can be pushed forward to Lg(g) = -gLLg E T,G to form a basis for T. G, satisfying the same Lie algebra (L) L,
Cij" & (g). The manifold of G together with the set of all G attached to each g forms the tangent bundle TG, a type of fiber bundle (Lee et al., 2009). L, is vector field on TG. The lift maps L; to an equivalent vector field on TS, which we will also denote by 14. Figure illustrates the flow of these vector fields on TG and TS.
3 Lie Algebra Convolutional Network
We can use the Lie algebra basis L, E g to construct the Lie group G with the exponential map. Similarly, we show that Lie algebras can also serve as building blocks to construct G-conv layers. We propose the Lie algebra convolutional network (L-conv). The key idea is to approximate the kernel x(u) using localized kernels which can be constructed using the Lie algebra (Fig. 2). This is possible because the exponential map is a generalization a Taylor expansion. We show that a G-conv whose kernel is concentrated near the identity can be expanded in the Lie algebra.
Let (u) E IR denote a normalized localized kernel, meaning Jo 6ggg)dg 1, and with support on a small neighborhood of size 17 centered around the identity (i.c., 5, + e' L,) - if ||e|| > r72). We pick 8, (ve) ~ 0n² - [| for ve = + L2 E T and 8,(()) for all other & T2G (0(-) being the Heaviside step function). Let Ko : G - R2mm 60 IRm be given by
[sOl (u) [w"] 5n (u (m(1- (1-
(6)
where w E Rmm @ A and e' E ARAA @ Rm are constants, and we choose [[<]]] (I+ (') - - Ly) = I lc L 0(72). Therefore,
< y.. Note that
f eddes, ((1 (())) =
The localized kernels Ko can used to approximate G-conv.
Linear expansion of G-cony with localized kernel. We can expand a G-conv whose kernel is (w) & in the Lie algebra of G to linear order. With v. I e' Li, have (see SI A)
Q(f(() = INO fl(g) -fo dunolv)
= 1 1n
dueno(ve)
wo f (edū,((t) ((()) + +l'gl
ag ((g) [(0)+ +
d
dg
wo 1+E'ghi dg.
f())+ 001))
(8)
with 8 E mg @ Rhth and सी E RR @ RT as before. Here is the integration measure on the Lie algebra g TiG' induced by the Haar measure dv, on G.
Interpreting the derivatives. In a matrix representation of G, we have gLو of = [g df og dga
Tr [gL, af] This can be written in terms of partial derivatives 0af(x) = Of /Ox" as follows. dg Using x" = x. we have dfggm) dgz
x 2.f0a(()) and so df dg
f.f(())) = gL.
[gL, 1 (a)) = [gT. vf(x))
Hence, for each I,, the pushforward glag-1 generates flow on S through the vector field i. = gLi- d/dg- i -=x000 (Fig. 1).

---

G-conv +p L-conv
1) Approximate with L-comws 14
2) ti, Lssing multi-ayer loyer
x(g)
Q
Q 2,0000 9K
0
G
Figure 2:
Sketch of the procedure for approximating G-conv using L-conv. First, the kernel is written as the sum of a number localized kernels иą with support around uy (left). Each of the Ki is then moved toward identity by composing multiple L-conv layers Qe ... (right).
Lie algebra convolutional (L-conv) layer. Equation 8 states that for kernel localized near the identity, the effect of the kernel can be summarized in 4 and E L,. Note that we do not need to perform the integral over G explicitly anymore. Instead of working with a kernel Ko. we only need to specify 8 and zτ. Hence, in general, we define the Lie algebra convolution (L-conv) as
@ff((z) w [I t f(x)
w [ ट" ماميا 0x] ((x
(10)
Being an expansion of G-conv, L-conv inherits the equivariance of G-conv, as we show next.
Proposition 1 (Equivariance of L-conv). With assumptions above, L-conv is equivariant under G.
Proof: First, note that the components of f. transform as [v=()= = [ v L ( while the partial transforms 1/00e [e '12 As result in L, [gL,,0] 0g all factors of cancel, meaning for u E (vxr) (2)). This is because of the fact that L, E TS is a vector field (i.e. 1-tensor) and, thus, invariant under change of basis. Plugging into equation 10, for w E G
w w --I([(())= =
= QW ax) w ( /w) s(w f(w
= w [1 [++1(110)] s(w w [1- - ( w. f(x) = Q/w (]( (11)
which proves L-conv equivariant.
Examples. Using equation 9 we can calculate L-conv for specific groups (details in SI A.2). For translations G = Tú = (12", +), we find the generators become simple partial derivatives L, = 0 (SI A.2.2), yielding f(x) " For 2D rotations (SI 1.2.1) the generator i. = (x0, yQ,) Oo. which is the angular momentum operator about the 7-axis in quantum mechanies and field theories. For rotations with scaling, = (2) x t we have two Li, one Le = Ôg from so(2) and a scaling with L, = I, yielding L, = dd, + yd,- = rå,. Next, we discuss the form of L-conv on discrete data.
3.1 Approximating G-cony using L-conv
L-conv can be used as basic building block to construct G-conv with more general kernels. Figure 2 sketches the argument described here (see also SI A.1).
Theorem (G-conv from L-convs). G-corv equation can be approximated using L-cony layers.
Proof: The procedure involves two steps, as illustrated in Fig. 2: 1) approximate the kernel using
localized kernels as the 6, in L-conv; 2) move the kernels towards identity using multiple L-cony layers. The following lemma outline the details.
O
Lemma (Approximating the kernel), Let the kernel K. G & with Sa [ggg < oo
be continuously differentiable with [dd<gg//g]) < 22, and with compact support over Go C G.
Let fx(g) = c, g) be a set of N kernels with support on an n neighborhood of uy E G. Then there exist Cj F @ F and us G such that is Σk=1 Ky approximates meaning Sc |l<[g) P < c² for arbitrary small R..
5

---

Proof: See SI A.1 for details. The intuition is similar to the universal approximation theorem for neural networks (Horikk et al.. 1989; Cybenko. 1989), only generalized a group manifold instead of R. Let Bo be the set of Ur = T + e² T.
with ||e]1 < "p. Choose set of us E G such that the neighborhoods B3 = ug Bg C G cover the support Gg of k. The bound [dd(gg/dgg
< <² means that on small enough neighborhoods B& C G, for any two u, E Bg we have [[<(u) ((t))><())l) k(v)ll snps², where |Gol is the volume of the support of K. Hence, for g E BR. k(g) can be approximated with K&s (g) = x(ux) 66, (uz g). with normalized localized kernels 5n(9), and any element us E Bg. We show that the approximation error of using A = Σx Kis to approximate K is bounded by < | Any desired error bound Ç can then be attained by choosing small enough 7) for neighborhood sizes.
Sa dg]]<((g) "[')]l²
Thus, we can approximate large class of kernels as x(g) = E. (g)(g) where the local kernels (x(g) cg (ug 9) have support only constants and (u) is as in equation 8. Using this, G-conv equation becomes
neighborhood of ug G. Here e R' ② Rm are
[x f(()= = Σ ck (u
u)
cal6, fl(gux).
(12)
The kernels Ky are localized around ug, whereas in L-conv the kernel is around identity. We can compose L-conv layers to move K3 from ug to identity.
Lemma 2 (Moving kernels identity). Kik can be moved near identity using multilayer L-conv.
Proof: In equation 12, write u = v. with U, = I E g. Using the definition equation 10 an L-cony layer Qe = II-el e' L, performs first order Taylor expansion (SI A. and so [,, 'v) = (, u) + ((2)). Thus, applying one L-conv layer moves the localized kernel along 8% on G. Writing ug as the product of set of small group elements Uy = II== Va, with vaa = + c L E g.
Defining L-conv layers Qa = I- - ca Lii, we can write
KA (g) R CsQp o................. ... o Q1 c 0,((9)
(13)
meaning Kj localized around uy can be written as a p layer L-cony acting on kernel 6, (g). localized around the identity of the group. With ||ea|] 17, the error in uy is 0(1
Thus, we conclude that any G-conv equation 3 can be approximated by multilayer L-conv. Further- more, for compact G, using the theorem in Kondor & Trivedi (2018), we can show that any equivariant feedforward neural network can be approximated using multilayer L-conv with nonlinearities.
Equivariance of nonlinearity. Pointwise nonlinearities give equivariant maps between scalar feature maps. To see this, let o R + R. We extend ø F + FF by applying a component-win Let S:S > F be scalar feature map (i.e., f(x:) 5 x)). Then
(x)))
g (a (z)
o (f) = a (g ())).
Since the composition of equivariant maps is equivariant, given equivariant linear mapping Q JS IrS (i.e. g Q = Q(g f'). the layer f - Q[J) is equivariant. Hence we have the corollary: Corollary 1. Assume G is compact and acts on S transitively. Then any equivariant feedforward neural network (FNN) can be approximated using multilayer L-conv with point-wise wise nonlinearities.
Proof: A FNN is defined as op o Fyl... (or o Fi [f]]((()) where F& are linear and 0s are point-wise nonlinearities. By Theorem 1 of Kondor& & Trivedi (2018), any linear layer in the equivariant FNN is G-conv, which by Theorem L-conv with nonlinearity can approximate any equivariant FNN.
be approximated by multilayer L-conv. Therefore, multilayer
Finally, our knowledge it is not known whether every equivariant function be approximated by equivariant FNN for a Lie group G. Hence, the corollary above is not a universal approximation theorem for equivariant scalar functions in terms of L-conv. However, it does show that multilayer I.-conv is equally expressive as other equivariant networks. Next, we discuss implementation details.
4 Discretized space and implementation: the tensor notation
In many datasets, such as images, f(xx) is not given continuous function, but rather as a discrete array, with S = {mo..... Tt--1) containing points. Each æt,, represents a coordinate in higher dimensional space, e.g. on a 10 x 10 image, 2o is (z, v) (0, 0) point and Zgg is (z, y) (9,9).

---

Feature maps and group action In the tensor notation, we encode 2n E S as the canonical basis
(one-hot) vectors in z E Rd with ı = ju (Kronecker delta). c.g. To = (1, .......... The features become f E F = R @ I. meaning d x m tensors, with f( = x = fp. Although is discrete, the group acting on FF can be continuous (e.g. image rotations). Any G S GLa(R) of the general linear group (invertible fgg zp) = 2 f. Ag E G, so that for w E G we have
matrices) acts on D. E R and f E F. We define
w. fa f(w xx) T
w.,win =[w Inl"f
(14)
Dropping the position Ty.. the transformed features are matrix product w = ---wi wIff f. We can write
G-conv this notation (SI B). Similarly, we can rewrite L-conv equation in the tensor notation. Defining U.
1 + E L;
Q(f(()) w (I ao (1 (I+ g² fwar -(x + + (gTa))o)) fwor.
(15)
Here, L, = gL,zg is exactly the matrix analogue of pushforward vector field L, in equation 9. The equivariance of L-conv in tensor notation is again evident from the 9" f. resulting in
Q(w f(() = x v. g w fW = Q[f((w g) = Q()
(16)
Tensor L-conv layer implementation using the global Lie algebra basis L.;
The discrete space L-conv equation 15 can be rewritten
@f = (f Life) wor.
Qifi -f
1E [L,K [w']
(17)
Where W Wo W E mm & mout and 20 R min R are trainable weights. The L, can be either inserted as inductive bias or they can be learned to discover symmetries.
To implement L-conv, note that the formula of equa- tion 17 is quite similar to Graph Convolutional Network (GCN) (Kipf & Welling, 2016). For each i, the shared convolutional weights are E'WOTT and
L-conv layer
feetures in
L, shared among features
M71
the aggregation function of the GCN, function of the graph adjacency matrix, L. L-conv. Thus, conv can be implemented as GCN modules for each L,, plus residual connection for the fWOT term.
d
f
in
d
W
Figure 3 shows the schematic of the L-conv layer. In naive implementation, L, can be general d x d matrices. However, being vector fields generated by the Lie algebra, f. has more constrained structure which allows them to encoded and learned using much fewer parameters than d x d matrix. Specif- ically, encoding the topology of S as a graph (see SI B. 1). the incidence matrix replaces partial derivatives (Schaub al., 2020) in equation and the Lg become weighting of the edges. This weighting is similar Gauge Equivariant Mesh (GEM) CNN (Cohen et al., 2019a). Indeed. in L-conv the lift a = 30 fixes the gauge by mapping neighbors of zo neighbors of x, Changing how the discrete S samples an underlying continuous space will change dn and hence the gauge.
Weights shared among clims.
L-conv layer architecture. Lu only act on the d flattened spatial dimensions, and W only act on the min input features and returns Mout output features. For each i, L- conv is analogous to a GCN with d nodes and Min features.
Figure 3:
Choosing the number of L,, Beside the width of W and टÍ, the number n& of L, is hyperpa- rameter in L-conv. For instance, if S is a discretization of n dimensional space the symmetry group likely G c GL,, (R) x T,, with nt, ~ O(1²). Note that n,, independent of the size d of the discretized space (e.g. number of pixels) and generally n² & d. Choosing n. larger than the true number of L. only results in an over-complete basis and shouldn't be a problem. We conducted small controlled experiments to verify how multilayer L-conv approximates G-cony (SI C).
7

---

Angle
Learned & (recursiee)
Learned L. Ifxed N100
Exent numerica
0.75
0.50
reour X
22-
0.00
-0.35
22
20
13 36
10
22
35
40
Figure 4: Learning the infinitesimal generator of SO(2) Left shows the architecture for learning rotation angles between pairs images (SI C.3). Next to it is the learned using recursive L-conv in this experiment. Middle L is learned using fixed small rotation angle 0 π T/10, and right shows L found using the numeric solution from the data.
Learning symmetries using L-conv. Rao & Ruderman (1999) introduced basic version of L- conv and showed that it can learn ID translation and 2D rotation. We conducted experiments to learn large rotation angle between two images (SI C), shown Fig. 4. Left shows the architecture for learning the rotation angles between pair of 7 x 7 random images f and R(0) f with 0 E T0,r/33, Second left is the learned L SO(2) using 3 recursive layer L-conv. Middle is the learned using L-cony with fixed small rotation angle 0 = ग/ 10 (SI C.2) and right is the exact solution While the middle L less noisy, does not capture weights beyond first neighbors of each pixel. (also see SI C for discussion symmetry discovery literature.)
R (Y (X""""X"""XX""""""""""""""""">>>
L-conv potentially replace other equivariant layers in neural network. We conducted limited ex- periments for this on small image datasets (SI D). L-conv allows one to look for potential symmetries data which may have been scrambled or harbors hidden symmetries.
Relation to other architectures
CNN. This special case of expressing G-conv as L-conv when the group continuous ID translations. The arguments here generalize trivially to higher dimensions. Rao & Ruderman (1999, sec. used the Shannon-Whitakker Interpolation (Whitaker, 1915) define continuous translation on periodic ID arrays as f ff)) Here g(=)² à/ - d/2 cos ( (z P pー v)) approximates the shift operator for continuous These g(=) form 1D translation group G as g(w)g(=) g(
with ((0) = 8- For any z = f E Z, 9N = g(> = 14) are circulant matrices that shift by as [g 8p-n. Thus, ID CNN with kernel size can be written suing as
k W + +00 =a " EignTI + + (
F(J)
(18)
(/==
where W, are the filter weights and biases. Iu can be approximated using the Lie algebra and written as multi- L-conv as in sec. 3.1. Using 9(0)² = 8(p v), the single Lie algebra basis [ 0.g(z) +00 acts as i.((z) x -z) (because f fā,5(= - =)) = 0 (v)). Its components are 1. = L(p v) Σ, 2np sin 2np (p v)). which are also circulant due to the 2
(p-22 v) dependence. Hence, [Lf]p = Σ, I(p- - v)f. = [[ fl, convolution. Rao & Ruderman (1999) already showed that this f. can reproduce finite discrete shifts 9p used in CNN. They used primitive version of L-conv with g, (T + v...V. Thus, L-conv approximate ID CNN. This result generalizes easily to higher dimensions.
Graph Convolutional Network (GCN). Let A be the adjacency matrix of a graph. In equation 17 if L. h(A), such Ż, 1-2 A/2 1/2, obtain a GCN (Kipf & Welling, 2016) (D سp Σ, Aup being the degree matrix). So in the special case where all neighbors each node < > have the same edge weight, meaning [..]]. [ Vu,p E< μ. >, equation 8 uniformly aggregating over neighbors and L-conv reduces to GCN. Note that this similarity is not just superficial. In GCN h(A) L is in fact Lie algebra basis. When L = h(A), the vector field is the

---

flow of isotropic diffusion df /t h(A), from each node to its neighbors. This vector field defines one parameter Lie group with elements g(t) exp[h(A)t). Hence, L-conv for flow groups with single generator are GCN. These flow groups include Hamiltonian flows and other linear dynamical systems. The main difference between L.-conv and GCN that I-ccovv can assign different weight to each neighbor of the same node, similar to GEM-CNN (Cohen et al., 2019a) with a fixed gauge set by 9u. Next, we discuss the mathematical properties of the loss functions for L-conv.
Group invariant loss
Loss functions of equivariant networks are rarely discussed. Yet, recent work by Kunin et al. (2020) showed the existence of symmetry directions the loss landscape. To understand how the symmetry generators in L-conv manifest themselves in the loss landscape, work out the explicit example of a mean square error (MSE) loss. Because G is the symmetry group. f and g should result in the same optimal parameters. Hence, the minima of the loss function need to be group invariant. One way to satisfy this is for the loss itself to be group invariant, which can be constructed by integrating over G (global pooling (Bronstein al., 2021)). A function / /= Sc dgF(g) is G-invariant (SI A.3). We can also change the integration S d''xr by change of variable dg/ dx (see SI A.3 for discussion on stabilizers).
MSE loss and Field Theory. The MSE is given by 7 T= d(1l where fa data samples and Q[f] is L-conv or another G-equivariant function. In supervised learning the input a pair fn. Yn- G can also act on the labels yn- We assme that y/, are either also scalar features Un S + Rm with group action g yn (2) yo ( (e.g. fn and yn are both images), that Vn are categorical. In the latter case g yn yn because the only representations of continuous G on discrete set constant. We can concatenate the inputs é, [fnlly..] with well-defined G action g On = [g fmlg- Dn]. The collection of combined inputs & = (Øi,..., ON )" is an (m my) x N matrix. Using equations and 9, the MSE loss with parameters W {W", ,c) becomes (SI 1.3.1)
[ dgC dg dgllmro [+ eccol|
1 W
dxx
Th m2 + 02 2, ha x + 0 (Ø" v'
(19)
Oz
Bg
Equation 19 generalizes the free field theories in physics (Polyakov, 2018). Here determinant of the Jacobian, W = W' and
òr
is the
äg
m2 = OT WT'W. wo
h h hdd(z) (z) = E m [L.1%
m
(20)
Note that h has feature space indices via |e moeles. las. with index symmetry hog h 8a F (i.e. is a ID scalar), ha becomes a a Riemannian metric for S. In general h combines 2-tensor h,s = a. 0,0g E TS @ TS with an inner product h²" has f on the feature space F.
hag
When
In field theory, the motivation is to preserve spatial symmetries for the metric h. In equation 19, h transforms equivariantly
2-tensor has (0- --- (v -0 h" for E G (SI A.3). The last term in equation 19 vanishes for many groups (SI A.3) and it is also absent in physics.
Robustness and Euler- Lagrange Equation. Equivariant neural networks more robust. To check this, we can quantify how the network would perform for an input ø' = o + SΦ which adds a small random perturbation dΦ to data point Φ. Robustness such perturbation would mean that, for optimal parameters W*, the loss function would not change, i.e. T W+== = [(; W*), requiring to be minimized around real data points Φ
This can be cast as variational equation [T W*] = O, which yield the familiar Euler- Guler-Lagraage (EL) equation (SI A.4). Therefore, for equivariant network to be robust, i.e. [6;; W1/000 0, we would require the data points á to satisfy the EL equations for optimal parameters W
Oc
ac =
Robustness to random noise
EL:
0. 000,00)
(21)

---

where the partial derivative terms appear because of the L-conv layer.
Equivariance and Conservation laws. Conserved currents, via Noether's theorem provide way find hidden symmetries (see also Kunin et al. (2020)). The idea is that the equivariance condition equation 2 can be written for the integrand of the loss, L(Ø, W If we write the equivariance equation for infinitesimal U.,, we obtain a vector field which is divergence free. Since G is the symmetry of the system, transforming an imput á + Φ by w E G the integrand should change equivariantly, meaning L(w Φ] w L[ø]. When robustness error is minimized as in equation 21, an infinitesimal R 7 + L4, with dep T.r0,, results in a conserved current (SI A.4)
Ôc 56 ac Noether current: J 0(0,6')
ax 6x
[ W 0 = - 0
(22)
The above equation shows that for equivariant networks with a given symmetry, the deviation in data along the symmetry direction (L,) yields divergence free current J known as Noether current. also provides an alternative means discover symmetry generators L, by minimizing J"]|- Note that this Noether current is the "stress-energy" tensor, associated with space (or space-time) variations dæ (Landau, 2013) (SI A.5). We can potentially design more general equivariant networks leading to other Noether currents.
7 Conclusion and Discussions
We propose the Lie algebra convolutional neural network (L-conv), an infinitesimal version of G- conv. L-conv layers do not require encoding irreps or discretizing the group, and can be combined to approximate any feedforward equivariant networks on compact groups. Additionally, L-coov's universal and simple structure allows us to discover symmetries from data. It is easy to implement, with formula similar to GCN. We validated that L-conv can learn the correct Lie algebra basis in synthetic experiment.
We discover several intriguing connections between L-conv and physics. Our derivation shows that equivariant neural networks based on L-conv lead to Noether's theorem and conservation laws. Conversely, we can also optimize Noether current discover symmetries. Furthermore, the current equivariance formulation only pertains to "spatial symmetries" (i.e. G acts on S). In physics, more general "internal symmetries" are quite common (e.g. particle physics). We can potentially design more general equivariant networks with L-ccov encoding such symmetries.
Our method also shed lights scientific machine learning, especially for physical sciences. Physicists generally use simple polynomial forms for the Lagrangian, or the loss function. These "perturbative" Lagrangian lead divergences in quantum field theory. However, it is believed the Lagrangian is more complicated. Hence, more expressive L-conv based models can potentially provide more advanced ansatze for solving scientific problems.
Acknowledgments and Disclosure of Funding
R. Walters is supported by a Postdoctoral Fellowship from the Roux Institute and NSF grants #2107256 and #2134178. This work supported in part by the U. S. Army Research Office under Grant 1911NF-20-1-0334, DOE ASCR 2493 and NSF Grant #2134274. N. Dehmamy and D. Wang were supported by the Air Force Office of Scientific Research under award number 1A9550-11-110354.
References
Anselmi, F., Evangelopoulos, G., Rosasco, L., and Poggio, T. Commetry-adaptee representation learning. Pattern Recognition, 86:201-208, 2019.
Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G. Learning invariances in neural networks. arXiv preprint arXiv:2010.11 2020.
Bogatskiy, A., Anderson, B., Offermann, J., Roussi, M., Miller, D., and Kondor, R. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pp. 902- PMLR, 2020.
10

---

Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint vrXiv:210413377. 2021.
Cohen, T. and Welling, M. Learning the irreducible representations of commutative lie groups. In International Conference Machine Learning, pp. 1755-1763, 2014.
Cohen, T. and Welling, M. Group equivariant convolutional networks. In International conference machine learning, pp. 2990-2999, 2016a.
Cohen, T., Weiler, M., Kicanaoglu, B., and Welling, M. Gauge equivariant convolutional networks and the icosahedral cnn. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2019a.
Cohen, T. S. and Welling, M. Steerable cnns. 2016b.
Cohen, T. S., Geiger, M., Köhler, J., and Welling, M. Spherical cnns. In International Conference on Learning Representations, 2018.
Cohen, T. S., Geiger, M., and Weiler, M. general theory of equivariant cnns on homogencous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019b.
Cybenko, Approximation by superpositions sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Finzi, M., Stanton, S., Izmailov, P., and Wilson, A. G. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint UrXiv:2002.12880 2020.
Finzi, M., Welling, M., and Wilson, A. G. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. arXiv preprint arXiv:2104.094 09949, 2021.
Gens, R. and Domingos, P. M. Deep symmetry networks. Advances in neural information processing systems, 27:2537-2544 2014.
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for
quantum chemistry. In International Conference on Machine Learning, pp. 1263-1272. PMLR, 2017.
Hall, B. Lie groups, Lie algebras, and representations: an elementary introduction, volume 222.
Springer, 2015.
Hartford, J., Graham, D., Leyton-Brown, K., and Ravanbakhsh, S. Deep models of interactions across sets. In International Conference Machine Learning, pp. 1909-1918. PMLR, 2018.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approxi- mators. Neural networks, 2(5):359-366, 1989.
Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. arXiv preprint 1609. 1600900900, 2016.
Kondor, R. and Trivedi, S. On the generalization of equivariance and convolution in neural networks to the action of compact groups. arXiv preprint arXiv:/802.03669. 2018.
Kondor, R., Lin, Z., and Trivedi, S. blebsch-gordan nets: a fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, pp. 10117-10126, 2018.
Kunin, D., Sagastuy- Brena, J., Ganguli, S., Yamins, L.., and Tanaka, H. Neural mechanies: Sym- metry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:20/2.04728. 2020.
Landau, I.. D. The classical theory of fields, volume Elsevier, 2013.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L., D. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.

---

LeCun, Y.. Bottou, L., Bengio. Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-233 1998.
Lee, M., Chow, B., Chu, S.-C., Glickenstein, D., Guenther, C., Isenberg, J., Ivey, T., Knopf, D., Lu,
P., Luo, F,, et al. Manifolds and differential geometry. Topology, 643:658, 2009.
Pfau, D., Higgins, I., Botev, A., and Racanière, Disentangling by subspace diffusion. arXiv preprint arXiv:2006.1299 2020.
Polyakov, A. M. Gauge fields and strings. Routledge, 2018.
Rao, R. P. and Ruderman, D. L. Learning lie groups for invariant visual perception. In Advances in neural information processing systems, pp. 810-816, 1999.
Ravanbakhsh, S. Universal equivariant multilayer perceptrons. arXiv preprint arXiv:2002.02912. 2020.
Ravanbakhsh, S., Schneider, J., and Poczos, B. Equivariance through parameter-sharing. In Proceed- ings of the 34th International Conference Machine Learning-Volle Volume 70, pp. org, 2017.
JMLR.
Schaub, M. T., Benson, A. R., Horn, P., Lippner, G., and Jadbabaie, A. Random walks on simplicial complexes and the normalized hodge l-laplacian. SIAM Review, 62(2):353-391, 2020.
Sohl-Dickstein, J., Wang, C. M., and Olshausen, B. A. An unsupervised algorithm for learning lic group transformations. arXiv preprint 101. 2010.
Wang, R., Walters, R., and Yu, R. Incorporating symmetry into deep dynamics models for improved generalization. In International Conference on Learning Representations, 2020.
Weiler, M. and Cesa, G. General e equuqiarat steerable cnns. Advances in Neural Information Processing Systems, pp. 14334-151 14345, 2019.
Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 3d steerable cnns: rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems, pp.
Learning
2018a
Weiler, M., Hamprecht, F. A., and Storath, M. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 449- 2018b.
Weinberg, S. The quantum theory fields, volume 3, Cambridge university press, 1995.
Whitaker, E. On the functions which are represented by the expansion interpolating theory. In Proc. Roy. Soc. Edinburgh, volume 35, pp. 181-194, 1915.
Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. Harmonic networks: Deep translation and rotation equivariance. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028 003--507.440 2017.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos. B., Salakhutdinov, R. R., and Smola, A. J. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.
Zhou, A., Knowles, T., and Finn, C. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933 2020.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A]. You are strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "g2-VrCnK_i4", "reviewer": "Reviewer_2v7e", "summary": "In the manuscript, the authors propose a method to estimate continuous symmetry of data through the estimation of Lie rings, rather than the Lie groups themselves.\nTo the best of my knowledge, there has never been a method for nonparametric estimation of Lie rings with linear representations, and I believe that the method is very novel and significant in the study of equivariant neural networks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As mentioned above, I believe that the method which is described in this manuscript has both significant novelty and importance that should be urgently shared with researchers in the same field.\nAlthough I acknowledge the novelty of the proposed method, the current manuscript is unclear what is the purpose of the research and the novelty of the whole research resulting from the development of the method. I think that the paper should be accepted after this ambiguity has been fixed.\nIn the following, I pointed out the unclear points according to the novelty or research purpose pointed out by the authors.\n\n\n1. We propose the Lie algebra convolutional network (L-conv), a building block for constructing group equivariant architectures.\n>The authors state that the estimation error of the Lie group by the proposed method is $O(η^{p+1})$.\nIn the estimation of Lie rings, only information in the neighborhood of a point is used.\nTherefore, the estimation error of the Lie group by the proposed method may be much larger than that of the direct estimation of the Lie group due to noise in the data, training error of DNN[1], or the large curvature of the manifold formed by the Lie group.\nIn this case, the accuracy of the task or the symmetry estimation will be lower than the previous studies that introduce the constraint of the Lie group or estimate the Lie group directly.\nCan you provide any discussion or numerical experiments that show that this does not occur?\n\n\n2. We show that multi-layer L-conv can approximate group convolutional layers, including CNN, and find graph convolutional networks to be a special case of L-conv.\n>This is a very interesting discussion, but the meaning of this discussion in this research is unclear.\nWhat advantage does this property give to L-conv as a network model of deep neural networks?\nIf the advantage includes higher performance compared to CNNs and GCNs, then the numerical results in the supplemental material D should be mentioned here as well.\n\n\n3. The Lie algebra generators in our L-conv can be learned to automatically discover symmetries.\n>As far as I know, there are methods that also learn Lie groups [2] or estimate Lie-ring from trained DNNs [1]. \nWhat is the advantage of the proposed method?\n\n\n4. We find that MSE loss function for L-conv generalizes important Lagrangians in physics.\n> I could not understand why \"The MSE is given by $I$\".\nAre you making the assumption that the error is given in quadratic form?\nIf so, can $I$ represent any quadratic-form likelihood function with symmetries?\n\n\n5. We find generalization error and equivariance in the loss landscape can be expressed as Euler-Lagrange equations and Noether currents, transferring ideas from physics to ML.\n> About the discussion about this novelty (from Line 283 to 296), it seems to be obvious that if the Lagrangian satisfies the principle of least action (Robustness to the perturbation $\\delta \\phi$), then it satisfies Hamilton's principle (variational principle).\nTherefore, it could not seem to be an element in itself to consist the novelty of the manuscript.\nAbout the discussion about this novelty (from Line 297 to 306), the discussion could not seem to be an element in itself to consist the novelty of the manuscript either since the discussion only contains a description of the basic knowledge of analytical mechanics. \nIn order to increase the importance of the discussion, I would like to see a more detailed description of how the obtained knowledge can be used in concrete ways.\nSpecifically, I did not feel the following statements were concrete enough.\n(Line 304) \"It would be interesting to see if these conserved currents can be used in practice as an alternative way for identifying or discovering symmetries. \"\n\n6. There are the other major comments about \"7 Discussion\" as described below:\nAuthors have pointed out the following as useful aspects of the findings of this research.\n(Line 325)\"But perhaps the most interesting discovery here is the connection made with symmetry encoding models in physics. This is significant for two reasons. \"\nHowever, I could not determine if the two points presented have significance for the following reasons, respectively.\n\n6-1. About (Line 326) \"One is that methods for dealing with symmetry used in physics acquire important meanings in ML, as our discussion on the loss function, generalization and conservation laws show. \"\n>It is true that paragraphs such as Line 307 discuss the correspondence between physics and machine learning, but it does not specifically describe the prospects of how they can be connected to the development of machine learning algorithms.\nTherefore, I can not understand there is any significance in this finding.\n\n\n6-2. About \"The second reason is for scientific machine learning. Physicists usually use the simplest forms for the loss function, similar to the MSE loss above. Many areas of physics suffer from this because these ansatze diverge in many cases or fit the data poorly. Hence, more complex L-conv loss functions can potentially serve as more advanced ansatze for solving physics problems.\"\n>I would like you to describe what kind of physical problems you can solve here as well.\nPersonally, since the estimation of the Lie ring is important in the conservation law estimation of dynamical systems, I am interested in whether the method can be applied to such problems.\nAlso, if the authors use quantum field theory as the target of the discussion, I would like you to discuss whether discrete symmetry can also be treated in the framework of the proposed method.\nThat is because quantum field theory has many discrete symmetries.\n\n>6-1 and 6-2 are related to the concerns in point 5.\n\n\nThere are some minor comments as described below:\n>1. I understand that the Lie rings estimated by the method are restricted to those realized by linear transformations.\nThis restriction is looser than that of the representation of Lie groups.\nIs there any advantage to this relaxation?\n\n>2. Figure 4 is not cited in the text.\n\n>3. (Line 197) Equation 58 is not found in the text.\nAlso, Equation 58 in Supplemental material does not seem to be the relevant equation.\n\n>4. (Line 262) \"assme\" -> \"assume\"  typo?\n\n>5. (Line 286) \"here s\" -> \"here is\"  typo?\n\n>6. (Line 86) $\\epsilon ^i$ seems to be undefined.\n\n\nReferences:\n\n[1] Mototake, Yoh-ichi. \"Interpretable conservation law estimation by deriving the symmetries of dynamics from trained deep neural networks.\" Physical Review E 103.3 (2021): 033303.\n\n[2] Zhou, Allan, Tom Knowles, and Chelsea Finn. \"Meta-learning Symmetries by Reparameterization.\" International Conference on Learning Representations. 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I described one of my concerns about the limitation of the method as major comment 1 on the Main review.\nIn addition to this, I point out that there is no discussion on the limitation of the method by restricting the Lie ring to realization by linear transformations. If possible, please discuss what kind of symmetries cannot be handled by the proposed method.\n\n\nI do not think there is a negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "more than 12 hours"}, "extra_text": {}}, {"review_id": "NZ42Jk8GMFk", "reviewer": "Reviewer_vpfc", "summary": "They propose a model for approximating invariant functions for Lie groups, the Lie algebra convolutional network (L-conv). While convolutional networks for ordinary groups require some kind of discretization, their model does not require it. Also, Lie algebra generators in our L-conv can be learned to automatically discover symmetries.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This was a difficult paper to judge. First of all, I can assure that the mathematical instrumentation of their method is solid.\nLieConv. and other group convolutions have the disadvantage that the integral over the Haar measure needs to be approximated by a discrete approximation when implemented, which prevents it from being an equivariant function in the strict sense. In their method, the convolution for the group is approximated using an equivariant function, and the model gives a strictly equivariant function.If you look at Theorem 1, you will also see that their model is an universal approximator for functions written in group convolution. If it is an universal approximator for equivariant functions, it should be clearly stated.\nI considered the experimental part of this paper to be weak. Although it has the strength of giving an exact invariant function, it is unclear in the current paper whether the previous methods or theirs are stronger for the actual invariant task. This is because their model also has a part that approximates the convolution, and it is not clear at this stage whether the approximation accuracy or the approximation accuracy by discretization is better.\nIt would be good to have some comparison experiments with Finzi's model, for example, to clarify this part.\nThere are no major problems with the structure of the paper.\nHere are some typos I found.\n105 expand\n197 L-conv equation 58", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20hours"}, "extra_text": {}}, {"review_id": "MgLJsMQTacd", "reviewer": "Reviewer_DTit", "summary": "The paper proposes a method to build equivariant neural networks by defining the layers in terms of the generators of the Lie group to which the NN is to be equivariant to. At the core lies the idea that any function (or conv kernel) with local support can be expanded in the Lie algebra basis via a Taylor approximation-like approach. Then, by observing that any function can be approximated with a basis of (shifted) local functions, it can be concluded that the presented approach in theory can approximate any group convolution. In addition, the authors the investigate the option to let go of the user-imposed constraint of being equivariant to a specific group, and instead let the neural network learn the generators (of the unknown group) in addition to the learnable weights that parametrize the layers. This is an open problem and the authors make promising steps in this direction.\n\nFinally, the authors draw connections to physics, but I must admit, this part was too much for me and I did not get it, but possibly could have by investing more time in the paper. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[overall impression]\nOverall the paper presents very interesting ideas and has high theoretical value. Especially the Lie algebra approach in which convolution kernels are treated as a superposition of localized functions is appealing and leads to a new class of implementations of group convolutions. Additionally it allows to further explore the option to learn the symmetries that are relevant for a task. The unique approach gives new insights in the field of equivariant deep learning and that's where the value of the paper is. Furthermore, the paper is packed with important relevant insights, but at the same time I consider this it weakness as it makes it hard to distill the important parts. I do believe the paper can still significantly improve from more intuitive explanations; after rereading the paper and writing the review I am starting to see the brilliance of it but it does require quite some effort. So despite it being a paper that is hard to comprehend (could be just me), the vastness of sound theoretical insights makes the paper in my opinion worthwhile a publication at NeurIPS.\n\n[readability/too much content]\nAs mentioned, my main concern is the density of the paper. In my opinion the paper is quite technical and there is simply too much information in it. It would have been nice if it were more focussed. As a reader I often struggled to grasp the general ideas as the emphasis is mostly on the technical details and I spend a lot of time on getting intuition or interpretation right. Providing layman's explanation/interpretation of the equations could improve readability of the paper (even for people already familiar to Lie group equivariant methods). More intuition probably comes at the cost of reducing the amount of information in the main paper. In my opinion, section 6 was a bit detached from the rest of the paper and was probably not necessary. I do appreciate such connections but it felt a bit too much (it could form the basis for a separate paper).\n\nMy detailed comments are as follows.\n\n[discretization issues?]\nAbstract, intro and Line 323: \"... to encode many irreps or discretizing the group\":\nA key feature of the method seems to be that it doesn't require group discretization as opposed to other methods, but I am not sure if this is really the case. The definition of the L-conv layer has an obvious dependency of g on the left-hand side of the equation. Doesn't this mean one has to evaluate it on a grid on the group G? I think here I misunderstood something as it seems that in Equation 16 the dependency on g is lost. If I am indeed mistaken here, please use my ignorance as queue that more intuition about the equations need to be provided in the paper. \n\nRegardless, the data is always discretized (see section 4 tensor notation) and so will the generators/left-invariant vector fields (or at least the regular representations/action of g on the vector representations of the data). So I don't understand the argument of not using a discretization as the proposed method still seems to be prone to discretization artifacts. If possible, try to address my concern regarding the discretization issue.\n\n[Lie conv methods are involved?]\nLine 32. \"however, their approach is quite involved, ...\" and line 39 \"and this task is quite tedious\". I disagree here, I think such Lie group based methods are in fact one of the easiest ways to build G-CNNs (especially compared to steerable G-CNNs) as it allows for a generic and modular code base in which only a few operations (group product, inverse and log) need to be defined after which implementations follow standard procedures. See e.g. the PoinConv method in Finzi et al. (already cited) or on a dense grid as in Bekkers et al. (missing cite, see ref below).\n\n*ref*\nErik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2019.\n\n[group quotients?]\nWith respect to the lifting, i.e., replacing points x=g x0 with g. I think it is important to remark that g may not be unique (e.g. when x comes from a homogeneous space that is a quotient of the group, e.g., when x \\in R^2 and G=SE(2)). It currently reads as if there is only one g for each x. If this is not the case, is this a problem?\n\n[Equation 6]\nEquation 6 is important but it is hard to follow what is precisely happening. Some intuition here may be helpful.\nIf I understand correctly the function f(g) is locally approximated with a first order Taylor expansion? \nAnd because the kernel is localized the evaluation boils down to multiplying this approximation locally with W^0 as this like the average value of that the kernel is assumed to have and can locally be assumed to be constant?\nThen d/dg is a bit abstract, this is \"just\" the differential of f, right? Which is a covector and multiplying it with g L_i filters out the i-th component?\n\nA thing that confused me is that now all of a sudden the integration measure moves up-front in the integral whereas in eq 3 and 4 it is at the end. Is there a particular reason for this?\n\n[Figure 2 is very insightful! ]\n\n[Related work]\nThe approach taken in this paper turns out to be very related to two particular works. Firstly, also in Bekkers' work on B-Spline CNNs on Lie groups, convolution kernels are expanded in a basis of highly localized functions, namely B-splines. See e.g. Figure 3 of that paper. The result of group convolution with such kernels can then be obtained by expnding the kernels and to the regular g-conv, or by shifting the underlying function to the original and take the inner products with the basis functions, which is similar to the work presented in this paper.\n\nSecondly, the work by Smets et al. on PDE based G-CNNs seems very related. In it, layers are defined by PDEs expressed in left-invariant vector fields, similar to this work. In particular, the convection term of their PDEs is computed by 1, constructing left-invariant vector fields from a given Lie algebra basis, 2 use them as differential operators (like in this paper equation 10) and 3, multiply them with corresponding weights (like in this paper). The viewpoint is however rather different (PDE vs local approximation viewpoint). \n\n*ref*\nSmets, B., Portegies, J., Bekkers, E., & Duits, R. (2020). PDE-based group equivariant convolutional neural networks. arXiv preprint arXiv:2001.09046.\n\n[deep L-conv networks?]\nIt is a bit unclear how the experiment with learning the symmetries could generalize to deeper architectures. The current experiments only use one group equivariant layer (as 3 recurring L-convs). Are such layers also suitable for constructing deep architectures?\nIf so, would this mean that the generators need to be the same throughout all layers?\n\n[practical details of learning the generators]\nLine 199: \"The L_i can be either inserted... or learned to discover symmetries\" In the latter case, how is L_i parametrized? This is generally a very large and dense matrix, or am I mistaken? Is there any constraint put on the L_i matrices that are to be learned?\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "-KgzrF818ub", "reviewer": "Reviewer_rszC", "summary": "The paper introduces an equivariant L-Conv layer, a building block that can be used to construct group equivariant layers. These L-conv layers are used to discover symmetries in data, and various connections between L-Conv and concepts in physics are made.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper lacks focus and it is difficult to see what the main contribution of the paper is. Each contribution in the list of contributions in the Intro is presented poorly, which makes it difficult to assess the strenghts and weaknesses of L-Conv and its use for discovering symmetries from data.\n\n- Majority of paper is on L-Conv, a new group equivariant layer, suggesting that this new methodology is the main focus and this is a methodology paper. If this were the case, there should be more details in the main paper about the practical benefits of L-Conv over baseline group equivariant layers such as G-conv, shown by experimental results. However all experimental results (except from a brief, insufficiently described result in Figure 3) are placed in the appendix, making the practicality of L-Conv difficult to assess. Furthermore, the results on images in Appendix D comparing to group equivariant baselines such as LieConv are disappointing - all models show test accuracy < 0.5, and LieConv seems to outperform L-Conv despite having much fewer parameters, except the rotated&scrambled image case which is a rather contrived scenario.\n\n- The title of the paper, on the other hand, suggests that symmetry discovery is a main component of the paper. However there is very little description of this in the main paper (a single paragraph in Section 3.1), and due to the brevity it is difficult to understand how symmetry discovery is happening and how the reader is supposed to interpret the learned L matrix in Figure 3. The task in Figure 3 seems to be a task trying to predict rotation angles between two rotated versions of the same input, which I'd argue is not really an instance of 'symmetry discovery'. Symmetry discovery usually refers to the case when given a G-equivariant/invariant task with unknown G, one has to narrow down the set of possible groups G. However for the task in Figure 3, any model that solves the task has to be rotation equivariant, so there is no discovery of symmetry involved here. It would be more relevant/interesting to see what L_i is learned by L-Conv when used on a more standard invariant/equivariant task such as image classification or object localisation, to see if L-Conv can learn translational/rotational symmetry. Furthermore when L_i is learned, the number of lie algebra generators (the number of L_i learned) determines what groups can/cannot be learned, which seems to be a limitation of the method for symmetry discovery.\n\n- Re connections between MSE loss function/generalisation error for L-Conv and concepts in Physics, it's not clear what are the insights gained from these connections. The paper merely points out these connections, but does not expand on what can be learned from such connections.\n\n- The main paper reads to be a summary of the appendix, with considerable overlap between the main paper and the appendix (e.g. Sections 3&6 <-> Appendix A, Section 4 <-> Appendix B.1). I find myself having to read the appendix to make sense of the main paper - this is another example showing that the paper is disorganised, since the roles of the main paper and the supplementary material are reversed.\n\nIntroduction of L-Conv layer in Section 3 is confusing.\n \n- Equation (6) is hard to follow. The equality in the first line seems to be from Taylor expansion (it'd be good to mention this explicitly for clarity), but the jump from line 1 to 2 in Equation (6) seems erroneous. Shouldn't the second line be W^0 f(g) + c \\bar{epsilon}^i g L_i df/dg instead? Hence the last line of equation (6), where you seem to have gone from the Taylor expansion back to the analytic expression, seems unjustified.\n\n- In Section 3, it's not clear at all how L-conv is implemented in practice (e.g. what delta_eta is and how W^0, \\bar{epsilon}_i are computed), and information on this only appears at the end of page6, towards the end of the paper, at which point I found myself having to go back to Section 3 to make sense of it. This is an instance showing that the paper is poorly organised and presented.\n\n- Section 3.1 on approximating G-Conv using L-conv doesn't seem to tell us anything about how L-Conv can be made to approximate G-Conv in practice. There is no information about how the local kernels kappa_k(g) should be chosen given a G-Conv kernel \\kappa(g), and there is no detail about the error introduced by taking a finite number of L-Convs to approximate a G-Conv. The statement \"L-Convs can approximate G-Conv\" doesn't mean much if there is no analysis on the approximation error.\n\n- tensor notation further makes things all the more confusing, and seems unnecessary since all \"tensors\" are just matrices or vectors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not mention any limitations of their work. There are clear limitations are in the lack of clarity and organisation in the presentation of the paper, which is making it difficult to assess the limitations of the proposed methods and ideas in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, the authors build group equivariant neural network layers. These are built from convolutions with sufficiently local functions that can be lifted from the Lie group to the Lie algebra. This approach is sufficiently general to encompass previous equivariant architectures. This approach also comes with a universal approximation property, which the authors need to highlight.

== Why accept this paper?
The paper offers a clearly creative and novel contribution, with a number of high value theoretical hindsights. The community will greatly benefit from having access to these ideas, and I look forward to seeing what and how the present authors, or other researchers, will build on top of those ideas.

== Why not a higher rating?
As mentioned by all reviewers, this paper is particularly hard to read. Part of the reason is that it deals with a difficult problem, but we can only praise the authors for this, not blame them. But another reason for the difficulty to read the paper is its very high density with a lack of focus and clarity. Following the discussion with reviewers, the authors received a number of suggestions to improve their paper, and they should implement as many of those suggestions as possible.
Decision: Accept (Poster)