Lie Algebra Convolutional Neural Networks with Automatic Symmetry Extraction
Authors: Nima Dehmamy
Yanchen Liu
Robin Walters
Rose Yu
Venue:  (2021)
Keywords: equivariance
group convolutional neural network
inductive bias
group equivariant architecture
Lie group
Lie algebra

Abstract: Existing methods for incorporating symmetries into neural network architectures require prior knowledge of the symmetry group. We propose to learn the symmetries during the training of the group equivariant architectures.  Our model, the Lie algebra convolutional network (L-conv), is based on infinitesimal generators of continuous groups and does not require discretization or integration over the group. We show that L-conv can approximate any group convolutional layer by composition of layers. We demonstrate how CNNs, Graph Convolutional Networks and fully-connected networks can all be expressed as an L-conv with appropriate groups. By allowing the infinitesimal generators to be learnable, L-conv can learn potential symmetries. We also show how the symmetries are related to the statistics of the dataset in linear settings.  We find an analytical relationship between the symmetry group and a subgroup of an orthogonal group preserving the covariance of the input. Our experiments show that L-conv with trainable generators performs well on problems with hidden symmetries. Due to parameter sharing, L-conv also uses far fewer parameters than fully-connected layers.

Under review as a conference paper at ICLR 2021
LIE ALGEBRA CONVOLUTIONAL NETWORKS WITH AUTOMATIC SYMMETRY EXTRACTION
Anonymous authors Paper under double-blind review
ABSTRACT
Existing methods for incorporating symmetries into neural network architectures require prior knowledge of the symmetry group. We propose to learn the symme- tries during the training of the group equivariant architectures. Our model, the Lie algebra convolutional network (L-conv), is based infinitesimal generators of continuous groups and does not require discretization or integration over the group. We show that L-conv can approximate any group convolutional layer by composition of layers. We demonstrate how CNNs, Graph Convolutional Networks and fully-connected networks can all be expressed as an L--conv with appropriate groups. By allowing the infinitesimal generators to be learnable, L-conv can learn potential symmetries. We also show how the symmetries are related to the statistics of the dataset in linear settings. We find an analytical relationship between the symmetry group and a subgroup of an orthogonal group preserving the covariance of the input. Our experiments show that L-conv with trainable generators performs well on problems with hidden symmetries. Due to parameter sharing, L-conv also uses far fewer parameters than fully-connected layers.
Many machine learning (ML) tasks involve data from unfamiliar domains, which may or may not have hidden symmetries. While much of the work equivariant neural networks focuses equivariant architectures, the ability of the architecture to discover symmetries in a given dataset is less studied. Convolutional Neural Networks (CNN) (LeCun et al., 1989; 1998) incorporate translation symmetry into the architecture. Recently, more general ways to construct equivariant architectures have been introduced (Cohen & Welling, 2a;b;; Cohen al., 2018; Kondor & Trivedi, 2018). Encoding equivariance into an MI. architecture can reduce data requirements and improve generalization, while significantly reducing the number of model parameters via parameter sharing (Cohen et al., 2019; Cohen & Welling, 6b; Ravanbakhsh et al., 2017; Ravanbakhsh, 2020). As a result, many other symmetries such as discrete rotations in 2D (Veeling et al., 2018; Marcos et al., 2017) and 3D (Cohen al., 2018; Cohen & Welling, 2016a) well as permutations (Zaheer et al., 2017) have been incorporated into the architecture neural networks.
Many existing works equivariant architectures finite groups such permutations in Hartford et al. (2018) and Ravanbakhsh et al. (2017) or discrete subgroups of continuous groups, such as 90 degree rotations in (Cohen al., 2018) dihedral groups D in Weiler & Cesa (2019). Ravanbakhsh (2020) also proved a universal approximation theorem for single hidden layer equivariant neural networks for Abelian and finite groups. General principles for constructing group convolutional layers were introduced in Cohen & Welling (2016b), Kondor & Trivedi (2018), and Cohen et al. (2019), including for continuous groups. A challenge for implementation is having to integrate over the group manifold. This has been remedied either by generalizing Fast Fourier Transforms (Cohen al.. 2018). or using irreducible representations (irreps) (Weiler et al., 201 8a) either directly as coefficients
spherical harmonics as in Worrall et al. (2017) or using more general (Kondor et al., 2018). Other approaches include discretizing the group as in Weiler et al. (201 Sa;b); Cohen & Welling (2016a), or solving constraints for cquivariant irreps as in Weiler & Cesa (2019), or approximating the integral by sampling (Finzi et al., 2020).
The limitations in all of the approaches above are that: 1) they rely on knowing the symmetry group a priori, and 2) require encoding the whole group into the architecture. For a continuous group, is not possible to encode all elements and have to resort to discretization or truncated sum over irreps. Our work attempts to resolve the issues with continuous groups by using the Lie algebra (the linearization of the group near its identity) instead of the group itself. Unlike the Lie group which is

---

Under review as a conference paper at ICLR 2021
infinite, the Lie algebra usually has finite basis (notable exception being Kac-Moody Lie algebras for 2D Conformal Field Theories (Belavin et al., 1984) in physics). Additionally, we show that the Lie algebra basis can be learned during training, through separate optimization process. Hence, our architecture, which generalizes a group convolutional layer, is potentially capable of learning symmetries in data without imposing inductive biases.
Learning symmetries in data was tackled in restricted settings of mostly commutative Lie groups as in Cohen & Welling (2014) and 2D rotations and translations in Rao & Ruderman (1999) and Sohl-Dickstein et al. (2010) or permutations (Anselmi et al., 2019). However, the symmetries learned by the architecture are not necessarily familiar spatial symmetries. As we show in the case of linear regression, the symmetries may correspond to transformations preserving the statistics of the data, Specifically, we show general relation between the symmetries of linear regression and a deformed orthogonal group preserving the covariance matrix. Such symmetries of the probability distribution and ways to incorporate them into the architecture were also discussed in Bloem- Reddy Teh (2019). The work that is closest in spirit and setup to ours is Zhou et al. (2020) which uses meta-learning to automatically learn symmetries. Although the weight-sharing scheme of Zhou et al. (2020) and their encoding of the symmetry generators is different, their construction does bear some resemblance to ours and we will discuss this after introducing our architecture.
Contributions Our main contributions can be summarized as follows
We propose a group equivariant architecture using the Lie algebra, introducing the Lie algebra convolutional layer (L-conv).
In L-conv the Lie algebra generators can be trained to discover symmetries, and it outper- forms CNN on domains with hidden symmetries, such rotated and scrambled images.
Group convolutional layers on connected Lie groups can be approximated by multi-layer L-conv, and conlecco CNN and graph convolutional networks are special cases of L-conv.
In linear regression, we show analytical relations between symmetries in and orthogonal groups preserving covariance of data.
EQUIVARIANCE IN SUPERVISED LEARNING
Consider the functional mapping y: f(x;) of inputs X (zI,..., to outputs Y (y1.... :y). We assume each input æ E mddxm where IR are the "space" dimensions and RT the "channels", and y Re (or 25 for categorical variables). We assume group G acts only on the space factor (Rd shared among channels) of æ through a d-dimensional representation Ta G - GLa(R) mapping each g an invertible d d matrix. The map Ta must be continuous and satisfy (a( = Ta(ww) for all u, E G (Knapp, 2013, IV.1). Similarly, let G act on via c-dimensional representation T.. To simplify notation, will denote the representations simply as Un = (u). A function f solving = f(x;) is said to be equivariant under the action of a group G by representations Te, To
v.y = u.f(x) = f (uxx) f(z) = uef (uj'z). x).
VuEG
Lie Groups and Lie Algebras The full group of invertible d d matrices over R. is the general linear group, denoted as (RR. It follows that every real d- dimensional group representation Ta(G) C GL. If "faithful representation" (i.e. (w)) o T(x) if A v), then G C GR) In our problem, we only know the group G through its linear action on X or on output h of layer I in neural network. Therefore we may assume the representation Ta, acting on hi with largest dj is faithful. GLa(R) is a Lie group and all of its continuous subgroups G are also Lie groups. Next, we will first briefly review the Lie algebra of Lie group and then use it to introduce our equivariant architecture.
Notation
Unless stated or obvious, a in A is an index, not an exponent. We write matrix products A B- Ea A B, Recall that Rdxm. For linear transformation A Rd. Rd2
2

---

Under review as a conference paper at ICLR 2021
acting on the spatial index the channel index, will one upper and one lower index in Ag = [A- h/.. We will use (a, b, c) for channels, and (4, 2, p) for spatial, and (1, k) for Lie algebra basis indices. We will occasionally keep explicit summation Σ, for clarity.
For any Lie group G C (R), group elements infinitesimally close to the identity element can be written as u = I+ L4- We can find basis L, which span the "Lie algebra" g of the group G, meaning they are closed under commutations
[Li Lnp = L,L, LuL, = Sij L*
(2)
with fiz called the structure constants of the Lie algebra. The L, are called the infinitesimal generators of the Lie group. They define vector fields spanning the tangent space of the manifold of G near its identity element I. For elements u E GG in the connected component G2 C G containing the identity, an exponential map exp: g + Gy can be defined such that u explt L]. For matrix groups, if G' is connected and compact, the matrix exponential defined through a Taylor expansion is such a map and it surjective. For most other groups (except (C, and nilpotent groups) it not surjective. Nevertheless, for any connected group ( iincluding G1) every element u can be written as a product u = II. it L] using the matrix exponential (Hall, 2015). We will use this fact to modify existing results about group equivariant architectures to introduce L-conv.
GROUP EQUIVARIANT ARCHITECTURE
Consider the case where the solution f yo f(x) is implemented as feedforward neural network. Denote the lincar output of layer / by hı (1 Rd with ho z₁. Kondor & Trivedi (2018); Cohen & Welling (2016a) showed that a feedforward neural network is equivariant under the action of a group G if and only if each of its layers implement group convolution (G-conv) given by
[ dpl (u
h (fi (h1-11
where du(u) is the Haar measure on the group manifold, my is appropriate dy-dimensional representation of G, and g: G - Rm xma+i is a set of convolution kernels, where my and m++1 are the number of input and output channels (filters). Here fi are point-wise activation functions. In the formalism of Kondor & Trivedi (2018), we may consider hy to be a map from the homogeneous space X; to R where X; is space such that the space of maps Lg[A] is parameterized by Rd,, To use equation 3 in practice, we need to make the integral over the group manifold tractable. Although, Finzi et al. (2020), which approximates the integral over G by sampling, approach the idea of working in the Lie algebra, they use a logarithm map to define the Lie algebra. We take a different approach and formulate the architecture in terms of the Lie algebra basis. We also allow the architecture to learn basis L, for the symmetries.
For continuous symmetry groups we will now show that using the Lie algebra can approximate G-conv without having to integrate over the full group manifold, thus alleviating the need to discretize or sample the group. The key point is the discreteness and finiteness of the Lie algebra basis for many Lie groups.
Proposition 1. Let G be Lie group and Ta G + Rd di di-dinensional representation of G. If a convolution kernel 9I: : G + Tm ixm; has support only on an infinitesimal ii neighborhood of identity, G-conv layer of equation 3 can be written in terms of the Lie algebra.
Proof: Lincarization over 1) yields i I+ L in equation 3, with L, E R4xd, The inverse is then UT R T e. I. ) Since gi has support only in an 1) neighborhood of identity, fixing basis Li, we can reparametrize gi + I.) = (i(c) a function over the Lie algebra 9I gg -> R I xm141 Dropping the layer index / for brevity, we have
* ((()) = [ 5(0))) ((T)((())))) ((T - ( (WOT- 17%T-W.-1 W. W.T) I.) h. VaS(h)
(4)
W g(e)dmie, w' f( dgle)d
(5)
3

---

Under review as a conference paper at ICLR 2021
where the number of W is n,, the number of L. When f(h is homogeneous function like ReLU or linear activation, h Vv.f(h) = ah) In general, since we are designing the layers, we can choose h (h) (hh) to be desired activation function.
Note that in G-conv, the convolutional kernels g(u) are the trainable weights. The difficulty with learning them is that one has to parametrize and integrate the convolution major advantage of working in the Lie algebra instead of G is that in equation the integrals of g(e) separate from f(i), leaving us with set of integrated weights W. This means that instead of the architecture needing to implement the integral Ja (u) [(hh)dj we only need to learn a finite number of convolutional weights W, like the filter weights in CNN. Additionally, Indeed, having learnable L, to learn hidden symmetries is one of main features we are proposing, as we show our experiments.
the group G. The
can learn Li-
L-conv Layer
The construction in equation 4 is at the core of our results. h E ( has components he where E {I,-.. , d
is the spatial index and {1,-.. the channel index. We define the Lie algebra convolutional layer (L-conv) as
nu
L-conv
(h(h) =o
Σ 0=0
Lg h W +6)
Fh =
Σ K.0.2
[.l] A [W']e ba
(6)
where is the activation function, Lo = and W' E 1gm:xmi-1 are the convolutional weights, and the second line shows the components (all symbols are indices, not exponents). Note that, similar discussion in Weiler et al. (2018a), 8a), arbitrary nonlinear activation a may not keep the architecture equivariant under G, but linear activation remains equivariant. Also, note that in equation 3, the activation d 1 of the pre- vious layer. In defining L-conv we are including the activation a after the convolution, which is why appears outside in equation 6 instead of as a (h).
L-conv layer
channels m
m
1, s,shared among channets
L
(ml)
h
h
W
Weighte sheared among dims..
Since in connected Lie groups larger group ele- ments can be constructed as u II. exp(ta (Hall, 2015) from elements near identity, it fol- lows that any G-conv layer can constructed from multiple L-cony layers on these groups.
L-conv layer architecture. L, only act on the spatial dimensions, and W only act the my feature. For each i, this action analogous Graph Convolutional Network with d nodes and my features per node.
Figure I:
Proposition 2. Any G-conv layer on con- nected Lie group can be approximated by multi-layer L-comv to arbitrary accuracy.
Proof: To show this we need to take two steps. First, we discretize support of g(u) in equation to infinitesimally small sets G = Ur Gt. each having support around an element uy E G.- This allows to write (f (h) Σx Amfg(uu)f(() h), where Apt Sa. du. Next, write write 4y = IT Va where
each uy as product. On connected groups for any 4ę E G, Va explta. L] ss (I + ta L) are elements close to identity. Then, f(uh) can be written as composition | G-conv layers, each with weights (u) 5(u v.) (defined using faithful representation of G)
[I vorh (F......
(a / dp du
[(u hh)
Since u, are near identity, each F,, can be converted to an L-conv layer with linear activation and weights W f. and W 1. Thus, the G-conv layer can be approximated as and composition
4

---

Under review as a conference paper at ICLR 2021
of L-conv layers. Each Gj can be considered one output channel for the multi-layer L-conv and g(ux) are weights of a final aggregation layer.
So far we have shown that L-conv layers could replace G-conv for connected groups. The advantage of using L-conv layers is that for many potential symmetry groups, such orthogonal and unitary groups, the number of generators Lg is finite and often a small number them may be sufficient. For instance, consider (0(2) rotations on p X p images. The flattened image has d p² dimensions and the number of GL,((R) generators is p². However the induced representation of 50(2) on IR has a single generator. In fact, in many domains we expect symmetries hidden in the data to be much lower in dimensionality than the data itself. Additionally, in the other extreme, where there is no restriction on the symmetry, meaning G GL), we observe that L-conv becomes a fully-connected layer, as shown next.
3. A fully-connected neural network layer can be written an L-conv layer using (RR) generators, followed by a sum pooling and nonlinear activation.
Proposition
Proof: The generators of (L) are one-hot E E Rddd matrices L, E(a,9) which are non-zero only at index = (a, 3) with elements written using Kronecker deltas
GL (R) generators L = [E(a,a)l = 6
(
Now, consider the weight matrix w E Rmxd and bias E R" of a fully connected layer acting on h I2 as F(h) aw h b). The matrix element can be written as
w E wg w11g E
ΣΣ 1 a.
whi (E Σ W -
meaning an L-conv layer with weights W ones) followed by pooling over is the same fully connected layer with weights w.
(a.3) wg L, (1 input channel, and 1 being a vector of
Thus, interestingly, more restricted groups, rather than large, meaning groups with fewer L,, lead to more parameter sharing. Indeed, we show below, the most restricted L-cony graph convolutional networks. It is also easy to see that familiar CNN can be encoded as L-conv. Note that, generally shift operators used in CNN are thought of as group elements for discrete permutation group. However, the same shift operators L, can be used to create continuous fractional shifts as (I + aL:)/ + 1). Since shifts commute with each other, they form basis for the Lie algebra of an Abelian subgroup of (R.)
Proposition 4. CNN is special case of L-conv where the generators are shift operators.
Proof: In ID CNN, for kernel size k we have l shift operators L, given by Li = 8p. Plugging this into equation and doing the sum over V, we recover the convolution formula
F(h) =a
Σ Ne Wiie + ba 4.C
(10)
In higher dimensional CNN, covers the relevant flattened indices for the required shifts. For instance, a p q image, when flattened becomes vector of length pq. To cover (2, 2) convolutional kernel, we need shifts to be by 0, 1, 9, q + pixels. Thus, we need four L, given by
L. 8 8n.--83 8:
[0, 1, 1]:
(11)
This means that the number of generators L, is the size of the convolutional kernel in CNN. General- ization to higher dimensions is straightforward.
In CNN, covering larger patches is equivalent g(u) covering a larger part of the group. But large convolutional kernels are not generally used. Instead, we achieve larger receptive fields by adding more CNN layers, similar to Proposition
We may also label them by single index like i (3d, but indices is more convenient.
5

---

Under review as a conference paper at ICLR 2021
Connection with Graph Convolutional Networks Finally, we note that the L-conv layer equa- tion has the structure of a Graph Convolutional Networks (GCN) Kipf & Welling (2016) with multiple propagation rules L; derived from graph adjacency matrices and ju, indexing graph vertices. Thus, an L-cony with a single I. is a GCN. In fact, there is deeper connection here, as we show now
Proposition 5. A GCN with propagation rule A E pdd is equivalent to an L-conv, with the group being ID flows generated by A A-1.
Proof: We can define linear flow similar to diffusion equation
dh(t) dt
h + 5t) (t)
X(A [))(t)
(12)
where À = 5t/dt sets the time scale. Thus L = X(A
I) the generator a 1D group of flows with elements = exp[tL]. subgroup of the group of diffeomorphisms on R with a single generator
L. Thus, a GCN with propagation rule A is an L-conv using Lo = I and L L with the same convolutional weights for Lo, L1-
Hence, the most restricted L-conv based on ID flow groups with a single generator are GCN. These flow groups include Hamiltonian flows and other linear dynamical systems. CNN can also be interpreted as a GCN with multiple propagation rules, where each shift operator is a subgraph of the grid network.
LEARNING POTENTIAL SYMMETRIES
When dealing with unknown continuous symmetry groups, it can virtually impossible to design G-conv, it relies the structure of the group. The Lie algebra, however, has much simpler, linear structure and universal for all Lie groups. Because of this, L-conv affords us with a powerful tool to probe systems with unknown symmetries. L-conv is generic weight-sharing ansatz and the number of L, can be expected to be small in many systems. This means that even we do not know L, it may be possible to learn the L; from the data. In fact, low-rank L. via SGD simultaneously with other weights yield impressive performance on data with hidden symmetries (Fig. 2), without needing any input about the symmetry.
show in our experiments, learning
Learning the L; We learn the L; using SGD, simulta simultaneously with W and all other weights. Our current implementation is similar to a GCN F(h) = aL, h W' b) where both the weights W and the propagation rule L; are learnable. When the spatial dimensions d of the input E Rd is large, e.g. a flattened image, the I, with d² parameters can become expensive learn. However, generators of groups are generally very sparse matrices. Therefore, we represent L; using low-rank decomposition L; U.V. An encoder V of shape ny, X d, d encodes n, matrices V,, and a decoder U of shape n d x dr. Here d is the input dimensions and d, the latent dimension with d, < d for sparsity. In order for the L; to form a basis for Lie algebra, they should closed under the commutation relations equation 2, well as orthogonal under the Killing Form (Hall, 2015, Chapter 6). These conditions can be added to the model as regularizers (see Appendix F.2), but regularization also introduces additional time complexity of On dR.d). d). which can be quite expensive compared to the O (ngd,d) of learning L, via SGD. Therefore, in the experiments reported here we did not use any regularizers for L,.
Comparison with Meta-learning Symmetries by Reparameterization (MSR) Recently Zhou al. (2020) also introduced architecture which learn equivariances from data. We would like to highlight the differences between their approach and ours, specifically Proposition in Zhou et al. (2020). Assuming discrete group G = [gi..... fully-connected layer, acting on œ E R as vec(W) = matrices" and a E IR are the "filter weights". Then they use meta-learning to learn UG and during the main training keep U fixed and only learn v. We may compare MSR to our approach by setting = s. First, note that although the dimensionality
9. .re)
they decompose the weights W e R of
where U E IR** are the "symmetry
U E Rgdddd seems similar to our L E ignxdxd the L; are matrices of shape d x d, whereas U has shape (nd) x d with many more parameters than L. Also, the weights of L-conv W E Rx xmm-1, with my being the number of channels, are generally much fewer than MSR filters v E Rdd Finally, the way in which Up acts data is different from L-conv, as the dimensions reveal. The prohibitively high dimensionality of U requires MSR to adopt sparse-coding scheme, mainly Kronecker decomposition. Though not necessary, too

---

Under review as a conference paper at ICLR 2021
choose sparse format for L,, finding that very low-rank L3 often perform best. A Kronecker decomposition may bias the structure of U as it introduces a block structure into it.
Contrast with Augerino In concurrent work, (Benton et al., 2020) propose Augerino, a method to Iearn invariances with neural networks. Augerino uses data augmentation to transform the input data, which means it is restricting the group be subgroup of the augmentation transformations. The data augmentation is written as 9, = cxp (2: L;) (equation (9) in Benton et al. (2020)), with randomly sampled &, [-1, 1]. O; are trainable weights which determine which L; helped with the learning task. However, in Augerino, L; are fixed a priori to be the six generators of affine transformations in 2D (translations, rotations, scaling and shearing). In contrast, our approach is more general. We learn the generators L; directly without restricting them to be known set of generators. Additionally, we not use the exponential map, hence, implementing L-conv very straightforward. Lastly, Augerino uses sampling to effectively cover the space of group transformations. Since the sum over Lie algebra generators is tractable, we do not need to use sampling.
Next, we show that in certain cases, such linear regression, the symmetry group can be derived analytically. While the results may not apply to more non-linear cases, they give us good idea of the nature of the symmetries we should expect L-conv to learn.
3.1 SYMMETRIES OF LINEAR REGRESSION
In the of linear regression can derive part of the symmetries explicitly, shown next. We absorb biases into the regression weights A as the last row and append a row of to the input.
Theorem 1. Consider a linear regression problem on inputs X R and labels Y E Rx like above. We are looking for the linear function Y = AX. For this problem is equivariant under group G. through two representations E TT(G) and un E (G) acting on and Y, respectively, it is sufficient for u and Uh satisfy
u HHu -H
u. Y -YX".
(13)
where H = is the covariance matrix of the input.
Proof: The equivariance condition equation 1 becomes
Ue Ny = AuX
= A = u.Au
(14)
Assuming that the number of samples is much greater than features, >> d. and unbiased data, X X will be full rank x d and that its inverse exists. The solution to the linear regression Y= AX
given by A Yx" r (x x )-1 Using the definition of covariance H above, the condition
equation 14 becomes A- = u Y x " (uX XXXX' xfu")-+ Thus, a sufficient condition for equation 14 hold is equation 13.
The first condition in equation 13 is unsupervised, stating that u preserves the covariance H, while the second condition is supervised, stating that u. and u preserve the coss-correlltton of input and labels YXT. Next, we show that the group satisfying equation 13 is related to an orthogonal group.
Corollary 1. The subgroup of symmetries of linear regression satisfying uHu" = H in equation 13 form an orthogonal group isomorphic to SO(d), with a Lie algebra basis given by
L = H 12 1/2
L E so(d)
(15)
Proof: multiplying "HuT = H by H 1/2 from both sides we see that v' = H- 1/2 1/2 satisfies u'ue = meaning u' E (O()). Expanding elements near identity we get R= 1 + L = 2/ (1+ + 1/2 Setting proves equation 15. (see Supp D.2 for more details)
H11/2
4 EXPERIMENTS
To understand precisely how L-conv performs in comparison with CNN and other baselines, conduct a set carefully designed experiments. Defining pooling for L-conv merits more research. Without pooling, cannot use L-cony in state-of-the-ar models for familiar problems such image
7

---

Under review as a conference paper at ICLR 2021
classification. Therefore, we the simplest possible models in our experiments: one two L-conv, or CNN, or FC layers, followed by a classification layer.
Test Datasets We use four datasets: MNIST, CIFARIO, CIFARIOO, and FNSS. To test the efficiency of L-conv in dealing with hidden or unfamiliar symmetries, we conducted our tests on two modified versions of each dataset: 1) Rotated: each image rotated by random angle (no augmentation); 2) Rotated and Scrambled: random rotations are followed by a fixed random permutation (same for all images) of pixels. We used 80-20 training test split on 60,000 MNIST and FashionMNIST, and on 50,000 CIFARIO and CIFARI00 images. Scrambling destroys the correlations existing between values of neighboring pixels, removing the locality of features in images. As result, CNN need to encode more patterss, as each image patch has a different correlation pattern.
Test Model Architectures We conduct controlled experiments, with one (Fig. 2) or two (Fig. 3) hidden layers being either L-cony or baseline, followed by a classification layer. For CNN, L-conv and L-conv with random Li, we used n m- = 32 for number of output filters (i.e. output dimension of W*). For CNN we used 3 x 3 kernels and equivalently used пy = 9 for the number of I, in L--covv and random I--covv As described in sec. 3, we encode I. as sparse matrices I,; = U,V; with hidden dimension d, 16 in Fig. 2 and d, = 8 in Fig. 3, showing that very sparse L,₃ can perform well. The weights W are each my x m4+1 dimensional. The output of the L--oov layer is d x m4+1. As mentioned above, we use two FC baselines. The FC in Fig. 2 and FC(~-L-conv) in Fig. 3 mimic L-conv, but lacks weight-sharing. The FC weights are W = ZV with V being (mr,d)) x and Z being (m4+1 x d) x d,, For "FC (shallow)" in Fig. 3, we have one wide hidden layer with u = RL-cooe/ (mde), where Nt is the total number of parameters in the L-conv model, m and (LLccon/(mmee)
the input and output channels, and dl is the input dimension. We experimented with encoding L, as multi-layer perceptrons, but found that a single hidden layer with linear activation works best. We also conduct tests with two layers of L-conv, CNN and FC (Fig. 3), with each L-conv, CNN and FC layer as descried above, except that we reduced the hidden dimension in L, to dj = 8.
Baselines We compare L-conv against three baselines: CNN, random L,, and fully connected (FC). Using CNN on scrambled images amounts to using poor inductive bias in designing the architecture. Similarly, random, untrained L; is like using bad inductive biases. Testing on random L; serves to verify that L-conv's 's performance is not due to the structure of the architecture. and that the L, in L-conv really learn patterns in the data. Finally, to verify that the higher parameter count in L-conv is not responsible for the high performance, we construct two kinds of FC models. The first type ("Fully Conn." " in Fig. 2 and "FC (~ L-conv)" in Fig. 3) is a multilayer FC network with the same input (d x mo). hidden (k X ny for low-rank L,) and output (d x m2) dimensions as L-conv, but lacking the weight-sharing, leading to much larger number parameters than L-conv. The second type ("FC (shallow)" in Fig. 3) consists of single hidden layer with width such that the total number of model parameters match L-conv.
Results Fig. 2 shows the results for single layer experiments. On all four datasets both in the rotated and the rotated and scrambled case L-conv performed considerably better than CNN and the baselines. Compared to CNN, L-conv naturally requires extra parameters encode T:,, but low-rank encoding with rank di < d only requires O(dad) parameters, which can be negligible compared to FC layers. We observe that FC layers consistently perform worse than L-conv, despite having much more parameters than L-conv. We also find that not training the L; ("Rand L-conv") leads to significant performance drop. We ran tests on the unmodified images as well (Supp. Fig 4), where CNN performed best, but L-conv trails closely behind CNN.
Additional experiments testing the effect of number of layers, number of parameters and pooling are shown in Fig. 3. On CIFAR100, we find that both FC configurations, FC(~L-ccov)) and FC(shallow) consistently perform worse than L-conv, evidence that L-conv's performance is nor due to its extra parameters. L-conv outperforms all other tested models on rotated and scrambled CIFAR 100. Without pooling, we observe that both L-conv and CNN do not benefit from adding a secnd layer. This can be explained by Proposition 2, which states that multi-layer L-conv is still encoding the same symmetry group G. only covering a larger portion of G. Our hypothesis is that lack of pooling is the reason behind this, we discuss next.
Pooling on L-cony On Default CIF,R100, CNN with Maxpooling significantly outperforms regular CNN and L-cony, indicating the importance of proper pooling. Interestingly, on rotated and scrambled
8

---

Under review as a conference paper at ICLR 2021
MNISTT Cony,
Hidden Layers
CIFAR10, or F.C. Layers
L-aane
L-sanr
L-cate
Lesnn
TeLt Accuracc
8449
8.82
6489
8375
Retated
Scrambled
Rotated
Strambled
Layers
Layers
Text ACCUACY
-092
Parameet
6.6
Rotated
Rof. Scrembled
Rotated
Scrwmiiedd
Figure 2: Results on four datasets with two variant: "Rotated" and "Rotated and scrambled". In all cases L-conv performs best. On MNIST, FC and CNN come close, but using more parameters.
CIFAR100 Rot. & Strambled
CIFAR100 Default
0.22
7x.10
Test Accuracy
6.20
6.18
Paramie
5w
6.16
3x
6.14
Test
6.12
1x10f
6.10
0.0
0x10t
Leom
Loow (ichhatoon
con- FSr (-com))
comm
Makk Foot
eodt
CNN
Lecanu
(ciphttioo)
for Caol Mad-Maohh Machhod can Max uod cay
comd
ChN
Loam FFCconemnn L-can
Ieg conel Cusa Mostou
Caol Mechool
can
Figure 3: Comparison of and two layer performance of L-conv (blue), CNN without pooling (orange), CNN with Maxpooling after each layer (green). fully connected (FC) with structure similar L-conv (red) and shallow FC, which has single hidden layer with width such that the total number of parameters matches L-conv (purple). The labels indicate number of layers and layer architecture (e.g. "2 L-conv" means two layers of L-conv followed by one classification layer). Left and middle plots show test accuracies CIFAR 100 with rotated and scrambled images, and on the original CIFARI00 dataset, respectively. The plot on the right show the number of parameters each model, which is the same for the two datasets.
CIFAR100, find that max-pooling does not yield any improvement. We believe the role of pooling is much more fundamental than simple dimensionality reduction. On images, pooling with strides blurs our low-level features, allowing the next layer encode symmetries at larger scale. Cohen & Welling 62)16a) showed relation between pooling and coset of subgroups and that strides are subsampling the group to subgroup C G. resulting outputs which are equivariant only under H and not the full G. These subgroups appearing at different scales in the data may be quite different.
However, a naive implementation of pooling on L--cnvv may involve three I., and be quite expensive (see Appendix C). Devising efficient and mathematically sound pooling algorithm for L-conv is future step we are working on.
DISCUSSIONS
We introduced the L-conv layer, a group equivariant layer based on Lie algebras of continuous groups We showed that many familiar architectures such as CNN, GCN and FC layers can be understood as L-conv. L-conv easy to setup and can learn symmetries during training. domains with hidden symmetries we find that an L-conv layer outperforms other comparable baseline layer architectures. L-conv can in principle be inserted anywhere in an architecture. L-conv can also be composed in multiple layers, though Proposition suggest it would be approximating the same symmetry group, and we did not observe significant improvements in performance in two layer tests. However, we believe this is due lack of pooling for L-conv. For CNN images, coarse boarse-graiiing (maxpooling with strides) allow the system to find features at a different scale, whose symmetry may be a subset of the lower scale (Cohen & Welling, 2016a) perhaps new symmetries. Thus, one future work

---

Under review as a conference paper at ICLR 2021
direction is defining proper pooling (see Supp C). Lastly, to ensure L, behave like Lie algebra basis, we need to include regularizers enforcing orthogonality among the Li, which another future direction.
REFERENCES
Fabio Anselmi, Georgios Evangelopoulos, Lorenzo Rosasco, and Tomaso Poggio. Symmetry-adapter representation learning. Pattern Recognition, 86:201-208, 2019.
Alexander Belavin, Alexander M Polyakov, and Alexander B Zamolodchikov. Infinite conformal symmetry in two-dimensional quantum field theory. Nuclear Physics B, 241(2)3333-38, 1984.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010. /00000000//88 2020.
Benjamin Bloem- Redyy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. arXiv preprint arXiv: 1xXiv:9901.00022 2019.
Taco Cohen and Max Welling. Learning the irreducible representations commutative lie groups. In International Conference on Machine Learning, pp. 1755-1763, 2014.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, 201
Taco Cohen and Max Welling. Steerable enns. arXiv preprint arXiv:1612.08498, 2016b.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130, 2018.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on ho- mogeneous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002.12880. 2020.
Brian Hall. Lie groups, Lie algebras, and representations: an elementary introduction, volume 222. Springer, 2015.
Jason Hartford, Devon R Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep models of interactions across sets. arXiv preprint arXiv: 1803.02879, 2018.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: aet::::090000000 2016.
Anthony W Knapp. Lie groups beyond an introduction, volume 140. Springer Science & Business Media, 2013.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. arXiv preprint / /802003690 2018.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gored nets: a fully fourier space spherical convolutional neural network. In Advances Neural Information Processing Systems, pp. 10117- 10126, 2018
Yann LeCun, Bernhard Boser. John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551) 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11) (6(11):2278-2222 1998.
10

---

Under review as a conference paper at ICLR 2021
Diego Marcos, Michele Volpi. Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5048-5057, 2017.
Rajesh PN Rao and Daniel L Ruderman. Learning lie groups for invariant visual perception. In Advances in neural information processing systems, pp. 810-816, 1999.
Siamak Ravanbakhsh. Universal equivariant multilayer perceptrons.. arXiv preprint arXiv:2002.02912, 2020.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing In Proceedings of the 34th International Conference Machine Learning- Vollum 70, pp. 2892- 2901. JMLR. org, 2017.
Jascha Sohl-Dickstein. Ching Ming Wang, and Bruno A Olshausen. An unsupervised algorithm for learning lie group transformations. arXiv preprint arXiv:1001.1021 2010.
Bastiaan Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In International Conference Medical image computing and computer-assisted intervention, pp. 210-218. Springer, 2018.
Maurice Weiler and Gabriele Cesa. General e (2) equ-euuiva steerable cnns. In Advances Neural Information Processing Systems, pp. 14345, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems, pp. 10381-10392, 2018a.
Maurice Weiler, Fred Hamprecht, and Martin Storath. Learning stecrable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 849-858, 2018b.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp- 3391-3401, 2017.
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933, 2020.


Reviews:
[{"review_id": "OYvz-Toq7qH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper meticulously builds a theoretical framework for Lie-algebra convolutional layers and then goes on to show how CNNs, GCNs and FC layers are a special case of L-convs. The paper also demonstrates how the underlying generators can be learnt from data and provide convincing supporting experimental results. The proposed L-conv layers also use much fewer parameters compared to previous works. \n\nKey strengths:\n\nThe theoretical framework developed in this paper, starting from Lie groups and equivariance and invariance definitions is very elegant and convincing. I checked the maths at each step and am convinced that it is correct, to the best of my knowledge. I did need to refer to Hall 2015 though. Intuitively as well as mathematically, it makes sense to me. The comparison to MSR (Zhou 2020) seems fair to me. \nThe experiments, though limited, in the main paper, are quite convincing. The experiments are cleverly constructed and provide enough justification to support the utility of L-conv layers in comparison to CNN and FC layers. \n\nQuestions:\n\nFor Figure 2: For CIFAR100 and FashionMNIST, CNN seems to do better on \"rotated+scrambled\" compared to \"rotated\". What is the reason behind that? This is not seen in any other method or dataset.  \n\nSuggestions for improvements:\n\n1. The paper inherently assumes familiarity with Lie groups/Lie algebra or even exp/log of matrices which I am familiar with, but not all readers will be. Therefore, instead of citing Hall 2015, it would be good to cite Sections within the textbook. This will aid uptake of an important mathematical sub-field. \n2. There is no mention of accompanying code in the manuscript. Would the authors consider making it available upon acceptance? It would help further research in this area. \n3. The section on linear regression (3.1) seems to occur again in an expanded form in the supplementary material (Sec C). The derivation in the supplementary material was a little bit clearer. \n4. It would be worthwhile checking the paper for typos. A few that I noted: larest, wight, \"a too many\"\n5. Figure 1 is not easy to interpret. A substantial caption would be beneficial. It's also not referenced in the text. \n\nOverall comments:\nThe main contribution of this paper is the development of the theoretical framework for Lie-algebra convolutions. The paper does so very convincingly and I regard this as an important contribution to the area of deep learning. This may open the door to the field using the correct inductive biases for many problems in vision, speech and physics. I enjoyed reading this paper, including the supplementary material that makes a start on imposing orthogonality during regularization for complex-valued neural networks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GdyH1eNDoqA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to making deep learning robust to arbitrary symmetries. To this end, the authors introduce a group equivariance architecture of neural networks which ensures that the function learned by the network has the equivariance property with respect to the symmetry group action (roughly speaking, it ensures that the symmetry affects the input and output of the function in the same way, or that the application of the symmetry does not influence the how the function works).\n\nThe building block of the group equivalence architecture is the L-conv layer, i.e. a layer which models a Lie algebra. The L-conv layer works somewhat like the convolutional layer, but instead of using the kernel mechanism, it multiplies the previous layer by L_i which is the basis spanning the Lie algebra around the corresponding Lie group’s identity element. The authors relate to the condition (provided in existing literature) for a feedforward network layer to be equivariant under a group action of G, and show that the L-conv layer satisfies this condition (Propositions 1 and 2).\n\nSubsequently, it is shown that fully connected layers, convolutional layers as well as graph convolutional networks can be expressed with L-conv layers. Further, the authors compare their approach to a very similar one submitted concurrently to ICLR. They also consider the special case of linear regression and show that the symmetry Lie group can be described analytically. Finally, experiments comparing the L-conv based architecture to CNNs and fully connected layers on standard deep learning benchmark data subject to rotation and random permutations are presented.\n\n\n*****Strengths:*****\n\nModelling arbitrary symmetries and equivariance / invariance relations with continuous groups is an important topic that has sparked much interest recently in various application areas from computer vision to computational chemistry. The work presented in this paper is certainly relevant.\n\nI found the paper well organised and easy to follow.\n\nWhile the approach presented here is similar to a couple of papers reviewed in related work (especially Zhou et al.), I think that the description of the symmetry relations in the language of Lie groups and Lie algebra bases is novel and well presented here.\n\n\n*****Weaknesses:*****\n\nThe main weakness of this paper, in my view, is that no description of L_i learning is provided. To some extent, I am not sure if I understand the process correctly after reading the paper. While the theory appears to guarantee that correctly trained L_i are able to produce L-conv layers acting as fully connected layers, convolutional layers or graph convolution networks while ensuring the equivariance property, no analysis is given as to how feasible it is to find the correct basis L_i. In my understanding, the entire contribution of the paper hinges on the ability to find L_i and thus it should be given more space in the paper. The only comment on this is given in the experiment section, where the authors mention an additional autoencoder being employed. Meanwhile, from section 2 and equation 6, one could guess that L_i could be learned directly by sgd. If the latter is not the case, I think a discussion as to why not would make the paper clearer. If there are other theoretical results on the structure / properties of Lie algebra bases that the authors make use of, I think they should be provided too.\n\nIn the experiment section, the only experiment concerns rotation as the modelled symmetry. As the goal of the paper is modelling arbitrary symmetries, i.e. all functions with the equivariance property wrt some Lie group, would it not be more convincing to devise an arbitrary set of transformations (satisfying equation 1) without a simple analytical form (as in the case of rotation) and show that one can guarantee robustness against this arbitrary set? The current experiment could be beaten by any approach parametrising rotational symmetry.\n\n\n*****Questions / feedback:*****\n\nSee weaknesses section.\n\n\n*****Typos:*****\n\nTheorem 1 (the third sentence) should be rephrased -> for this problem to be equivariant?\n\nEquation 15: so -> SO\n\nSection 5: fing -> find\n\n\n*****Post Rebuttal*****\n\nI would like to thank the authors for their comments. They clarified most of my doubts regarding the paper, most importantly about learning the basis $L_i$. While I agree with reviewer 3 that there is room for improvement when it comes to experimental evaluation, I appreciate the changes made during the rebuttal period and keep my original score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GZr5KPY-OoH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper presents a new approach for building group-equivariant neural networks. The authors propose *L-conv*, a layer which is equivariant to transformations from a group $G$ in the neighborhood of identity. They show that a network with a sufficient number of such layers is $G$-equivariant as a whole. Additionally, the authors propose to learn the structure of the group directly from the training data instead of incorporating it in the network *a priori*.\n\n### Strengths:\n1. The paper is interesting and is easy to follow. The problem the authors are solving is clearly stated from the very beginning and is described in the title. \n2. Proposition 1 is insightful. Equations 4 and 5 contain a novel parametrization of convolutional kernels. It also allows considering standard convolution from a different perspective.\n3. Proposition 2 reveals a useful connection between the  Lie algebra-Lie group correspondence and a network of *L-conv* layers.\n\n### Weaknesses:\n1. **Experiments**.  The conducted experiments do not demonstrate the advantage of the proposed models over other models that use the power of data symmetry. The advantage of the proposed models over conventional CNN is not properly demonstrated.\n    * The decision of using very shallow networks is not clear. An experiment with a 1-layer CNN is not directly generalizable to deeper networks. \n    * Paragraph **Learning $L_i$ during Training** does not contain enough information for a clear understanding of the chosen models. A diagram of the chosen models or a table could make it more clear.\n    * Figure 2, MNIST. The *L-conv* model contains 2 times more parameters than the standard CNN. It demonstrates $\\approx 95$ \\% while the standard CNN demonstrates $\\approx 93$ \\%. It is not clear whether the improvement is caused by the increased number of trainable parameters or by the proposed layer. The same argument is applicable to the CIFAR10/100 and the FashionMNIST experiments.\n    * There are no demonstrations of the learned generators $L$. A demonstration of it will make it possible to understand whether the learned structure matches the known structure of the data. For example the rotation operator for Rotation MNIST. \n    * It is worth understanding whether a network of *L-conv* layers converges to a group equivariant network or proposition 2 is purely theoretical. \n2. Paragraph **Notation** and the proposed notation itself do not seem useful. \n    * The Einstein summation rule is used only in Equation 4 which can be easily rewritten with $\\sum$. The rewritten version will be more readable. \n    * Equation 2 is not used in the paper at all.  \n    * Some letters are used multiple times while meaning different things. For example $f$ is a structure constraint in Equation 2, it is a neural network in paragraph 1 of Section 2, and it is a point-wise activation function later in the same Section.\n\n### Decision\nTo sum up, the paper proposes an interesting method for building group equivariant neural networks which allows for estimating symmetries from the data instead of incorporating it in the network *a priori*. However, the experimental results are not sufficient for proving the advantage of the proposed approach. The theoretical contribution of the paper is not sufficient for considering it as pure-theoretical. The paper seems raw for acceptance at the current stage.\n\n\n# Revision\nAfter a fruitful conversation with the authors I am changing my decision.\nThe conducted experiment presented in Figure 4 confirms an advantage of L-conv.\nThe paper has a room for improvement. However, it contains valuable discoveries and can be clearly placed in the field of symmetry-aware models. The proposed approach is well-described and is properly evaluated on shallow networks. While the generalization of the results to deeper models is not possible (and the reasons are described by the authors correcly), the approach is interesting by itself and is important to the field.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision for this paper is quite difficult: the methodological ideas are interesting, such as learning the generators directly, but the experimental results are relatively weak. Perhaps learning the generators is too unconstrained. Moreover, the proposed 'L-conv' builds heavily on prior work such as the 'LieConv' model of Finzi et. al, which is not made very transparent in the current narrative. And LieConv does provide better performance than L-Conv --- this should also be made clear in the text. This was a very difficult call, and after extensive discussion, the decision is intended to be in the long term best interests of the paper. The ideas are interesting and warmly appreciated, the reviewers appreciated aspects of the response, and the project is sufficiently promising that it was felt that their impact would be much greater if the experimental execution were strengthened, such that the project largely terminating at this point would do it a disservice in the long run. There was a general feeling that this is still a 'work in a progress' and was somewhat rush-written. The authors are strongly encouraged to continue pursuing this work, strengthening the experiments and narrative,  as above.
Decision: Reject