Efficient Equivariant Network
Authors: Lingshen He
Yuxuan Chen
Zhengyang Shen
Yiming Dong
Yisen Wang
Zhouchen Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: equivariance
efficient

Abstract: Convolutional neural networks (CNNs) have dominated the field of Computer Vision and achieved great success due to their built-in translation equivariance. Group equivariant CNNs (G-CNNs) that incorporate more equivariance can significantly improve the performance of conventional CNNs. However, G-CNNs are faced with two major challenges: \emph{spatial-agnostic problem} and \emph{expensive computational cost}. 
In this work, we propose a general framework of previous equivariant models, which includes G-CNNs and equivariant self-attention layers as special cases. Under this framework, we explicitly decompose the feature aggregation operation into a kernel generator and an encoder, and decouple the spatial and extra geometric dimensions in the computation. Therefore, our filters are essentially dynamic rather than being spatial-agnostic. 
We further show that our \emph{E}quivariant model is parameter \emph{E}fficient and computation \emph{E}fficient by complexity analysis, and also data \emph{E}fficient by experiments, so we call our model $E^4$-Net. Extensive experiments verify that our model can significantly improve previous works with smaller model size.
Especially, under the setting of training on $1/5$ data of CIFAR10, our model improves G-CNNs by $5\%+$ accuracy,
while using only $56\%$ parameters and $68\%$ FLOPs.

Efficient Equivariant Network
Lingshen He', Yuxuan Chen', Zhengyang Shen', Yiming Dong', Yisen Wang Zhouchen Lin 12,44 Key Laboratory of Machine Perception (MOE), School of Artificial Intelligence, Peking University "Institute for Artificial Intelligence, Peking University
School of Mathematical Sciences and LMAM, Peking University
Pazhou Lab, Guangzhou 510330, China i ingshenhe@pku. edu. en, yuxuan chen1997@oure shenzhyēpku. edu. en, yimingdong milouuloce com,
com,
yisen wangēpku. edu, en, zlin@pku. edu.
Abstract
Convolutional neural networks (CNNs) have dominated the field of Computer Vi- sion and achieved great success due to their built-in translation equivariance. Group equivariant CNNs (G-CNNs) that incorporate more equivariance can significantly improve the performance of conventional CNNs. However, G-CNNs are faced with two major challenges: aratial-agnostie problem and expensive computational cost. In this work, we propose general framework of previous equivariant models, which includes G-CNNs and equivariant self-attention layers special cases. Un- der this framework, we explicitly decompose the feature aggregation operation into kernel generator and encoder, and decouple the spatial and extra geometric dimensions in the computation. Therefore, our filters are essentially dynamic rather than being spatial-agnostic. We further show that our Equivariant model is parameter Efficient and computational Efficient by complexity analysis, and also data Efficient by experiments, so we call our model - Extensive experiments verify that our model can significantly improve previous works with smaller model size. Especially, under the setting of training on 1/5 data of CIFAR1O, our model improves G-CNNs by 5%+ accuracy, while using only 56% parameters and 68% FLOPs.
Introduction
In the past few years, convolutional neural networks (CNNs) have been widely used and achieved
superior results on multiple vision tasks, such as image classification [3]] 55 51 22), semantic segmentation [ and object detection 44 compelling explanation of the good performance of CNNs is that their built-in parameter sharing scheme brings in translation equivariance: shifting an image and then feeding it through CNN layer is the same as feeding the original image and then shifting the resulted feature maps. In other words, the translation symmetry is preserved by each layer. Motivated by this, Cohen and Welling (9] proposed Group Equivariant CNNs (G-CNNs), showing how convolutional networks
be generalized to exploit larger groups of symmetries. Following G-CNNs, researchers have designed new neural networks that are equivariant to other transformations like rotations (9 6l 24 49] and scales 665 533 However, G-CNNs still have two main drawbacks: 1) In the implementation, G-CNNs would introduce extra dimensions to encode new transformations, such rotations and scales, thus have very high computational cost. 2) Although G-CNNs achieve group equivariance by sharing kernels, like vanilla CNNs, they lack the ability to adapt kernels to diverse feature patterns with respect to different spatial positions, namely, the spatial-agnostic
problem [68 39 70 71 54 67 36
*Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Some previous works focus on solving these two problems. Cheng et al. 4 proposed to decompose the convolutional filters over joint steerable bases to reduce model size. However, it essentially G-CNNs which still have the inherent spatial-agnostic problem. To incorporate dynamic filters, one solution is introducing attention mechanism into each convolution layer G-CNNs without disturbing inherent equivariance [48 The cost is that they introduce extra parameters and increase the complexity of space and time. Another solution is to replace group convolution layers with stand- alone self-attention layers by designing specific position embedding to ensure equivariance 447 26). However, the self-attention mechanism suffers from quadratic memory and time complexity, because it has to compute the attention score at each pair of inputs.
Actually, Cohen et al. 7. Kondor al. [29] and Bekkers [I] revealed that an equivariant linear layer is essentially a convolution operation. Inspired by this, we further discover that general feature-extraction layer, either linear or non-linear, being equivariant is equivalent to that the feature aggregation mechanism between each pair of inputs only depends on the relative positions of these two inputs. Based on this observation, we propose a generalized framework of previous equivariant models, which includes G-CNNs and equivariant attention networks as special cases. Under this generalized framework, we design a new equivariant layer to conquer the aforementioned difficulties. Firstly, to avoid quadratic computational complexity, the feature aggregation operator is explicitly decomposed into a kernel generator and an encoder which takes one single feature as the input. Since our kernels are calculated based on input features, they are essentially dynamic rather than being spatial- -antialaggnostcc. In addition, we decouple the feature aggregation mechanism across spatial and extra geometric dimensions to reduce the inter-channel redundancy in convolution filters 4 and further accelerate computation. Extensive experiments show that our method can process data very efficiently and perform significantly better than previous works using lower computational cost. As our method is parameter Afficient, computational Afficient, data Efficient and Equivariant, we name our new layer as layer
We summarize our main contributions as follows:
We propose generalized framework of previous equivariant models, which includes G- CNNs and attention-based equivariant models as special cases.
Under the generalized framework, we explicitly decompose the feature aggregation operator into kernel generator and encoder, and further decouple the spatial and extra geometric dimensions to reduce computation.
Extensive experiments verify that our method is also data efficient and performs competi- tively with lower computational cost.
2
Related Work
Vanilla CNNs 34
naturally translation equivariant. More symmetries considered to be
exploited into the network for different tasks, such as rotations over plane 9
66
35|
12
61
59 49
37
52
4 41
10
58 21 241, rotations over 3D space [62 16 57 64
14 60
15
50
28
6 scaling
65
40,
53 46,, symmetries on manifold [8] [1. and other general symmetry groups 17 56 18]
These works accomplish equivariance by constraining the linear mappings in layers, followed by pointwise non-linearities to enhance their expressive power. In general, researchers [29] 7 pointed out that equivariant linear mapping can always be written G-CNNs in practice. However, their theory is still limited to linear cases.
convolution-lin integral, i.e.,
point out the spatial-agnostic problem of CNNs and attention achieve impressive results on various vision tasks, researchers start to consider non- linear equivariant mapping. Romero al. [48 45) directly reweighted the convolution kernels with attention weights generated by features and obtained non-linear equivariant models. However, compared with G-CNNs, these methods introduce extra parameters and operations, resulting in an even heavier computational burden. Also, some works [47] 26 23 proposed group equivariant self-attention 43 [3.. Fuchs et al. [19] incorporated self atf-attention into 3D equivariant networks and proposed (E(3)-Tranffomeet However, since their filters are essentially calculated
As works 68
39
70
71
54
67
36
mechanisms 25
63
13
20
based on pair of inputs, the computational complexity is quadratic.
In this work, we further extend the linear equivariant theory to more general situation, including lon-linear cases. Under the framework, we design a new equivariant layer to solve both the spatial-

---

agnostic problem in convolution -basdd equivariant models and heavy computation cost problem in most equivariant models.
3 A Unified Framework of Previous Group Equivariant Models
In this section, we first briefly review two representative group equivariant models: the linear model G-CNNs [9), and the non-linear model equivariant self-attention [47 26). Then, we propose general framework of previous equivariant models based on the inner relationship among these specific models.
3.1 Equivariance
Equivariance indicates that the outputs a mapping transform in predictable way with the transfor- mation of the inputs. Formally, a group equivariant map v/ satisfies that
Yu E g, TD [Tu[f]] = [T]]
(1)
where G is transformation group, f is an input feature map, and Tu and Th are group actions, indicating how the transformation u acts on the input and output features, respectively. Besides, since we hope that two transformations u, U E G acting on the feature maps successively is equivalent to the composition of transformations we Tu To
acting on the feature maps directly, we require that Twe. where uv is the group product of u and v. The same is the case with T.
Now examine the specific form of the transformation group G. In this work, focus the analysis of 2D images defined on RR2 Consequently, we are most interested in the groups of the form IR2 x A, resulting from the semi semi-product (>4) between the translation group IR²2 and group A acts on TR?2, e.g., rotations, scalings and mirrorings. This family of groups is referred to as affine groups and their group product rule is:
(Ius (I) av) (I + Quioes a,,a),,
where (Zu, a.) and v = (Ir, an), in which Tu. E RR2 and au, A. For ease of implementation, following (9). we take A as the cyelic group C4 or the dihedral group D4, then g becomes p4 or p4m. As for the group action, we employ the most common regular group action in this work, i.e.,
[.) f(u-'o))
(3)
Here, we only care about the group action over the feature maps defined on G. because we always use lifting operation to lift the input images defined on IR² to the feature maps on G, where the equivariance can be preserved properly, will be shown in Section 3.2
3.2 G-CNNs
Let p()) x - C and W - RC+IXCC be the input feature and the convolutional filter in the 1-th layer, respectively, where Cj denotes the channel number of the I-th layer. X' is taken as I2 for the first layer, and taken as G for the following layers. Then for any g E G. the group convolution
29 of f()) and W on 9 at g is given by
1/- W(g Wig (d)p
(((+))()) v (()
(4)
where va()) is the Haar measure. When X' is discrete, Eqn. 44 can be rewritten
p((+1) (g) Σ W (gg ")))
(5)
TEX
G-CNNs essentially generalize the translation equivariance of conventional convolution to a more general group G.
In fact, the first layer maps the 2D images to function defined on G, while the following layers map one feature map on G to another. As result, the computational complexity of the first layer and the following layers are of the order (122 and (A*a respectively, where & the kernel size in the spatial space. As result, G-CNNs have much larger computational cost when A is large, especially for the intermediate layers. In this work, we employ the first layer of G-CNNs as a lifting operation, and focus on reducing the computation of the latter layers.
3

---

3.3 Equivariant Attention Networks
Group Equivariant seff-Attention (G-SA) 47 26 is a representative method of equivariant attention networks, whose form can be simplified as follows:
(())()))
Σ Softmax ( ( (b)) (f (T) + Pg-ig)]hy (P)) DEG
where hy : Ci + RC1+1 and hg CR - TRd are the embedding functions of values, querys and keys, respectively, which are neural networks in the most general case. is the dimension of the low dimensional embeddings, and Pg ig E Igd encodes the relative positions of the query (()) (g) and the key f()) (g)..
3.4 Generalized Equivariant Framework
As more and more group equivariant structures emerge, researchers start to deduce the most general cquivariant structures. To this end, Cohen et al. (7]. Kondor et al. [29] and Bekkers [ proposed general theory of linear group equivariant structures, which indicates that G-CNNs are the most general equivariant linear layers. Besides, a lot of non-linear equivariant structures appear recently, such equivariant self-attention layers 47 26 framework.
This motivates us investigate more general
In all, with only slight modification, most of layers in neural network can be viewed as kind of aggregation of pair-wise feature interaction as follows:
(g)
Σ H (g). GE9
where the feature aggregation operator (. RC x RC + R is a mapping indexed by pair of location g and g. which describes how aggregate the input feature pair f(g) and f(g). In general, the above layer is not equivariant. However, we can find a general constraint for H (f (g), (1) (g) to make this layer equivariant over G.
Theorem The layer formulated as Eqn. 7 is group equivariant if and only if there is a mapping Hg RC x RCi- + R which is indexed by a single group element 3. such that, VP(i) and
Vg. 9 E G. the layer satisfies:
Σ H (P (g), (9) (G))
Σ H.
(1),
(8)
Proof + Firstly, Vu,g and 7 E g.
(1) = -(l+1)(u = Σ
H g (f00 p)
3E9
On the other hand,
Σ H (T. (g), TT) (7) Σ Hyg (1 9), (u) = Σ (r (u- (())))()))))))) (7).
JEG
DEG
Geg
As TTt(++)(())
= =Ezeg H ( (1) T,f (g)
(P((), g.
Σ
H Hl y.ug (1()) (u 9), ((() (g) Σ
Hu (w-g)),) (
gEG
nEG
Let g > ug, we get:
VP() g. u,
Etag (f (g), (F)) (T))
gEg
gEG

---

then, we let u to be geg" g=
AP ,9,
Σ
He - - ((()))
p ())
Has ()) ( ( ()).
GEg
BEG
We denote H. (: ())) as He, ) we can get exactly the Eqn.(8)
* This is obvious. Q.E.D
From the theorem, we can get group equivariant layer:
((+
Σ GEG
which is also the only equivariant form of Eqn. 77. Actually, the above theorem also reveals the essence of equivariance in previous works, i.e., if the relative positions of (g1-91) ) and (g2-92) are the same, i.e., 91 -1 'gi- 92 92 = g. the feature pairs located the two tuples should be processed equally. In other words, we should employ the same function Hg to act on these two input feature pairs.
From this perspective, we can readily see that both the kernel sharing used in G-CNNs, Eqn. 4 and the relative position encoding adopted in the G-SA, Eqn. (6), utilizes the above rule. According to Theorem [] designing group equivariant layer becomes much more easily and flexibly than ever, as we only need to design a new function Hg. In addition, the new formulation provides a more general perspective on the group equivariant layer, i.e. sharing the parameters of function Hlg which generalizes the kernel sharing schemes in G-CNNs. Based on the above understanding, we can see that if we replace the feature vector in the right hand side of Eqn. (9) with the local patch at group element and g. respectively, it is still equivariant. Proposition 1
The following layer equivariant,
p((+t)())
(g)
(10)
JEG
where for i == 1,2, the FN,ts) denote the local patches of g. in which Ni(g) represent g's neighbor- hood (9g'lg' E [e(e) and N((e) is the predefined neighborhood of the identity element e E G.
One remarkable advantages of introducing local patch is that contains more semantic information than feature vector. Notice, we acquire the local patches by concatenating features in the neighbor- hoods of g and ÿ predefined order on NG (e) and N2(e) respectively, i.e., J'(g") is concatenated at the same place in Fxi(g) fg "='1 g') in (e)- We denote the concatenation operator as U. and will discuss the above in detail in Section 4.1] which shows that concatenating features can not only make our framework more flexible, but also help to reduce the computational burden of our newly proposed equivariant layer.
Efficient Equivariant Layer
A straight-forward and easy case of Eqn. 110 is to adopt Hg. Vg G, as multi-layer perceptron (MLP), where the subscript ğ is used to identify different MLPss However, Eqn. (10), we have to compute mapping from two high dimensional vectors another high dimensional one for each input pair of and g. which very expensive. A similar issue cxists in computing the attention score in self-attention. To deal with this problem, we decompose H into the following form to reduce the computation, i.e.,
vg E,
Hz(x, v) = Kg(x) o V
(11)
where @ means dlement-wise product, and Kg RGNNG() + RC+1 is a kernel generator and V (e + Ci++ is an encoder. We use to denote the numbers of elements in set. Hence, we can compute Kz(x) and Vyy) separately. In addition, to further save computation, we split the kernel into several slices along the channels, such that K3 is shared across these slices,
5

---

(g)
Resize
F(++1) (g)
MLP
Linear
Transform,
MLP Resize
Transform,
MLP
Resize
Linear
Transform
0
0
Resize
MLP
Linear Transform
U Rolation-wise concel
Element-wiie product
Spalial-wise aggregation
An example of our E llyer on p4 group. We firstly concatenate features along rotation dimension in a predefined order on the C4 and then pass them through the MLP and the linear layer to generate kernel and encode features, respectively. After this, the element-wise product is carried out to compute H. Hgg and finally spatial-wise aggregation is performed to acquire the output. Note that when computing output features at different rotation dimensions, the generated kernel should be rotated by specific degree to keep the correct relative position.
Figure 1:
Le., LA & Lij C4+1, K = (mod s), where s is the number of slices, and and are channel indexes. The K3 is essentially dynamic filters which adaptive to features around avoiding the spatial-agnostic problem in G-CNNs. Unlike conventional dynamic filters, which are matrices, the output of Kg is vector, which can be viewed as depth wise kernel [5]. This can decouple channel dimension with spatial dimension during feature aggregation reduce the computational cost. Position information is implicitly encoded in the organized output form of kernel generator, rather than using explicit positional embedding in the group self-attention layer 26
In practice, we can view the whole kernel family {Kgheeg as the output of a single mapping, i.e., K: (eM + [1 Then, we resize the output of K to be [ x C++1 matrix, with different represent different Kg. Namely, if we adopt K as MIP, the computations and parameters used for hidden layer are shared across K3 for different ÿ. which is another merit of the Eqn. [T]. However, there is still large search space for Hg. as Eqn 11 is only special structue of Hg. leave a more complete study of Hl in the future work.
4.1 Implementation on Affine Group
In this section, design very efficient equivariant layer based Eqn. I² x A. The computation the operator is:
(II) for affine group
p()
Σ gEN(g)
K Lo
U (() g'EN((
ov
U
p (
(12)
Following the standard practice in computer vision, aggregation is done only on the local neighbor- hood of g. (g).. To save computation, we choose \((g) to be only spatial-wise neighborhood, i.e.,

---

N(9) = (g(u, | 22,, where St R² and C is the identity element of group A. However, aggregating information along spatial neighborhood only discards the information interaction along A, which could lead to drop in performance [32. We alleviate the issue by concatenating the feature map along A, i.c., we choose (g) and N2 ( to be {g(0, a) la A). The order of concatenation is predefined on A. As will be shown in the later experiments, this concatenation does not introduce much computation but can significantly improve performance. Compared to group convolution, such design enables to decouple the feature aggregation across the spatial dimension and the A dimension to further reduce computational cost.
In practice, we adopt the K as a two layer MLP: ((x) = We where W1 E RC///xCi |A|, W2 E [RRlsxxi/rr, and r is the reduction ratio which saves both parameters and computation, is the number of slices defined before. For 2D images, Q is usually adopted k x k square mesh grids and [Ω]] = k², where k is the kernel size. We simply adopt the encoder V as a linear transform: Vyy) = Way, where W3 E IR [1++11)]]A]] For better illustration, we visualize a concrete layer of Eqn. 122) by choosing as p4 in Figure 1.
4.2 Computational Complexity Analysis
In practice, the feature map is defined on discrete mesh grids. We use h and w to denote the height and the width of mesh grids. As the numbers of the input and output channels are usually the same, we assume C
C(+1
Parameter Analysis The number of learnable parameters of -ayle 12) csk2/r /r. As < c, parameter counts are dominated by the first term when A: not too large, and increasing kernel size will not significantly increase parameter counts, which is shown in later experiments. The parameters count of group convolution layer is c2 |AA. Notice that (1+1/r) < k2 and s/r < clA,, parameters count of our layer is significantly less than that of group convolution layer.
ea 1/r) +
Time Complexity Analysis The FLOPs of E El-layer and group convolution layer are (1 A/r) hw + (1 + c/AAbe and k2 | respectively. Similarly, (1 1/r) < k22 and (1 + s/r) < clA), the FLOPs -lyye is significantly lower than that of group convolutional layer.
can be observed that both parameter count and FLOPs of our Ea-layer are composed two terms, one depending 1:2 and the other not relying on k; which is result of disentangling across spatial dimension with both channels and A during feature aggregation.
5 Experiments
In this section, we conduct extensive experiments study and demonstrate the performance of our model. The experimental results show that our model has a greater capacity than the group- convolution-based one in terms of parameter efficiency, computational efficiency, data efficiency and accuracy. On the MNIST-rot dataset, we detailedly study the effect hyperparameters on the number of parameters, computation FLOPs and performance of model. All the experiments done the GeForce RTX 3090 GPU.
5.1 Rotated MNIST
The MNIST-rot dataset 33 is the
Table 1: Test error on rot-MNNST(wwihh standard deviation under
most widely used benchmark 5 random seed variations)
test the equivariant models. It
Model
Test error (%)
Params
FLOPs
contains 62k 28 28 randomly ro- tated gray-scale handwritten dig- its. Images in the dataset are split into 10k for training, 2k for vali- dation and 50k for testing. Ran- dom rotation of digits and only 20 percent of training data of the standard MNIST dataset increases the difficulty of classification.
p4 SA [7
2.5410.052
44.67K
400M
p4 CNN 9
1.79++00043
77.54K
46.2M
P4 CCC 45
0.69+
73.13K
27.0M
E NNet (Ours)
1.29-10.023
18.8K
17M
E (Ou())
1.17:10019
41.1K
36.9M
7

---

For fair comparison, we keep both training settings and architectures of our model as close as possible to previous works [7 In addition, we adopt the p4 group to construct all our models in this section. In our first experiment, we adopt our - given in the supplementary material to make a comparison to previous works. This is very lightweight model which contains only 18.8K learnable parameters. It is composed of one group convolutional layer which lifts the image to the p4 group, six E layyass and one fully connected layer. Two 2 x max-pooling layers are inserted after the first and the third E llyer to downsample feature maps. The last E - is followed by a global max group pooling layer [9) which takes the maximum response over the entire group, to ensure the predictions invariant to rotations.
Our model trained using the Adam optimizer 27 for 200 epochs with batch size of 128. The learning rate is initialized as 0.02 and is reduced by 10 at the 60th, 120th and 160th epochs. The weight decay set as 0.0001 and no data augmentation is used during training. The results are listed in T Our models significantly outperform G-CNNs [9] using only about 25% parameters and 40% FLOPs. For G-SA (7]. which is group equivariant stand-alone self-attention model, even performs inferiorly to G-CNNs with much more computational cost. The a-p4-CNN model 45 further introduces the attention mechanism to group convolution along both spatial and channel dimensions to enhance the expressiveness of G-CNNs, while our E*tNet still significantly outperforms it with less computational cost. We also experiment with a larger model to further demonstrate the capacity of our model, which is listed in the last line of Table I
Ablation Study of Concatenation: In the E+- layer (12), we introduce the concatenation oper-
Table 2:
The effect of concatenation
Concate
Test error (%)
Params
FLOPS
ation to enable the disentanglement across the rotation and the spatial information interaction.
None
0.10:
9.9K
8.9M
only K
1.96::0.045
14.4K
13M
To study the importance of concatenation, we carry out experiments on the case that neither K3 nor V Eqn.
only V
1.52:10.036
14.4K
13M
K&V
1.29:-0.023
18.8K
17M
12) use concatenation, i.e., (1 g. (2) g. As shown in the first line of Table 2 this leads to significant drop in performance. This is because if aggregation in Eqn. (12) is done merely in the spatial neighborhoods without concatenation, there is no information interaction along the rotation dimensions. We also experiment the cases using concatenation only in K3 or V, and the performance of both is better than the case without concatenation but is still inferior the case with concatenation in both Kg and V. This further illustrates the importance of concatenation along A.
Elpperaaraeeees Analysis: We investi-
Table 3: Hyperparameters Analysis
gate the effect of various hyperparameters used in the Elylayer. The reduction ratio and the slice number 8 in the K3 and kernel size l control the computations and param- eters of the layer. Based on the baseline model, vary the three hyperparameters respectively. As shown in the Table [ im- provement is observed when decreasing the reduction ratio and increasing the slice num- ber, with the cost of computational burden increasing. Especially, the improvement of over 4 and r 1 over marginal, which is attributes to redundancy in the kernel 4 In conclusion, appropriately increasing the reduction ratio r and decreasing the slice number can help to reduce computational cost while preserving performance. Keeping other hyperparameters fixed, we study the effect of kernel size on our model. In Table 33] the performance peaks when kernel size equals 7. In general, larger kernel size leads to improved performance due to larger receptive field. In addition, as explained in Section 4.2 increasing kernel size does not dramatically increase parameters and FLOPs as standard convolution.
Hyperparam
Test error (%)
Params
FLOPSs
s=1
1.45++0.022
16.3K
14.9M
s=2
0.29++00023
18.8K
17M
s=4
1.24:10.026
23.9K
21.2M
r=1
1.29110.023
18.8K
17M
2
0.33
13.0K
12M
r=4
1.37+
10.1K
9.5M
k=3
1.46+00.031
15.6K
14.3M
k=5
1.29+000233
18.8K
17M
is
k=7
1.27+00021
23.8K
21.1M
5.2 Natural Image Classification
In this section, we evaluate the performance of our model on the two common natural image datasets, CIFARIO and CIFARIO0 (30]. The CIFAR-10 and the CIFAR100 datasets consist of 32 32 images
8

---

Table 4: Test error on CIFARIO and CIFARI00. (with standard deviation under 5 random seed variations)
Model
CIFARIO (%)
CIFARIO0 (%)
Params
FLOPs
R18
9.7
3440077
11M
0.56G
p4-R18
7.53+0021
27.96:t0.56
11M
2.99G
p4-E R18((urs))
6.72:10.14
26.59
5.8M
1,85G
p4m-R18 R18
5.83+0.17
24.95=:0.42
10.8M
5.63G
p4m- E R18((urs)
4.961
22.18::0.46
6.0M
3,87G
belonging to 10 and 100 classes, respectively. Both of the datasets contain 50k training data and 10k testing data. Before training, images are normalized according to the channel means and standard deviations.
In this experiment, we adopt ResNet-18 22 as the baseline model(short as R18), which is composed of an initial convolution layer, followed by 4 stage Res-Blocks and one final classification layer. Following the standard practice in (9) we replace all the conventional layers with p4 (p4m) convo- lutions R18 and increase the width of each layer by VI (v8) to keep the learnable parameters approximately the same. We denote the resulting models p4-R1B 18 (p4m-R R 18). We replace the second group convolution layer in each Res-Block of p4-R18 (44m-R18) with our E' E'ylayer, resulting in the p4-E R18 [p4m-E'RR R18). For a fair comparison, all the above models are trained under the same training settings. We use the stochastic gradient descent with an initial learning rate of 0.1, Nesterov momentum of 0.9 and weight decay of 0.0005. The learning rate is reduced by 5 at 60th, 120th, and 160th epochs. Models are trained for 200 epochs using 128 batch size. No data augmenlation is used during training to illustrate data efficiency of our model.
The classification accuracy, parameters count and FLOPs of all models on CIFARIO and CIFARIO0 are reported in Table 4 We can see that models incorporating more symmetry achieve better performance, i.e., R18 S p4-R18 S p4m-R18. Our p4 and påm models significantly outperform their counterparts on both CIFARIO and CIFAR100. Furthermore, our model decreases the parameter count and FLOPs by 45% and 32%, respectively. Notice that the model size reduction is purely caused by the introduction of our la-lerss, as topological connections and width of each layer of E* model and its counterparts are the same.
Data Efficiency: To further study the performance of our model, we train all the models listed in Table 4 on CI- FARIO with different sizes of training data. To be specific, we consider settings, where 1k, 2k, 3k, 4k and 5k train- ing data of each class are randomly sampled from the
RIS
pd. RIR
p4es--- R18
23
M4-EXx19
Pfm--ERR18
CIFARIO 10 training set. Testing is still performed on the original test set of CIFAR10. Other training settings are identical to the above. We visualize the results in Figure
Test eror (54)
is observed that the performance gap between p4, p4m and R² models tend increase as we reduce the training data. This is mainly because that the prior that the label invariant to rotations more important when training data are fewer. The trend is also observed in the gap be- tween our models and their counterparts. For instance, the gap between p4m-E3 E R188 and p4m-R R18 18 is 0.87% when training data of each class 5k, while it is enlarged 5.22% when training data of each class is reduced to 1k. Especially, we observe the line of R4- intersects with the one of p4m-R18, which further indicates that our model is much more data efficient than G-CNNs. As indicated above, symmetry prior is more important when training data are fewer, and the data efficiency our model implies that p4-EAR EA-ERRI R188 and p4m-E EAR88 R18 can better exploit the symmetry of data.
1k
Triining per cluss
Figure 2: Trend of test error(%) on vari- ous training data sizes.

---

Limitation and Future Work
From the theory perspective, although we extend the general equivaraint framework from linear cases to common non-linear cases, there's s two limitations on the generalization: 1) we only focus on layers with such pair-wise interactions proposed in Eqn. and higher-order interactions cases are not included. 2) We only consider regular group action in this framework, which is special case of general group actions. We leave extending this equivariant framework these cases as future work.
From the practice perspective, we only give special implementation of Eqn.(TO) in an intuitive insight, and further exploration the space of equivariant map in demand. An alternative is to exploit searching algorithms from neural architecture search [42] 38 69] to find more powerful and efficient model. Besides this, our E -laer slower than G-CNN despite less FLOPs due to convolutions are optimized by many speedup libraries. Our layer is implemented only in a naive way, that is, using the unfold operation followed by summation operation for the aggregation step. the future, we will try to implement a customized CUDA kernel for GPU acceleration to reduce training and inference time of our model.
Conclusions
In this work, propose general framework of group equivariant models which delivers unified understanding on the previous group equivariant models. Based on the new understanding, we propose novel efficient and powerful group equivariant layer which serve convolutional layers. Extensive experiments demonstrate the El-layer is more powerful, parameter efficient and computational efficient than group convolution layers and their variants. Through side by side comparison with G-CNNs, we demonstrate our E -lyya can significantly improve data efficiency of equivariant models, which show great potential for reducing the cost of collecting data.
drop-in replacement for
Acknowledgment
Zhouchen Lin was supported by the NSF China (No.s 6253300 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606) and Project 2020BD006 supported by PKU-Baidu Fund. Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund.
References
[1] Erik Bekkers. B-spline enns on lie groups In JCLR, 2019.
[2] Erik Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In MICCAI, 2018.
[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deepl.ab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 834-848, 2017.
[4] Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. RotDCF: Decomposition of convolutional filters for rotation-equivariant deep networks. In ICLR, 2018.
[5] François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.
[6] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In ICLR, 2018.
[7] Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogencous spaces. In NeurIPS, 2019.
[8] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral CNN, In ICML, 2019.
10

---

[9] Taco S Cohen and Max Welling. Group equivariant convolutional networks. In ICML, 2016.
[10] Taco S Cohen and Max Welling. Steerable CNNs. In ICLR, 2017.
[11] Pim De Haan, Maurice Weiler, Taco S Cohen, and Max Welling, Gauge equivariant mesh cmns: Anisotropic convolutions on geometric graphs. In ICLR, 2021.
[12] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting eyclic symmetry in convolutional neural networks. In ICML, 2016.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Carlos Esteves, Christine BlenchBlanchett, Ameesh Makadia. and Kostas Daniilidis. Learning
SO(3) equivariant representations with spherical CNNs. In ECCV, 2018.
[15] Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross- domain 3d equivariant image embeddings. In ICML,
[16] Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniilidis. Equivariant multi-view networks. In ICCV, 2019.
[17] M Finzi, S Stanton, P Izmailov, and AG Wilson. Generalizing convolutional networks for equivariance to Lie groups on arbitrary continuous data. In ICML, 2020.
[18] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups In ICMI, 2021.
[19] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. SE(3)- Transformers: 3D roto-translation equivariant attention networks. In NeurlPS, 2020.
[20] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? In ICLR, 2021.
[21] Simon Graham, David Epstein, and Nasir Rajpoot. Dense Steerable Filter Cons for Exploiting Rotational Symmetry in Histology Images. IEEE Transactions Medical Imaging, 2020.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[23] Lingshen He, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. In NeurIPS, 2021.
[24] Emiel Hoogeboom, Jorn WT Peters, Taco S Cohen, and Max Welling. HexaConv. In ICLR. 2018.
[25] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excinee networks. In CVPR. 2018.
[26] Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. LieTransformer: Equivariant self-attention for lie groups. ICML, 2021.
[27] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR. 2015.
[28] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-Gordan Nets: a fully fourier space spherical convolutional neural network. In NeurIPS, 2018.
[29] Risi Kondor and Shubhendu Trivedi. On the generalization equivariance and convolution in neural networks to the action of compact groups In ICML, 2018.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.

---

[32] Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Mare Pollefeys. TIPPOOLING: transformation-nvve invariian pooling for feature learning in convolutional neural networks. In CVPR. 2016.
[33] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures problems with many factors of variation. In ICML, 2007.
[34] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[35] Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In NeurlPS, 2018.
[36] Duo Li, Jie Hu, Changhu Wang. Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen.
Involution: Inverting the inherence of convolution for visual recognition. In CVPR, 2021.
[37] Junying Li, Zichen Yang, Haifeng Liu, and Deng Cai. Deep rotation equivariant network. Neurocomputing, 2018.
[38] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018.
[39] Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. ECCV, 2020.
[40] Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in enns with vector fields. arXiv preprint a 2018.
[41] Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In ICCV, 2017.
[42] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICML, 2018.
[43] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019.
[44] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[45] David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In ICML, 2020.
[46] David W Romero, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Wavelet networks: Scale equivariant learning from raw waveforms. arXiv preprint arXiv:2006.05259, 2020.
[47] David W Romero and Jean- Jean-Baptist. Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. In ICLR, 2021.
[48] David W Romero and Mark Hoogendoorn. Co-attentive equivariant neural networks: Focusing cquivariance on transformations co-occurring in data. In ICLR, 2020.
[49] Zhengyang Shen, Lingshen He, Zhouchen Lin, and Jinwen Ma. PDO-eConvs: Partial differential operator based equivariant convolutions. In ICML, 2020.
[50] Zhengyang Shen, Tiancheng Shen, Zhouchen Lin, and Jinwen Ma. PDO-eS2CNNs: Partial differential operator based equivariant spherical CNNs. In AAAI, 2021.
[51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
12

---

[52] Bart Smets, Jim Portegies, Erik Bekkers, and Remco Duits. Pde-based group equivariant convolutional neural networks. arXiv preprint 1xXiv:2001.0904: 2020.
[53]
Ivan Sosnovik, Michal Szmaja, and Amoll Smeulders. Scale-equivvarian steerable networks. In ICLR, 2019.
[54] Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz. Pixel-adaptive convolutional neural networks. In CVPR, 2019.
[55] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[56] Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. In ICML, 2019.
[57] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an canslltion-equivariiin neural networks for 3d point clouds. In NeurIPS, 2018.
[58] Bastiaan S Veeling. Jasper Linmans, Jim Winkens, Taco S Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In MICCAI, 2018.
[59] Maurice Weiler and Gabriele Cesa. General E(2)-equivariant steerable CNNs. In NeurIPS, 2019.
[60] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. In NeurlPS, 2018.
[61] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In CVPR, 2018.
[62] Marysia Winkels and Taco S Cohen. Pulmonary nodule detection in scans with equivariant cnns. Medical Image Analysis, 2019.
[63] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018.
[64] Daniel Worrall and Gabriel Brostow. CubeNet: Equivariance to 3D rotation and translation. In ECCV, 2018.
[65] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In NeurlPS, 2019.
[66] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Har- monic networks: Deep translation and rotation equivariance. In CVPR, 2017.
[67] Jialin Wu, Dai Li, Yu Yang. Chandrajit Bajaj, and Xiangyang Ji. Dynamic filtering with large sampling field for convnets. In ECCV, 2018.
[68] Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. In NeurIPS, 2019.
[69] Yibo Yang, Hongyang Li, Shan You, Fei Wang, Chen Qian, and Zhouchen Lin. Ista-nas: Efficient and consistent neural architecture search by sparse coding. In NeurIPS, 2020.
[70] Yikang Zhang, Jian Zhang, Qiang Wang, and Zhao Zhong. Dynet: Dynamic convolution for accelerating convolutional neural networks. arXiv preprint arXiv:2004.1069, 2020.
[71] Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuan Yang. Decoupled dynamic filter networks. In CVPR, 2021.
13


Reviews:
[{"review_id": "kozGE1lUeRu", "reviewer": "Reviewer_y9u4", "summary": "This paper presents a general framework for group equivariant models which generalizes group equivariant CNNs (G-CNNs) and equivariant self-attention layers. For practical use, the paper proposes a particular framework, called E4-Net, which can significant improve previous G-CNNs with smaller model size.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Two main contributions of the paper are as follows:\n- The authors propose a general framework of group equivariant models which generalizes G-CNNs and equivariant self-attention layers. The authors suggest a layer $\\psi:\\mathbf{f}^l \\mapsto \\mathbf{f}^{l+1}$ where $\\mathbf{f}^{l+1}(g)$ is defined to be a sum over $\\tilde{g} \\in \\mathcal{G}$ of values of a function, say $H$, at $g,\\tilde{g},\\mathbf{f}^l(g)$ and $\\mathbf{f}^l(\\tilde{g})$. It is proved in Theorem 1 of the paper that $\\psi$ is equivariant if and only if $H$ can be viewed as a function at $g^{-1}\\tilde{g},\\mathbf{f}^l(g)$ and $\\mathbf{f}^l(\\tilde{g})$.\n- A special case for the function $H$ is used to construct E4-Net, a new group equivariant layers. In E4-Net, the function $H$ is considered to be a point-wise product of a kernel generator $K_{g^{-1}\\tilde{g}}(\\mathbf{f}^l(g))$ and an encoder $V(\\mathbf{f}^l(\\tilde{g}))$. Experiments on Rotated MNIST and CIFAR shows that E4-Net outperforms classical G-CNNs with less numer of parameters and time complexity.\n\nThe results are novel and interesting. The paper is well-written. Besides, I have some comments as follows:\n\n1. The construction for the kernel generator $K$ is described in detail in page 5. But I do not see any clear construction for the encoder $V$. It seems to me from Figure 1 that the encoder $V$ is chosen to be a linear map. If yes, is there any reason for choosing $V$ to be a linear map rather than an MLP?\n\n2. An explanation for the relation of E4-layer with G-Conv layer and G-SA layer is needed.\n\n3. In experiments, comparisons of the accuracy of E4-Net with G-CNNs are reasonable. Why do not you compare with G-SA neural nets?\n\nMinor comments:\n- page 4, line 132: \"querys\" --> \"queries\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "T8H-IkGN53", "reviewer": "Reviewer_F1d8", "summary": "This paper introduces a functional abstraction for group-convolutional networks employing first- and second-order features. Based on this functional abstraction, a new functional form for second-order group-convolutional networks is devised, which can be made more parameter-efficient than existing approaches.\nExperiments are performed on rotated MNIST and CIFAR, evaluating error, number of parameters and certain ablations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I enjoyed reading about the generalization idea, I feel that there is a certain amount of imprecision hampering it.\n\nSome general comments and questions:\n\n- The statement that the authors \"propose a generalized framework of equivariant models\" could be construed as having found the most general formulation. This is certainly not the case, given higher-order interactions that cannot be framed this way.\n\n- The \"spatial-agnostic problem\" is not a problem. It is just an architecture choice.\n\n- Most CNNs, and incidentally the network proposed in this paper, are not actually equivariant to pixel shifts due to the pooling\n\n- the formulation of the kernels with neighborhoods \\mathcal N(g) is a bit vague. It should be made clear that the precise indexing of the elements of the neighborhood is important if one does not want to resort to permutation invariant functions such as averaging. It is mentioned briefly when incorporating group \\mathcal A in the example. A further question would be: It would seem that these neighborhoods need to exhibit some form of transformability as well. Can this be simply characterized? At some points, MLPs are proposed to handle whole neighborhoods. How would these be made to transform correctly?\n\n- The experiments on CIFAR are a bit contrived -- please compare to fully data-augmented CNNs. If this approach is to be shown to be useful, it should be able to perform at the level of data augmentation using the symmetries encoded in the network. Data augmentation may also help in the proposed architecture because of the lack of shift-invariance below aggregated stride size.\n\n- A demonstration of object recognition on ImageNet would truly show this approach to be efficient. This benchmark is also a form of water-shed - many architectures work on CIFAR that do not work on ImageNet\n\n- the symmetry groups chosen are not very interesting. It would have been nice to see some more non-trivial rotation at least.\n\n- style suggestion: s/Besides, //g", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not really discussed.\nI do not foresee potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7nil985bMC_", "reviewer": "Reviewer_3YmE", "summary": "The paper describes a generalization of group equivariant neural networks that typically rely on linear operators such as the group convolution, or pseudo linear operators such as attentive group convolution. The authors take a look at the aggregation function of the convolution operator (which is usually linear via a kernel times feature values followed by sum aggregation) and replaces it with a non-linear function that generally could depend not just on relative positions, but also on the central and neighbor feature values. The result is a more non-linear operator than group convolutions, which are a special case of this framework.\n\nThe paper is accurate and has an appropriate experimental section with decent ablation studies. The proposed work compares favorably compared to the baselines both in terms of accuracy, parameters and nr of operations (FLOPs).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper describes a great idea of modifying the main workhorse of G-CNNs (the group convolution) to something non-linear. Also the paper focusses on efficiency which is a relevant aspect of G-CNNs. The paper performs a decent set of abblation studies, though these are done on MNIST, so it may not generalize to more challenging datasets, but the method is further validated on CIFAR10 and CIFAR100. Since the paper focussed on efficiency it was a pity not to see the method being applied to problems that actually require efficient implementations (such as e.g. imagenet). I think this is one of the main limitations of the paper, but up to some other concerns (see below), I think it is a great paper.\n\nI think he paper could mainly improve by presenting additional intuition behind the proposed method and discuss the main mechanism that motivates why it should work better (see second upcoming concern). This could possibly be achieved by explicitly framing conventional methods as special cases of this framework, which enables to precisely point to differences that could explain why the current method should work better. I find this important particularly because the method is presented as a generalization of other works, and it would be great to explicitly see related works  as special cases. I think this would help interpretation. \n\nAll in all I have the following comments:\n\n[Thm 1 + proof | concern about input dependency on g and \\tilde{g}]\nI think the construction is not very transparent with respect to the dependency of of H to it’s inputs. Namely in equation 8, the inputs of H_{g,\\tilde{g}} are denoted as arbitrary open “slots” (with \\cdot) for arbitrary inputs. However, these inputs in practice do depend again on g and \\tilde{g}, as given in equation 7. With this in mind I don't think the proof is formally correct and is in my opinion even misleading. Though when explicitly taking the dependency into account the theorem and proof is valid . After 154 I suggest not writing \\cdot for the inputs but f(g) and f(\\tilde{g}) respectively. (Or please correct me if my concern is ill founded).\n\nRelated to this concern: In equation 11 x and y still depend on g or \\tilde{g}. This dependency seems important but is currently obscured, is this a problem?\n\n[related work and interpretation of the generalization]\nThen, the proof of thm. 1 follows the exact same structure as thm.1 of [Bekkers] and reference [23] for the attentive version. It would be great if these works and other related works could be discussed in more detail when it comes to claims regarding this work generalizing many group equivariant works. Just to be clear, I do not dispute this claim, it is very well founded, it is just that the paper contains sentence such as “this work… delivers a unified understanding on the previous group equivariant models” (line 357). In my opinion, the understanding of this unified view could be better delivered as the equation is quite abstract and I think can be better understood with some specific (mathematically precises) special cases such as g-conv or attentive g-conv.\n\nFor example, an additional intuitive explanation of equation 12 would be very helpful. Why is this layer more powerful than standard convs?\n\nref:\n[Bekkers] Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2019.\n\n[solution to the spatial agnostic problem?]\nThen I have doubts regarding the claim of solving “the spatial-agnostic problem”. I do not understand how this problem is solved. By construction equivariant methods have to be spatial agnostic otherwise they can’t be equivariant and thus operations only depend on relative positions/transformations. The only difference now is that the convolution kernels are conditioned on the feature values at the central or neighbor locations. Maybe it is mentioned somewhere in other words, but do I understand correctly that merely the dependency of the conv kernel (as in deformable convs), or aggregation function on local features solves the spatial agnostic problem? My apologies for possibly misinterpreting parts of the paper, but perhaps the notions of equivariance vs “conditioning” can be decoupled and explicitly discussed in terms of the spatial agnostic problem. For me this part was slightly confusing.\n\n[Practical question]\nHow is the aggregation function/convolution kernel parametrized. In continuous kernel convolutions such as e.g. PointConvs [Wu et al.] or LieConvs [Finzi et al.], the kernel is parametrized by an MLP (though it does not depend on neighborhood feature values). Then for every possible g^{-1}\\tilde{g} the kernel is defined. Is this also the case in this paper, or is the kernel (e.g. in eq 12) indexed for a finite set of possible values for g^{-1}\\tilde{g}? (made possible because only a discrete group is considered)\n\nrefs\n[Wu et al.] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2019.\n[Finzi et al.] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165–3176. PMLR, 2020.\n\n[Minor comments. ]\nWhen equation 10 is presented I didn’t understand why this property was important. But then later on this is used for efficient implementations where the sub-group part is handled in one step by aggregating neighborhood information. Perhaps the section about eq 10 could be better introduced with such a motivation.\n\nLine 217. What is “s”. I think this symbol is not defined.\nLin 292. Same here, I don’t understand what a “slice numer s” is.\n\n\nRegarding implementation of eq 12. The output of V represents transformed feature values of some neighboorhood, and it should follow the same ordering as the kernel values outputted by K. Is this automatically obtained or does one have to practically code this?\n\nLine 320, why is only the second group conv layerin a res-block replaced and not all?\n\nSection 5.2. Since the focus is on efficiency I was a bit dissapointed not to see any application to large datasets that actually require efficiency. Where there still practical limitations?\n\nThe FLOP analysis is great, still I would like to get a sense of computation times. For example, I can imagine that all the stacking of features and what not leads to quite some overhead which, despite less FLOPs, doesn’t lead to a reduction in computation time. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The ablation study is done on MNIST and conclusion may not generalize to more challenging datasets. Otherwise I see no concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have written a very convincing rebuttal that leans me toward acceptance of this paper.
Decision: Accept (Poster)