Efficient Equivariant Network
Year: 2022

Abstract:
Convolutional neural networks (CNNs) have dominated the field of Computer Vision and achieved great success due to their built-in translation equivariance. Group equivariant CNNs (G-CNNs) that incorporate more equivariance can significantly improve the performance of conventional CNNs. However, G-CNNs are faced with two major challenges: spatial-agnostic problem and expensive computational cost. In this work, we propose a general framework of previous equivariant models, which includes G-CNNs and equivariant self-attention layers as special cases. Under this framework, we explicitly decompose the feature aggregation operation into a kernel generator and an encoder, and decouple the spatial and extra geometric dimensions in the computation. Therefore, our filters are essentially dynamic rather than being spatial-agnostic. We further show that our Equivariant model is parameter Efficient and computational Efficient by complexity analysis, and also data Efficient by experiments, so we call our model E-Net. Extensive experiments verify that our model can significantly improve previous works with smaller model size. Especially, under the setting of training on 1/5 data of CIFAR10, our model improves G-CNNs by 5%+ accuracy, while using only 56% parameters and 68% FLOPs.

heading: 1 Introduction
text: In the past few years, convolutional neural networks (CNNs) have been widely used and achieved superior results on multiple vision tasks, such as image classification [31, 55, 51, 22], semantic segmentation [3], and object detection [44]. A compelling explanation of the good performance of CNNs is that their built-in parameter sharing scheme brings in translation equivariance: shifting an image and then feeding it through a CNN layer is the same as feeding the original image and then shifting the resulted feature maps. In other words, the translation symmetry is preserved by each layer. Motivated by this, Cohen and Welling [9] proposed Group Equivariant CNNs (G-CNNs), showing how convolutional networks can be generalized to exploit larger groups of symmetries. Following G-CNNs, researchers have designed new neural networks that are equivariant to other transformations like rotations [9, 61, 24, 49] and scales [65, 53]. However, G-CNNs still have two main drawbacks: 1) In the implementation, G-CNNs would introduce extra dimensions to encode new transformations, such as rotations and scales, thus have a very high computational cost. 2) Although G-CNNs achieve group equivariance by sharing kernels, like vanilla CNNs, they lack the ability to adapt kernels to diverse feature patterns with respect to different spatial positions, namely, the spatial-agnostic problem [68, 39, 70, 71, 54, 67, 36].
∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Some previous works focus on solving these two problems. Cheng et al. [4] proposed to decompose the convolutional filters over joint steerable bases to reduce model size. However, it is essentially G-CNNs which still have the inherent spatial-agnostic problem. To incorporate dynamic filters, one solution is introducing attention mechanism into each convolution layer in G-CNNs without disturbing inherent equivariance [48, 45]. The cost is that they introduce extra parameters and increase the complexity of space and time. Another solution is to replace group convolution layers with standalone self-attention layers by designing a specific position embedding to ensure equivariance [47, 26]. However, the self-attention mechanism suffers from quadratic memory and time complexity, because it has to compute the attention score at each pair of inputs.
Actually, Cohen et al. [7], Kondor et al. [29] and Bekkers [1] revealed that an equivariant linear layer is essentially a convolution-like operation. Inspired by this, we further discover that a general feature-extraction layer, either linear or non-linear, being equivariant is equivalent to that the feature aggregation mechanism between each pair of inputs only depends on the relative positions of these two inputs. Based on this observation, we propose a generalized framework of previous equivariant models, which includes G-CNNs and equivariant attention networks as special cases. Under this generalized framework, we design a new equivariant layer to conquer the aforementioned difficulties. Firstly, to avoid quadratic computational complexity, the feature aggregation operator is explicitly decomposed into a kernel generator and an encoder which takes one single feature as the input. Since our kernels are calculated based on input features, they are essentially dynamic rather than being spatial-agnostic. In addition, we decouple the feature aggregation mechanism across spatial and extra geometric dimensions to reduce the inter-channel redundancy in convolution filters [4] and further accelerate computation. Extensive experiments show that our method can process data very efficiently and perform significantly better than previous works using lower computational cost. As our method is parameter Efficient, computational Efficient, data Efficient and Equivariant, we name our new layer as E4-layer.
We summarize our main contributions as follows:
• We propose a generalized framework of previous equivariant models, which includes GCNNs and attention-based equivariant models as special cases.
• Under the generalized framework, we explicitly decompose the feature aggregation operator into a kernel generator and an encoder, and further decouple the spatial and extra geometric dimensions to reduce computation.
• Extensive experiments verify that our method is also data efficient and performs competitively with lower computational cost.

heading: 2 Related Work
text: Vanilla CNNs [34] are naturally translation equivariant. More symmetries are considered to be exploited into the network for different tasks, such as rotations over plane [9, 66, 35, 12, 61, 59, 49, 37, 52, 4, 41, 2, 10, 58, 21, 24], rotations over 3D space [62, 16, 57, 64, 14, 60, 15, 50, 28, 6], scaling [65, 40, 53, 46], symmetries on manifold [8, 11], and other general symmetry groups [17, 56, 18]. These works accomplish equivariance by constraining the linear mappings in layers, followed by pointwise non-linearities to enhance their expressive power. In general, researchers [29, 7, 1] pointed out that an equivariant linear mapping can always be written as a convolution-like integral, i.e., G-CNNs in practice. However, their theory is still limited to linear cases.
As works [68, 39, 70, 71, 54, 67, 36] point out the spatial-agnostic problem of CNNs and attention mechanisms [25, 63, 43, 13, 20] achieve impressive results on various vision tasks, researchers start to consider non-linear equivariant mapping. Romero et al. [48, 45] directly reweighted the convolution kernels with attention weights generated by features and obtained non-linear equivariant models. However, compared with G-CNNs, these methods introduce extra parameters and operations, resulting in an even heavier computational burden. Also, some works [47, 26, 23] proposed group equivariant self-attention [43, 13]. Fuchs et al. [19] incorporated self-attention into 3D equivariant networks and proposed SE(3)-Transformers. However, since their filters are essentially calculated based on a pair of inputs, the computational complexity is quadratic.
In this work, we further extend the linear equivariant theory to a more general situation, including non-linear cases. Under the framework, we design a new equivariant layer to solve both the spatial-
agnostic problem in convolution-based equivariant models and heavy computation cost problem in most equivariant models.

heading: 3 A Unified Framework of Previous Group Equivariant Models
text: In this section, we first briefly review two representative group equivariant models: the linear model G-CNNs [9], and the non-linear model equivariant self-attention [47, 26]. Then, we propose a general framework of previous equivariant models based on the inner relationship among these specific models.

heading: 3.1 Equivariance
text: Equivariance indicates that the outputs of a mapping transform in a predictable way with the transformation of the inputs. Formally, a group equivariant map Ψ satisfies that
∀u ∈ G, Ψ [Tu[f ]] = T ′u[Ψ[f ]], (1) where G is a transformation group, f is an input feature map, and Tu and T ′u are group actions, indicating how the transformation u acts on the input and output features, respectively. Besides, since we hope that two transformations u, v ∈ G acting on the feature maps successively is equivalent to the composition of transformations uv ∈ G acting on the feature maps directly, we require that TuTv = Tuv , where uv is the group product of u and v. The same is the case with T ′u. Now we examine the specific form of the transformation group G. In this work, we focus on the analysis of 2D images defined on R2. Consequently, we are most interested in the groups of the form G = R2 o A, resulting from the semi-product (o) between the translation group R2 and a group A acts on R2, e.g., rotations, scalings and mirrorings. This family of groups is referred to as affine groups and their group product rule is:
uv = (xu, au)(xv, av) = (xu + auxv, auav), (2)
where u = (xu, au) and v = (xv, av), in which xu, xv ∈ R2 and au, av ∈ A. For ease of implementation, following [9], we take A as the cyclic group C4 or the dihedral group D4, then G becomes p4 or p4m. As for the group action, we employ the most common regular group action in this work, i.e., Tu[f ](v) = f(u−1v). (3) Here, we only care about the group action over the feature maps defined on G, because we always use a lifting operation to lift the input images defined on R2 to the feature maps on G, where the equivariance can be preserved properly, as will be shown in Section 3.2.

heading: 3.2 G-CNNs
text: Let f (l) : X → RCl and W : G → RCl+1×Cl be the input feature and the convolutional filter in the l-th layer, respectively, where Cl denotes the channel number of the l-th layer. X is taken as R2 for the first layer, and taken as G for the following layers. Then for any g ∈ G, the group convolution [29, 7, 1] of f (l) and W on G at g is given by
f (l+1)(g) = Ψ[f (l)](g) = ∫ X W (g−1g̃)f (l)(g̃)dµ(g̃), (4)
where µ(·) is the Haar measure. When X is discrete, Eqn. (4) can be rewritten as f (l+1)(g) = ∑ g̃∈X W (g−1g̃)f (l)(g̃). (5)
G-CNNs essentially generalize the translation equivariance of conventional convolution to a more general group G. In fact, the first layer maps the 2D images to a function defined on G, while the following layers map one feature map on G to another. As a result, the computational complexity of the first layer and the following layers are of the order O(k2|A|) and O(k2|A|2), respectively, where k is the kernel size in the spatial space. As a result, G-CNNs have a much larger computational cost when A is large, especially for the intermediate layers. In this work, we employ the first layer of G-CNNs as a lifting operation, and focus on reducing the computation of the latter layers.

heading: 3.3 Equivariant Attention Networks
text: Group Equivariant Self-Attention (G-SA) [47, 26] is a representative method of equivariant attention networks, whose form can be simplified as follows:
f (l+1)(g) = ∑ g̃∈G Softmaxg̃[h T Q(f (l)(g))(hK(f (l)(g̃)) + Pg−1g̃)]hV (f (l)(g̃)), (6)
where hV : RCl → RCl+1 , and hQ, hK : RCl → Rd are the embedding functions of values, querys and keys, respectively, which are neural networks in the most general case. d is the dimension of the low dimensional embeddings, and Pg−1g̃ ∈ Rd encodes the relative positions of the query f (l)(g) and the key f (l)(g̃).

heading: 3.4 Generalized Equivariant Framework
text: As more and more group equivariant structures emerge, researchers start to deduce the most general equivariant structures. To this end, Cohen et al. [7], Kondor et al. [29] and Bekkers [1] proposed a general theory of linear group equivariant structures, which indicates that G-CNNs are the most general equivariant linear layers. Besides, a lot of non-linear equivariant structures appear recently, such as equivariant self-attention layers [47, 26]. This motivates us to investigate a more general framework.
In all, with only slight modification, most of layers in a neural network can be viewed as a kind of aggregation of pair-wise feature interaction as follows:
f (l+1)(g) = ∑ g̃∈G Hg,g̃(f (l)(g), f (l)(g̃)), (7)
where the feature aggregation operator Hg,g̃(·, ·) : RCl × RCl → RCl+1 is a mapping indexed by a pair of location g and g̃, which describes how to aggregate the input feature pair f(g) and f(g̃). In general, the above layer is not equivariant. However, we can find a general constraint for Hg,g̃(f
(l)(g), f (l)(g̃)) to make this layer equivariant over G. Theorem 1 The layer formulated as Eqn.(7) is group equivariant if and only if there is a mapping H̃ĝ : RCl × RCl → RCl+1 which is indexed by a single group element ĝ, such that, ∀f (l) and ∀g, g̃ ∈ G, the layer satisfies:∑
g̃
Hg,g̃(f (l)(g), f (l)(g̃)) = ∑ g̃ H̃g−1g̃(f (l)(g), f (l)(g̃)) (8)
Proof ⇒ Firstly, ∀u, g and g̃ ∈ G, Tuf (l+1)(g) = f (l+1)(u−1g) = ∑ g̃∈G Hu−1g,g̃(f (l)(u−1g), f (l)(g̃)).
On the other hand,∑ g̃∈G Hg,g̃(Tuf (l)(g), Tuf (l)(g̃)) = ∑ g̃∈G Hg,g̃(f (l)(u−1g), f (l)(u−1g̃)) = ∑ g̃∈G Hg,ug̃(f (l)(u−1g), f (l)(g̃)).
As Tuf (l+1)(g) = ∑ g̃∈G Hg,g̃(Tuf (l)(g), Tuf (l)(g̃)),
⇒ ∀f (l), g, u, ∑ g̃∈G Hg,ug̃(f (l)(u−1g), f (l)(g̃)) = ∑ g̃∈G Hu−1g,g̃(f (l)(u−1g), f (l)(g̃)).
Let g → ug, we get: ∀f (l), g, u, ∑ g̃∈G Hug,ug̃(f (l)(g), f (l)(g̃)) = ∑ g̃∈G Hg,g̃(f (l)(g), f (l)(g̃)).
then, we let u to be g−1, ∀f (l), g, ∑ g̃∈G He,g−1g̃(f (l)(g), f (l)(g̃)) = ∑ g̃∈G Hg,g̃(f (l)(g), f (l)(g̃)).
We denote H̃g−1g̃(·, ·) as He,g−1g̃(·, ·), we can get exactly the Eqn.(8) ⇐ This is obvious. Q.E.D
From the theorem, we can get a group equivariant layer: f (l+1)(g) = ∑ g̃∈G H̃g−1g̃(f (l)(g), f (l)(g̃)), (9)
which is also the only equivariant form of Eqn. (7). Actually, the above theorem also reveals the essence of equivariance in previous works, i.e., if the relative positions of (g1,g̃1) and (g2,g̃2) are the same, i.e., g1−1g̃1 = g−12 g̃2 = ĝ, the feature pairs located at the two tuples should be processed equally. In other words, we should employ the same function H̃ĝ to act on these two input feature pairs.
From this perspective, we can readily see that both the kernel sharing used in G-CNNs, Eqn. (4), and the relative position encoding adopted in the G-SA, Eqn. (6), utilizes the above rule. According to Theorem 1, designing a group equivariant layer becomes much more easily and flexibly than ever, as we only need to design a new function H̃ĝ. In addition, the new formulation provides a more general perspective on the group equivariant layer, i.e., sharing the parameters of function H̃ĝ , which generalizes the kernel sharing schemes in G-CNNs. Based on the above understanding, we can see that if we replace the feature vector in the right hand side of Eqn. (9) with the local patch at group element g and g̃, respectively, it is still equivariant. Proposition 1 The following layer is equivariant,
f (l+1)(g) = ∑ g̃∈G H̃g−1g̃(FN1(g),FN2(g̃)) (10)
where for i = 1, 2, the FNi(g) denote the local patches of g, in which Ni(g) represent g’s neighborhood {gg′|g′ ∈ Ni(e)} and Ni(e) is the predefined neighborhood of the identity element e ∈ G.
One remarkable advantages of introducing local patch is that it contains more semantic information than feature vector. Notice, we acquire the local patches by concatenating features in the neighborhoods of g and g̃ in a predefined order onN1(e) andN2(e) respectively, i.e., f(g′) is concatenated at the same place in FN1(g) as f(g−1g′) in FN1(e). We denote the concatenation operator as ⋃ , and will discuss the above in detail in Section 4.1, which shows that concatenating features can not only make our framework more flexible, but also help to reduce the computational burden of our newly proposed equivariant layer.

heading: 4 Efficient Equivariant Layer
text: A straight-forward and easy case of Eqn. (10) is to adopt H̃ĝ, ∀ĝ ∈ G, as a multi-layer perceptron (MLP), where the subscript ĝ is used to identify different MLPs. However, in Eqn. (10), we have to compute a mapping from two high dimensional vectors to another high dimensional one for each input pair of g and g̃, which is very expensive. A similar issue exists in computing the attention score in self-attention. To deal with this problem, we decompose H̃ into the following form to reduce the computation, i.e.,
∀ĝ ∈ G, H̃ĝ(x, y) = Kĝ(x) V (y) (11)
where means element-wise product, and Kĝ : RCl|N1(e)| → RCl+1 is a kernel generator and V : RCl|N2(e)| → RCl+1 is an encoder. We use | · | to denote the numbers of elements in a set. Hence, we can compute Kĝ(x) and V (y) separately. In addition, to further save computation, we split the kernel into several slices along the channels, such that Kĝ is shared across these slices,
i.e., ∀ 1 ≤ i, j ≤ Cl+1, Kiĝ = K j ĝ if i ≡ j (mod s), where s is the number of slices, and i and j are channel indexes. The Kĝ is essentially a dynamic filters which is adaptive to features around g, avoiding the spatial-agnostic problem in G-CNNs. Unlike conventional dynamic filters, which are matrices, the output of Kĝ is a vector, which can be viewed as a depth wise kernel [5]. This can decouple channel dimension with spatial dimension during feature aggregation to reduce the computational cost. Position information is implicitly encoded in the organized output form of our kernel generator, rather than using explicit positional embedding in the group self-attention layer [26, 47].
In practice, we can view the whole kernel family {Kĝ}ĝ∈G as the output of a single mapping, i.e., K̃: RCl|N1(e)| → R|G|Cl+1 . Then, we resize the output of K̃ to be a |G| × Cl+1 matrix, with different rows represent different Kĝ. Namely, if we adopt K̃ as an MLP, the computations and parameters used for hidden layer are shared across Kĝ for different ĝ, which is another merit of the Eqn. (11). However, there is still a large search space for H̃ĝ, as Eqn (11) is only a special structue of H̃ĝ, we leave a more complete study of H̃ĝ in the future work.

heading: 4.1 Implementation on Affine Group
text: In this section, we design a very efficient equivariant layer based on Eqn. (11) for affine group R2 oA. The computation of the operator is:
f (l+1)(g) = ∑
g̃∈N (g)
Kg−1g̃  ⋃ g′∈N1(g) f (l)(g′)  V  ⋃ g̃′∈N2(g̃) f (l)(g̃′)  . (12) Following the standard practice in computer vision, aggregation is done only on the local neighborhood of g, N (g). To save computation, we choose N (g) to be only spatial-wise neighborhood, i.e.,
N (g) = {g(v, eA) | v ∈ Ω}, where Ω ∈ R2 and eA is the identity element of group A. However, aggregating information along spatial neighborhood only discards the information interaction along A, which could lead to a drop in performance [32]. We alleviate the issue by concatenating the feature map along A, i.e., we choose N1(g) and N2(g) to be {g(0, a)|a ∈ A}. The order of concatenation is predefined on A. As will be shown in the later experiments, this concatenation does not introduce much computation but can significantly improve performance. Compared to group convolution, such a design enables us to decouple the feature aggregation across the spatial dimension and the A dimension to further reduce computational cost. In practice, we adopt the K̃ as a two layer MLP: K̃(x) = W2Relu(W1x), where W1 ∈ RCl/r×Cl|A|,W2 ∈ R|Ω|s×Cl/r, and r is the reduction ratio which saves both parameters and computation, s is the number of slices defined before. For 2D images, Ω is usually adopted as a k × k square mesh grids and |Ω| = k2, where k is the kernel size. We simply adopt the encoder V as a linear transform: V (y) = W3y, where W3 ∈ RCl+1×Cl|A|. For better illustration, we visualize a concrete layer of Eqn. (12) by choosing G as p4 in Figure 1.

heading: 4.2 Computational Complexity Analysis
text: In practice, the feature map is defined on discrete mesh grids. We use h and w to denote the height and the width of mesh grids. As the numbers of the input and output channels are usually the same, we assume Cl = Cl+1 = c.
Parameter Analysis The number of learnable parameters of E4-layer (12) is c2|A|(1 + 1/r) + csk2/r. As s c, parameter counts are dominated by the first term when k is not too large, and increasing kernel size will not significantly increase parameter counts, which is shown in later experiments. The parameters count of group convolution layer is c2k2|A|. Notice that (1+1/r) k2 and s/r c|A|, parameters count of our E4-layer is significantly less than that of group convolution layer.
Time Complexity Analysis The FLOPs of E4-layer and group convolution layer are (1 + 1/r)c2|A|2hw + (1 + s/r)k2c|A|hw and k2c2|A|2hw, respectively. Similarly, as (1 + 1/r) k2 and (1 + s/r) c|A|, the FLOPs of E4-layer is significantly lower than that of group convolutional layer.
It can be observed that both the parameter count and FLOPs of our E4-layer are composed of two terms, one depending on k2 and the other not relying on k, which is a result of disentangling across spatial dimension with both channels and A during feature aggregation.

heading: 5 Experiments
text: In this section, we conduct extensive experiments to study and demonstrate the performance of our model. The experimental results show that our model has a greater capacity than the groupconvolution-based one in terms of parameter efficiency, computational efficiency, data efficiency and accuracy. On the MNIST-rot dataset, we detailedly study the effect of hyperparameters on the number of parameters, computation FLOPs and performance of our model. All the experiments are done on the GeForce RTX 3090 GPU.

heading: 5.1 Rotated MNIST
text: The MNIST-rot dataset [33] is the most widely used benchmark to test the equivariant models. It contains 62k 28×28 randomly rotated gray-scale handwritten digits. Images in the dataset are split into 10k for training, 2k for validation and 50k for testing. Random rotation of digits and only 20 percent of training data of the
standard MNIST dataset increases the difficulty of classification.
For a fair comparison, we keep both training settings and architectures of our model as close as possible to previous works [9, 47]. In addition, we adopt the p4 group to construct all our models in this section. In our first experiment, we adopt our E4-Net given in the supplementary material to make a comparison to previous works. This is a very lightweight model which contains only 18.8K learnable parameters. It is composed of one group convolutional layer which lifts the image to the p4 group, six E4-layers and one fully connected layer. Two 2× 2 max-pooling layers are inserted after the first and the third E4-layer to downsample feature maps. The last E4-layer is followed by a global max group pooling layer [9], which takes the maximum response over the entire group, to ensure the predictions invariant to rotations.
Our model is trained using the Adam optimizer [27] for 200 epochs with a batch size of 128. The learning rate is initialized as 0.02 and is reduced by 10 at the 60th, 120th and 160th epochs. The weight decay is set as 0.0001 and no data augmentation is used during training. The results are listed in Table 1. Our models significantly outperform G-CNNs [9] using only about 25% parameters and 40% FLOPs. For G-SA [47], which is a group equivariant stand-alone self-attention model, even performs inferiorly to G-CNNs with much more computational cost. The α-p4-CNN model [45] further introduces the attention mechanism to group convolution along both spatial and channel dimensions to enhance the expressiveness of G-CNNs, while our E4-Net still significantly outperforms it with less computational cost. We also experiment with a larger model to further demonstrate the capacity of our model, which is listed in the last line of Table 1.
Ablation Study of Concatenation: In the E4layer (12), we introduce the concatenation operation to enable the disentanglement across the rotation and the spatial information interaction. To study the importance of concatenation, we carry out experiments on the case that neither Kĝ nor V in Eqn. (12) use concatenation, i.e.,
N1(g) = g, N2(g̃) = g̃. As shown in the first line of Table 2, this leads to a significant drop in performance. This is because if aggregation in Eqn.(12) is done merely in the spatial neighborhoods without concatenation, there is no information interaction along the rotation dimensions. We also experiment the cases using concatenation only in Kĝ or V , and the performance of both is better than the case without concatenation but is still inferior to the case with concatenation in both Kĝ and V . This further illustrates the importance of concatenation along A.
Hyperparameters Analysis: We investigate the effect of various hyperparameters used in the E4-layer. The reduction ratio r and the slice number s in the Kĝ and kernel size k control the computations and parameters of the layer. Based on the baseline model, we vary the three hyperparameters respectively. As shown in the Table 3, improvement is observed when decreasing the reduction ratio and increasing the slice number, with the cost of computational burden increasing. Especially, the improvement of s = 2 over s = 4 and r = 1 over r = 2 is marginal, which is attributes to redundancy
in the kernel [4]. In conclusion, appropriately increasing the reduction ratio r and decreasing the slice number s can help to reduce computational cost while preserving performance. Keeping other hyperparameters fixed, we study the effect of kernel size on our model. In Table 3, the performance peaks when kernel size equals 7. In general, a larger kernel size leads to improved performance due to a larger receptive field. In addition, as explained in Section 4.2, increasing kernel size does not dramatically increase parameters and FLOPs as standard convolution.

heading: 5.2 Natural Image Classification
text: In this section, we evaluate the performance of our model on the two common natural image datasets, CIFAR10 and CIFAR100 [30]. The CIFAR-10 and the CIFAR100 datasets consist of 32× 32 images
belonging to 10 and 100 classes, respectively. Both of the datasets contain 50k training data and 10k testing data. Before training, images are normalized according to the channel means and standard deviations.
In this experiment, we adopt ResNet-18 [22] as the baseline model(short as R18), which is composed of an initial convolution layer, followed by 4 stage Res-Blocks and one final classification layer. Following the standard practice in [9], we replace all the conventional layers with p4 (p4m) convolutions in R18 and increase the width of each layer by √ 4 ( √
8) to keep the learnable parameters approximately the same. We denote the resulting models as p4-R18 (p4m-R18). We replace the second group convolution layer in each Res-Block of p4-R18 (p4m-R18) with our E4-layer, resulting in the p4-E4R18 (p4m-E4R18). For a fair comparison, all the above models are trained under the same training settings. We use the stochastic gradient descent with an initial learning rate of 0.1, a Nesterov momentum of 0.9 and a weight decay of 0.0005. The learning rate is reduced by 5 at 60th, 120th, and 160th epochs. Models are trained for 200 epochs using 128 batch size. No data augmentation is used during training to illustrate data efficiency of our model.
The classification accuracy, parameters count and FLOPs of all models on CIFAR10 and CIFAR100 are reported in Table 4. We can see that models incorporating more symmetry achieve better performance, i.e., R18 ≤ p4-R18 ≤ p4m-R18. Our p4 and p4m models significantly outperform their counterparts on both CIFAR10 and CIFAR100. Furthermore, our model decreases the parameter count and FLOPs by 45% and 32%, respectively. Notice that the model size reduction is purely caused by the introduction of our E4-layers, as topological connections and width of each layer of E4 model and its counterparts are the same.
Data Efficiency: To further study the performance of our model, we train all the models listed in Table 4 on CIFAR10 with different sizes of training data. To be specific, we consider 5 settings, where 1k, 2k, 3k, 4k and 5k training data of each class are randomly sampled from the CIFAR10 training set. Testing is still performed on the original test set of CIFAR10. Other training settings are identical to the above. We visualize the results in Figure 2.
It is observed that the performance gap between p4, p4m and R2 models tend to increase as we reduce the training data. This is mainly because that the prior that the label is invariant to rotations is more important when training data are fewer. The trend is also observed in the gap between our models and their counterparts. For instance, the gap between p4m-E4R18 and p4m-R18 is 0.87% when training data of each class is 5k, while it is enlarged to 5.22% when training data of each class is reduced to 1k.
Especially, we observe the line of p4-E4R18 intersects with the one of p4m-R18, which further indicates that our model is much more data efficient than G-CNNs. As indicated above, symmetry prior is more important when training data are fewer, and the data efficiency of our model implies that p4-E4R18 and p4m-E4R18 can better exploit the symmetry of data.

heading: 6 Limitation and Future Work
text: From the theory perspective, although we extend the general equivaraint framework from linear cases to common non-linear cases, there’s two limitations on the generalization: 1) we only focus on layers with such pair-wise interactions proposed in Eqn.(7), and higher-order interactions cases are not included. 2) We only consider regular group action in this framework, which is a special case of general group actions. We leave extending this equivariant framework to these cases as future work.
From the practice perspective, we only give a special implementation of Eqn.(10) in an intuitive insight, and further exploration in the space of equivariant map is in demand. An alternative is to exploit searching algorithms from neural architecture search [42, 38, 69] to find a more powerful and efficient model. Besides this, our E4-layer is slower than G-CNN despite less FLOPs due to convolutions are optimized by many speedup libraries. Our layer is implemented only in a naive way, that is, using the unfold operation followed by a summation operation for the aggregation step. In the future, we will try to implement a customized CUDA kernel for GPU acceleration to reduce training and inference time of our model.

heading: 7 Conclusions
text: In this work, we propose a general framework of group equivariant models which delivers a unified understanding on the previous group equivariant models. Based on the new understanding, we propose a novel efficient and powerful group equivariant layer which can serve as a drop-in replacement for convolutional layers. Extensive experiments demonstrate the E4-layer is more powerful, parameter efficient and computational efficient than group convolution layers and their variants. Through a side by side comparison with G-CNNs, we demonstrate our E4-layer can significantly improve data efficiency of equivariant models, which show great potential for reducing the cost of collecting data.

heading: Acknowledgment
text: Zhouchen Lin was supported by the NSF China (No.s 61625301 and 61731018), NSFC Tianyuan Fund for Mathematics (No. 12026606) and Project 2020BD006 supported by PKU-Baidu Fund. Yisen Wang is partially supported by the National Natural Science Foundation of China under Grant 62006153, and Project 2020BD006 supported by PKU-Baidu Fund.

Review summaries:
This paper introduces a functional abstraction for group-convolutional networks employing first- and second-order features. Based on this functional abstraction, a new functional form for second-order group-convolutional networks is devised, which can be made more parameter-efficient than existing approaches. Experiments are performed on rotated MNIST and CIFAR, evaluating error, number of parameters and certain ablations.

The paper describes a generalization of group equivariant neural networks that typically rely on linear operators such as the group convolution, or pseudo linear operators such as attentive group convolution. The authors take a look at the aggregation function of the convolution operator (which is usually linear via a kernel times feature values followed by sum aggregation) and replaces it with a non-linear function that generally could depend not just on relative positions, but also on the central and neighbor feature values. The result is a more non-linear operator than group convolutions, which are a special case of this framework.  The paper is accurate and has an appropriate experimental section with decent ablation studies. The proposed work compares favorably compared to the baselines both in terms of accuracy, parameters and nr of operations (FLOPs).