EXPLOITING LEARNED SYMMETRIES IN GROUP EQUIVARIANT CONVOLUTIONS
Attila Lengyel

Jan van Gemert

Computer Vision Lab, Delft University of Technology, The Netherlands
ABSTRACTGroup Equivariant Convolutions (GConvs) enable convolutional neural networks to be equivariant to various transformation groups, but at an additional parameter and computecost. We investigate the filter parameters learned by GConvs
and find certain conditions under which they become highly
redundant. We show that GConvs can be efficiently decomposed into depthwise separable convolutions while preserving equivariance properties and demonstrate improved performance and data efficiency on two datasets. All code ispublicly available at github.com/Attila94/SepGrouPy.
Index Terms— group equivariant convolutions, depthwise separable convolutions, efficient deep learning
1. INTRODUCTION
Adding convolution to neural networks (CNNs) yields translation equivariance [1]: first translating an image x and thenconvolving is the same as first convolving x and then translating. Group Equivariant Convolutions [2] (GConvs) enableequivariance to a larger group of transformations G, includingtranslations, rotations of multiples of 90 degrees (p4 group),and horizontal and vertical flips (p4m group). Equivarianceto a group of transformations G is guaranteed by sharing parameters between filter copies for each transformation in thegroup G. Adding such geometric symmetries as prior knowledge offers a hard generalization guarantee to all transformations in the group, reducing the need for large annotateddatasets and extensive data augmentation.
In practice, however, GConvs occasionally learn filtersthat are near-invariant to transformations in G. An invariant filter is independent of the transformation and will for
GConvs yield identical copies of the transformed filters in
the consecutive layer, as shown in Fig. 1. This implies parameter redundancy, as these filters could be represented bya single spatial kernel. We propose an equivariant pointwise
and a depthwise decomposition of GConvs with increased parameter sharing and thus improved data efficiency. Motivatedby the observed inter-channel correlations in learned filtersin [3] we explore additionally sharing the same spatial kernelover all input channels of a GConv filter bank. Our contributions are: (i) we show that near-invariant filters in GConvs
This project is supported in part by NWO (project VI.Vidi.192.100).

Input

Layer 1

FM1

0
cout

Layer 2

1
cout

FM2

0
cout

Layer 3

1
cout

g0

g1

g2

g3

g0

g1

g2

g3

Fig. 1: Filters and feature maps of a GConv architecture
trained on Rotated MNIST. Rotation invariant filters in Layer
2 result in identical feature maps FM2 (green) and cause
Layer 3 to learn identical weights along the group dimension
g (blue). In contrast, non-symmetric filters in Layer 2 (red)result in non-identical filters in Layer 3 (brown).

yield highly correlated spatial filters; (ii) we derive two decomposed GConv variants; and (iii) improve accuracy compared to GConvs on RotMNIST and CIFAR10.
2. RELATED WORK
Equivariance in deep learning. Equivariance is a promisingresearch direction for improving data efficiency [4]. A variety of methods have extended the Group Equivariant Convolution for the p4 and p4m groups introduced in [2] to larger
symmetry groups including translations and discrete 2D rotations [5, 6], 3D rotations [7, 8, 9], and scale [10, 11]. Here,
we investigate learned invariances in the initial GConv framework [2] for the p4 and p4m groups, yet our analysis extends
to other groups where invariant filters exist.
Depthwise separable decomposition [12]. These decompose a multi-channel convolution into spatial convolutions ap-

plied on each individual input channel separately, followed bya pointwise (1x1) convolution. Depthwise separable convolutions significantly reduce parameter count and computation
cost at the expense of a slight loss in representation power
and therefore generally form the basis of network architectures optimized for efficiency [13, 14, 15]. The effectivenessof depthwise separable convolutions is motivated [3] by theobserved inter-channel correlations occurring in the learned
filter banks of a CNN, which is quantified using a PCA decomposition. We do a similar analysis to motivate and derive
our separable implementation of GConvs.
3. METHOD
3.1. Group Equivariant Convolutions
Equivariance to a group of transformations G is defined asΦ(Tg x) = Tg0 Φ(x),∀g ∈ G,(1)

where Φ denotes a network layer and Tg and Tg0 a transformation g on the input and feature map, respectively. Note thatin the case of translation equivariance T and T 0 are the same,
but in general not need to be. To simplify the explanation,
we first focus on the group p4 of translations and 90-degree
rotations, but extend to larger groups later.
Let us denote a regular convolution asll+1Xn,:,:=

CX

ll
Fn,c,:,:∗ Xc,:,:,(2)cwith X the input and output tensors of size [C l , H, W ], whereC l is the number of channels in layer l, H is height and W is
width, and F the filter bank of size [C l+1 , C l , k, k], with k
the spatial extent of the filter.
In addition to spatial location, GConvs encode the addedtransformation group G in an extra tensor dimension suchthat X becomes of size [C l , Gl , H, W ], where Gl denotes
the size of the transformation group G at layer l, i.e. 4 forthe p4 group. Likewise, GConv filters acting on these featuremaps contain an additional group dimension, yielding a filter bank F l of size [C l+1 , C l , Gl , k, k]. As such, filter banks
in GConvs contain Gl times more trainable parameters compared to regular convolutions. A GConv is then performedby convolving over both the input channel and input groupdimensions C l and Gl and summing up the outputs:ll+1
Xn,h,:,:
=

l

C X
G
X
c

l
l
F̃n,h,c,g,:,:∗ Xc,g,:,:
.

(3)

g

Here F̃ l denotes the full GConv filter of size [C l+1 , Gl+1 ,C l , Gl , k, k] containing an additional dimension for the output group Gl+1 . F̃ l is constructed from F l during each forward pass, where Gl+1 contains rotated and cyclically permuted versions of F l (see [2] for details). Note that input

images do not have a group dimension, so the input layer has
1
1
Gl =1 and Xc,g,:,:reduces to Xc,:,:, whereas for all followingllayers G =4 for the p4 group (and Gl =8 for p4m).

3.2. Filter redundancies in GConvs
A rotational symmetric filter is invariant to the relative orientation between the filter and its input. Thus, if the filter kernels
in the group dimension of a p4 GConv filter bank F l are rotational symmetric and identical, the resulting feature maps will
also be identical along the group dimension due to the rotation and cyclic permutation performed in constructing the full
filter bank F̃ l . As a result, the filters in the subsequent layer
acting on these feature maps receive identical gradients and,given same initialization, learn identical filters. This is illustrated in Fig. 1 where a p4 equivariant CNN is trained on Rotated MNIST. The first layer contains a single fixed rotation
invariant filter. All layers have equal initialization along thegroup dimension and linear activation functions. The filtersin layer 2 converge to be identical along the group dimension.
Furthermore, the filter kernels in the second layer belonging
to the first output channel (green) are also rotational symmetric, resulting in identical feature maps in FM2 (green) andconsequently the filters learned in the first input channel oflayer 3 (blue) become highly similar. This is in contrast to
the non-symmetric filters in layer 2 (red), resulting in nonidentical filters in layer 3 (brown).
Even non-rotational symmetric filters can induce filter
correlations in the subsequent layer. For instance, an edge
detector will result in inverse feature maps along the groupdimension, i.e. g 0 ≈ − g 2 and g 1 ≈ − g 3 and the filters acting on these feature maps will receive inverse gradients andconsequently converge to be inversely correlated. Inversely
correlated filters can be decomposed into the same spatial
kernel multiplied by a positive and negative scalar.
Upon visual inspection of the learned filter parameters ofa regular p4 equivariant CNN we observe that, even without
any fixed symmetries or initialization and with ReLU activation functions, the filter kernels tend to be correlated along
the group axis. To quantify this correlation we perform a PCA
decomposition similar as in [3]. We reshape the filter bank F
to size [C l+1 × C l , Gl , k 2 ] and perform PCA on each set offilters Fn,:,: for all n ∈ [1, C l+1 × C l ], where for each n wehave Gl features with k 2 samples. This results in Gl principal components of size k 2 , with PC1 being the filter kernel
explaining the most variance within the decomposed set. We
perform this decomposition for all layers in a p4 equivariant
network. Fig. 2 shows the ratio of the variance explained by
PC1 for each layer (after the input layer), before and after
training. In many cases a substantial part of the variance is
explained by a single component, demonstrating a significant
redundancy in filter parameters.

Layer 2

40
init.
trained 20
mean
0 0.0

0.5

Layer 3

1.0 0.0

Layer 4Layer 5Layer 6Layer 70.5

1.0 0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
variance explained by PC1
Fig. 2: Ratio of variance explained by the first principal component when decomposing a filter kernel along the group dimension,
before (blue) and after (red) training on Rotated MNIST. Redundancy in filter parameters increases as the network converges.
Substituting F̃ l in Eq. (3) and rearranging yields

3.3. Separable Group Equivariant Convolutions
To exploit the correlations in GConvs we decompose the filterbank F l into a 2D kernel K that is shared along the groupdimension, and a pointwise component w which encodes theinter-group correlations:lll
Fn,c,g,:,:= Kn,c,:,:· wn,c,g.

ll+1
Xn,h,:,:
=

=

(4)

(5)

where Th denotes the 2D transformation corresponding to
output group channel h and w̃l contains copies of wl that are
cyclically permuted along the input group dimension. A naive
implementation would be to precompute F̃ and perform a regular GConv as in Eq. (3). Alternatively, for better computational efficiency we can substitute the filter decomposition in
Eq. (5) into the GConv in Eq. (3) and rearrange as follows:
l

=

l

C X
G
X
c

g

l

l

C X
G
X
c

l
l
l
Xc,g,:,:∗ Th (Kn,c,:,:) · w̃n,h,c,g



(6)


l
l
l
Xc,g,:,:· w̃n,h,c,g∗ Th (Kn,c,:,:
)

=

l

l

C X
G
X



(12)


l
l
Xc,g,:,:· w̃n,h,c,g∗ Th (Kn,:,: )

(13)

g

(7)

(14)

with
l

l
X̃n,h,:,:
=

l

C X
G
X
c


l
l
Xc,g,:,:· w̃n,h,c,g
.

(15)

g

This way the GConv essentially reduces to an inverse depthwise separable convolution with Eq. (15) being the pointwiseand Eq. (14) being the depthwise component. This variant isnamed gc-GConv after the summation variables in Eq. (15).
While the g and gc decompositions may impose too stringent restrictions on the hypothesis space of the model, the improved parameter efficiency, as detailed in section 3.4, allows
us to increase the network width given the same parameterbudget resulting in better overall performance.

g

3.4. Computation efficiency

l

C
X

g

l
l
l
Xc,g,:,:∗ Th (Kn,:,:) · w̃n,h,c,gll= X̃n,h,:,:∗ Th (Kn,:,:)lllF̃n,h,c,g,:,:= Th (Kn,c,:,:) · w̃n,h,c,g,

l+1
Xn,h,:,:=

cc

The full GConv filter bank is then constructed as

l

C X
G
X

l
l
X̃n,h,c,:,:∗ Th (Kn,c,:,:
)

(8)

c

with
l

l
X̃n,h,c,:,:
=

G
X


l
l
Xc,g,:,:· w̃n,h,c,g
.

(9)

g

Expanding the dimensions of w̃l to [C l+1 , Gl+1 , C l , Gl , 1, 1]
we can implement Eq. (9) as a grouped 1 × 1 convolution
with C l groups, followed by a grouped spatial convolution
with C l+1 × Gl+1 groups, as given in Eq. (8). We refer to thisseparable GConv variants as g-GConv, denoting the summation variable in Eq. (9).
Alternatively, we share the spatial kernel K along both thegroup and input channel dimension by decomposing F l as:lll
Fn,c,g,:,:= Kn,:,:· wn,c,g,lllF̃n,h,c,g,:,:= Th (Kn,:,:) · w̃n,h,c,g
.

(10)
(11)

The decomposition of GConvs allows for a theoreticallymore efficient implementation, both in terms of the number of stored parameters and multiply-accumulate operations(MACs). As opposed to the [C l × Gl × k 2 × C l+1 ] parameters in a GConv filter bank, g- and gc-GConvs require only
[C l × C l+1 × (Gl + k 2 )] and [C l+1 × (C l × Gl + k 2 )],respectively. Similarly, a regular GConv layer performs[C l ×Gl ×k 2 ×W ×H ×C l+1 ×Gl+1 ] MACs, whereas g- andgc-GConvs do only [C l ×C l+1 ×Gl+1 ×W ×H ×(Gl +k 2 )]and [C l+1 × Gl+1 × W × H × (C l × Gl + k 2 )], assuming’same’ padding. This translates to a reduction by a factor of
1
1
1
1
k2 + Gl and k2 + C l ×Gl , both in terms of parameters and
MAC operations. The decrease in MACs comes at the cost ofa larger GPU memory footprint due to the need of storing intermediate feature maps, as is generally the case for separable
convolutions. Separable GConvs are therefore especially suitable for applications where the available processing power isthe bottleneck as opposed to memory.

Network

Test errorw

Param.

MACs

Z2CNN [2]c-Z2CNN
P4CNN [2]g-P4CNN [ours]
gc-P4CNN[ours]
g-P4CNN [ours]
gc-P4CNN [ours]

5.20 ± 0.110
4.64 ± 0.126
2.23 ± 0.061
2.60 ± 0.098
2.88 ± 0.169
1.97 ± 0.044
1.74 ± 0.070

20
57
10
10
10
17
30

25.21 k25.60 k24.81 k8.91 k
3.42 k25.26 k24.64 k2.98 M
4.14 M
11.67 M
4.37 M
1.80 M
12.34 M
13.01 M

SFCNN [6]DREN [17]H-Net [18]α-P4CNN [19]a-P4CNN [20]0.71 ± 0.022
1.56
1.69
1.70 ± 0.021
2.06 ± 0.043

10
-

25 k
33 k
73.13 k20.76 k-4. EXPERIMENTS
4.1. Rotated MNIST
We construct a g-separable (Eqs. 8-9) and gc-separable(Eqs. 14-15) version of the P4CNN architecture [2] andevaluate on Rotated MNIST [16]. Rotated MNIST has 10classes of randomly rotated handwritten digits with 12k train
and 60k test samples. We set the width w of the g-P4CNN
and gc-P4CNN networks such that the number of parameters
are as close as possible to our Z2CNN and P4CNN baselinesof 10 and 20 channels, respectively. We follow the training
procedure of [2] and successfully reproduced the results.
Table 1 shows the test error averaged over 5 runs. Both gand gc-P4CNN significantly outperform the regular P4CNN
architecture and perform comparably or better than otherarchitectures with a similar parameter count. Both g- andgc-P4CNN also outperform a depthwise separable versionof Z2CNN (c-Z2CNN), validating that GConvs are more
efficiently decomposable than regular convolutions. Additionally, we evaluate data-efficiency in a reduced data setting.
As Fig. 3a shows, both g- and gc-P4CNN consistently outperform P4CNN. Sharing the same 2D kernel in a GConv filterbank is thus a strong inductive bias and improves the model’s
sample efficiency. The test error as a function of number ofparameters is also shown in Fig. 3b. Separable GConvs do
better for all model capacities.

Z2
c-Z2
P4
g-P4 [ours]
gc-P4 [ours]

16
14
12
10
8
6
4
2

8

Z2
c-Z2
P4
g-P4 [ours]
gc-P4 [ours]

7
Test error (%)

Test error (%)

Table 1: Test error on Rotated MNIST - comparison with z2
baseline and other p4-equivariant methods. w denotes network width. Separable GConv architectures perform better
compared to regular GConvs (upper part) and comparable toother equivariant methods (lower part).

6
5
4
3
2

2

4
6
8 10
Training samples (k)120

0.1 0.2 0.3 0.4 0.5 0.6
Parameters (M)

(a)

(b)

Fig. 3: Test error on Rotated MNIST for varying training set(a) and model sizes (b). Architectures with separable GConvs
perform consistently better.
Table 2: Test error on CIFAR10 - comparison with otherp4m-equivariant methods. gc-p4m-ResNet44 performs best.
Network

CIFAR10 CIFAR10+ Param.

ResNet44† [2]p4m-ResNet44‡ [2]αF -p4m-ResNet44 [19]a-p4m-ResNet44 [20]g-p4m-ResNet44 [ours]
gc-p4m-ResNet44 [ours]
†‡

13.10
8.06
10.82
9.12
7.60
6.72

7.66
5.78
10.12
6.09
5.43

2.64M
2.62M
2.70M
2.63M
1.78M
1.88M

Unable to reproduce results from [2]: 9.45 / 5.61† , 6.46 / 4.94‡ .

cluding random horizontal flips and random translations of
up to 4 pixels. Our gc-p4m-ResNet44 outperforms all othermethods using less parameters, as shown in Table 2. Also ina low data regime using only 20% of the training samples our
gc-p4m architecture outperforms the regular p4m networkwith an error rate of 13.43% vs. 14.20%.

5. DISCUSSION

Our method exploits naturally occurring symmetries in
GConvs by explicit sharing of the same filter kernel along
the group and input channel dimension using a pointwise anddepthwise decomposition. Experiments show that imposing
such restriction on the architecture only causes a minor performance drop while allowing to significantly reduce the network parameters. This in turn (i) improves data efficiency and(ii) allows to increase the network width for the same parameter budget resulting in better overall performance. Sharing the
spatial kernel over only the group dimension (g) proves less4.2. CIFAR 10
effective than additionally sharing over input channels (gc) as
Similarly, we perform a benchmark on the CIFAR10 dataset [21] the latter also efficiently exploits inter-channel correlations
using a p4m equivariant version of ResNet44 as detailed
in the network. This allows to further increase the networkin [2]. CIFAR 10+ denotes moderate data augmentation inwidth and thereby its representation power.

6. REFERENCES[1] Osman Semih Kayhan and Jan C van Gemert, “Ontranslation invariance in cnns: Convolutional layers can
exploit absolute spatial location,” in Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2020, pp. 14274–14285.
[2] Taco Cohen and Max Welling, “Group equivariant convolutional networks,” New York, New York, USA, 20–
22 Jun 2016, vol. 48 of Proceedings of Machine Learning Research, pp. 2990–2999, PMLR.
[3] Daniel Haase and Manuel Amthor, “Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets,” in Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] M. Rath and A. Condurache, “Boosting deep neuralnetworks with geometrical prior knowledge: A survey,”
ArXiv, vol. abs/2006.16867, 2020.
[5] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen
A. J. Eppenhof, Josien P. W. Pluim, and Remco Duits,“Roto-translation covariant convolutional networks for
medical image analysis,” in Medical Image Computing
and Computer Assisted Intervention – MICCAI 2018.
2018, pp. 440–448, Springer International Publishing.
[6] Maurice Weiler, Fred A. Hamprecht, and Martin
Storath, “Learning steerable filters for rotation equivariant cnns,” in Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), June
2018.
[7] Marysia Winkels and Taco S. Cohen, “Pulmonary nodule detection in ct scans with equivariant cnns,” Medical
Image Analysis, vol. 55, pp. 15 – 26, 2019.
[8] Daniel E. Worrall and Gabriel J. Brostow, “Cubenet:
Equivariance to 3d rotation and translation,” in Computer Vision - ECCV 2018 - 15th European Conference,
Munich, Germany, September 8-14, 2018, Proceedings,
Part V, 2018, pp. 585–602.
[9] Maurice Weiler, Mario Geiger, Max Welling, Wouter
Boomsma, and Taco S Cohen, “3d steerable cnns:
Learning rotationally equivariant features in volumetricdata,” in Advances in Neural Information Processing
Systems. 2018, vol. 31, pp. 10381–10392, Curran Associates, Inc.
[10] Daniel Worrall and Max Welling, “Deep scale-spaces:
Equivariance over scale,” in Advances in Neural Information Processing Systems. 2019, vol. 32, pp. 7366–
7378, Curran Associates, Inc.

[11] Ivan Sosnovik, Michał Szmaja, and Arnold Smeulders,“Scale-equivariant steerable networks,” in International
Conference on Learning Representations, 2020.
[12] Laurent Sifre and Prof Stéphane Mallat, “Ecole polytechnique, cmap phd thesis rigid-motion scattering forimage classification author:,” 2014.
[13] A. Howard, Mark Sandler, G. Chu, Liang-Chieh Chen,
B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Quoc V. Le, and H. Adam, “Searching for mobilenetv3,” 2019 IEEE/CVF International Conference
on Computer Vision (ICCV), pp. 1314–1324, 2019.
[14] Mingxing Tan and Quoc Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,”
in Proceedings of the 36th International Conference on
Machine Learning. 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 6105–6114, PMLR.
[15] Mirgahney Mohamed, Gabriele Cesa, Taco S. Cohen,
and Max Welling, “A data and compute efficient designfor limited-resources deep learning,” 2020.
[16] Hugo Larochelle, Dumitru Erhan, Aaron Courville,
James Bergstra, and Yoshua Bengio, “An empiricalevaluation of deep architectures on problems with manyfactors of variation,” in Proceedings of the 24th International Conference on Machine Learning, New York,NY, USA, 2007, ICML ’07, p. 473–480, Association for
Computing Machinery.
[17] Junying Li, Zichen Yang, Haifeng Liu, and Deng Cai,
“Deep rotation equivariant network,” Neurocomputing,vol. 290, pp. 26–33, 2018.
[18] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow, “Harmonic networks: Deep translation and rotation equivariance,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[19] David Romero, Erik Bekkers, Jakub Tomczak, andMark Hoogendoorn, “Attentive group equivariant convolutional networks,” in Proceedings of the 37th International Conference on Machine Learning. 13–18 Jul
2020, vol. 119 of Proceedings of Machine Learning Research, pp. 8188–8199, PMLR.
[20] David W. Romero and Mark Hoogendoorn, “Coattentive equivariant neural networks: Focusing equivariance on transformations co-occurring in data,” inInternational Conference on Learning Representations,2020.
[21] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton,“Cifar-10 (canadian institute for advanced research),” .