Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups
Authors: David M Knigge
David W. Romero
Erik J Bekkers
Venue: ICLR 2022 Submitted (2022)
Keywords: Group equivariance
separable convolutions
group equivariant neural networks

Abstract: Group convolutional neural networks (G-CNNs) have been shown to increase parameter efficiency and model accuracy by incorporating geometric inductive biases. In this work, we investigate the properties of representations learned by regular G-CNNs, and show considerable parameter redundancy in group convolution kernels. This finding motivates further weight-tying by sharing convolution kernels over subgroups. To this end, we introduce convolution kernels that are separable over the subgroup and channel dimensions. In order to obtain equivariance to arbitrary affine Lie groups we provide a continuous parameterisation of separable convolution kernels. We evaluate our approach across several vision datasets, and show that our weight sharing leads to improved performance and computational efficiency. In many settings, separable G-CNNs outperform their non-separable counterpart, while only using a fraction of their training time. In addition, thanks to the increase in computational efficiency, we are able to implement G-CNNs equivariant to the $\mathrm{Sim(2)}$ group; the group of dilations, rotations and translations.  $\mathrm{Sim(2)}$-equivariance further improves performance on all tasks considered.

Under review as a conference paper at ICLR 2022
EXPLOITING REDUNDANCY: SEPARABLE GROUP CONVOLUTIONAL NETWORKS ON LIE GROUPS
Anonymous authors
Paper under double-blind review
ABSTRACT
Group convolutional neural networks (G-CNNs) have been shown to increase parameter efficiency and model accuracy by incorporating geometric inductive biases. In this work, we investigate the properties of representations learned by reg- ular G-CNNs, and show considerable parameter redundancy in group convolution kernels. This finding motivates further weight-tying by sharing convolution kernels over subgroups. To this end, we introduce convolution kernels that are separable over the subgroup and channel dimensions. In order to obtain equivariance to arbitrary affine Lie groups provide continuous parameterisation of separable convolution kernels. We evaluate our approach across several vision datasets, and show that our weight sharing leads to improved performance and computational efficiency. In many settings, separable G-CNNs outperform their non-separable counterpart, while only using fraction of their training time. In addition, thanks to the increase in computational efficiency, we are able to implement G-CNNs equiv- ariant to the Sim(2) group: the group of dilations, rotations and translations of the plane. Sim (2) (e)equvvariiane further improves performance on all tasks considered, and achieves state-of-the-an performance on rotated MNIST.
INTRODUCTION
Minsky & Papert (1988) suggest that the power of the perceptron comes from its ability to learn to discard irrelevant information. In other words; information that does not bear significance to the current task does not influence representations built by the network. According to Min- sky & Papert (1988), this leads to definition of perceptrons in terms of the symmetry groups their learned representations are invariant to. Progress in geometric deep learning has shown the power of pro-actively equipping models with such geometric structure as inductive bias, reduc- ing model complexity and improving generalisation and performance (Bronstein et al., 2017). An early example of such geometric inductive bias at work can be seen in the convolutional layer in a CNN (LeCun et al., 1998). CNNs have been instrumental in conquering computer vision tasks, and much of their success has been attributed to their use of the convolution operator, which commutes with the action of the translation group. This property, known as equivari- ance to translation, comes about as result of the application of the same convolution kernel throughout an input signal, enabling the CNN to learn to detect the same features at any location in the input signal, directly exploiting translational symmetries that naturally occur in many tasks.
Although invariance object-identity preserving transformations has long been recognised desirable model characteristic in machine learning literature (Kondor, 2008; Cohen, 2013; Sifre & Mallat, 2014), only recently Cohen & Welling (2016a) 6a) introduced the Group Equivariant CNN (G-CNN) as natural extension of the CNN (LeCun et al., 1998), generalising its equivariance properties to group actions beyond translation. The layers of a G-CNN are explicitly designed to be equivariant to such transformations, hence the model is no longer burdened with learning invariance to transformations that leave object identity intact. has since been shown that equivariant deep learning approaches may serve as solution in fields that as of yet remain inaccessible to machine learning due to scarce availability of labelled data, or when compact model design due to limited computational power is required (Winkels & Cohen, 2018; Linmans et al., 2018; Bekkers, 2019).
Complexity and redundancy issues impeding regular group convolutions A growing body of work shows applications of G-CNNs consistently and decisively outperforming classical CNNs

---

Under review as a conference paper at ICLR 2022
G
|G]
H
W
a) G
b R
c) by H - d) : -
Figure 1: In group convolutions on affine Lie groups, a feature map f defined over the group, depicted in Fig. a) is convolved with a filter k shown in Fig. b) of size |G| and |G] the size the group. We propose factorising this convolution into a convolution over the group with kernel ky shown in Fig. e) with size |G| 1 x 1, followed by a convolution over the spatial dimensions with kernel kga shown in Fig. d) with size x x S.
8, where is the kernel size,
(Worrall et al., 2017; Weiler et al., 2018a; Bekkers et al., 2018; Esteves et al., 2018; Bekkers, 2019; Worrall & Welling, 2019; Sosnovik et al., 2021b). However, a practical challenge impeding application to larger groups is the computational complexity of regular group convolutions, which scales exponentially with the dimensionality of the group. Furthermore, Lengyel van Gemert (2021) show that group convolution filters in the original formulation of the G-CNN by Cohen & Welling (2016a) exhibit considerable redundancies along the group axis for the p4m and z² groups. Similar observations motivated depthwise separable convolutions (Chollet, 2017), which not only increased parameter efficiency but also model performance; observed correlations between weights explicitly enforced with further parameter sharing through the of kernels separable along spatial and channel dimensions. We address the observations of redundancy along with the scalability issues of regular G-CNNs in their current form. Our paper contains the following contributions:
We introduce separable group convolutions for affine Lie groups RR x H, sharing the kernels for translation elements E along subgroup elements h E H. See Fig. for an overview.
We propose the use SIREN (Sitzmann et al., 2020) as kernel parameterisation in the Lie algebra imposing fixed number of parameters per convolution kernel, regardless of the resolution at which this kernel is sampled, and ensuring smoothness over the Lie group.
Separable group convolutions allow us to build Sim Sim(2)CCNNs which we thoroughly experiment with. We show equivariance Sim(2) increases accuracy over range vision benchmarks.
To achieve equivariance to continuous affine Lic groups, we propose a random sampling method over subgroups H for approximating the group convolution operation.
First, we position this work within the area of equivariant deep learning by giving an overview of related works, and explaining which current issues separable group convolutions, and show how they may be applied to continuous groups. Lastly, we apply these ideas by experimenting with implementations for noto-translations in 2D (SE(2)), dilation and translation in 2D (R22 x +R+ and dilation, rotation and translation in 2D (Sim(2)).
addressing with this work. We derive
RELATED WORK
Group equivariant convolutional neural networks Broadly speaking, research on G-CNNs can be divided into two approaches. First, Regular G-CNNs use the left-regular representation of the group of interest to learn representations of scalar functions over the group manifold, or a quotient space of the group. The left-regular representation acts on the convolution kernels, yielding an orbit of the kernel under the group action. Convolving the input using these transformed filters, feature map defined over the group obtained at each layer. This approach most naturally extends the conventional CNN, where convolution kernels are transformed under elements of the translation group. Regular G-CNNs have been implemented for discrete groups (Cohen & Welling, 201 Winkels & Cohen, 2018; Worrall & Brostow, 2018), compact continuous groups (Marcos et al., 2017; Bekkers et al., 2018) and arbitrary non-compact continuous Lie groups (Bekkers, 2019; Finzi et al., 2020; Romero et al., 2020). However, practical implementations for continuous groups often require
2

---

Under review as a conference paper at ICLR 2022
some form of discretisation of the group, possibly introducing discretisation artefacts, and requiring choice of resolution over the group. For the second class, steerable G-CNNs, representation theory is used to compute basis of equivariant functions for given group, which are subsequently used to parameterise convolution kernels (Cohen & Welling, 2016b; Weiler et al., 201 Sosnovik et al., 2019; 2021a). Although steerable G-CNNs decouple the cardinality of the group from the dimensionality of the feature maps, this approach is only compatible with compact groups.
The current paper may, in approach, be compared to Bekkers (2019) and Finzi et al. (2020), who define convolution kernels on the Lie algebra of continuous groups to enable convolutions on their manifold. Similarly, we make use of the Lie algebra and exponential map to obtain convolution kernels on the group, but separate the kernels by subgroups.
Bekkers (2019) defines a set of basis vectors in the Lie algebra, which, when combined with the exponential map, allow for the identification of group elements by vector in IR". Subsequently, set of B-splines is defined on the algebra, which form basis to expand convolution kernels in. A linear combination of these bases creates locally continuous function on the Lie algebra defining a convolution kernel and its behaviour under transformations of the group. Although this method allows for direct control over kernel smoothness, the learned convolution filters are limited in their expressivity by their basis functions. Finzi et al. (2020) instead use an MLP to learn convolution kernels the Lie algebra, which in addition allows them to handle point cloud data. The MLP is constructed to learn kernel values at (arbitrary) relative offsets in the Lie algebra. In contrast, we propose to SIRENS (Sitzmann et al., 2020) parameterise convolution kernels, they have been shown to outperform other forms of MLPs in parameterising convolution kernels (Romero et al., 2021), and offer more explicit control over kernel smoothness: desirable property for addressing discretisation artefacts that occur when modelling features on continuous groups (see App. A.6).
Separable filters in machine learning In image processing, spatially separable filters have long been known to increase parameter- and computational efficiency, and learning such constrained filters may even increase model performance (Rigamonti et al., 2013). In Sifre Mallat (2014), authors investigate SE(2)-invariant feature learning through scattering convolutions, and propose separating the group convolution operation for afiin groups into cascade of two filterings, the first along the spatial dimensions R", and the second along subgroup dimension H. From this, authors derive a separable approach to the convolution operation with learnable filters as used in CNNs. This formulation has since been named the deptiivise-eparrable convolution (Chollet, 2017), special case of the Network-In-We In-Network principle (Lin et al., 2013) which forms the basis for the success of the Inception architectures (Szegedy et al., 2015). In depthwise separable convolutions, each input channel first gets convolved using a (set of) kernel(s) with limited spatial support. Afterwards, a 1x1 convolution is used to project the feature set detected in the input channels to the output space. Chollet (2017) speculates that the Inception architectures are successful due to the explicit separation of spatial and channel mapping functions, whereas in conventional CNNs, kernels are tasked with simultaneously mapping inter-channel and spatial correlations.
Haase Amthor (2020) argue that the original formulation of the segthwise-separabble convolution reinforces inter-kernel correlations, but does not in fact leverage intra-kernel correlations. Subse- quently, they propose an inverse ordering of the operations given cepthhwss-ssparrable convolutions, sharing the same spatial kernel along the input channels, and show convincing results. Extending this investigation of learned convolution filters to the original G-CNN (Cohen & Welling, 201 6a), Lengyel & van Gemert (2021) remark on the high degree of correlation found among filters along the rotation axis, and propose to share the same spatial kernel for every rotation feature map. We attempt to generalise this approach, proposing separable convolutions on arbitrary affine Lie groups.
BACKGROUND
In the following section we give the theoretical background for implementing group convolutions for arbitrary affine Lic groups. We assume familiarity with the basics of group theory and provide the relevant concepts in Appx. A. I and Appx. A.4. For simplicity of notation, we initially assume that input signal/feature map f has single channel.
Lifting convolutions To preserve information on the pose of features in the input, an equivariant convolution operation is achieved by lifting a function from the input space to (a homogeneous space of) the group. As we are interested in real signals, specifically image data living on R², we assume the Lie group of interest H is taken in semidirect product with the domain of our data; G=R² x H.
3

---

Under review as a conference paper at ICLR 2022
In group convolutions, a given kernel is left-acted by all transformations in G, thereby generating signal on the higher dimensional space G instead of RR. Hence, the output feature maps disentangle poses through domain expansion, e.g. positions plus rotations scales. For given group element E H, kernel k; and location o in the input domain R22 the lifting convolution is given by:
(f *utting k)(() f f (z)) x) dē.
where y= h) and kn Tes det Cr [k] is the kernel k TR2 - IR transformed via the action of group element via Ci [k] (a:) == k<h -lzx, 'x), and with det h the determinant of the matrix representation of h that acts R. The output of lifting convolutions yields G-feature map with the original two spatial input dimensions (TR2), and an additional group dimension (H). See Fig. 10.
Group convolutions Now that the data is lifted the domain of the group, we continue with group convolutions in subsequent layers. Given a kernel & (now a function on G), Haar measures dğ and dh on the group G and sub-group H respectively, group convolutions are given by:
(f group k)) = [c f(0)k(g -g) dg = =So ((T) dg
[ / f( h) [h|
dē dh
JR
[ [ f(e, (i x), ga
h) [h]
dā dh.
(2)
Evaluating this convolution for every group element g E G, we again obtain a function defined on G. As we know G=RR² H, we can factor this operation into transformation of a kernel & by group element E H, kn Cx.k), followed by a convolution at every spatial location in f. See Fig. 11.
Achieving invariance Using lifting and group convolution operations, can construct convolutional layers that co-vary with the action of the group and explicitly preserve pose information throughout the representations of the network. In most cases, we ultimately want representation that is invariant to transformations of the input in order to achieve invariance to these identity-preserving transformations. This is achieved by aggregating the information at all group elements in feature map with an operation invariant to the group action, e.g., max-, mean- or sum sum-projection. In practice, this is done after the last group convolution, and is followed by one or more fully connected layers.
4 SEPARABLE GROUP CONVOLUTIONS ON LIE GROUPS
Redundancies in group convolution filters Similar to Haase & Amthor (2020); Lengyel & van Gemert (2021) we investigate parameter efficiency of learned convolution kernels to motivate separa- ble filters. We train an SB(()-equivariant CNN on the Galaxy 10 dataset (see Sec. 5 for experimental details) and analyse the resulting group convolution kernels. We apply PCA by treating the val- ues of the group convolution kernel for each subgroup element h E H distinct spatial kernels with l x features. The ratio of variance explained by the first principle component gives an indication of the variability of the group convolution kernel along the subgroup axis. If the ratio of explained variance is high, the distinct spatial convolution kernels along the subgroup axis are well-characterrised by single shared kernel. In Fig. 2, results are shown for the group convolution layers in our SLE(2)-CNN before and after training. We find that during the training process, redun- dancy along the subgroup axis increases considerably. This motivates sharing single spatial kernel along the subgroup elements, which can be achieved by separating the group convolution operation into a convolution over the subgroup H, followed by a convolution over the spatial domain R.2.
Separable group convolutions Let us assume that the convolution kernel k G + R in Eq. 2 is separable. That is, kkg)==ka=( ( In this case, we can derive the factorised separable group convolution as (see Appx. A.2 for the full derivation):
( group kg) = [ / f(Ø,
f( To (#= - -x), 1i) [M
dā ch
= [ [ f(t h) 1 R) ah kg (h 1(x - xz)) [7
dā.
(3)
Here Kn is a convolution kernel over the group H, and kga is a convolution kernel over the spatial domain R22 We set kP= kig, with ki constant along IR2 and h.g. constant along H. This can
4

---

Under review as a conference paper at ICLR 2022
16384 samples
16384 samples
16384 samples
32768 samples
65536 samples
Trained Untrained
Trained Untrained
Trained Untrained
Trained Untrained
Trained
Untrained
2
2
2
2
8.0
0.5
1.0 0.0
0.5
1.0 8.0
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
Figure 2: A set of histograms showing redundaney in learned group convolution kernels. On the x-axis is the ratio of variance explained by the first principal component when applying PCA on the set of spatial kernels along the group axis of a group convolution kernel. Y-axis shows the proportion of kernels with this explained variance ratio, where all bins sum to 1. The number of spatial kernels is listed in the title of each subfigure. Throughout the training process, redundancy in the group convolution kernels increases. Left to right: subsequent layers in the network.
be thought of parameterising a convolution kernel k over the group G by sharing same spatial kernel kige weighed by a value k (h) at every input group element h E H, see Fig. 12. Importantly, this factorisation greatly increases the efficiency of the group convolution operation. Since we can precompute the inner integral in Eq. 3, convolving over a single channel of feature map I of size H| x goes from O(1 H xxxxxkk to O( xxxxx + k²))). See Fig. 13 for a visual intuition of the separable group convolution.
Defining convolution kernels on Lie algebras In order to perform lifting- and group convolutions (Eqs. 1, 2), we need to evaluate our convolution kernels k at relative offsets g' on the group. As in Bekkers et al. (2017); Weiler et al. (2018b); Bekkers 20: Finzi et al. (2020), we express our group convolution kernel & in analytical form, as function of relative group elements yielding kernel values. Motivated by findings in Romero et al. (2021). we use a Sinusoidal Representation Network (SIREN) (Sitzmann al., 2020) kernel parameterisation. SIRENS lead to great performance improvements over MI.s with ReL.U, LeakyReI.U and Swish when parameterising convolution kernels (we replicate this comparison for G-CNNs in Appx. C.4). Furthermore, the SIREN offers explicit control of kernel smoothness through a frequency multiplier parameter wy: an important property discussed in Appx. A.6. Since the affine Lie group can be eocledcii,ean, and neural networks are functions generally defined over euclidean spaces, we resort to defining the kernel function on the Lie algebra of our group of interest (Finzi et al., 2020). The kernel function kp maps points in the Lie algebra (which may be associated with the relative offsets on the group we are convolving over by the exponential map) to kernel values, kP g - RR. For given element y, we have: k: (g) (S))=S g), see Fig. 14. The separable kernels follow the same principle, separated by subgroups, see Appx. B.I.
Approximating equivariance for compact continuous and non-compact continuous groups In the case of small discrete subgroups H, such the group C4 of rotations by 90 degrees, we are able to perform lifting- and group convolution operations exactly equivariant to the action of the group, as the integral given in Eq. 2 is tractable. We consider image data in particular, which is defined over discrete grid z², a subgroup of IR2 making both integrals over the group discrete sums:
(f * &group k)(g)=
E ((lt, (e- h),h- h)
1 AdAh,
(4)
#E2? hEH
with Aë and Ah denoting the volume elements corresponding to the grid points. In the case of continuous groups, it is possible to either make discretisation of the subgroup H, or to approximate the group convolution by means of random sampling. To obtain volumetrically uniform sampling grid over the group, we sample a set of n equidistant points in the Lie algebra as in Bekkers (2019), and map those to the group using the exponential map to obtain grid H := [he,,.., hn]] see Fig. 15. For noncompact such as the dilation group R+, we localize the support the Lie algebra. In the case of compact continuous groups such SO(2) we approximate the integral by convolving over a uniformly spaced grid H. which is perturbed by left multiipllicctionn with a uniformly randomly sampled group element h ~ dh, i.e., h, + He := h. H.. By uniform sampling of H we obtain left-invariant Ah which only scales the overall convolution result, allowing us to omit it from 4. We further let A = 1 and obtain the discrete separable group convolution for continuous groups:
5

---

Under review as a conference paper at ICLR 2022
(f group kg) ~ Σ
E
h(a,h)k(h- 22)) 10), 1
#E22 NEH.
Σ
Σ
LieN. (at h) (h (h- (hi) kigz (h -1 ( x)).
(5)
BE22
During training and inference, randomly sampling over the rotation group yields an unbiased esti- mation (Wu et al.. 2019), making the network approximately equivariant to the continuous group SO(2). Discretisation makes the network exactly equivariant only to a discretised subgroup of SO(2), biasing the network to be equivariant to fixed subset of transformations. In our experiments we show that randomly sampling outperforms discretisation when modelling equivariance to SO(2).
Channel support of separable group convolution kernels In CNNs (and G-CNNs), a feature map
I' at layer generally has multiple channels C²: f IR2 + RC" Factorising the group convolution operation into distinct subgroup and spatial dimensions presents us with choice: can either (1) define both kif and kga over the input channels, or choose to (2) reduce the support of our spatial kernel single channel, and share reweighing of along the input channels an inverse depth-wiss separable manner, similar to Haase & Amthor (2020). An ablation study detailed in Appx. C.2 shows that on a fixed parameter budget, additional depthwise separation (2) consistently outperforms (1) defining both kernels over the input channels. As such, we will keep to the additional depthwise separation and refer to this implementation as the separable group convolution.
Expressivity of separable group convolutions Separable convolution kernels are strictly less ex- pressive than their non-separable counterpart, but to what extent would this limit the expressivity of their learned representations? In separable group convolutions, we are sharing weighted version of a single spatial kernel along the input group axis. In contrast, we could view non-separable group convolutions as having the ability to learn distinct spatial configurations of features along the group axis. For a visual example, see Appx. A.5. Although this reduction in expressivity could theoretically prove limiting in the application of G-CNNs on vision problems, in the next section we experimentally show that separable group convolutions in fact often outperform their non-separable counterpart.
EXPERIMENTS
We empirically motivate the use of seperable group convolutions by studying the performance of three G-CNNs that incorporate equivariance to three distinct groups. As our goal is to isolate the effect of separating the group convolution, we use the same shallow ResNet architecture throughout all experiments, only varying the sampling resolution over the group, see Appx. B. for details. After the last convolution block, we apply max-projection over the remaining group dimensions to achieve an invariant representation. We experiment with three different groups acting on R2: the roto-translation group (SE(2), the translation-dilation group (R2 x R+) and the group of rotations, dilations and translations (Sim(2)) For SE((), we first evaluate the influence of random sampling versus discretisation of S2).). Next, we assess the difference in performance of separable and non-separable group convolutions transformed MNIST variants; MNIST-rot, MNIST-scale for SE(2) and R22 x R* respectively. Due to the reduction in computational complexity separable group convolutions bring, we able model equivariance to higher-dimensional groups: we implement two (im(2)-CNNN and evaluate them on MNIST-rot-scale. Lastly, to investigate the ad advantages cquivariance brings in more complex problem settings, we experiment with all setups on three vision benchmark datasets; CIFARIO, CIFAR 100 (Krizhevsky et al., 2009) and Galaxy 10 (Leung & Bovy, 2019). See Appx. B.2 for more details on the datasets and training regimes used, and Appx. C.1 for an additional experiment empirically validating the equivariance properties of our models.
Discretisation versus random sampling over compact subgroups We investigate random sampling to approximate the group convolution. We trained SF( on rotated MNIST for different resolutions over SO(2), using random sampling and discretisation. Results are shown in Fig. 3. We can clearly the advantage of approximating the group convolution integral through random sampling. likely attributable to the fact that through random sampling we obtain an unbiased estimator for the convolution, whereas discretisation equates biased sampling of the rotation group.
Separable and non-separable G-CNN performance on MNIST variants Following Cohen & Welling (2016a); Weiler et al. (2018b); Sosnovik et al. (2019); Finzi et al. (2020) we conduct experiments on MNIST variants with separable and non-separable implementations of equivariant models for each of the three groups. We discuss results per dataset in-depth.

---

Under review as a conference paper at ICLR 2022
bacokip ap Doomrorr or
Baselin Separsbbee
300
Beeelie Saparec
Teat aroe
No reparable
22
1010
12 14
12
0010
group clonarts
gww.
Figure 3: Test error vs. (O(2) resolution for separa- ble SE(2)-CNNs on MNIST- rot, discretisation vs. ran- dom sampling.
Figure 4: Test error ver- sampling resolution of SO(2) on MNIST-rot for separable and non-separable SF(22-CNNS.
Figure 5: Test error ver- sus sampled extent of the dilation group for separable and non-separable R x R CNNs.
Rotated MNIST The SE(2)-CNNs are evaluated rotated MNIST, the standard benchmark for rotation equivariant models. To assess the influence of sampling resolution over the group, we vary the number of sampled group elements from 1 to 20. The group convolution is approximated through random sampling. Results are shown in Fig. 4. We see the influence of increasing the resolution over the rotation group from I to 4 elements, after which the non-separable model saturates, and performance starts to drop when further increasing the number of sampled rotations. This is in line with findings by Bekkers et al. (2018), who note that as the group resolution increases, so does the possibility for overfitting on specific spatial configurations. The separable implementation saturates around 7 elements, but performance does not drop significantly beyond this point. This may imply that the reduction in kernel expressivity also has a regularising effect that benefits generalisation. In this experiment, separable group convolutions decisively outperform the non-separable variant.
Scaled MNIST The 2² -CNNs are evaluated on MNIST scale. The group convolution integral is approximated through discretisation. In this experiment, we retain the same resolution over the dilation group in all experiments, but instead vary the value which the group is truncated, ranging from 1.0 at a grid of 1 element in the Lie algebra to 3.0 at a grid of 20 group elements. As in Sosnovik al. (2019), reduced performance, likely because of increase in equivariance error due to truncation of the group. Therefore, we limited support of the group convolution kernel on I+ to two neighbouring elements for all experiments in this paper. Results are shown in Fig. 5. Here, non-separable group convolutions outperform the separable variant, suggesting that the ability of non-separable group convolutions to model spatial feature patterns over different scales is beneficial for the scaled MNIST dataset.
found that the inter-scale interactions in the group convolution operation
Scaled rotated MNIST Lastly, the Sim (im(2)-CNN is benchmarked on MNIST-rot-scall We limit the sampling resolution at each layer of the G-CNN to 2,4,6 and 8 elements for each subgroup. We test two implementations: (1) separable, where the group convolution is factorised into two convolutions, one over k + SS( and one over IR2 and (2) H-separable, which factorises the convolution into three steps: a convolution over RR, one over SO(2), and one over IR?. As shown in Fig. 6 the separable implementation turned out to be unstable for low SO(2) resolutions. We suspect that an approximation of S0(2) of only 2 rotation elements, paired with possible aliasing effects and equivariance error occurring over the dilation group, impedes the model from learning robust representations. For higher resolutions over SO(2), this effect seems to decrease. The results for the //-spparable implementation (Fig. 7), highlight much clearer increase in performance as the resolution over both groups increases, which suggests decoupling these information mappings stabilises the learning process.
Application to computer vision benchmarks To evaluate the value of cquivariance to different transformations groups in natural image classifcation, we apply the SE _²2 x and Sim(2)- CNNs to three vision datasets. For SE(2) and IR2 xR+ we use resolutions of 4 and 8 group elements, and truncate the scale group at value of /3. as we found this to work well for our experiments. For S2m( we use a resolution of elements over the SO(2) group and elements over the R2² R++ subgroups and again truncate IR at v3. On CIFARIO, as GPU memory allows, we also experiment with using resolutions of 8 group elements for both subgroups. As baseline, we implement the same architecture of our nen-separable G-CNN, but restrict it to equivariance to translations. Results are summarised in Table 1 and exhibit the following general pattern: For most groups, separable group convolutions outperform non- nen-sapaee ones, and in all experiments best performance achieved by (im(()-CNNN))) Next, discuss results per dataset in-depth.
7

---

Under review as a conference paper at ICLR 2022
8.19 +.042
20.32
13.00
7:3g
20.15
6.56
+0.07
=0.08
+0.30
+0.87
6.49
gntp 8000
05 0t0 doeeee 5092
21.19
#:0.27
6:37
6.70
=0.32
4.99
4.54
40..33
5.4
20.45
4.8
2.0.25
6.64
2
6.75
+:0.25
5.00
=0.11
4.88
2:0.3
4.33
+:0.08
3.65
4.2
3.85
3.69
3.62
3.28
+.0.4
+0.07
+0.24
20.18
10.18
20.19
grou
no. group eterrntt
Figure 6: Test error ver- sus resolution over rotation and scale group for separable Sim(2) CNN.
Test error versus resolution over rotation and scale group for H-separable Sim(2) CNNN.
Figure 7:
Figure Process time per epoch (one pass over training and test sets) seconds for different resolutions H.
Table 1: Test accuracy (%) on different vision benchmark datasets. Best performance per group convolution implementation is underlined in blue. Best overall performance is boldfaced, Separable along rotation and scale dimensions.
GROUP
No. ELEM
SEPARABLE
CIFARIO
CIFARI00
GALAXY 10
X
74.50(+0.144
(=1.28)
85.58(=0:45)
78.30(10.75)
48.11 (=0.25)
86.96(00.07)
SF(2)
X
76.81(+0.57)
$1.67 (=0.60)
84.97(=0.07)
8
80.89(+0088)
50.19 (=0.40)
86.61(=0.38)
X
81.44(+:0.22)
44.80 (=0.56)
15.13
4
/
83,08(
52,50 (=0.60)
84.82(=0:25)
R R
X
80.56(1:0882)
(4.18(20.92)
84.22(0022))
/
$3.6001-0.23
53,26 (=0.110)
85. 00.06)
/
86.63/=0.600
49,93 (=0.86)
94.90
Sim
/1
85.65(100666
54.62 (=1.4)
87.45(=0.3)
89.38(
8 8
87.64(10.11)
Buseline
77.79(10.27)
47.91(=1.50)
85.10(1::422
CIFARIO First, we evaluate our models on the CIFAR10 dataset (Krizhevsky et al., 2009). Results in Table show that on CIFARIO, the non-separable variant of our (EEN)-CNN is outperformed by the translation equivariant baseline. With separable group convolutions and group resolution of 8 elements, SE(2)-CNF show improved performance. Interestingly, R x + -CNNs outperform both seems scale invariance is better inductive bias than rotation invari-
baseline and SE(2) (2(2)-CNNs, ance in CIFARIO. Combining scale and rotation invariance ultimately leads to the best performance, with separable (im and resolution of scale and 8 rotation group elements achieving 89.38% test accuracy. The H -eseparabe Sim Sim(2)-Ch) (2)-CNN, which models no interactions between scale and rotation, performs slightly worse.
CIFARIO0 To assess the impact of separating spatial and group information in more complex problem settings, we experiment with CIFARIO0 Notably, the non-separable IR² - perform well under baseline. We discuss this particular finding in Sec. 6. For SE(2)-CNNs, non-separable convolutions outperform separable ones, suggesting the model benefits from modelling distinct spatial configurations at different poses. Again, best performance is achieved by the Sim(2)-CNN.
Galaxy10 Lastly, we apply our models in the domain of astro-photography, on the Galaxy dataset Leung & Bovy (2019). Rotation and scaling symmetries exist naturally in this dataset, making it an interesting application for our invariant models. Here, results show little difference between performance of non-separable and separable approaches. Rotation invariance seems to be the stronger inductive bias in this dataset, but again, combining rotation and scale invariance leads to the best performance, with the H -ssearrbbee Sim(2)-CN) CNNN achieving 87.45% test accuracy. The higher separable Sim(2) test error seems to indicate that interactions between scale and rotation features for this dataset impede model performance, again, possibly due overfitting.
8

---

Under review as a conference paper at ICLR 2022
Table 2: Test error (%) on rotated MNIST for separable G-CNNs in comparison to other equivariant
baselines: G-CNN (Cohen & Welling. 201 6a), H-Net (Worrall et al., 2017), RED-NN (Salas et al., 2019), LieConv (Finzi et al., 2020), SFCNN (Weiler et al., 2018b), E(2)-NN (Weiler & Cesa, 2019). t Separable along dilation and rotations dimensions. + Train-time augmentation by continuous rotations.
Baseline Methods
Separable G-CNNs (Ours)
G-CNN
H-Net
RED-NN
LieConv
SPCNN
SFCNN,
ECZ)NN,
SE(2)
SE(2),
Sim(2))
Sim
2.28
1.69
1.39
1.24
0.88
0.714
0.68
0.89+.008
0.66+
0.66+.c00
0.59+.002 +.00
Table
Test error (%) on CIFARIO with AlI-CNN-C architecture, for our separable group convolu- tions in comparison to other baselines: All-CNN-C (Springenberg al., 2014), p4- and p4m-G-CNN (Cohen & Welling, 2016a), i Separable along dilation and rotations dimensions. + Train-time augmentation by random horizontal flips and random cropping. 1-Simm (1) (2)-CNNs where the SIREN hidden size in units. 6-Sim (2)im(2)-Cur) (2) have approximately equal numbers of parameters to the original AlI-CNN-C.
Baseline Methods
Separable G-CNNs (Ours)
A param.
1.37m param.
1 1.2m parmn.
1.14m panm.
1.30m param.
3.20m param
AB-CNN-C: p1-G-CNN
P4-G-CNN_
pln-G-CNN
plm-G-CNNN
3-Sim((;)
5-Sim(3))
6Siim(2))
8-Simi33)
16-Sim(3)
16-Simm23)
9,95
7,67
8.50
5.50
Comparison in training time between separable and non-separable group convolutions As one of the main motivators for the separable group convolution is its theoretical increase in computational efficiency, a comparison is made in training time between separable and non-separable G-CNNs. We tracked processing times per epoch, which consists of train- and inference procedure, for the rotated MNIST experiments, and show them in Fig. 8. Separable group convolutions present a remarkable decrease in inference time, which may be leveraged to allow for model equivariance to larger groups
SOTA on rotated MNIST To compare separable G-CNNs with related work, we finetune our models to competitive performance. With minimal adjustments to model and training regime, listed in App. B.2, we are able to achieve state of the art performance on MNIST-rot, see Tab. 2.
Competitive performance on CIFAR10 We compare performance of our approach in a larger model. To this end, like (Cohen & Welling, 201 6a), we re-implement the AlI-CNN-C architecture by Springenberg et al. (2014), using our Sim(2) convolution layers as drop-in replacement. To also compare performance absolute numbers of trainable parameters, we train three configurations, see App. B.2. We show that Sim(2)-CNNs, with SIREN hidden sizes of 6 and 5 units, containing similar or smaller numbers of trainable parameters to the original AII-CNN-C. improve model accuracy compared to CNNs G-CNNs equivariant to Sim(2)-subgroup I² and p4, and with data augmentation 6-Sim(2)-CNM) also outperform p4m-G-CNNs, see Tab. 3. Increasing SIREN hidden size 16 leads to even more significant improvements in performance.
DISCUSSION & FUTURE WORK
On implicit kernel representations Some of the non-separable configurations generated results below baseline, which is unexpected given that G-CNNs usually improve upon regular CNNs. We conjecture that these results are caused by the SIREN parameterisation, which take as input coordinate vector of different units (mixing both spatial and sub-group h coordinates). This could limit kernel expressivity, as sharing of a frequency multiplier wi sets kernel smoothness to be identical along spatial and subgroup dimensions. The fact that for the separable variant we use two distinct SIRENs would lift this restriction. Despite useful advantages of MLP parametrisations for the kernels, such as their flexibility and ease of implementation for arbitrary Lie groups, remains an open problem to have full control over their smoothness; ideally one band-limits the kernel MLPs to the discretised resolution on each axis, which would be an important direction for future work.
Conclusion Motivated by observed redundancies in learned group convolution filters, we introduced separable group convolutions, a computationally efficient implementation of regular group convo- lutions. We showed that separable group convolutions not only drastically increase computational efficiency, but in many settings also outperform their non- non-separable counterpart. Furthermore, we demonstrated the value of separable group convolutions larger groups; Sim -CNNs decivisely outperform models only equivariant to subgroups of Sim(2), clearly reinforeing equivariance means of model generalisation.
solution for modelling equivariance to

---

Under review as a conference paper at ICLR 2022
REFERENCES
Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2019.
Erik J Bekkers, Marco Loog, Bart M Haar Romeny, and Remco Duits. Template matching via densities on the rran-trannlation group. IEEE transactions on pattern analysis and machine intelligence, 40(2):452-466 2017.
Erik Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In Interna- tional conference on medical image computing and computer-assisted intervention, pp. 440-448.
Springer, 2018.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Taco Cohen. Learning transformation groups and their invariants. Master's thesis, University of Amsterdam, 2013.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, PMLR, 2016a.
Taco Cohen and Max Welling. Steerable cnns. arXiv preprint arXiv: 2016b.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Danilidis. Learning so (3) equivariant representations with spherical enns. Proceedings of the European Conference on Computer Vision (ECCV), pp. 68, 2018.
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. International Conference Machine Learning, pp. 3165-3176. PMLR, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick. Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: fttv:/770.0000 706.02667 2017.
Daniel Haase and Manuel Amthor. Rethinking depthwise separable convolutions: How intra-kernel correlations lead to improved mobilenets. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14600-14609, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1411v:1412.6980 2014.
Imre Risi Kondor. Group theoretical methods in machine learning. PhD thesis, Columbia University, 2008.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2324 1998.
Attila Lengyel and Jan C van Gemert. Exploiting learned symmetries in group equivariant convolu- tions. xixiv preprint arXiv:2106.04914, 2021.
Henry W Leung and Jo Bovy. Deep learning of multi-element abundances from high-resolution spectroscopic data. Monthly Notices of the Royal Astronomical Society, 483(3):3255-3277, 2019.
10

---

Under review as a conference paper at ICLR 2022
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv:1312.4400. 2013.
Jasper Linmans, Jim Winkens, Bastiaan S Veeling, Taco Cohen, and Max Welling. Sample effi- cient semantic segmentation using rotation equivariant convolutional networks. arXiv preprint 2018.
Diego Marcos, Michele Volpi. Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5048-5057, 2017.
Marvin L Minsky and Seymour A Papert. Perceptrons: expanded edition, 1988.
Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2754-2761, 2013.
David Romero, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Wavelet networks: Scale equivariant learning from raw waveforms. arXiv preprint arXiv:2006.0525% 2020.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.026//1 2021.
Rosemberg Rodriguez Salas, Petr Dokládal, and Eva Dokladalova. Red-nn: Rotation-equivariant deep neural network for classification and prediction of rotation. 2019.
Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for texture classification. arXiv preprint 1:03. 2014.
Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint (rXiv:0006009611, 2020.
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. arXiv preprint arXiv:1910.1109) 2019.
Ivan Sosnovik, Artem Moskalev, and Amool Smeulders. Disco: accurate discrete scale convolutions. arXiv preprint 106.v:2/06.02233, 2021a.
Ivan Sosnovik, Artem Moskalev, and Amold WM Smeulders. Scale equivariance improves siamese tracking. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2765-2774, 2021b.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1e.. 22066 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Maurice Weiler and Gabriele Cesa. General -22)eequivaria steerable cnns. arXiv preprint 19Xiv:991.0022. 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02547. 8a.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation cquivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018b.
Marysia Winkels and Taco S Cohen. 3d g-cnns for pulmonary nodule detection. arXiv preprint 1804. 04656, 2018.
11

---

Under review as a conference paper at ICLR 2022
Daniel E Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In
Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-584, 2018.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint arXiv; 16697, 2019.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetor, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9621-9630, 2019.
A THEORY AND BACKGROUND
A.1 GROUP THEORETIC PRELIMINARIES
In this section, group theoretical prerequisites used throughout the paper are briefly refreshed. This by no means intended as an exhaustive exposition of the fundamentals of group theory, we only introduce those concepts relevant to the current work.
Group. group defined by a set of group elements, along with a binary operator G - G, called the group product. The group product defines way to combine each pair of elements 91. 92 E G. For the binary operator to be considered a group product, it needs to satisfy four constraints:
1. Closure. G is closed under ;; for all 91, E G we have 91 92 E G.
2. Identity. There exists an identity element e s.t. for each E G. we have e . C g-
3. Inverse. For every element g E G we have an element g -1 G. s.t. 9 g e.
4. Associativity. For every set of elements 91 92. G, we have (g1 g2) 93 91 (g2 93).
Lie groups. A Lie group is a group of which the elements form smooth manifold. Since the group itself is not necessarily a vector space, combination of elements through addition or subtraction is not defined. However, to each Lie group G. algebra g may be associated, given by the tangent space of the Lie group at the identity T, GG). The Lie algebra may be interpreted as a vector space of infinitesimal generators of the group, set of elements from which we can obtain the group G, by repeated application.
Exponential and logarithmic map. The exponential map exp : g G is function mapping elements from the Lie algebra to the group. For many transformation Lie groups of interest this map surjective, and it possible to define inverse mapping: the logarithmic map which maps from the group to the algebra.
Semi-direct product groups. In practice, we are often only interested in data defined over IRd, and hence in this paper only consider affine Lie groups of the form G = Rd x H, where Rd is the translation group d dimensions and H a transformation Lie group of interest.
Group action. A group G may have an action a given space X. Given group element g E G and a set X, the group action 'T deñines what happens to any element of X' when we apply the transformation given by element g to it. This action is given by:
T G x X' + X' and T X X
(6)
such that for any two elements g. h E G, we can combine their actions into a single action; Ty = Tg Th. To avoid clutter, write the action (x)) g z. Note that the action of group G on domain X' also extends to functions defined on this domain, treated next.
Left-regular representations We extend the group action X' to square integrable functions defined on X; (X). Intuitively, any group action on X' induces an action on functions on X; as elements of the set X' are transformed, function on X is dragged along. Commonly, this is
12

---

Under review as a conference paper at ICLR 2022
expressed through the left-regular representations. Imagine we have function; f: -> R. Let's say we want to reason about the function ff after transformation by group element r; let us denote this transformed function f". We may inspect the value of this function for a given element of X by reasoning backwards from our transformed function. For example, to obtain the value of f' for the transformed element a', find what the value of was for before applying r. This is done by applying the inverse of action r to the transformed element a'. For any element of the set of square-integrable functions on X; f L2(X) the left-regular representation of g is given by:
L f f' and for d' (x)): (' (TT=:
Equivariance. An operator is equivariant with respect to group, if it commutes with the action of the group. For an operator & (x( + LY):)::
Ag E G:: Lg o + $ Cg.
A.2 DERIVING SEPARABLE GROUP CONVOLUTIONS
If we set the kernel & to be separable, meaning we parameterise it by multiplying kernel ky which is constant along the spatial domain and kernel lgg which is constant along the group domain:
k(g)) k(x;, h)
kga (())
(9)
We can derive separable group convolutions as:
(f *group k)(g) = JC f(ā)k(g .9) du(
[ h) -Ln -h--ih-af k( h)
dā dì
[n|
[
/ f(t,h)k(b 10alh--(6- ( x)) 1t),
dā dh
[7
H
1
((e (h (E (h h)
dē dh
[/]
[ I f (( Ti)kze (h -1:0-1110 2 x)) [7o
dā dh
[
[/1 [(ar, ((e,)))kn( πo) au kga -1 (ix xz)
dē.
(10)
[
A.3. EXAMPLES OF DIFFERENT LIE GROUPS
We implemented models equivariant to three different groups: SE(2), IR2 x IR+ and Sim In this section, we describe these groups more detail, and give definitions for the logarithmic map required in obtaining G-CNNs equivariant to these groups (Bekkers, 2019).
The translation group 2² The translation group in two dimensions RR, has group product and inverse for two elements I.g' = x' E R
g g (x + x')
(11)
T.
(12)
With logarithmic map:
log g = x.
(13)
The rotation group S0( The rotation group in two dimensions describes the set of continuous rotation transformations of the plane, and consists of all orthogonal matrices R with determinant 1.
Its group product and inverse for two elements g = Rg.g' R2 E SO(() is given by:
9.9' g Ro
Ro+o'
(14)
1 R
(15)
13

---

Under review as a conference paper at ICLR 2022
With logarithmic map:
0 mod 2m
-0 mod 2xx
log
(16)
The dilation group R The group of dilation transformations R+ has a group product and inverse that, for two elements g 5,9 = R+ given by:
g = 88'
(17)
h- s
(18)
The logarithmic map is given by:
logg g In 8.
(19)
The Special Euclidean group SE( The Special Euclidean group dimensions describes the set of geometric transformations that are formed by combinations of rotations and translations in two dimensions. Each group element can be parameterised by two variables and x, describing the rotation angle and translation vector, product and inverse are given by:
x) G. For two elements 9. g E SEE()), the group
9 g (x, 0) (x',0) (To(x') + x, 0')
(20)
g (-T_o(x),
(21)
As we can sec, to combine the two elements g.g' we apply the action of the rotation part of to the translation of g' before we combine with the translation of g. we combine elements semi-directly. SE(2) is semidirect product of the translation group IR² and rotation group SO(2). We write this as SE(2) R.2 (2).). To simplify implementation, we separate the logarithmic map into logarithmic maps for (2) and IR² in our implementation.
The
group I2a x IR+ Another group of interest to our research topic is translationddilation group I2² x R+, the group of translation and dilation transformations. Dilations occur frequently in natural images, in the form of scaling transformations objects and scenes, as the distance between camera and object differs between images. Group elements parameterised by scaling factor s and translation element x, = (x, 8) E G. For two elements g', group product and inverse given by:
g = (x,8) 8) (x's")
= (Tx(x') + x, ss')
(22)
g (-T,-1 (x),
(23)
To simplify implementation, we separate the logarithmic map into logarithmic maps for R+ and R22 our implementation.
The Similarity group S2m(2) The similarity transformation group is the semi-direct product of the roto-translation group SE(2) and the isotropic scaling group R* and defines dilation-rote wot---ransoaaoioon transformations in two dimensions. Each group element can be parameterised by three variables 0, and x. For two elements g, E Sim(2) the group product and inverse are given by:
g g' () ((x',0))
((T,(To(x))) +x, T.')) + 0), s8') ((T.(TT..)) (x)) xx(x' s5')
((T,(To(x') +)
(24)
g (TT))
(25)
In Eq. 24 we use the fact that the isotropic dilation group has no action on the rotation group. This consequence of the fact that I and SO(2) are both abelian groups, and may be taken in direct product to create the group of dilation-rotation transformations. We again separate logarithmic maps by subgroup.
14

---

Under review as a conference paper at ICLR 2022
a (2,2)
R?
Cx (k)
* {
}
(k)) E ER IR2
y
= y
x
fin
K
fout GGRR
Figure The convolution operation in CNN. In upper section of the figure above, a spatial kernel ie is transformed by group element (2222) R22 to yield Lu (k:). In the lower section, h is transforned under the action of each element of (a discretisation of) the group R², yield set K of translated copies of k. Convolving over Sin with each of these copies yields a new feature map Sout defined over G R22
A.4 VISUALISATIONS OF OPERATIONS IN (SEPARABLE) G-CNNS
To ease the reading experience of this work, in this section we provide some additional group theoretic perspective on regular CNNs, and additional visualisations and for number of operations used in (separable) group convolutions.
CNNs from group theoretic perspective We give brief treatment of ordinary CNNs The ordinary convolution operation used in neural networks requires definition of a convolution kernel k on R.? as we are modulating signal f which itself lives R.2. The kernel le is applied location in the input space R² to again yield function over R² Intuitively, this is the same as saying (1) we transform set of kernels K [L (k) R²). and (2) apply the convolution operation f using this set of transformed kernels K. By rying the kernel weights used throughout the translation group, learned
on every
convolution kernel under the action of every group element æ E IR2,, to obtain
features automatically generalised over spatial positions. This intuition is visualised in Fig. 9.
Lifting convolution In Fig. 10 we show a visualisation of the lifting convolution for the SE(2) group.
Group convolution In Fig. 11 we show a visualisation the group convolution for the SE(2) group.
Separable group convolution kernel In Fig. 12 show visualisation of separable group convolution kernel.
Separable group convolution In Fig. 13 we show visualisation of the separable group convolution operation.
Defining convolution kernels on Lie groups In Fig. 15 we show how obtain grid the Lie group (3)3)/SO) by mapping from its algebra. In Fig. 14 we show how we subsequently obtain a kernel on this group by defining SIREN its algebra.
15

---

Under review as a conference paper at ICLR 2022
8go
(2,2)
X
k: k:R² +R
R2
: SO(2)
L Low (k)
* ([co(Co()))EER [R] 10 SO(2)
H
y
fin : IR? +
K
fout G
R
Figure 10: An example of the lifting convolution for G = SE(2) R2 x SO(2). In the upper section of the above figure, a spatial kernel is transformed by group element (æ,0) SEE(2 with æ = (2, 2),0 0 = 90" to yield L Lo.0 (k). In the lower section, is transformed under the action of each element of (a discretisation of) the group SE(2). to yield set K of translated and rotated copies of Convolving fon with each of these copies yields new feature map fout defined over G = SF(2).
16

---

Under review as a conference paper at ICLR 2022
0g0
H
(2222)
AH
X
0c
y
:: G + R
RR2
:= S0(2)
Crian Loso (k)
H
[(C(Co())) s
H
y
fin : G + R
K
fout G + R
Figure 11: An example of the group convolution for discretisation of) G SE(() = IR SO(2). In the upper section of the above figure, a group convolution kernel & (defined over G) is transformed by group element (x, 8) SE(2) with (2, 0 90" yield C Lego (k). In the lower section, & transformed under the action of each element of (a discretisation of) the group SE(2), to yield set K of translated and rotated copies of k. Convolving fin with each of these copies yields
new feature map four defined over G SE(2).
H
y =
y
x
KH HHRR R
kRz² 2 R
k G - R
Figure 12: The separable group convolution kernel parameterises the kernel is G R as ky kg.. kn is defined over the subgroup H, with no spatial extent, whereas kga is defined only on R² with extent along H, weighting each instance of kga by its corresponding value for ky. In practice, is computationally advantageous apply the convolution operations in sequence.
the group. The full group convolution kernel k is obtained by repeating kga
17

---

Under review as a conference paper at ICLR 2022
H
8g
H
(2.2)
k
Cana (kH)
X
0c
y
y
R
H = S0(2)
x
kig
IR?
IR
Lagn Loza (kize)
H * E @ SO(2)
H
(Co R 10 s0(2)
y
fi G
K
Sont G -
Figure 13: An example of the separable group convolution for (a discretisation of) G SE(2)
R x S0(2). In the upper section of the above figure, a kernel k (defined over H) and kernel
kga (defined over R2) are transformed by group element (æ:, 0) SE(2) with (2, 2), 0 90 to yield L. La (kn) and Lu Lasp (kgz). In the lower section, kn and kga are transformed under the action of each element of (a discretisation of) the group SEE(). to yield sets K of transformed copies of 2 and a kga of translated and rotated copies of Convolving f.. with K h and Kga sequentially yields a feature map fout defined over G = SE(2).
18

---

Under review as a conference paper at ICLR 2022
k:p
log
G
Figure 14: An example of a localised kernel on the quotient group (0(3)/SO(2). Al- though the SIREN parameterising the kernel is defined on g. we can associate the kernel values sampled on g with the relevant ele- ments g' E G through use of the grid H. de- fined G, which we map g via the loga- rithmic map.
Figure 15: An example of a local kernel grid on the quotient group SO(3)/SO(2) We obtain volumetrically constant grid on G by sampling a set of equidistant points in and mapping them to G via the exponential map. The grid on G, show in this figure, then serves input for the SIREN defined on g via the logarithmic map.
0o
2270
g0
I
y
*i fting
H
Ag0
gg
Oo
x
y
x
fm
fort
-
R
Figure 16: An example of configuration of features four single separable group convolution kernel unable to represent.
A.5 EXPRESSIVITY LIMITATIONS OF SEPARABLE GROUP CONVOLUTIONS
As discussed, separable group convolution kernels are strictly less expressive than their non-separable counterpart. For example, in the case of the roto-translation group SE(2), this would enable non- separable group convolution kernels to learn to represent features that are built of different spatial configurations at different orientations.
We draw up a simplified example illustrated in Fig. 16. Assume we have an elementary feature type e, and a spatial kernel l with which we can recognise this feature type its canonical pose fy. In our input fin. we have three instances of the feature one under the canonical pose 6,, and two under 90º rotation. Applying the lifting convolution using kernel for the group G R2a C4 of translations and 90" rotations yields feature map fout defined over G, with spatial feature maps for lo.,, ..., O270. The spatial feature map for contains response single spatial position. In contrast, the feature map f contains response at two spatial positions. The spatial configurations for the feature maps along are different. A single conventional group convolution kernel could learn to recognise these distinct spatial configurations along the subgroup axis, whereas separable group
19

---

Under review as a conference paper at ICLR 2022
convolution kemel could not, since it simply repeats (a weighted version of) the same spatial kernel kga along the group axis.
Although this reduction in expressivity could theoretically prove limiting in the application of G- CNNs on vision tasks, our experiments show that in practice this rarely seems a problem, and may even help prevent overfitting.
A.6 KERNEL SMOOTHNESS FOR GROUP CONVOLUTIONS ON CONTINUOUS GROUPS
As mentioned, using SIRENs are able to explicitly control kerne smoothness. We briefly elaborate on the importance of kernel smoothness in G-CNNs. In conventional CNNs, weights at distinct spatial locations generally initialised independently. Because the kernels are only transformed using discrete translation operations æ E z²² translation equivariance is ensured by virtue of using the exact same weights values throughout all spatial locations.
In G-CNNs for continuous groups, the kernel is transformed under actions of a continuous transfor- mation group of interest H to obtain equivariance. However, in our convolution operation we are still using a discretised kernel; we are required to sample kernel values at different grid points for different elements h E H. We are no longer able to simply reuse the same weight values throughout the group as with regular CNNs. To this end, we define our kernels in an analytical form, which we can trivially evaluate at arbitrary grid points. Because our grid has a fixed resolution, the kernels sampled from this analytical form are susceptible to aliasing effects; the analytical kernel function may exhibit higher frequencies than can be captured in the discretisation the kernel. We visualise the effects of aliasing in Fig. 17.
In short, for continuous groups, the group action transforms a signal smoothly as the group is traversed. To prevent discretisation artefacts, want kernel to exhibit the same smoothly transforming behaviour, hence we use SIRENs, as they offer explicit control over the smoothness of kernels in their analytical form.
B ADDITIONAL EXPERIMENTAL DETAILS
B. ARCHITECTURAL DETAILS AND PARAMETERISATION
Model architecture As architecture for our experiments we use a simple ResNet model (He et al., 2016). We use single lifting convolution with 32 output channels, followed by two residual blocks of 2 group convolutions. The first block has 32 output channels, the second has 64 output channels. After the first residual block we apply max-pooling with kernel size 2 over the spatial dimensions of the feature map. After the last residual block, we apply max pooling over remaining spatial and subgroup dimensions, followed by two linear layers with batchnorm and ReLU in between. An overview is given in Fig. 18.
blocks and random sampling on the group In our residual blocks (He et al., 2016), we subsequently convolve the input to the block xn, by two group convolutions, gconv_1 and gconv 22, yielding Four and apply elementwise addition with the input Fi, followed by ReLU activation.
Group convolution
When approximating the group convolution through random sampling, we must take care to define the input and output of the two group convolution layers on the same grid over the group as the skip-connection to ensure a well-defined equivariant group convolution block. This may be done by adding group shortcut layer which maps from the set of input elements the group to the set of output elements. We implement this as a group convolution with 1 x spatial extent. The group shorteut layer thus simultaneously serves of gconv_1 to the output space of gconv_2 2 and maps from the input grid on H of the first group convolution to the output grid on H of the second group convolution.
channelwise projection from the input space
SIREN architecture and separable kernel parameterisation All our kernels are parameterised by SIREN (Sitzmann et al., 2020). In SIREN, output y for layer 1 and input ---1 is defined by:
y' sin l' + bl)
(26)
20

---

Under review as a conference paper at ICLR 2022
Hg
9.
h/1
H
Example of aliasing effects in analytical kernel parameterisations. Assume we are performing a lifting convolution to the SE(2) group. We evaluate two variants of our kernel function; kf exhibiting low frequencies over the kernel domain, and ky exhibiting high frequencies over kernel domain. We evaluate these functions for spatial grid Hg. to obtain a kernel under the identity rotation, and for slightly rotated version of this grid Hyg. We see the kernel sampled from ky changes smoothly, whereas the high frequency components in Kj lead to considerably different results for the two grids.
Figure 17:
Input
Group come: in: q. Loonv. blook in out q. q_oomv blook.ooo out k_stbe:
BatchNorm
InstanceNorm
Group g.
ReLu
ReLU
Droup comy: Loooory.bbe bww.bblcccccd out convooo.. www.bboooooooo out steer 7
Group conv block 32 out: 32 k kusibe
InstarceNorm
ReLU
BetchNorm
Betchhorm
MaxFoolR
ReLU
Linear 255 out: 10
Figure 18: Architecture used over all experiments.
21

---

Under review as a conference paper at ICLR 2022
Table 4: Number of trainable parameters for different implementations. For all groups and datasets. these numbers are kept constant.
NON- SEP-RAAB
SEPARABLE
H-SEPARABLE
742k
803k
864k
In this equation, wp acts as a multiplier for low dimensional frequencies found in the input domain (the grid of relative offsets the group), which explicitly introduces higher frequencies, allowing the neural net to learn high-frequency functions (such as kernels). We found a value for wp of 10 to work well in all our experiments. For the SIREN, we used an architecture of two hidden layers of 64 units. In non-separable G-CNNs we have single SIREN with a final layer mapping to a vector Re X In separable G-CNNs, we use two SIRENs, the first mapping to a function the Lie algebra of the subgroup H; R XCout, and the second mapping to RR Formulating the kernel k for group element g = (ar, h) in terms of kH, kgga, input channel i and output channel j. and logarithmic map H logn, we obtain:
k = ktH (logn 2
(27)
Lastly, for H-separable Sim IN-NN we use three SIRENs, the first mapping to a function the Lie
algebra of the subgroup R+; R K the second mapping (2) to R and the third mapping R2 to Root one mapping to SO(2). Formulating the kernel & for group element g ((ar, 0), s)
in terms of kso(2), kg-, liga, input channel and output channel j, logarithmic maps logR and
logso(2) on IR+ and S0(2) respectively we obtain:
k ()) k (logæ+ s) (logsor2) (x)
(28)
Model sizes In Tab. 4, we report the number of trainable parameters for each model configuration. Throughout our experiments, we kept the number of channels in all our configurations constant, to fairly compare the expressivity of the learned representations in non-separable and separable group convolutions. This has as effect that the number of parameters in the separable implementations larger than for non-separable implementations, due to our use of separate SIRENs to parameterise kernels over the different subgroups. To ensure that the difference in number of trainable parameters does not influence the comparison between separable and non-separable group convolution layers, we explicitly chose to over perr-aaameterise our SIREN architecture, as shown in an additional ablation in App. C.3. We only change SIREN hidden size when comparing our models to baselines proposed in related works as detailed in App. B.2.
B.2 EXPERIMENTAL DETAILS
Here, we list training regimes for all experiments. We keep these consistent as possible, as we are only interested in the effect of separating the group convolution operation, and in isolating the effect of incorporating equivariance into our models. We list information all datasets, and any dataset-specific model configuration details we use in our experiments here.
Optimizer All architectures are trained with Adam optimisation (Kingma Ba, 2014), and 1e weight decay. All models are trained on a single Titan V.
Rotated MNIST The 62.000 MNIST images (LeCun et al., 1998) are split into a training, validation and test set of 10.000, 2.000 and 50.000 images respectively, and randomly rotated to orientations between [0, 2T)). Note that in (Weiler al., 2018b: Weiler & Cesa, 2019), the rotated MNIST dataset is augmented during training by transforming images with random continuous rotations. We only use train-time augmentation for the state-of-the-are results obtained in Tab. 2, as detailed in App. B.2. All models trained on rotated MNIST, except for the state-of-the-at runs detailed in B.2, are trained for 200 epochs with a batch size of 128 and learning rate of 10-
Scaled MNIST The 62.000 MNIST images again split into training, validation and test set of 10.000, 2.000 and 50.000 images respectively, but now randomly scaled by factor [0.3, 1] and padded with zeros to retain the original resolution. No aateentgmentssoons of any kind are used in the experiments on scaled MNIST. All models trained on scaled MNIST are trained for 200 epochs with batch size of 128 and learning rate of 1 10-1
22

---

Under review as a conference paper at ICLR 2022
Scaled rotated MNIST The 62.000 MNIST images again split into training, validation and test set of 10.000, 2.000 and 50.000 images respectively, but now randomly scaled by factor [0.3, 1], padded with zeros to retain the original resolution and randomly rotated by orientations between [0, (T). For the experiments on scaled rotated MNIST we do not use data-augmentation of any kind. All models trained on rotated scaled MNIST are trained for 200 epochs with a batch size of 128 and learning rate of 1 10-
We evaluate our models on the CIFARIO dataset, containing 62.000 32 32 color images in 10 balanced classes (Krizhevsky et al., 2009). All models trained on CIFAR10 10 are trained for 200 epochs with batch size of 128 and a learning rate of 1 10
CIFARIO
CIFARI00 We evaluate our models on the CIFARI00 dataset, containing 62.000 32 x 32 color images in 100 balanced classes (Krizhevsky et al., 2009). All models trained on CIFARIO are trained for 200 epochs with batch size of 128 and a learning rate of 1 10-4
Galaxy 10 This dataset contains 21785 69x69 color images of galaxies divided into 10 unbalanced classes. For Galaxy 10. we limit batch size to 32 images, and scale learning rate accordingly to 2.5 10-5, as suggested by Goyal et al. (2017).
Achieving SOTA on rotated MNIST To compare performance of our separable regular G-CNNs with previous work, we apply our implementation of SF(() and Sim(2)-CNNs to achieve state of the art results on the rotated MNIST dataset, discussed in Sec. 5.
We make some slight adjustment to our architecture and training regime: we reduce the SIREN to a single hidden layer of 64 units. The convolution over SO(2) is approximated using random sampling, while the convolution over RR is approximated using a discretisation with a truncation of the scale group at v3. We use batch size of 64, and a learning rate of 10-4 and train for 300 epochs. To fairly assess the impact of equivariance, we train number of models without continuous rotation augmentations.
To compare with previous SOTA, achieved by Weiler & Cesa (2019), we also train models with continuous rotation augmentations. Weiler & Cesa (2019) does not specify performance results for models without train-time augmentations, therefore we also compare with the prior SOTA by Weiler et al. (2018b), which trains models both with and without augmentations.
We show results in Tab. 2. With SE(2)-CNNs we achieve best performance with a resolution of 20 elements over SO(2). For the (im we achieve best performance with a resolution of 12 elements over SO(2) and 5 elements over R+ These results show that without train-time augmentation
outperform the previous SOTA by Weiler & Cesa (2019), which was trained with continuous rotation augmentations. This improvement further increases when we add rain-time augmentation by continuous rotations, to a test error of 0.59%.
Achieving competitive performance on CIFARIO To further compare the viability and perfor- mance of our separable approach to group convolutions, following Cohen & Welling (2016a) we re-implement the AlI-CNN-C proposed by (Springenberg et al., 2014), using our separable group con- volution layers as drop-in replacement for the regular convolution layers. All experiments described in this section use this exact architecture.
In this experiment we looked to isolate the power of our separable group convolution layers in larger-scale models, which why we deliberately chose a large but relatively simple architecture. We keep the number of channels constant with the original implementation by (Springenberg et al., 2014). Because of the way our kernels are parameterised, it is hard to make a direct comparison in model performance in terms of representation expressivity, while keeping the number of trainable parameters equal to the original model. Therefore, we train three configurations, with different SIREN hidden sizes.
To obtain a model with approximately the same number of trainable parameters, for the first configu- ration we limit the hidden size of our SIRENs 6 to arrive 1.33m parameters, where the original model has 1.4m. Note that this limitation likely has considerable implications for the expressivity of the sampled group convolution kernels.
To push our separable group convolution layers further, we also train a model configuration with a SIREN hidden size of 5 units, arrive at a model with 1.14m parameters (19% smaller than the original model by Springenberg et al. (2014)).
23

---

Under review as a conference paper at ICLR 2022
1.0
1.0
Test soarrac
0.8
accuracy
0.8
0.8
elementts 8 elements
0.8
elemennts
Depemmnts
0.4
0.2
90
180 Rotation
270
380
90
180 Rotation
270
360
Figure 19: Test error vs rotation angle of MNIST test set, when trained upright MNIST, for separable SE(2)-CNN.
Figure 20: Test error vs rotation angle of MNIST test set, when trained on upright MNIST, for non-separable SE(2)-
Lastly, to see how our layers would fare when unimpeded by limitations SIREN size, we train a configuration with SIREN hidden size of 16 units. Note that this model contains 3.22m parameters (representing an 128% increase in parameter count to the original model).
All models trained with resolution of elements randomly sampled over SO(2) and discreti- sation of the scale group of 3 elements truncated at v3. We train all of these configurations both with and without data augmentation by padding the original image by at most 6 pixels and randomly cropping the original resolution, and random horizontal flips. All models are trained for 300 epochs with learning rate of 1 10-4.
Results shown in Tab. 3. We see that for approximately equal parameter counts, our separable group convolution layers equivariant to Sim(2) outperform the original CNN baseline and the p4-G-CNN. Without data augmentation, our model with hidden units is outperformed by the p4m-G-CNN which is also equivariant to reflections. With augmentation (containing reflection augmentations) our model outperforms all baseline models we compare to by a significant margin. The smaller configuration with 5 units seems to limit kemne expressivity somewhat, although without data augmentation it still outperforms the translation equivariant original implementation of the AlI-CNN-C. It seems data augmentation throws the model off in this particular configuration. The larger configuration with 16 hidden units, both with and without data augmentation, significantly outperforms all baselines and other configurations.
C ADDITIONAL EXPERIMENTS
C.1 VALIDATING MODEL INVARIANCE TRANSFORMATION GROUPS
Following Weiler et al. (2018b), we empirically assess the level of model-invariance to rotation transformations, by training on upright MNIST, and subsequently assessing performance on test set of MNIST images that have been transformed by a subgroup element E H.
Each model is trained the MNIST training set containing 60,000 images, and tested 10,000 transformed images. Performance tested for rotations over a range between [0, 2mr, in 100 steps. The group convolution is approximated through random sampling. Results are shown in figures 19 and 20 for 1, 2 and 8 group elements for the separable and non-separable variants. We can see that equivariance error very similar for separable and non-separable variants, and greatly reduces with increased resolution over the group. We do note that for 8 elements, the non-separable group convolution achieves slightly better performance.
C.2 SUPPORT OF THE SEPARABLE CONVOLUTION KERNELS OVER THE CHANNEL DIMENSIONS
As explained in Sec. 4, choose to either define both ky and kg the input channel dimension, or incorporate depthwise separability as well; additionally sharing the same spatial kernel kga over the input channels well as the group input elements. In additional ablation study
24

---

Under review as a conference paper at ICLR 2022
Baseline
Group separable
Group+dephhvise separable
Test arror
Test ersterro emor
20
Baseline
Group separable
Grouptdeplhuwsse separable
no. group elements
no. group elements
Figure 21: Results for depthwise group sep- arable and group separable S CNN rotated MNIST for different resolutions on the group.
Figure 22: Results for depthwise group sep- arable and group separable I2² x - on CIFAR 10 for different resolutions on the group.
find that any resolution above | 1, additional depthwise separability outperforms defining both kernels over the input channel dimensions
fixed parameter budget.
For this experiment on rotated MNIST, we use the same depthwise group separable SF((2) -F(2)-CNN CNN as in Sec. 5, with the architecture described in App. B.1. For the non-depthwise group separable -CN),, we use single residual group convolution block of 48 channels instead two of 32 and 64 channels respectively to arrive approximately the same number of parameters (815k for the implementation without depthwise separability, 803k for the implementation with depthwise separability). We train the models for 150 epochs, using adam optimisation, with learning rate of 1.
We repeat the same experiment for IR2 R CIFAR10, for resolution of 2, and 4 elements. Architectures and training regimes used for the non-depthwise group separable and depthwise group separable experiments are identical to above.
Results, shown in Fig. 21 and Fig. 22 for rotated MNIST and CIFAR10 respectively, highlight that additional depthwise separability outperforms non-depthwise group separable convolutions fixed parameter budget, which is why we stick with additional depthwise separation of the spatial kernel, and name this approach the separable group convolution.
C.3 HOw DOES SIREN ARCHITECTURE INFLUENCE PERFORMANCE?
As explained, keep the architecture of SIRENs constant all experiments. Because multiple SIRENs in separable architectures, and we want to isolate the effect of separability of the group convolution operation, so as not to advantage the separable implementations by their increased parameter numbers. To this end, we look for the largest size of hidden layer that does not negatively impact performance in the non-separable implementation.
deliberately choose over-parameteric the SIREN architectures,
We assess the impact of the size of the hidden layers in our SIREN on performance on rotated MNIST for separable and non-separable SF(2)-CNN implementations. We sample at resolution over S0( of 8 elements. Interestingly, results figure 23 show that even with very small SIRENs hidden size of 4 or 8 performance of the non-separable implementations is remarkable. Separable implementations clearly underperform for small SIREN sizes, leading us to hypothesize that the restriction in kernel expressivity due separability paired with prammtttrtttnnoooooonnnn is detrimental to performance in these configurations.
For non-separable implementations, performance barely increases from 16 hidden units on, and starts to decrease after a size 64 hidden units. For this reason, we choose a size of 64 hidden units in all SIRENs in experiments.
25

---

Under review as a conference paper at ICLR 2022
Test accuracy MNIST-rot, SE(2)-CNN
Test accuracy
0.98
Our models
Separable
0.96
Nonseparable
16
32
64
128
256
512
SIREN hidden
Figure 23: Test accuracy of SE(2)-CNNs on rotated MNIST for different sizes hidden layers in the SIREN parameterisation.
Test accuracy on CIFAR10, 82 + CNNN
Test accuracy
0.8
Our models
Separable
0.7
Nonseparable
126
SIREN hidden size
Figure 24: Test accuracy R² + -CNNs on CIFARIO for different sizes hidden layers in the SIREN parameterisation.
We repeat the same experiment for R² x -CNNS on CIFARIO, and visualise the results in Fig. 24. Here, performance for very small SIREN configurations is less impressive, but we still a similar saturation around a SIREN size of 64 hidden units.
C.4 COMPARING ACTIVATION FUNCTIONS FOR CONTINUOUS CONVOLUTION KERNELS
To assess the impact of our of SIRENs the performance of our separable G-CNNs, perform an additional ablation on rotated MNIST in which we compare with other activation functions. We train separable SE(2)-CNNS with resolution 8 group elements SO(2), and only vary activation functions in the MLP parameterising the convolution kernlss. The convolution on SÖ(2) is approximated through random sampling.
We look at other activation functions used in parameterising convolution kernels; ReLU (Wu et al., 2019), LeakyReLU, and SiLU (Finzi al., 2020), see Tab. 5. These results conclusively show the power of sine as activation function, one of the reasons we chose to use SIRENs in our work.
Table 5: Test accuracy (%) of SE(2)-CNNs rotated MNIST for different activation functions used in parameterisation the convolution kernel.
ACTIVATION
SEPARABLE
TEST ACCURACY
0.9855 (+.0012)
Sine
/
0,9906 (=0002
0,9721 (t.0003)
ReLU
0.9743 (+.0034)
0.9722 (+.0052)
LeakyReLU
0.9788 (+.0013)
0.9651 (+.0012)
Swish
0.9595 (+0045))
26

---

Under review as a conference paper at ICLR 2022
Sampling method over x group, MNIST-scale
10
Baseline
Discrefisation Random sampling
6010
Test efror
10
3x100
2
12
20
No, group elements
Figure 25: Test error (%) of R22 x R*-CNNs on scaled MNIST for random sampling versus discreti- sation of the scale group.
16384 samples
16384 samples
16384 samples
32768 samples
65536 samples
60
60
60
Trained
Trained Untrained
Trained
Trained Untrained
Untrained
Untrained
Untrained
100
40
40
40
50
8.0
0.1
0.2 0.0
0.1
0.2 8.0
0.1
0.2
8.0
0.1
0.2
8.0
0.1
0.2
Figure 26: set histograms showing variance along the subgroup axis separable group convolutions. In contrast Fig. 2, here show the variability of the convolution kernel along the subgroup axis. This may be seen as an inverse measure redundancy. Here we can see that contrary in non-separable group convolution kernels, variability along the group axis increases over the training process, indicating that redundancy decreases. Separable group convolutions thus solve the redundancy issues in group convolution kernels.
C.5 RANDOM SAMPLING OVER NON-COMPACT GROUPS
As explained section 4, for non-compact groups we approximate the group convolution through discretisation of the transformation group of interest. Motivation for this decision is the fact that in non-compact groups, we have to deal with boundary effects of truncating the group, introducing equivariance еrror into the convolution operation. For dilation, further motivation for approximation through discretisation is the loss of information that occurs in for example natural images, as result of downscaling signal on fixed resolution grid. Random sampling over the dilation group would have as effect that the representation built by group convolution layer contains different spatial resolutions of information at every sampling step. This results in subsequent layers receiving noisy information at each iteration, which would impede model performance, with training being highly unstable for low resolutions over R
To empirically reinforce this hypothesis, we performed the same experiment with our I2 x R on MNIST-scale described in section with random sampling over the dilation group. Results, shown in figure 25, highlight that indeed, random sampling the dilation group leads to significantly worse performance. We also noticed great instability during the training process, seen in the variance of the results achieved with random sampling.
27

---

Under review as a conference paper at ICLR 2022
C.6 Do SEPARABLE GROUP CONVOLUTIONS REDUCE REDUNDANCY IN G-CNNS?
As explained, convolving with separable group convolution kernels is analogous to a group convolu- tion with kernel which shares a reweighting of single spatial kernel along the group axis. This approach was motivated by redundancy observations in regular group convolution kernels along the group axis, seen in Fig.
To assess whether the separable group convolution solves these redundancy issues in G-CNNs, we may want perform a similar test of the variability of our introduced separable implementation along the group axis. Note that, if we were to reconstruct the full group convolution kernel k from ky and kig and apply the same PCA variance test as used to obtain Fig. 2. we would find that each group convolution could be fully explained by the first principal component (corresponding to the reshared spatial kernel). Instead, we measure the variability of our separable kernels by assessing the variance of kh. which lies along this axis. Results are shown in figure 26 for the separable SE(2)-CNN trained on Galaxy 10 dataset with a resolution of 8 elements on (2( Here, we observe the variance increasing in all layers over the training process. This indicates that indeed, separable group convolutions reduce parameter redundancy in G-CNNs.
28


Reviews:
[{"review_id": "xFQ9BH0hCHu", "reviewer": "Reviewer_4A1c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nS1: The paper is well-written, the method introduced makes sense and seems to work well in practice. \n\n*Weaknesses*\n\nW1: The submission has somewhat limited novelty. It can be seen as an application of depthwise convolutions to group-CNNs. Depthwise convolutions are quite popular since at least 2017 (Xception, Mobilenets); and the (non-separable) group-CNN utilized seems the same as Finzi et al [1], with sinusoidal activations. More importantly, the idea of separating the group-convolution along the group dimension has also appeared in Lengyel and van Gemert [2], although for a different flavor of G-CNN. \n\nW2: I see several issues with the experimental section. The most problematic is that all comparisons are against the paper's own baselines. While this makes for fair comparisons in the sense that the data pipeline, architectures and training schedule are all the same (although I have reservations about the number of parameters, see W3), I'm not sure if the conclusions hold in general. For example Weiler and Cesa [3] seems to show significantly better performance than the proposed method on rotated MNIST and CIFAR10/100; can the proposed method improve those results? If not, what would be the applications where the method is useful? Finzi et al [1] also show favorable results on a molecular property prediction, how does the submission fare in that task? When comparing with Finzi et al, I believe there should also be an ablation to disentangle the effects of the separable convolutions and the use of SIREN instead of a regular MLP.\n\nW3: Please report the number of parameters for each model trained. If I understand correctly, the number of channels per layer is kept constant, so the non-separable models have several times more parameters than the separable, is that correct? In that case could the lower performance on non-separable be explained by overfitting or slower convergence? I think both constant number of channels per layer or constant number of parameters are informative and should be reported (Weiler and Cesa [3] do something similar for CIFAR). \n\nW4: In Table 5, I believe a few experiments are missing and would be needed to disentangle the effects of the separation along the group dimension and the channel dimensions. I suggest to show, for each group, the performance when the convolutions are separable over the channel (depthwise) but not the group dimension. And for the baseline, it would be interesting to also see the performance for the separable depthwise version. \n\n*Questions*\n\nQ1: For the rotated MNIST, it is shown that approximating the convolution with random samples is superior, however the other MNIST variations seem to fall back to the discretization. Why is this the case? Do the random sampling approximation perform worse on the higher dimensional groups?\n\nQ2: The SIM(2) MNIST experiment is described as limited to 2, 4, 6, or 8 elements for each subgroup. Does this also refer to the way the dataset is constructed, or is it created with random rotations and scaling sampled from the continuous internal?\n\n*References*\n\n[1] Finzi et al, \"Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data\", ICML'20.\n\n[2] Lengyel and van Gemert, \"Exploiting Learned Symmetries in Group Equivariant Convolutions\", ICIP'21.\n\n[3] Weiler and Cesa, \"General E(2) - Equivariant Steerable CNNs\", NeurIPS'19.         ", "summary_of_the_paper": "The paper proposes to use separable convolutions along the group dimension in the type of group CNNs proposed by Finzi et al [1]. The motivation is the same as the popular depthwise separable convolutions for conventional CNNs: reducing parameter redundancy to increase efficiency and accuracy. Experiments on rotated MNIST, CIFAR10/100, and Galaxy10 show that the method outperforms the non-separable versions both in accuracy and speed.\n", "summary_of_the_review": "I believe the paper is not there yet, mainly because of limited novelty when compared to [1, 2] (W1), lack of comparison against the state-of-the-art of the tasks proposed (W2), and not very convincing experimental results due to lack of details (W3) and ablations (W4).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oUlRMSoE_x", "reviewer": "Reviewer_m8cZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The reviewer is very interested in this research work. This paper explores the shortcomings of existing work in depth, discusses separable convolution kernels, and does the parameterization of different groups more comprehensively, which  is a good research guide. In addition, the authors also conducted experiments on many vision datasets to illustrate the superiority of the proposed method and achieved performance improvements.\n\nHowever, there exist some confusing parts or weaknesses to be further strengthened.\n1. The biggest shortcoming of this paper is its limited innovation. As stated in this paper, the proposed method is more like a promotion of Cohen2016, by using a strategy similar to ‘depthwise separable convolution’ in Xception. It seems that this work depends on the existing method heavily and its original parts bring weak contributions.\n\n2. The theoretical foundation of this paper is not solid. It is worth mentioning that the authors explored arbitrary affine Lie groups (three different groups: SE2, dilation +translation, and dilation +Sim(2)). But as far as the reviewer knows, this is not entirely applicable. For example, correlation is an equivariant map for the translation but NOT for the rotation group. Why can the authors directly consider SE(2) and Sim (2) in convolution (both containing Rotation)? In addition, when proving translation equivariant, it is necessary to extend f and k to group G, otherwise g^(-1)x does not belong to group G and cannot go on. The reviewer has not seen relevant explanations. Regarding Sim(2), it has a totally different Lie algebra expression from SE(2)’s due to the scaling, and there has not existed a discussion on specific forms. \n\n3. The motivation of the proposed method comes from parameter redundancy, but from Figure 1, the reviewer found that the metric is F-test (variance ratio), which is usually used to analyze statistical models that use more than one parameter to determine whether all or part of the parameters are suitable for estimating the population. And the robustness of F-test is extremely dependent on the samples with a normal distribution. In the case of limited samples, the distribution is often biased, which tends to reduce the power of such a test, leading to incorrect statistical inference.\n\n4. The authors used SIREN as kernel parameterization in the Lie algebra. But as far as the reviewer knows, this work proves the superiority of using periodic activation functions (such as sine), which is, however, irrelevant to this paper from the perspective of the reviewer. Much more details should be provided here.\n\n5. Another concern of the reviewer is that evaluation results of this paper are not convincing to the reviewer. Traditional CNN-based methods have achieved 95+% accuracy on CIFAR10, and regular GNNs (p4m & Z2) can also reach 90+%. The accuracy results of this paper are still inferior to 90% with a 77% baseline. It violates the claim that \"G-CNNs usually improve upon regular CNNs\" stated in this paper.\n\n6. Some parts are not novel at all, such as the section “Depthwise separability”. The reviewer suggests the authors to include more relevant references (Xception or more). \n", "summary_of_the_paper": "This paper discusses considerable parameter redundancy in regular group convolution networks and then proposes separable convolution kernels to share the weights over the subgroups. Besides, the authors explored the equivariance of three different groups and presented a continuous parameterization scheme. Evaluations on several datasets show that their proposed method gains improved performance and computational efficiency.", "summary_of_the_review": "The research field of this paper is very interesting, but unfortunately there remain many shortcomings in theoretical and experimental results, which need to be forcefully addressed. The reviewer therefore thinks that the paper is not good enough to be published on ICLR.\n\nAfter reading the authors' rebuttal, I increase my rating a bit. But I am still not contented with the stuff in this submission, which should undergo a thorough revision to convince and be easily understood by the readers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "PTQfuw_hx3E", "reviewer": "Reviewer_hC39", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are as follow :\n\n1/ Clarity of the theoretical and empirical parts\n\n2/  Experimental results are promising\n\n3/ The use of SIREN  to parametrize convolutional kernel is a tricky idea and quite original since it is originally proposed for coordinates space data. As a result, a given layer has a fixed number of parameters regardless the resolution of data\n\n4/ Approximating the group convolution integral through random sampling shows a clear advantage over discretization schemes. It allows to mitigate some artifacts of raw data \n\nThe weaknesses are as follow :\n\n1/ The proposed model is quite generic. In order to show its scalability, one may wonder if the proposed model could be extended to large scale vision tasks : 3D rendering, Video classification and also to other domains like physical processes where group of symmetries are important.\n\n2/ Lack of comparison with state-of-the art models\n\n3/ Need a further theoretical study \n\n\nSmall typos in appendix, before equation (9) \"the group group\"\n\n", "summary_of_the_paper": "The paper adresses redundancy in group convolutional filters and the scalability of group ConvNets. It is achieved by introducing a separable group convolution for Affine Lie group which allows to share the kernels for translation elements.", "summary_of_the_review": "Following the aforementioned consideration, l recommend to accept the paper. The proposed research direction is promising and could open perspectives to several domains, mainly physical process tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N1kqS_Ta9qT", "reviewer": "Reviewer_xJAv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a solid paper to explore the combination of group convolutions and techniques employed in popular CNN architectures. The ideas of how we can achieve $Sim(2)$-equivariant on separable convolution operations are clearly explained, they are helpful in designing kernels with better performance. Besides, the paper itself is educational to readers not familiar with this field, I recommend accepting this paper to let more researchers benefit from the ideas behind G-CNNs.\n\nI have a few questions to ask:\n\n1. The motivation is explained as the group convolutions learn redundancies. Is this only found in G-CNNs or all types of CNNs? Do separable convolution operations address this problem finally? Can we get a figure similar to Figure 2?\n\n2. Although the utilization of SIREN is the highlight of the paper, the motivation behind is unexplained and unclear. Why the group convolution kernels can be represented implicitly? Is SIREN the only choice?  Do larger SIREN networks provide better parameterization?\n\n3. Why does random sampling over subgroups provide better performance? I am baffled by these results, if we can design a discretization sampling method that has similar anti-aliasing effects?", "summary_of_the_paper": "The paper builds group convolutional neural networks based on the depth-wise separable convolution operations, which are commonly seen in modern CNNs. The authors demonstrate that the $Sim(2)$-equivariant can be achieved in such separable convolution operations. In the implementation, the authors borrow the SIREN approach for parameterizing the proposed group convolutions. Finally, the experimental analysis shows good improvement over other types of group convolutions.", "summary_of_the_review": "The bad thing of such mathematical ideas inspired paper is that the experimental performance are always far from sota. But I think the performance is not the only thing, it is not bad to accept this paper for me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study separable convolutions in the group-convolutional setting, and describe experiments showing them to be more computationally efficient without loss of performance in the setting of some group-augmented MNISTs, and show some promising results on un-augmented CIFAR10, CIFAR100, and Galaxy10.  The reviewers are mixed; some of the reviewers have concerns about the completeness of the experiments and the novelty of the work, and in particular to what extent the experiments support the specific novelties claimed.  The authors have made some updates to address this in the revision, but my opinion is that the authors should resubmit to the next venue after further experiments and exposition to clarify.
Decision: Reject