Learning Equivariances and Partial Equivariances From Data
Authors: David W. Romero
Suhas Lohit
Venue: ICLR 2022 Submitted (2022)
Keywords: group equivariance
learning equivariances from data
partial equivariance
group convolutional networks.

Abstract: Group equivariant Convolutional Neural Networks (G-CNNs) constrain features to respect the chosen symmetries, and lead to better generalization when these symmetries appear in the data. However, if the chosen symmetries are not present, group equivariant architectures lead to overly constrained models and worse performance. Frequently, the distribution of the data can be better represented by a subset of a group than by the group as a whole, e.g., rotations in $[-90^{\circ}, 90^{\circ}]$. In such cases, a model that respects equivariance partially is better suited to represent the data. Moreover, relevant symmetries may differ for low and high-level features, e.g., edge orientations in a face, and face poses relative to the camera. As a result, the optimal level of equivariance may differ per layer. In this work, we introduce Partial G-CNNs: a family of equivariant networks able to learn partial and full equivariances from data at every layer end-to-end. Partial G-CNNs retain full equivariance whenever beneficial, e.g., for rotated MNIST, but are able to restrict it whenever it becomes harmful, e.g., for 6 / 9 or natural image classification. Partial G-CNNs perform on par with G-CNNs when full equivariance is necessary, and outperform them otherwise. Our method is applicable to discrete groups, continuous groups and combinations thereof.

Under review as a conference paper at ICLR 2022
LEARNING EQUIVARIANCES AND PARTIAL EQUIVARIANCES FROM DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
Group equivariant Convolutional Neural Networks (G-CNNs) constrain features to respect the chosen symmetries, and lead to better generalization when these symme- tries appear in the data. However, if the chosen symmetries are not present, group equivariant architectures lead to overly constrained models and worse performance. Frequently, the distribution of the data can be better represented by subset of a group than by the group as a whole, e.g. rotations in [-90", 90% In such cases, model that respects equivariance partially is better suited represent the data. Moreover, relevant symmetries may differ for low and high-level features, e.g., edge orientations in face, and face poses relative to the camera. As result, the optimal level of equivariance may differ per layer. In this work, we introduce Partial G-CNNs: a family of equivariant networks able learn partial and full equivariances from data at every layer end-to-end. Partial G-CNNs retain full equivariance whenever beneficial, e.g., for rotated MNIST, but are able to restrict it whenever it becomes harmful, e.g., for 6 / 9 or natural image classification. Partial G-CNNs perform on par with G-CNNs when full equivariance necessary, and outperform them otherwise. Our method is applicable to discrete groups, continuous groups and combinations thereof.
INTRODUCTION
The translation equivariance of Convolutional Neural Networks (CNNs) (LeCun et al., 1998) has proven to be an important inductive bias for good generalization on vision tasks. This achieved by restricting learned representations respect the translation symmetry observed in visual data, such that if an input is translated, its feature representation is also translated, but not modified in any other way. Group equivariant Convolutional Neural Networks (G-CNNs) (Cohen & Welling, 2016) extend equivariance to other symmetry groups. Analogously, they restrict the learned representations to respect the symmetries in the group considered such that if an input is transformed by an element in the group, e.g., a rotation, its feature representation is also transformed, e.g., rotated, but not modified.
However, the group to which G-CNNs are equivariant must be fixed prior training, and imposing cquivariance to group larger than the symmetries present in the data leads to over-constraining and worse performance (Chen et al., 2020). The latter results from difference in the distribution of the data, and the family of distributions the model can describe. Consequently, the group must be selected carefully, and it should correspond to the transformations that appear naturally in the data.
Interestingly, the distribution of the data can frequently be better represented by subset of the group than by the group itself, e.g., rotations in [-90", .909]. For instance, natural images are much more likely to show an elephant standing straight or slightly rotated than an elephant upside-down. In some cases, group transformations can even change the desired model response, typical example the classification of the numbers 6 and 9, whose defining factor is their pose. In both examples, the distribution of the data is better represented by a model that respects rotation equivariance partially. That is, model equivariant to some, but not all rotations.
In addition, the optimal family of full partial equivariances may change per layer. This results from changes in pose likelihoods for low and high-level features in the data, e.g., the orientations of edges in an human face, and the pose of human faces relative to the camera. Whereas detecting a face may benefit from partial rotation equivariance, detecting edges benefits from full rotation equivariance.

---

Under review as a conference paper at ICLR 2022
These observations indicate that constructing model with different levels of partial / full equivari- ances at each layer may be advantageous. Weiler & Cesa (2019) empirically show that manually selecting the level of equivariance different layers can lead to accuracy improvements. However, manually defining layer-wise levels of equivariance is at best a difficult and time-consuming task.
In this work, introduce Partial Group equivariant Convolutional Neural Networks: new family of equivariant methods able to learn partial and full equivariances directly from data at every layer. This is achieved by defining learnable probability distribution over group elements at each group convolutional layer in the network. Instead of sampling group elements uniformly from the group during the group convolution as normally done- -, Partial G-CNNs sample them from the learned distribution. This allows them to adjust their level of equivariance during training. We note that our partial equivariance framework allows for both equivariance forgetting, and full equivariance. The former is obtained by collapsing the distribution along certain group dimension, e.g., the rotation axis, and the later results from keeping the learned distribution uniform over the entire group.
We evaluate Partial G-CNNs illustrative toy tasks and on challenging benchmark datasets. We show that whenever full equivariance is beneficial, e.g., for rotated MNIST, Partial G-CNNs remain fully equivariant. However, whenever harmful, our models restrict equivariance to a subset of the group transformations, e.g., to make 6 6/9 classification possible or to improve on natural image classification. Our method improves upon conventional group equivariant networks when equivariance reductions are advantageous, and matches their performance whenever their design is optimal. To the best of our knowledge, this work is the first approach explore partial group equivariances, and to provide practical implementation to learn them directly from data in an end-to-end manner.
In summary, our contributions are:
We present a novel architectural design for equivariant networks, with which group convolutional networks are able to learn layer-wise partial full equivariances from data.
We empirically show that partial equivariance allows our networks to improve upon conventional G-CNNs for tasks for which full cquivariance is harmful. However, whenever beneficial, our models learn stay fully equivariant and match the performance of G-CNNs.
RELATED WORK
Equivariance learning. Learning equivariant mappings from data has been explored by (i) meta- learning means to iterate between the learning of free weights and a weight-tying matrix that encodes the symmetry equivariances of the network (Zhou et al., 2020), or by (ii) learning the Lie algebra generators of the group jointly with the parameters of the network (Dehmamy et al., 2021). However, both approaches utilize the same learned symmetries across layers. In addition, MSR (Zhou al., 2020) is only applicable to (small) discrete groups, and requires longer training times. L-Conv (Dehmamy et al., 2021), on the other hand, is only applicable to continuous groups and is not fully compatible with current deep learning components, e.g., pooling, normalization.
Contrary to these approaches, our method learns different levels of full/partial equivariance at every layer, is fully compatible with current deep learning components, and is applicable for discrete groups, continuous groups and combinations thereof. It is important to mention, however, that Zhou et al. (2020); Dehmamy et al. (2021) aim to learn the structure of the group from scratch. Our method, on the other hand, starts from (very) large group and allows layers in the network to constrain their equivariance levels whenever necessary in order to fit the distribution of the data better.
Invariance learning. Learning the right amount of invariance from data has been explored by (f) learning a probability distribution over data augmentations, e.g., rotation, scaling, and passing the responses of the network to the augmented inputs through a voting mechanism (Benton et al., 2020), or by (ii) minimizing the lower bound on the marginal likelihood of the problem to infer the right amount of rotation invariance in the weights of network (van der Ouderaa & van der Wilk, 2021).
Contrary to these approaches, our method learns full partial equivariant representations, and is able to modify the extent of equivariance at every layer. Most similar to approach is Augerino (Benton et al., 2020), whose main goal is to approximate invariance to global input transformations. However, Augerino requires passing several augmented versions of an input during inference, and only works with continuous transformations. Partial G-CNNs on the other hand, are also capable of defining distributions on discrete sets of transformations.
2

---

Under review as a conference paper at ICLR 2022
To the best of our knowledge, our work is the first approach to explore partial group equivariances, and to provide practical implementation to learn them from data end-to and-to-end.
Group equivariant neural networks. The seminal work of Cohen & Welling (2016) has inspired several methods equivariant to many different groups. Existing convolutional methods show equivari- ance planar rotations (Dieleman et al., 2016; Worrall et al., 2017: Weiler et al., 201 spherical rotations (Weiler et al., 201 8a; Cohen et al., 201 Esteves et al., 2019;;b; 2020), scaling (Worrall & Welling, 2019; Sosnovik et al., 2019; Romero et al., 2020b), and more general symmetry groups (Romero et al., 2020a; Bogatskiy et al., 2020; Finzi et al., 2021). Group equivariant self-attention has also been proposed (Fuchs et al.. 2020; Romero & Cordonnier, 2020; Hutchinson et al., 2021).
Common to all these approaches is that they require fixing the symmetry group to which the model will be fully equivariant prior to training. In contrast, our method allows us define layers approximately equivariant to subsets of the group, and to learn these subsets during training.
3 PRELIMINARIES: GROUP EQUIVARIANCE AND THE GROUP CONVOLUTION
Group equivariance is the property of a map to respect the transformations in group. We say that map equívariant a group if whenever the input transformed by elements of the group, the output of the map is equally transformed but not modified. Formally, for a group G with elements g é G acting on set XX, and map - X + Xt we say that á is equivariant to G if:
Group equivariance:
x) g- o(z), Vx E Vg E G.
(1)
For example, the convolution of signal f: R and kernel & : + is translation equivariant because Le * f)= * Ltf, where L, translates the function by t: (z)(=)=((==)))) -f--). t). That is, if the input is translated, its numerical descriptors via the convolution are also translated but not modified.
The group convolution. In order to construct neural networks equivariant to a given group G. it crucial define an operation that respects the symmetries in the group. The group convolution generalizes the convolution to be equivariant to general symmetry groups. Formally, for any e G. the group convolution of a signal f G + R and a kernel s G IR is given by:
h(u)= (v+ (u)= = (1o' f(vv) (v)
(2)
where ptp()) is the (invariant) Haar measure of the group. The group convolution is G-equivariant. That is, it holds that, for all u, W G:
+ (w) = Lu * (u), with fu)((w)=((="))))
The lifting convolution. Regularly, the imput of neural network is not readily defined on the group of interest G. but on a sub-domain XX, i.e., f: :X - R. As a result, in order to use group convolutions, we first require an operation that lifts the input signal from its domain X to the group G This operation is called a lifting convolution. For any u G, the lifting group convolution is defined as:
huw)= (v) s (u) - S, ())(() duc V X,, u G.
(3)
Practical implementation of the group convolution. The group convolution is defined on a continu- ous domain. Consequently, it cannot be computed in finite time, and thus must be approximated. Two main strategies exist to approximate group convolutions with regular group representations: (i) group discretization (Cohen & Welling, 2016), and (ii) Monte Carlo approximations (Finzi et al., 2020).
Group discretization approximates the group convolution by using a fixed discretization of the group, e.g.. the rotation group as the set of rotations by 90 degrees. Unfortunately, this approximation is only equivariant to the transformations in the discretization, and not to the underlying continuous group. better solution is proposed by Finzi et al. (2020). Finzi et al. (2020) propose a Monte Carlo approximation of the group convolution by sampling transformations uniformly from the entire group during each forward pass. Formally, this approximation is given by:
h(u,) ()44) Σ, (0))lu))( (u))(()))(()))))
where (wr). {ry} are the discretizations of the input and output domain of the operation, respectively.
Importantly, this approximation requires the convolutional kernel 1a be defined on the continuous group. As the domain cannot be enumerated, & cannot be parameterized by independent weights. Instead, Finzi et al. (2020) parameterize convolutional kernels via small neural networks. With this parameterization, all elements 27 u, can be mapped to well-defined kernel value.
3

---

Under review as a conference paper at ICLR 2022
4 PARTIAL GROUP EQUIVARIANT NETWORKS
In this section, we introduce our approach. First, we define the concept of partial group equivariance, and generalize the group convolution the partial equivariant case a constructive manner. Next, we illustrate how group subsets can be learned with learnable probability distributions on the group, and provide concrete examples for discrete groups, continuous groups and combinations thereof. We conclude by presenting the partial group convolution and the architecture of Partial G-CNNs.
4.1 PARTIAL GROUP EQUIVARIANCE
Contrary to group equivariance, say that map o is partially equivariant to G. if the map is cquivariant to transformations in a subset of the group S c G. but not to the whole group G:
( z) 9 ((x), Vz X,, V S.
Different from equivariance to a subgroup of G: a subset of the group which also fulfill the group axioms, we do not restrict the subset S to be itself group.
As will we show in Sec. 4.2, partial equivariance can only be obtained in an approximate manner. This is because the set 8 is not necessarily closed under group actions. Partial equivariance is related to soft invariance (van der Wilk et al., 2018): the property of a map to be approximately invariant. We opt for the word partial in the equivariant setting, make clear that it defines subset of the group.
4.2 FROM GROUP CONVOLUTIONS TO PARTIAL GROUP CONVOLUTIONS
In this section, generalize the group convolution describe partial equivariances. Vital to analysis is the equivariance proof of the group convolution (Cohen et al., 20 2018a). In addition, we must distinguish between the domains of the input and output signals of the group convolution, i.c. the domains of f and h in Eq. 2. This distinction is important because they might differ for partial group convolutions. From here on, we refer to these as input domain and output domain.
Proposition 4.1. The group convolution Geqquivariant in the sense that for all u, U, w E G: (+/ Lf (u) (w fu), with L (u)= u).
[ vbu (b((=')) uu) dup(w)=
Proof. (v + (w))) = 5q
= ( ff=+f (w (u).
From the first to the second equality, we use the change of variables - v. We can do this because the group convolution is map from the group to itself, and thus if w, u G. so does w Since the Haar measure is an invariant measure on the group, we have that pe (v)==Hg (7), for all v, e G. O
Going from the group to subset S. Crucial to the proof of Proposition 4.1 is the fact the group convolution is an operation from input signals to output signals on the group. As a result, w u is member of the output domain for any w G applied to the input domain. Consequently, group transformation applied to the input can be reflected by an equivalent transformation on the output.
Now, consider the case in which the output domain is not the group G. but instead arbitrary subset 8 c G. e.g., rotations in [- 2 ]. Following the proof of Proposition 4.1 with u E S, and v E G, we observe that the operation is equivariant to transformations w E G as long as w is member of 8. However, -1 does not belong to the output domain S, the output of the operation cannot reflect an equivalent transformation to that of the input, and thus equivariance is not guaranteed.
Note, however, that equivariance is only obtained Eq. 6 holds for all elements in the output domain. That is, if wt u is member of Ss for all elements c 8. This is not the case in general, as the output domain 8 not necessarily closed under group transformations. It is this difference, fact, what Partial G-CNNs make use of to disrupt full equivariance whenever necessary (Fig. 1).
Quantifying the equivariance difference. Importantly, we can precisely quantify how much the output response will change for any input transformation given output domain S. Intuitively, this difference is given by the difference in the parts of the output feature representation that go in and out of S by the action of the imput transformation. Naturally, the larger the input transformation and the smaller the size of S the larger the equivariance difference will be. Fig. illustrates this effect. A formal discussion and the derivation of the equivariance difference are provided in Appx. B.
4

---

Under review as a conference paper at ICLR 2022
Input
-150P
0"
D
D
D
D
W
G-ONN
60"-
W
180"
Subeet
Group 0
Figure 1: Partial group convolution. In a group convolution, the domain of the output closed under group transformations of the input. Consequently, all response components are part of the output regardless of the transformation applied to the input. Differently, the output domain S partial group convolution is not necessarily closed under input transformations. As result, the feature representation captured within S will change for different transformations of the input. In Fig. 1, for instance, the output feature representation of the input (outlined in red) gradually leaves S for different transformations of the input. For large transformations (180% here), the responses within & change entirely. This difference allows partial group convolutions distinguish among input transformations.
Shiee A
Sebee gC
Sebwe
Tapai
IRF
Khdppt
Figure Partial group convolution on group subset. Contrary to the group convolution, partial group convolutions can receive an input whose domain not the entire group G. but a subset thereof 8 Consequently, the input feature representation falling within ((1) may change for different input transformations (top). As a result, the output of the partial group convolution may show variations for different input transformations even if the output domain of the group convolution the group G (bottom row). Similar to the case in Fig. 1, this difference grows proportionally to the strength of the input transformation. This can be seen in the red-outlined feature representations at the bottom.
Going from a subset s(1) to another subset 52(2) Now, consider the case which the domain of input and the output of the group convolution are both subsets of the group, i.e., v (()) and é 822), Analogous to the previous case, equivariance to input transformations
G holds at positions u e ((2) for which - are also members of 8(2) Now however, the input domain not restricted to be closed. Consequently, the feature representation of the input will differ for different group transformations. This induces an additional difference the output response.
To see this, consider the partial group convolution in Fig. 2. For different input transformations, the feature representations within {(1) change. As result, the output feature representation might change for different input transformations even if the output domain is the group itself, i.e., S(2)_
In summary, considering group convolution from a subset s(1) to a subset ((2) has two sources of equivariance difference. The first, comes from considering a subset (22) as the output domain, and equivalent to the difference illustrated in the previous case. The second, comes from considering subset ((1) in the input domain. This allows the input features to change transformation, and the size of the input domain (()) This in turn induces additional difference in the output response. A formal treatment the second difference term provided Appx. B.2.
function of the input
Going from subset S to the group G. Finally, we also consider the case in which the input domain is a subset of the group and the output domain the group itself G. Following the previous analysis, that, this operation is equivariant to all input transformations G, as is
5

---

Under review as a conference paper at ICLR 2022
always member of the output domain for all values of u G. Nevertheless, the input features can change for different imput transformations, and thus the difference illustrated in the previous case can also emerge here. This case can be seen as an special case of the previous section, where = and thus the equivariance difference induced by considering subset in the output domain is zero.
Interestingly, the lifting convolution is an special case of this instance with S=X'. Note however, that the lifting convolution is exactly equivariant. This is because the input domain of the operation is X.. Consequently, the imput domain remains equal for all input transformations.
4.3 LEARNING GROUP SUBSETS VIA PROBABILITY DISTRIBUTIONS ON GROUP ELEMENTS
So far, we have discussed the properties of the partial group convolution without providing details on how to learn group subsets during training. In this section, we describe our approach and provide specific examples for discrete groups, continuous groups, and combinations thereof.
Vital our approach is the Monte Carlo approximation to the group convolution presented in Sec. 3:
(vb² (v:) = Σ, (0 u4) (z))))))()))
As shown in Appx. C, this approximation is equivariant to G in expectation if the elements in the input and output domain, { ( {uz} are uniformly sampled from the Haar measure, i.c., u, 1. ~ pe(()
Approach. Our main observation is that can prioritize sampling specific group elements during the group convolution by learning a probability distribution p()) over the elements of the group.
When group convolutions draw elements uniformly from the entire group, each group element is drawn with equal probability and thus, the resulting approximation is fully equivariant expectation. However, can also define different probability distribution with which some samples are drawn with larger probability. For instance, we can sample from a predefined region, e.g., rotations in J. by defining a probability distribution p(g) which uniform in this range, but zero otherwise. In the limit, we can forget a certain equivariance by letting this distribution collapse to single point, e.g., the identity, along the corresponding group dimension.
In other words, learning proper probability distribution p(g) on the group can be used to effectively learn subset of the group. Based on this intuition, we define a probability distribution p(g) on the output domain of the operation, i.e., on {ur}, in order to learn a subset of the group 8(2) upon which partial equivariance is defined. Note that we only need to learn distribution on the output domain for each layer. This is because group convolutional layers are applied sequentially, and thus the subset of the input domain is learned by a distribution on the output domain of the previous layer.
4.3.1 DEFINING p(g) FOR DISCRETE, CONTINUOUS AND MULTI-DIMENSIONAL GROUPS
Probability distributions for one-dimensional continuous groups. We take inspiration from Augerino (Benton et al., 2020). an use the reparameterization trick (Kingma & Welling. 2013) to parameterize continuous distributions. In particular, we use the reparameterization trick on the Lie algebra of the group (Falorsi et al., 2019) to define a distribution which is uniform over a connected set of group elements [u_ ....., C,, ... u.. but zero otherwise. To this end, define distribution u- 1] with learnable u on the Lie algebra g. and map it to the group via the pushforward of the exponential map exp: g + G. This give us distribution which is uniform over connected set of elements [2_
.,,.., ). but zero otherwise."
For instance, we can learn a distribution on the rotation group SO(2), which is uniform between [-0, 0] and zero otherwise by defining probability distribution 0- [-1, with Icarnable 0 on the Lie algebra, and mapping to the group. If we parameterize group elements as scalars g [-T, T) the exponential map is the identity, and thus p(g)= - (20)))(((000) (-), If we sample group elements from this distribution during the calculation of the group convolution, the output domain will only contain elements in |-0,0 0) and the output feature map will be partially equivariant.
Probability distributions for one-dimensional discrete groups. We can define a probability distri- bution on discrete group as the probability of sampling from all possible element combinations.
Finzi et al. (2020) show a similar result where ut and by are the same points and thus ny ~ me(.) suffices. Notte that an exp-pushforwarded local uniform distribution is locally equivalent to the Haar measure, and thus can still use the Haar measurement for integration on group subsets.

---

Under review as a conference paper at ICLR 2022
For instance, for the mirroring group {1, -1,, this distribution assigns probability to each of the combinations {0, 0}, {0,1}. (1,0), {1, 1} indicating whether the corresponding element is sampled not. For group with elements {e. (e.......... In.. however, this oblige to sample from 2 elements, which is computationally expensive and potentially unstable to train. Instead, we define element-wiss Bernoulli distributions of sampling the elements J. The probability distribution on the group can be then formulated as the joint probability of the element-wise Bernoulli distributions ple.g1........ 9n) I[;))((;;..
the elements (. ). -. and learn the probabilities (p<)
p(g:).
To learn the element-wise distributions, we make use of the Gumbel-Softmax trick (Jang et al., 2016; Maddison al., 2016), and use the Straight-Thron sampling. If all the probabilities {p,=1), the group convolution will be fully equivariant. However, whenever some probabilities start declining, group equivariance becomes partial. In the limit, if all probabilities (p:=0), the operation effectively forgets this equivariance.
Humbel-Softmat to propaptee through
Probability distributions for multi-dimennional groups. dileiidimensional groups are important for the applications we consider. For instance, the orthogonal group 0(2) is parameterized by rotations and mirroring, or the dilation-rotation group by scaling and rotations. In such cases, we construct the probability distribution over the entire group as combination of independent probability distributions along each of these axes. For a group G with elements g decomposable p(9)=I][= p(d)),
along dimensions g=(d1...,dd, we decompose the probability distribution where the probability p(d:) is defined given the type of space, i.e., continuous or discrete. For instance, for the orthogonal group O(2) defined by rotations and mirroring m, i.e., 9 m)) r e SO(2), m {+1}, we define the probability distribution on the group as p(g)=p(r)-p(m)) where p(r) a continuous distribution, and p(m) is discrete one.
4.4 THE PARTIAL GROUP CONVOLUTION
Let 8(1), (()) be subsets of group & and p(u) be uniform in S(2) and otherwise. The partial group convolution from function f (()) - IR to a function h: s(() R is given by:
huu)= (v> (u) fSo) p(u)x(( (v) (v); u ((2) 8()..
The cases described in Sec. 4.2 can be written as special cases of Eq. 7. In practice, Eq. 7 is computed with a Monte Carlo approximation as described in Algorithm (Appx. D).
4.5 PARTIAL GROUP EQUIVARIANT NETWORKS
We built upon the work of Finzi et al. (2020) and construct base G-CNNs equivariant continuous and discrete groups. This is achieved by combining kernel parameterizations on Lie groups, and actions of discrete groups on the group repre- sentation of the kernels. Moreover, lifting layer of Finzi et al. (2020) by convolutional layer with lifting convolutions (Bq. 3). Inspired by Romero et al. (2021), we parameterize convolutional kernels as implicit neural representations with SIRENs (Sitzmann et al., 2020). The proposed parameterization shows better expressivity and convergence speed, and outperforms ReLU-, Leaky ReL.U ReI.U and Swish-MLP parameterizations used so far for continuous G-CNNs, e.g., Schütt al. (2017); Finzi et al. (2020), Tab. 4.
Hozz
HaxPoolBd
Htrn
replace the isotropic
Devm
Partin
Norm
GlobalMecPool
Figure 3: Partial G-CNN architecture.
Our network structure is shown in Fig. 3, and varies only in the number of blocks and channels.
5 EXPERIMENTS
Experimental details. We parameterize all our convolutional kernels as a 3-layer SIREN (Sitzmann et al., 2020) with 32 hidden units. All our networks are constructed with two residual blocks of 32 channels each, batch normalization (Ioffe & Szegedy, 2015) and follow the structure shown in Fig. 3. Our networks intentionally selected to be simple to better assess the effect of partial equivariance. Importantly, we avoid learning probability distributions along the translation group, and assume all spatial positions to be sampled. This allows us fast PyTorch primitives in implementation.
7

---

Under review as a conference paper at ICLR 2022
Lifting
1st. Block
2nd. Block
Final Layer
[-,,a]
[-7,-]
[-,,=]
[-x,a]
[-0.89x, 0.89m]
[-0.25m, 0.25m]
Figure Learned full/partial equivariances for MNIST6- 180 with a Partial SE(2)-
Table 1: Test accuracy on MNIST6-188 and MNIST6-M.
BASE GROUP
MNIST6-180 (%)
BASC
MNIST6-M Acc. (%)
BASE GROUP
MNIST6-M
MODEL
G-CNN
SE(2)
50.0
Mirroring
50.0
E(2)
50.0
CNN
100.0
100.0
Toy tasks: MNIST6-180 and MNIST6-M. First, we validate the ability of our approach to solve situations in which full equivariance impedes solving task. To this end, construct datasets: MNIST6-180, and MNIST6-M, by extracting 6 numbers from the MNIST dataset (LeCun et al., 1998), and modifying them either by 180 rotations, or mirroring. The original numbers get the label "original", and the transformed data the label "transformed". The goal is to separate these two classes.
G-CNNs are unable solve tasks for which discrimination among group transformations are required. As result, SE(2)- (22)--CNN are unable to separate MNIST6- 180 classes, and E () -CNNs are unable to separate MNIST6-M classes (Tab 1). Partial G-CNNs with base groups (E()). F(2), on the other hand, seamlessly solve the tasks. Interestingly, we observe that the original MNIST6 class already contains some mirrored numbers, e.g., the input used in Fig. and thus some class overlap exists in MNIST6-M. Nevertheless, Partial G-CNNs are able to solve the task to perfection.
We can also visualize the learned full/partial equivariances at every layer of a network. Fig. illustrates the learned equivariances for Partial SE(2) -NNN with 2 residual blocks trained MNIST6-180. In order to solve the task, the network learns to progressively reduce its equivariance function of depth, up final range of [-0.25m, 0. 0.25m.. Interestingly however, the network does not disrupt equivariance directly at the beginning of the network, but instead decides to stay fully equivariant in the first four layers. We hypothesize that this is because these layers extract low- and mid-level features, which might appear at arbitrary poses. As a consequence, it is advantageous to preserve full equivariance this hierarchical level. The same behaviour is also present in E(2)-CNNs. However, we observe that discrete distributions are more unstable to train, and other patterns also emerge that, for example, interrupt full equivariance the second or third layer of the network.
Benchmark image datasets. Next, we validate our approach on the following benchmark image recognition datasets: rotated MNIST (Larochelle al., 2007), CIFAR- 10 and CIFAR- (Krizhevsky et al., 2009). Additional results on PatchCamelyon (Veeling et al., 201 8) can be found in Appx. F. We construct Partial G-CNNs with base groups SE(2) and E(2). and varying number of elements used in the Monte Carlo approximation, and compare them equivalent ResNets and G-CNNs.
Our results are shown in Table Partial G-CNNs are competitive with G-CNNs where full- equivariance is needed, i.e., RotMNIST and PatchCamelyon. However, for tasks in which the data does not naturally exhibit full rotation equivariance, i.c.. CIFAR-10 10 and CIFAR-100 -100 Partial G-CNNs consistently outperform their full equivariant counterparts. Our results support the usage of Partial G-CNNs for settings with both partial and full equivariance.
In addition, validate claim that SIRENs are better suited parameterize group convolutional kernels than existing alternatives. As shown in Tab. 4, SE(2)-CNNs with SIREN kernels consis- tently outperform SE(2)-CNN: with kernel parameterizations via ReLU, LeakyReLU and Swish nontinearities on all the image benchmarks considered. Our results support the usage of SIRENs as parameterization for continuous group convolutional kernels.
DISCUSSION
Memory consumption in partial equivariant settings. G-CNNs fix the number of samples with which the group convolution is approximated prior to training. G-CNNs use the same number of samples for all group convolutional layers regardless of the symmetries present in the data. Partial
8

---

Under review as a conference paper at ICLR 2022
Table 2: Test accuracy on vision benchmark datasets.
BASE GROUP
No. ELEMENTS
PARTIAL EQUIV.
CLASSIFICATION ACCURACY (%)
MODEL
ROTMNIST
CIFAR-IO
CIFAR-I00
ResNet
T(2)
1
97.23
83.11
47.99
X
99.10
83.73
52.35
4
V
99.13
86.15
53,91
SE(2)
X
99.17
86.08
55.55
8
/
99.23
88.59
57.26
G-CNN
X
99.24
86.59
51.55
16
/
99.18
$7.45
57.31
x
98.14
85.55
54,29
8
97.78
89.00
55.22
R(2)
x 98.35
88.95
57.78
16
/
98.58
90.12
61.46
G-CNNs the other hand, able dynamically adjust the number of samples used in order approximate partial group convolutions. In particular, we fix a maximum number of samples prior training, and restrict the amount of samples used at every layer based on the subset of the group learned. For instance, a Partial (E(2)-CNE) CNN with a learned distribution p(u)=2l( [[-1,1] only uses half of the elements used in the corresponding G-CNN. This efficiency in memory leads to important reductions in training time for Partial G-CNNs on partial equivariant settings.
Better kernel parameterization. Inspired by Romero al. (2021), we replace the Swish kernel parameterization proposed by Finzi et al. (2020) by SIREN Sitzmann et al. (2020). We observe that this parameterization consistently leads to better results and faster convergence.
Sampling per batch element. In our experiments, we sample once from the learned distribution p(u) at every layer, and use this sample for all elements the batch. better estimation of the probability distribution could be obtained by drawing different sample for each element in the batch at every layer. Although this method may benefit the learned distributions, comes at large memory cost. Consequently, all our reported results obtained using our original formulation.
Partial equivariances for other group representations. Our theory of learnable partial equivari- ances is directly applicable to methods using regular group representations, c.g., Cohen & Welling (2016); Romero & Cordonnier (2020); Hutchinson et al. (2021). An extension for other kind of group representations such irreducible representations in steerable CNNs, e.g., (Worrall al., 2017; Weiler et al., 2018h 8b; Weiler & Cesa, 2019) is an interesting research direction, as irreducible representations allow for exact equivariance to continuous groups.
Unstable training on discrete groups. Although discrete groups can be successfully modelled in partial equvariance settings, we observe that the discrete distribution used here is not very stable to train. Finding good parameterizations to learn discrete distributions is an active field of research (Hoogeboom et al., 2019; 2021), and advances in this field could lead to more stable training of partial equivariances on discrete groups
7 CONCLUSION AND FUTURE WORK
We presented Partial Group equivariant Convolutional Neural Networks (Partial G-CNNs): new family of equivariant methods able to learn partial and full equivariances directly from data at every layer. Partial G-CNNs match the performance of G-CNNs in full equivariant settings, and outperform G-CNNs in settings in which partial group equivariance better represents the data. By learning different levels of equivariance group equivariance in early layers to efficiently learn low-level features appearing at arbitrary poses. However, they are able to restrict whenever necessary to discriminate among high-level features.
every layer, Partial G-CNNs are able to maintain full
Our method entirely relies good parameterization of the probability distribution defined over group elements. Finding better representations and learning methods for these distributions, specially for distributions discrete variables. is promising direction for future research. In addition, an equally interesting future research direction is the extension of partial equivariances to neural architectures handling other kind of group representations such steerable CNNs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010. 1882, 2020.
Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pp. 992-1002. PMLR, 2020.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21 21(245):1-71, 2020.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pp. 2990-2999, PMLR, 2016.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns homogeneous spaces. arXiv preprint fttv:/8111000/1/ 20.8a.
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1881.110130, 2018b.
Nima Dehmamy, Yanchen Liu, Robin Walters, and Rose Yu. Lie algebra convolutional neural networks with automatic symmetry extraction, 2021. URL
forum? dorume2.d=cTQn2PLIOo)
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolu- tional neural networks. International conference on machine learning, pp. 1889-1898. PMLR, 2016.
Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning, pp. 1812- 1822. PMLR, 2019a.
Carlos Esteves, Yinshuang Xu, Christine Allen- Blenc-Blanchette, and Kostas Daniilidis. Equivariant multi- view networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1568-1577, 2019b.
Carlos Esteves, Ameesh Makadia, and Kostas Danilidis. Spin-weighted spherical cmns. arXiv preprint arXiv:2006. 2020.
Luca Falorsi, Pim de Haan, Tim Davidson, and Patrick Forré. Reparameterizing distributions on lie groups In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2244-3253. PMLR, 2019.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data. In International Conference on Machine Learning, pp. 3176-3176. PMLR, 2020.
Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivari- multilayer perceptrons for arbitrary matrix groups. arXiv preprint 10X44:2004.09459 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-tsssfform::s: 3d
roto-translation equivariant attention networks. arXiv preprint 200003000..1003, 2020.
Emiel Hoogeboom, Jorn WT Peters, Rianne den Berg, and Max Welling. Integer discrete flows and lossless compression. arXiv preprins arXiv: (005:0/9900073777 2019.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. arXiv preprint arXiv:2/02.05339, 2021.
Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: equivariant self-attention for lie groups. In International Conference Machine Learning, pp. 4533-4543. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. PMLR. 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 16I 144, 2016.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hugo Larochelle, Dumitru Erhan. Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp. 473-480, 2007.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2333 1998.
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00770 2016.
David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pp. 3188-8199. PMIR, 2020a.
David W Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977 2020.
David W Romero, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Wavelet networks:
Scale equivariant learning from raw waveforms. arXiv preprint (.XXi:2006605259, 2020b.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611, 2021.
Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: continuous-fillt convolutional neural network for modeling quantum interactions. arXiv preprint (wwXiv:1706.88550 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Ivan Sosnovik, Michal Szmaja, and Arood Smeulders. Scale-equivariant steerable networks. arXiv preprint arXiv: 11093, 2019.
Tycho F.A van der Ouderaa and Mark van der Wilk. Learning invariant weights in neural networks. Workshop in Uncertainty & Robustness in Deep Learning, ICML, 2021.
Mark van der Wilk, Matthias Bauer, ST John, and James Hensman. Learning invariances using the marginal likelihood. arXiv preprint arXiv: 05563,, 2018.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equiv- ariant cnns for digital pathology. In International Conference Medical image computing and computer-ussisted intervention, pp. 210-218. Springer, 2018
Maurice Weiler and Gabriele Cesa. General e( -e)ueuurvai steerable cnns. 1..0021 2019.
arXiv preprint
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv: 20.8a.
11

---

Under review as a conference paper at ICLR 2022
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849 858, 2018b.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint arXiv: 1697, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterizztion. arXiv preprint arXiv:2007.02933 2020.
12

---

Under review as a conference paper at ICLR 2022
APPENDIX
A GROUPS AND GROUP ACTION
Group theory is the mathematical language that describes symmetries. The core mathe- matical object is that of group, and defines what it means for something to exhibit symmetries. Specifically, a group consists of a tuple (G.-) - -set of transformations G. and binary operation with the following properties: (i) closure, i.e., 91 O2 = 9s G. Vg,, 92 é G (ii) associativity, i.e., gi (g2 93) (9) 92). . 93 for all 91, 92: 93 G. (iii) the existence of an identity element e e G. such G.
Groups:
that 9 e= g g. and (iv) the existence of an inverse و & for all
Group action: One can define the action of the group & on a set XX This action describes how group elements g E ạ modify the set I' when the transformation is applied. For instance, the action of elements in the group of planar rotations & SO(2) on an image I X -written Ox-, depicts how the image changes when the rotation 0 is applied.
B FORMAL TREATMENT OF THE EQUIVARIANCE DIFFERENCE IN PARTIAL GROUP CONVOLUTIONS
B. PARTIAL GROUP CONVOLUTIONS FROM THE GROUP G TO A SUBSET S
The partial group convolution from signals on G to signals on a subset S can be interpreted as group convolution for which the output signal outside of S is set to zero. Consequently, we can calculate the equivariance difference Degulv in the feature representation, by calculating the difference on the subset 8 of group convolution with a trou-trranffrmmed input (ffie ()) and a group convolution with canonical input proceeded by the same transformation on 8, i.e., L (f v))
The equivariance difference given by the effect of a subset in the output domain Aopui given by:
Ss Li (2) duc(u) S,(0* L. Luf)(u)
Aout
[o (1b f) (w) (u) S (v Aw (w) d))(()))
[ f)( duc(u) - -frs (v+ f)) d (uc
[S Smax + fuu (u)
(w f) (u)) 1
Smin
wimmin
(v> (u) dug(uu) f
* ()) dun(()) (()))
emin
018maax
(vb * (u)ddug(u f
Smann
(+* (onn)dua(w)
Srain
w8min
|I W (bb d))(u))
Emax
Jusmin (v+ -
Semin
(f)) f))) duc(w))
From the first line to the second we take advantage of the equivariance property of the group convolution: (faff ()) = =L Lf * (u), and account for the fact that only the region within is visible at the output. We use the change of variables u = w u from the second to third line. We specify the houndaries of S from the third to the fourth line. In the fifth line we separate the integration over 8 as a sum of two integrals which depict the same range. In the last line, we cancel out the overlapping parts of the two integrals to come to the final result.
In conclusion, the equivariance difference induced by subset g(2) on the domain of the output out is given by the difference between the part of the representation that leaves the subset S, and equiv the part that comes to replace it instead. This behaviour is illustrated in Figure 1.
13

---

Under review as a conference paper at ICLR 2022
B.2 PARTIAL GROUP CONVOLUTIONS FROM A SUBSET SQA) TO SUBSET (
To isolate the effect of having a group subset as domain of the input signal f. we first consider the domain of the output to be the group, i.e., 5(2)=G- The equivariance difference is given by the difference across the entire output representation of the group convolution calculated on input subset (1)) with canonical input f, and with group transformed input f..f-
The equivariance difference A given by the effect of subset in the input domain is given by:
Aequiv 166 Is 6i b(
So u) u) duc (v)
| (b ((r (u)()) J v/ ( u)s( dus(v) dun
1 S bi w(= [f()) f(w f(w ')] duc(
In other words, the equivariance difference induced by a subset ((1) the domain of the input eguiv is given by the difference in 5(1) of the canonical input f. and the part that comes to replace it when the input is modified by a group transformation w.
C EQUIVARIANCE PROPERTY OF MONTE-CARIE APPROXIMATIONS
Consider the Monte- approximation depicted in Eq. 4:
(1b4 (u:) Σ, (()
For a transformed version of the f. f we can show that the Monte-Cario approximation of the group convolution equivariant in expectation. The proof follows the same steps than Finzi et al. (2020). However, the last step of the proof has different reason, resulting from the fact that input and output elements can sampled from different distributions.
For transformed version of the f. we have that:
fwf = E, M0y wi) (()] (0)((w"2)))((()) ())))(
= E, u.) 4 (va u. L'b (u:))
From the first to the second line, we use the change of variables ū; way and the fact that, group ele- ments in the input domain are sampled from the Haar measure for which it holds that jag ( However, from the second the third line, we must also assume that this holds for the output domain. That is, that the probability of drawing w-lu, uy is equal to that of drawing uj. This is of particular importance for the partial equivariance setting, which this might not be the case.
D ALGORITHM FOR MONTE- CARLO APPROXIMATION OF THE PARTIAL GROUP CONVOLUTION
Algorithm 1: The Partial Group Convolution Layer
Inputs: position, function-value tuples on the group or a subset thereof (1y, f(U))). Outputs: convolved position, function-value tuples the output group subset {ut, (u+) {ua} ~ >pp):: //* Sample elements from p (u)
for u, { [u:) do
h(u)) E, (15 4. (t,))) ; / Compute group convolution (Eq. 4)
end
return {u, 1( (u,))
14

---

Under review as a conference paper at ICLR 2022
Table 3: Image recognition accuracy on PatchCam dataset.
BASE GROUP
No. BLEMENTS
PARTIAL EQUIV.
CLASSIFICATION ACCURACY
MODEL
PATCHCAM (%)
ResNet T(2)
1
67.59
x
89,87
8
/
89.07
G-CNN SF(2)
X
89.71
16
90.31
X
89.77
E(2)
16
88.13 13
E ARCHITECTURE AND HYPERPARAMETER DETAILS
We note that all the hyperparameters were chosen based on the best performance for the fully- equivariant networks and the same parameters were used for training the proposed Partial G-CNNs.
For all the experiments shown in this paper, we use residual networks (shown the main paper in Figure 3) with an initial lifting convolutional layer followed by 2 ResBlocks with full/partial group convolutional layers (or regular convolutional layers for the non-equivariant ResNet baselines), For all the networks for all the datasets, we use 32 feature maps in the hidden layers. We employ Batch Normalization and ReLU non-linearities as shown in the figure. For MNIST6-M and MNIST6- 800 max-pooling is performed after each of the ResBlock partial/full group convolutional layers. In the case of rotMNIST, max- paollion is performed after the lifting convolutional layer and the first group convolutional layers. For CIFAR-10 10 and CIFAR- 100, use max- max-pooling layers after each of the group convolutional layers. Finally, for PatchCamelyon, we apply the max-pooling operation after the lifting convolution well as the both the group convolutional layers. At the end of the network, a global max-pooling layer is used to create the invariant before classification.
The networks for MNIST6-180, MNIST6-M, rotMNIST, CIFAR-10 and CIFAR-100 are trained for 300 epochs and the networks for PatchCamelyon are trained for 30 epochs. We use a batch size of 64 for all networks. We use an initial learning rate of 3 and Adam optimizer with cosine annealing with 5 epochs of linear warm-up. In the case of CIFAR- 10, CIFAR- 100 and PatchCamelyon datasets, we also use a weight decay of 0.0001. For roMNIST, CIFAR-10. CIFAR- 100 and PatchCamely we also use different learning rate of 1e 4 for only the parameters of the probability distributions over the group elements.
F ADDITIONAL EXPERIMENTAL RESULTS
Classification results on PatchCamelyon. Table 3 shows the results obtained for ResNet, G-CNNs and Partial G-CNNs on the PatchCamelyon dataset (Veeling et al., 2018). Partial G-CNNs match the performance of G-CNNs this full equivariant setting, as expected.
We observe that the learned distributions over the group elements for rotMNIST and PatchCamelyon are exactly consistent with full-equivariance, and for CIFAR- 10 and CIFAR-I00, they clearly show partial equivariance
Evaluation of existing kernel parameterizations. We observe that SIREN kernels consistently lead to better accuracy than other existing kernel parameterizations. Our results are shown in Tab. 4.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of kernel parameterizations.
No. BLEMENTS
KERNEL TYPE
CLASSIMCATION ACCURACY (%)
MODEL
ROTMNIST
CIFAR-10
CIFAR-100
ReLU
96.49
59.95
28.01
LeakyReLU
94.47
56.19
27.36
Swish
94.41
66.
34,20
SIREN
83.73
$2.35
ReLU
97.73
68.29
37.81
SE(2)-CNN
LeaL
97.65
68.95
36.30
Swish
97.72
69.
34,10
SIREN
99.17
86.08
$5.55
ReLU
98.49
66.84
37.72
LeskyReLU
98.53
68.00 68.01
38.29
Swish
98.55
65.99
37.72
SIREN
99,24
86.68
$1.51
16


Reviews:
[{"review_id": "tNTCBkQ6ye", "reviewer": "Reviewer_3sxh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\nThe paper addresses the important problem of learning symmetries from data. One of the weaknesses of group equivariant CNNs is that one has to assume what is the group that acts on the dataset, which might be hard to know in practice. Commonly, the chosen group will be too restrictive and still miss relevant transformations. The paper proposes a way to start from some group of transformations and learn to prune it differently at each layer. This is potentially more flexible and is shown to outperform the baselines where the original group is fixed throughout. \n\n# Weaknesses\n\nIf I understand correctly, the partial equivariance as defined is an approximate equivariance to a subset of the group. In this case, the average along the group dimension will not be invariant to transformations from the subset. Furthermore, once equivariance is broken in some layer, it can never be recovered, so the effect of this approach is to monotonically restrict equivariance on each subsequent layer.\n\nOn one hand, this is satisfying because the benefits of reducing or breaking equivariance at some mid or final layer have been observed before (eg the group restriction in Weiler and Cesa [1]). On the other hand, what the method can learn from data seems to be straightforward to implement with either a group restriction or including coordinate values on feature channels like in Fuchs et al [2], while the proposal seems also prone to converging to undesirable models such as breaking equivariance too early or having a more restrictive layer following a less restrictive.  \n\n1) Figure 4 shows what I would expect as a reasonable output, where the equivariance is broken at the final layers to make use of the input pose. Is the same observed for CIFAR10/100? Please report similar figures for the SE(2) case.\n   \n2) Results would be more convincing if compared with the aforementioned handcrafted methods of breaking equivariance. How would something simple like dropping from E(2) to T(2) on the last layer, or using a fully connected layer instead of the final global max pooling perform on CIFAR10/100? These would highlight the advantages learning partial equivariances per layer. \n   \n3) The Monte Carlo approximation of the group convolution integral causes some equivariance error. The group pruning to achieve partial equivariance compounds on this error, even when only the group subset is considered. I appreciate the error expressions in Appendix B, but I believe such errors should be quantified and compared with the baselines (both non-equivariant and fully equivariant). Without these numbers it is hard to judge whether (or at what layer) we can consider the equivariance approximate or broken. \n\nI have a separate concern regarding novelty. The paper seems heavily inspired by Benton et al [3], who, despite focusing on learning invariances from data, also show an extension to learning global equivariances. I believe the differences and similarities between both approaches should be discussed more thoroughly and that quantitative comparison under fair conditions is also necessary. Benton et al [3] also show results for rotated MNIST and CIFAR10. \n   \n# Questions\n\n1) Please elaborate on which discrete groups are unstable to train as mentioned in section 6. Table 2 shows the best results for E(2) G-CNN, which has a discrete component. Was the instability observed in this model or others?\n\n# Minor\n\nFigures 1 and 2 are hard to read. It would be much easier to compare the feature maps and see the differences if they were rotated to the same pose. \n\n# References   \n\n[1] Weiler and Cesa, \"General E(2) - Equivariant Steerable CNNs\", NeurIPS'19.\n\n[2] Fuchs et al, \"SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks\", NeurIPS'20.\n\n[3] Benton et al, \"Learning Invariances in Neural Networks\", NeurIPS'20.\n\n[4] Finzi et al, \"Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data\", ICML'20.\n", "summary_of_the_paper": "The paper proposes to learn partial equivariances from data by training a group equivariant CNN (G-CNN) where the group elements are pruned at each layer. For learning the pruning operation, the paper leverages the idea of Finzi et al [4] to approximate the group convolution with a Monte Carlo integral, and the idea of Benton et al [3] to use the reparametrization trick to learn the bounds of an uniform distribution from where the group elements are sampled. Results show that the partial equivariance performs outperforms the full equivariance when the data itself is not fully rotational-equivariant. ", "summary_of_the_review": "I think the paper addresses an important problem and proposes a seemingly reasonable method that might be useful in practice. However, in the current state I'm not sure if there is enough evidence to demonstrate the value of the paper (see detailed review). I would be happy to increase my rating if such evidence is presented during the discussion phase. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RrPJfdg56-I", "reviewer": "Reviewer_RivC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n- Partial equivariance is an interesting yet under-explored problem. Existing G-CNN literature mainly focuses on full equivariance, and partial equivariance is seldom explored. This paper has demonstrated that studying partial equivariance is of practical relevance: It can improve performance upon conventional G-CNNs when full equivariance is harmful.\n- The proposed partial group convolution is a simple yet effective solution. This paper proposes a solution that only requires a small modification of existing G-CNN architectures, and the authors have empirically shown that the solution is effective.\n- Clear messages from the experiments. I like the experiments section, because the take-away messages from these experiments are very clear, and the main claims of this paper are empirically well supported: \n  - (Figure 4) Different layers may require different levels of equivariance.\n  - (Table 2) Partial G-CNNs improve performance when full equivariance is harmful, e.g. CIFAR10, CIFAR100, and still match the performance of G-CNNs when full equivariance is beneficial, e.g. rot-MNIST\n  - (Table 4) SIREN kernel parameterisation is significantly better compared to other choices.\n\n**Weaknesses**:\n\nMy main concern of this paper is its presentation and technical statements, which is detailed below:\n- Misleading terminology/notations. Even though the overall idea is simple and clear to me. I still find the technical parts very hard to follow and check their correctness, mainly due to the somewhat misleading usage of terminology and notations. Below are a few examples:\n  - The input/output space of group convolutions are functions on groups rather than the groups themselves, and the groups are the input space of a particular input/output function. This paper very often does not distinguish the two, hence causing confusion.\n  - Does $p_i$ in Section 4.3.1 (the discrete group case) define a categorical distribution over groups? If that is the case, then the probabilities should sum to 1? But then the paper set all the probabilities $p_i$ to $1$. I might misunderstand something here.\n- Unclear illustration: Figure 1 and Figure 2 are hard for me to understand. A few suggestions to improve clarity:\n  - Provide a mathematical definition of equivariance error.\n  - In Figure 1, images on the right are feature maps (functions on groups) rather than group elements. So better not annotate them as group elements and group subsets, or explain what’s the correspondence between group elements and feature maps?\nImages in Figure 2 might be too small.\n  - The text explanation for these two figures also needs lots of work.\n\n \n**Questions**:\n\n- If there is an image dataset, for some of the data, the corresponding group subset is $[-60, 60]$, for the rest, the corresponding group subset is $[0, 90]$, what would the overall learned group subset be?\n- In the definition of partial group convolution, can we constrain the distribution over groups to be a uniform distribution over a subset? That is to say, group elements are either in or out of the subset, and all elements in the subset are equally likely. For me, it seems to be a more ‘natural’ definition of ‘partial’ group convolution, unless there are cases when you would prefer a more flexible distribution $p(g)$?\n\nI will give a negative score for now because the technical part needs significant work before publishing. But I think it still solves an interesting problem with a simple solution, and the experiments are sufficient. If the authors are willing to give a list of changes they will make to the mathematical formulation and statements, I am happy to raise my score to the positive side.\n", "summary_of_the_paper": "**Summary and Contributions**:\n\nThe paper proposes partial group convolution that can learn layer-wise partial equivariance from data. The main idea is to use the Monte Carlo approximation to the group convolution as in LieConv (Finzi 2020), but sample group elements from a learned distribution over groups rather than a uniform one.\n\nThey empirically show that allowing the network to learn partial equivariance improves performance upon conventional G-CNNs when full equivariance is harmful, and still match the performance of G-CNNs when full equivariance is beneficial.\n", "summary_of_the_review": "The paper studies an interesting problem proposes a simple solution and has sufficient empirical support for main claims. However, the technique parts and over clarity needs significant work before publishing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "PeJdDO9gJnE", "reviewer": "Reviewer_8mxH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n  1. The writing is clear and easy to follow. The relationship with relevant works is discussed sufficiently.\n  2. The proposed partial equivariance is novel and several different cases of the subsets are properly discussed (e.g., group to subset, subset to group, etc.)\n  3. Experiments show certain improvements compared with enforcing equivariance on the whole group.\n\n## Weaknesses\n  1. One major concern is that the partial equivariance seems not motivated properly. It is not clear why and when the partial equivariance will be needed in the natural datasets. All experiments on MNIST are artificially constructed in favor of the partial equivariance, and the experiment on CIFAR only show marginal improvement on classification accuracy which cannot prove that the CIFAR dataset demonstrates partial equivariance property. More analysis/visualization/experiments/concrete examples should have been included to motivate the need for partial equivariance and show its specific benefits. \n  2. Another concern is related to the first one: The experiments are conducted only on small datasets and are artificially constructed in favor of the partial equivariance. More experiments on larger and natural datasets are needed.", "summary_of_the_paper": "This paper proposes to learn partial equivariance where the equivariance property is constrained to a subset of the considered group. Specifically, they propose to learn such subset instead of predefining it, by parameterizing the sampling probability of the MC sampling. Experiments on variants of MNIST and CIFAR10 demonstrate that the partial equivariance brings better performance than whole group equivariance in terms of classification accuracy.", "summary_of_the_review": "In summary, although the proposed partial equivariance is novel, it's not well motivated and the current experiments/analysis do not prove the benefits of partial equivariance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ihMfYruLB", "reviewer": "Reviewer_xU2c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\nThe main strength of the paper is that idea is conceptually simple and well executed. Indeed, enforcing strict equivariances is often a strong constraint in learning problems and as a result the ability to toggle the amount of equivariance needed should prove beneficial. The writing in this paper is also well done for the most part and the overall presentation is of high quality.\n\n**Weaknesses**\nIn my opinion there are a few weaknesses in this paper that I believe the authors could address. \n\n***Presentation Issues:***\nStarting with presentation, I found the notation to be unclear or inconsistent at times. For example, in Eqn 3. $f$ as previously defined is a map from $G\\to \\mathbb{R}$ but $v \\in X$. The lifting operation as a result is bit confusing. I'm aware of the lifting operation as defined in Kondor & Trivedi 2018 [1], can you please comment whether this is the same thing being done here? Also, Eqn 3. assumes a continuous/infinite group and discrete/finite groups this integral becomes a sum. Although this is obvious, I feel like the authors should mention this point as the first G-Conv paper was on finite groups. With regards to Fig 1 & 2, I tried my very best but I still couldn't discern what to look for. I appreciate the authors outlining the important samples in red but still it wasn't clear to me. I feel like a better and fine grained qualitative description is needed here as at the moment it takes too much effort to understand what should be a simple figure. Moreover, proposition 4.1 is not novel but the current presentation is a bit ambiguous on this point. I'm sure the authors will agree that this has been proven in other work (e.g. [2]) so it would be necessary to make this more explicit and cite the correct papers. Finally, with regards to the experiments a quantification of the error is needed (e.g. +/- with a statistical test) as this is standard protocol. \n\n***Technical Issues:***\nThis paper has a few other technical issues. First, it seems like a large body of Steerable G-CNNs are missed. This is important as the equivariance is guaranteed and not in expectation like in computing the integral. Furthermore, from the experiments it is unclear if the authors used Steerable G-CNNs for their baselines. I suspect that this is not the case as the results do not match [3] for RotationMNIST and CIFAR10. Also, do note that in [3] the authors achieve a variable level of equivariance at different depths which is in line with partial equivariance as proposed here. Admittedly, these are not the same but a similar idea was attempted in [3] and proved useful this diminishes the novelty of this work. If the baselines are indeed not G-Steerable with mixed equivariance then I feel like this is a must have and is quite a relevant comparison. Lastly, I found the quantification of the equivariance error an interesting discussion worthy of the main paper. However, I do not get the main point here as breaking exact equivariance to the whole group is desirable which is the premise of this paper. What should a reader take away from this error quantification if the error is not a bug but a feature? Finally, if I understood correctly the paper claims that we are learning a distribution over a subset but it seems like it is always the uniform distribution and what we instead learn is the domains or support of this distribution over the group. Is there a case where it would be beneficial if the learned distribution is not uniform?\n\n\n[1] Kondor, Risi, and Shubhendu Trivedi. \"On the generalization of equivariance and convolution in neural networks to the action of compact groups.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Cohen, Taco, Mario Geiger, and Maurice Weiler. \"A general theory of equivariant cnns on homogeneous spaces.\" arXiv preprint arXiv:1811.02017 (2018).\n\n[3] Weiler, Maurice, and Gabriele Cesa. \"General $ E (2) $-Equivariant Steerable CNNs.\" arXiv preprint arXiv:1911.08251 (2019).", "summary_of_the_paper": "This paper introduces the notion of partial equivariances which are defined as being equivariant with respect to a subset of a group. The authors also propose to learn the domain of this subset on the fly as opposed to enforcing this prior to training. The authors validate their proposed model on simple datasets that require different degrees of equivariance such as RotationMNIST, CIFAR10, and CIFAR100. ", "summary_of_the_review": "Overall the paper is of high production quality but misses the mark on claimed novelty. The biggest potential weakness is the omission of G-Steerable CNNs as baselines which are quite relevant here. There are a few minor presentation issues but these can be fixed with a revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed learning partial and full equivariances from data in an end-to-end way. The problem studied is an important issue of existing equivariant neural networks which always assume full equivariance. However, the paper only got 3 "marginally below threshold" and 1 "marginally above threshold" even after rebuttal. The major challenges include technical parts being hard to follow due to multiple reasons, unsatisfactory paper writing (the updated version has undergone restructuring), the issue of "breaking equivariance" after multiple layers, some important experiments (such as comparing with Steerable G-CNNs) missing, etc. After rebuttal only reviewer RivC raised his/her score to "marginally above threshold". Such scores are difficult to justify acceptance. The AC appreciated the authors for making great efforts on rebuttal and revising the manuscript. However, from the review comments it is clear that the paper still needs further revision (many of those clarified for the reviewers, e.g., explaining distribution for the discrete groups and adding more experiments, should be included in the manuscript). So the AC deemed that the paper is not ready for publication.
Decision: Reject