Learning Partial Equivariances From Data
Authors: David W. Romero
Suhas Lohit
Venue: NeurIPS 2022 Accept (2022)
Keywords: partial equivariance
soft equivariance
group equivariance
group convolutional neural networks

Abstract: Group Convolutional Neural Networks (G-CNNs) constrain learned features to respect the symmetries in the selected group, and lead to better generalization when these symmetries appear in the data. If this is not the case, however, equivariance leads to overly constrained models and worse performance. Frequently, transformations occurring in data can be better represented by a subset of a group than by a group as a whole, e.g., rotations in $[-90^{\circ}, 90^{\circ}]$. In such cases, a model that respects equivariance partially is better suited to represent the data. In addition, relevant transformations may differ for low and high-level features. For instance, full rotation equivariance is useful to describe edge orientations in a face, but partial rotation equivariance is better suited to describe face poses relative to the camera. In other words, the optimal level of equivariance may differ per layer. In this work, we introduce Partial G-CNNs: G-CNNs able to learn layer-wise levels of partial and full equivariance to discrete, continuous groups and combinations thereof as part of training. Partial G-CNNs retain full equivariance when beneficial, e.g., for rotated MNIST, but adjust it whenever it becomes harmful, e.g., for classification of 6/9 digits or natural images. We empirically show that partial G-CNNs pair G-CNNs when full equivariance is advantageous, and outperform them otherwise. Our code is publicly available at www.github.com/merlresearch/partial_gcnn .

some cases, Learning Partial Equivariances from Data
David W. Romero' Vrije Universiteit Amsterdam Amsterdam, The Netherlands d. www.rroguzmaa@vu.....
Suhas Lohit Mitsubishi Electric Research Laboratories Cambridge, MA, USA slohit@merl.. com
Abstract
Group Convolutional Neural Networks (G-CNNs) constrain learned features respect the symmetries in the selected group, and lead to better generalization when these symmetries appear in the data. If this is the case, however, equivariance leads to overly constrained models and worse performance. Frequently, transfor- mations occurring in data can be better represented by subset of a group than by a group as whole, e.g., rotations in [-90", ]90"]. In such cases, model that respects equivariance partially is better suited represent the data. In addition, relevant transformations may differ for low and high-level features. For instance, full rotation equivariance useful to describe edge orientations in a face, but partial rotation equivariance is better suited to describe face poses relative to the camera. In other words, the optimal level equivariance may differ per layer. this work, we introduce Partial G-CNNs G-CNNs able to learn layer-wise levels of partial and full equivariance discrete, continuous groups and combinations thereof as part of training. Partial G-CNNs retain full equivariance when beneficial, e.g., for rotated MNIST, but adjust whenever becomes harmful, c.g., for classification of 9 digits or natural images. We empirically show that partial G-CNNs pair G-CNNs when full equivariance is advantageous, and outperform them otherwise."
1 Introduction
The translation equivariance of Convolutional Neural Networks (CNNs) [31] has proven an important inductive bias for good generalization vision tasks. This is achieved by restricting learned features to respect the translation symmetry encountered in visual data, such that an input is translated, its features are also translated, but not modified. Group equivariant CNNs (G-CNNs) [7] extend equivariance to other symmetry groups. Analogously, they restrict the learned features to respect the symmetries in the group considered such that if an input is transformed by an element in the group, e.g., rotation, its features are also transformed, e.g., rotated, but not modified.
Nevertheless, the group to which G-CNNs equivariant must be fixed prior to training, and imposing equivariance to symmetries not present the data leads to overly constrained models and worse performance [6]. The latter comes from difference in the data distribution, and the family of distributions the model can describe. Consequently, the group must be selected carefully, and should correspond to the transformations that appear naturally in the data.
Frequently, transformations appearing in data can be better represented by subset of a group than by group whole, c.g., rotations in [-90", 1.0 J. For instance, natural images much more likely show an elephant standing straight or slightly rotated than an elephant upside-down. group transformations even change the desired model response, e.g., in the classification of the digits and 9. In both examples, the data distribution is better represented by a model that respects rotation equivariance partially. That is, model equivariant to some, but not all rotations.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Work done at Mitsubishi Electric Research Laboratories.
"Our code is publicty available htta://cw/eenieeeeeerrrrrrraaaaa

---

Input
-100"
-1ar
-NC
OF
-37
OF
INT
135
1907
TNF
0"
D
U
W
W
G-ONN
60"
R
180
Subsel &
Group G
Figure 1: Partial group convolution. a group convolution, the domain of the output is the group G. Consequently, all output components are part of the output for any group transformation of the input. In partial group convolution, however, the domain of the output is learned subset 8 and all values outside of 8 are discarded. As a result, parts of the output in partial group convolution can change for different group transformations of the input. In this figure, the output components within for 0" rotation (outlined in orange) gradually leave 8 for stronger transformations of the input. For strong transformations -here a 180 rotation- the output components within S are entirely different. This difference allows partial group convolutions to distinguish among input transformations. By controlling the size of S, the level of equivariance of the operation can be adjusted.
Moreover, the optimal level of equivariance may change per layer. This results from changes in the likelihood of some transformations for low and high-level features. For instance, whereas the orientations of edges in an human face are properly described with full rotation equivariance, the poses of human faces relative to the camera are better represented by rotations in a subset of the circle.
The previous observations indicate that constructing a model with different levels of equivariance each layer may be advantageous. Weiler and Cesa [46] empirically observed that manually tuning the level of equivariance different layers leads to accuracy improvements for non-fully equivariant tasks. Nevertheless, manually tuning layer-wise levels of equivariance is not straightforward and requires iterations over several possible combinations of equivariance levels. Consequently, it is desirable to construct model able to learn optimal levels of equivariance directly from data.
In this work, we introduce Partial Group equivariant CNNs (Partial G- G-CNNs): family of equivariant models able to learn layer-wise levels of equivariance directly from data. Instead of sampling group elements uniformly from the group during the group convolution learn a probability distribution over group elements at each group convolutional layer in the network, and sample group elements during group convolutions from the learned distributions. By tuning the learned distributions, Partial G-CNNs adjust their level of equivariance at each layer during training.
in G-CNNs- Partial G-CNNs
We evaluate Partial G-CNNs on illustrative toy tasks and vision benchmark datasets. We show that whenever full equivariance is beneficial, c.g., for rotated MNIST, Partial G-CNNs learn to remain fully equivariant. However, if equivariance becomes harmful, c.g., for classification of 6 digits and natural images, Partial G-CNNs learn to adjust equivariance to a subset of the group to improve accuracy. Partial G-CNNs improve upon conventional G-CNNs when equivariance reductions advantageous, and match their performance whenever their design optimal.
In summary, our contributions are:
We present novel design for the construction of cquivariant neural networks, with which layer-wise levels of partial or full equivariance can be learned from data.
We empirically show that Partial G-CNNs perform better than conventional G-CNNs for tasks for which full equivariance is harmful, and match their performance if full equivariance is beneficial.
2 Background
This work expects the reader to have a basic understanding of concepts from group theory such as groups, subgroups and group actions. Please refer to Appx. A if you are unfamiliar with these terms.
Group equivariance. Group equivariance is the property of a map to respect the transformations in a group. We say that map is equivariant to a group if whenever the input is transformed by elements of the group, the output of the map is equally transformed but not modified. Formally, for a group G with elements g G acting set X,, and map Φ X. X., we say that ø is equívariant to if:
ő(g x) g. o(z), Vx E X,, Vg E G.
(1)

---

For example, the convolution of a signal R R and a kernel sh R + R is translation equivariant because L.( * f)= L.f. where L, translates the function by t: (xf(x)=f(z -)). That is, if the input is translated, its convolutional descriptors are also translated but not modified.
The group convolution. To construct neural networks equivariant to group G. we require an operation that respects the symmetries in the group. The group convolution is such a mapping, It generalizes the convolution for equivariance to general symmetry groups. Formally, for any E G. the group convolution of a signal f G + - and a kernel &b G IR is given by:
h(u)= = (+b x (u)) ee u) due(v),
(2)
where (.).) is the (invariant) Haar measure of the group. The group convolution G-equivariant in the sense that for all u, U, G. it holds that:
(++ fu) fw f)(,,, with fe fw)(()))(w=")))
The lifting convolution. Regularly, the input of neural network is not readily defined on the group of interest G. but on a sub-domain thereof XX, i.e., I - R. For instance, medical images are functions on RR² although equivariance to 2D-translations and planar rotations is desirable. In this case .R= and the group of interest is G-SF(2). Consequently, we must first lift the input from X to G in order to use group convolutions. This achieved via the lifting convolution defined as:
(v +in ()) f.. (( (() duc(v); u G, UE X.
(3)
Practical implementation of the group convolution. The group convolution requires integration over continuous domain and, in general, cannot be computed in finite time. As a result, it is generally approximated. Two main strategies exist to approximate group convolutions with regular group repre- sentations: group discretization [7] and Monte Carlo approximation [16]. The former approximates the group convolution with a fixed group discretization. Unfortunately, the approximation becomes only equivariant to the transformations in the discretization and not to the intrinsic continuous group.
A Monte Carlo approximation,
the other hand, ensures equivariance -in expectation- to the continuous group. This is done by uniformly sampling transformations {uy}, {u:} from the group during each forward pass, and using these transformations approximate the group convolution as:
(4) Note that this Monte Carlo approximation requires the convolutional kernel eb to be defined on the continuous group. As the domain cannot be enumerated, independent weights cannot be used to parameterize the convolutional kernel. Instead, Finzi et al. [16] parameterize it with a small neural network, i.e., (M)=MM((x). This allows them map all elements 07 u, defined kernel value.
( & (u.) Σ, v(15' u)) f(; (at (vz)..
3 Partial Group Equivariant Networks
3.1 (Approximate) partial group equivariance
Before defining the partial group convolution, we first formalize what we mean by partial group equivariance. We say that map & is partially equivariant to G. if it equivariant transformations in subset of the group 8 c G. but not necessarily to all transformations in the group G. That is, if:
((g.x)= = g (x((z) Wx E X, E
(5)
Different from equivariance to subgroup of G aa subset of the group that also fulfills the group axioms-, we do not restrict the subset 8 to be itself a group.
As explained in detail in Sec. 3.3, partial equivariance holds, in general, only approximately. and it is exact only if S is a subgroup of G. This results from the set & not being necessarily closed under group actions. In other words, partial equívariance is relaxation of group equivariance similar to soft invariance [44]: the property of a map to be approximately invariant. We opt for the word partial in the equivariance setting to emphasize that (approximate) partial group equivariance arises by restricting the domain of the signals in group convolution to a subset, i.c., a part, of the group.
3.2 The partial group convolution
((2) be subsets of group @ and p(u) be a probability distribution on the group, which is non-zero only on s(2), The partial group convolution from a function f s() - R to function
Let 8()),
3

---

h: (2) + R is given by: h(u)= (1b * (u) SSau p(u) (u) (w) dup((); u u E
(6)
In contrast to group convolutions whose inputs and outputs are always defined on the entire group, i.c. f.h: G R. the domain of the input and output of the partial group convolution can also be subsets of the group. By learning these subsets, the model can become (i) fully equivariant (8(1) (8))), ( (it) partially equivariant ( CG)) or (iii) forget some equívariances (s(22) subgroup of Q).
3.3 From group convolutions to partial group convolutions
In this section, we show how group convolutions can be extended to describe partial equivariances. Vital to our analysis is the equivariance proof of the group convolution [7, 9]. In addition, we must distinguish between the domains of the input and output of the group convolution, i.e. the domains of f and h in Eq. 2. This distinction is important because they may be different for partial group convolutions. From here on, we refer these as the input domain and the output domain.
Proposition 3.1. Let fw)
The group convolution is Geeqiivariant in the sense that:
(v Lu (u) Lw * f)),,, for all u. v. W.W.e. E G.
Proof. [9] (b * fw Le ((r''))f(u) (ddc()))) = So (t "w (u)(')'))(((dddu ())(d)d(c())
= = * f) u)) Lw f)).
In the first line, the change of variables v- is used. This is possible because the group convolution is map from the group to itself, and thus if w. U. G. so does w- Moreover, as the Haar measure is an invariant measure on the group, we have that pg(r)= (7), ) for all v, 0 G. O
Going from the group G subset S. Crucial to the proof of Proposition 3.1 is the fact the group convolution is an operation from functions on the group to functions on the group. As a result, w u a member of the output domain for any w é G applied to the input domain. Consequently, group transformation applied to the input can be reflected by equivalent transformation on the output.
Now, consider the case in which the output domain is not the group C, but instead an arbitrary subset 8 c G. e.g., rotations in - 5.. Following the proof of Proposition 3.1 with u E 8, and v E G. we observe that the operation is equivariant to transformations E G as long as uu is member of S. However, if w- does not belong to the output domain S, the output of the operation cannot reflect an equivalent transformation to that of the input, and thus equivariance is not guaranteed (Fig. 1). By tuning the size of S. partial group convolutions can adjust their level of equivariance.
Note that equivariance is only obtained if Eq. 7 holds for all elements in the output domain. That is, if w-1 " a member of 8, for all elements é S. For partial group convolutions, this is, in general, not the case as the output domain 8 is not necessarily closed under group transformations. Nevertheless, we can precisely quantify how much the output response will change for any input transformation given an output domain 8. Intuitively, this difference is given by the difference in the parts of the output feature representation that go in and out of 8 by the action of input group transformations. The stronger the transformation and the smaller the size of 8, the larger the equivariance difference in the output is (Fig. 1). The formal treatment and derivation are provided in Appx. B.I.
Going from a subset (()) to another subset (2), Now, consider the case in which the domain of the input and the output are both subsets of the group, i.e., e (1)) and e (2)) Analogous the G holds at positions e (22) for which
previous case, equivariance to input transformations " are also members of (2)) Nevertheless, the input domain is not longer restricted to be closed, and thus the input can also change for different group transformations.
To see this, consider partial group convolution from an input subset ((1) to the group & (Fig. 2). Even if the output domain is the group, differences in the output feature map can be seen. This results from differences observed in the input feature map f for different group transformations of the input.
Similar to the previous case, we can precisely quantify how much the output response changes for an arbitrary subset s'(1) in the input domain. Intuitively, the difference is given by the change in the parts of the input feature representation that go in and out of ((2) by the action of the input group transfor- mation. The stronger the transformation and the smaller the size of ((1), the larger the difference in the output is (Fig, 2). The formal treatment and derivation of this quantity is provided in Appx. B.2.

---

0 Exput
3P Expux
0f W0--appx
OCeor
GCierr
Stroect (01
GCwer
Salbet !!!
Group G
Commn C
Comre
Figure The effect group subsets the input domain. Partial group convolutions can receive input functions whose domain is not the group & but subset 8(1), e.g., a mid layer of Partial G-CNN. Consequently, even if the output domain is the group, i.e. {(2)=G, a partial group convolution does not produce exactly equivariant outputs by design. The difference from exact equivariance in the output response is proportional to the strength of the input transformation and the size of ((1),
Special case: lifting convolutions. To conclude, we note that the lifting convolution (Eq. 3) is special case of partial group convolution with 8()= and (Z) Note however, that the lifting convolution fully group equivariant. This is because the domain of the input -X- is closed under group transformations. Hence no input component leaves X' for group transformations of the input.
3.4 Learning group subsets via probability distributions group elements
So far, we have discussed the properties of the partial group convolution in terms of group subsets without specifying how these subsets can be learned. In this section, describe how this can be done by learning a certain probability distribution on the group. We provide specific examples for discrete groups, continuous groups, and combinations thereof.
Vital our approach is the Monte Carlo approximation to the group convolution presented in Sec.
(b+ (u,)) Σ, (1)
As shown in Appx. C, this approximation is equivariant to G in expectation if the elements in the input and output domain are uniformly sampled from the Haar measure, i.e., Uy U Ng (-).²
Approach. Our main observation is that we can prioritize sampling specific group elements during the group convolution by learning a probability distribution p(u) over the elements of the group. When group convolutions draw elements uniformly from the group, each group element is drawn with equal probability and thus, the resulting approximation is fully equivariant in expectation. However, we can also define a different probability distribution that draws some samples with larger probability. For instance, we can sample from a certain region, e.g., rotations in [-3, #J. by defining a probability distribution on the group p(u) which is uniform in this range, but zero otherwise. The same principle can be used to forget an equivariance by letting this distribution collapse to single point, e.g., the identity, along the corresponding group dimension.
In other words, learning a probability distribution p(u) on the group that is non-zero only in subset of the group can be used to effectively learn this subset. Specifically, we define a probability distribution p( ()) on the output domain of the group convolution in order to learn a subset of the group s(2) upon which partial equivariance is defined. Note that we only need to define a distribution on the output domain of each layer. This is because neural networks apply layers sequentially, and thus the distribution on the output domain of the previous layer defines the input domain of the next layer.
Distributions for one-dimensional continuous groups. We take inspiration from Augerino [3], an use the reparameterization trick [26] to parameterize continuous distributions. In particular, we use the reparameterization trick on the Lie algebra of the group [15] define distribution which is uniform over a connected set of group elements [ E, ... w... but zero otherwise. To this end, we define uniform distribution the group via the pushforward of the exponential map exp: G. This give us a distribution which uniform over a connected set of elements [u- .....,,,, ....,, u.. but zero otherwise."
[-1, 1]) with learnable u on the Lie algebra g. and map it to
For instance, we can learn a distribution on the rotation group SO(2), which is uniform between [-0,0 0] and zero otherwise by defining uniform probability distribution 2/ [110 with learnable 9 on the Lie algebra, and mapping it to the group. If we parameterize group elements as scalars g [[
,, T), the exponential map is the identity, and thus (()) (11, 1)). If we sample group
FFinzi et al. [16] show similar result where i; and ny are the same points and thus by ~ (=(-)) suffices. "Note that an exp- pap-pporhwarred local uniform distribution is locally equivalent to the Haar measure, and thus can still use the Haar measurement for integration on group subsets.
5

---

elements from this distribution during the calculation of the group convolution, the output domain will only contain elements in [-8,0) and the output feature map will be partially equivariant.
Distributions for one-dimensional discrete groups. We can define a probability distribution on a discrete group as the probability of sampling from all possible element combinations. For instance, for the mirroring group {1, -1}. this distribution assigns probability to each of the combinations {0, 0J. {0,1), (1, {1, 1} indicating whether the corresponding element is sampled (1) or not (0). For a group with elements (e, II.............................. In J. however, this means sampling from 2 elements, which computationally expensive and potentially difficult to train. To cope with this, we instead define element-wist Bernoulli distributions over each of the elements .. -9n J. and learn the probability p. of sampling each element g.. The probability distribution on the group can then be formulated as the joint probability of the element-wise Bernoulli distributions (1e,91... .n.) = (9:)).
To learn the element-wise Bernoulli distributions, the Straight-Throe
the Gumbel-Softmas trick [25, 35], and use Gumbel-Softmax to back-propagate through sampling. If all the probabilities are equal to 1, i.e. 1. the group convolution will be fully equivariant. Whenever probabilities start declining, group equivariance becomes partial, and, in the limit, if all probabilities become zero, i.e. (p:=0)ā=1 then only the identity is sampled and this equivariance is effectively forgotten.
Probability distributions for multi-dimensional groups. There exist several multi-diimessinal groups with important applications, such as the orthogonal group 0(2) -parameterized by rotations and mirroring-, or the dilation-rotation group -parameterized by scaling and rotations-.
For multi-dimensione groups, we parameterize the probability distribution over the entire group as combination of independent probability distributions along each of the group axes. For a group & with elements و decomposable along dimensions g (d,,.., ..., d,), we decompose the probability distribu- tion as: p(g)= [2= p(d),, where the probability p(d.) is defined given the type of space - continuous or discrete-. For instance, for the orthogonal group O(2) defined by rotations and mirroring m, i.e. 9 (r.m) m). r e SO(2), m E [+13, we define the probability distribution on the group as p(g)= p(r) p(m), where p(r) is a continuous distribution, and p(m) is discrete one defined above.
3.5 Partial Group Equivariant Networks
To conclude this section, we illustrate the structure of Partial G-CNNs. We build upon Finzi al. [16] and extend their con- tinuous G-CNNs to discrete groups. This is achieved by param- eterizing the convolutional kernels on (continuous) Lie groups, and applying the action of discrete groups directly on the group representation of the kernels.
Lid toonv
MaxPoclRd
Sure
ReLD
Rew
Parstal
In addition, we replace the isotropic lifting of Finzi et al. [16] with lifting convolutions (Eq. 3). Inspired by Romero et al. [39], we parameterize convolutional kernels as implicit neural representations with SIRENs [41]. This parameterization leads to higher expressivity, faster convergence, and better accuracy than ReLU, LeakyReLU and Swish MLP parameterizations used so far for continuous G-CNNs, e.g., [40, 16] -see Tab. 7, [27]-. The architecture of Partial G-CNNs is shown in Fig. 3.
Surr
DALI
Partial OO0om
Norn
ReLU
ClobalMaPPool
Liower
Figure 3: The Partial G-CNN.
4 Related work
Group equivariant neural networks. The seminal work of G-CNNs [7] has inspired several methods cquivariant to many different groups. Existing methods show equivariance to planar [11, 50, 48], spherical rotations [47, 8, 12-14], scaling [49, 42, 38], and other symmetry groups [36, 4, 18]. Group cquivariant self-attention has also been proposed [19, 37, 23]. Common to all these methods is that they are fully equivariant and the group must be fixed prior to training. Contrarily, Partial G-CNNs learn their level of equivariance from data and can represent full, partial and no equivariance.
Invariance learning. Learning the right amount of global invariance from data has been explored by learning a probability distribution over continuous test-time augmentations [3], or by using the marginal likelihood of a Gaussian process [44, 43]. Contrary to these approaches, Partial G-CNNs aim to learn the right level of equivariance every layer and do not require additional loss terms. Partial G-CNNs relate to Augerino [3] in the form that probability distributions are defined on

---

Table 2: Augerino vs. Partial G-CNNs. Terms in
Table 1: Results on MNIST6-11 180 and MNIST6-M. parentheses show accuracy of Partial G-CNNs.
BASE GROUP
DAFASET
G-CNN
PARTIAL G-CNN
BASE
No.
CLASSIFICATION ACCURACY (S)
SE(2)
MNIST6-180
50.0
100.0
GROUP
ELEMS
CIFARIO
CIFAR
Mirroing
MNIST6-M
50.0
100.0
SE(2)
99.17 (99.23)
82.38 (88.99
52.98 (57.26)
B(2)
MN
50.0
100.0
16
99.25 (99.18)
82.53 (88.59)
51.47 (57.31)
MNIST6-M
50.0
100.0
B(2)
8
99.12 (97.78)
84.29 (89.00)
52.59 (55.22)
(98.35)
83.54 (90.12)
54.76 (6146)
continuous groups. However, Partial G-CNNs are intended to learn partial layer-wise equivariances and are able to learn probability distributions on discrete groups. There also exist learnable data augmentation strategies [34, 20, 33, 5] that can find transformations of the input that optimize the task loss. We can also view Augerino as learning a smart data augmentation technique which we compare with. In contrast to these methods, Partial G-CNNs find optimal partial equivariances at each layer.
Equivariance learning. Learning equívariant mappings from data has been explored by meta- learning of weight-tying matrices encoding symmetry equivariances [51, 1] and by learning the Lie algebra generators of the group jointly with the parameters of the network [10]. These approaches utilize the same learned symmetries across layers. MSR [51 J] is only applicable to (small) discrete groups, and requires long training times. L-Conv [10] is only applicable to continuous groups and is not fully compatible with current deep learning components, e.g., pooling, normalization. Unlike these approaches, Partial G-CNNs can learn levels of equivariance at every layer, are fully compatible with current deep learning components, and are applicable for discrete groups, continuous groups and combinations thereof. We note, however, that Zhou et al. [51]. Dehmamy et al. [10] learn the structure of the group from scratch. Contrarily, Partial G-CNNs start from a (very) large group and allows layers in the network to constrain their equivariance levels to better fit the data. Finzi et al. [17] incorporate soft equivariance constraints by combining outputs of equivariant and non-equivariant layers running in parallel, which incurs in large parameter and time costs. Differently, Partial G-CNNs aim learn from data the optimal amount of partial equivariance directly on the group manifold.
Experiments
Experimental details. We parameterize all our convolutional kernels as 3-layer SIRENs [41] with 32 hidden units. All our networks eexeept for the (partial) group equivariant 13-layer CNNs [29] used in Sec. 5.1- are constructed with 2 residual blocks of 32 channels each, batch normalization [24] following the structure shown in Fig. 3. Here, we intentionally select our networks to be simple as to better assess the effect of partial equivariance. We avoid learning probability distributions on the translation part of the considered groups, and assume all spatial positions to be sampled in order to use fast PyTorch convolution primitives in our implementation. Additional experimental details such as specific hyperparameters used and complementary results can be found in Appx. E, F.
Toy tasks: MNIST6- 180 and MNIST6-M. First, we validate whether Partial G-CNNs can learn partial equivariances. To this end, we construct two illustrative datasets: MNIST6 180, and MNIST6-M. MNIST6- 80 is constructed by extracting the digits of the class 6 from the MNIST dataset [31], and rotating them on the circle. The goal is to predict whether the number is a six, i.e., a rotation in [-90%, 90%] was applied, or a nine otherwise. Similarly, we construct MN/S76-M by mirroring digits over the axis. The the goal is to predict whether a digit was mirrored or not.
As shown in Tab. 1, G-CNNs are unable to solve these tasks as discrimination among group transfor- mations is required. Specifically, SE(2)-CNNs are unable to solve MNIST6- 180, and Mirror-CNi -G-CNNs equivariant to reflections- are unable to solve MNIST6-M. Furthermore, E(2)-CNNs cannot solve any of the two tasks, because E(2 -CNNs incorporate equivariance to both rotations and reflections. Partial G-CNNs, on the other hand, easily solve both tasks with corresponding base groups. This indicates that Partial G-CNNs learn to adjust the equivariance levels in order to solve the tasks.
In addition, we verify the learned levels of equivariance for Partial SE(2)-CN) on MNIST6- 180. To this end, we plot the probability of assigning the label to test samples of MNIST6 1 180 rotated on the whole circle. Fig 4 shows that the network learns to predict "6" for rotated samples in [-90", 90%], and "9" otherwise. Note that Partial G-CNNs learn the expected levels of partial equivariance without any additional regularization loss terms to encourage them -as required in Benton et al. [3]-.
7

---

1.0
on MNIST6-180 Partial G-CNNs become equivariant to rotations the semi-circlee Table 3: Accuracy on vision benchmark datasets.
6
fullequiv
BASE GROUP
CLASSIRIATTION ACCURACY (8)
Predicted probabiliyy of
0.8
ELEMS
EQUIV.
ROTMNIST
CIFARIO
CIFARIO0
0.6
T(2)
97.23
83.11
0.4
99.10
83.73
52.35
99.13
86.15
0.
SE(2)
99.17
86,08
55.55
99.23
$8.59
$7.26
0.0
99.24
86,39
51.55
-100
-90
99.18
89.11
57.31
Input rotation
98.14
85
54.29
Figure 4: Learned equivariances for "6"
(2)2)
97.78
89.00
55.22
57.78
98.58
90.12
61.46
Regular G-CNNs cannot solve the task.
Benchmark image datasets. Next, validate Partial G-CNNs on classification datasets: RotMNIST [30], CIFAR-10 10 and CIFAR-100 [28]. Results on PatchCam [45] can be found in Appx. F.
We construct Partial G G-CNNs with base groups SE(2) and E(2), and varying number of elements used in the Monte Carlo approximation of the group convolution and compare them to equivalent G-CNNs and ResNets (equivalent to T(2)-CNNs). Our results (Tab. 3) show that Partial G-CNNs advantageous (rotated MNIST and Patch-
are competitive with G-CNNs when full-equivariane Camelyon). However, for tasks in which the data does not naturally exhibit full rotation equivariance (CIFAR-1O 10 and CIF, Partial G-CNNs consistently outperform fully equivariant G-CNNs.
The need for learning partial eyer-wiie equivariances. Next, we evaluate (i) the effect of learning partial equivariances instead of soft invariances, and (if) the effect of learning layer- wise levels of equivariances instead of a single level of partial equivariance for the entire network.
For the former, compare Partial G-CNNs to equivalent ResNets with Augerino [3] (Tab. 2). We extend our strategy to learn distributions on discrete groups to the Augerino framework to allow it to handle groups with discrete components, e.g., F(2). For the latter, we construct regular G-CNNs and replace either the final group convolutional layer by a T(2) convolutional layer, or the global max pooling layer the end by learnable MLP (Tab. 4). If determining the level of equivariance the end of the network is sufficient, these models should perform comparably to Partial G-CNNs.
Tab. 2 shows that Augerino is competitive to Partial G-CNNs rotated MNIST, but falls behind by large margin on CIFAR-10 10 and CIFAR- 100. This result can be explained by how these datasets are constructed. RotMNIST is constructed by rotating MNIST digits globally, thus it is not surprising that a model able to encode global invariances can match Partial G-CNNs. The invariance and equivariance relationships in natural images, however, Consequently, tackling different levels of equivariance at each layer using Partial G-CNNs leads to benefits over using single level of global invariance for the entire network.
complex, as they be local well.
Although the T(2) and MLP alternatives outlined before could solve the MNIST-I 80 and MNIST-M toy datasets, we observed that Partial G-CNNs perform consistently better the visual benchmarks considered (Tab. 4). This indicates that learning layer-wise partial equivariances is beneficial over modifying the level of equivariance only at the end of the model. In addition, it is important to highlight that Partial G-CNNs can become fully-, partial-, and non-equivariant during training. Alternttve models, on the other hand, are either unable to become fully equivariant (T(2) models) or very unlikely to do so in practice (MLP models).
SIRENs as group convolution kernels. Next, we validate SIRENs as parameterization for group convolutional kernels. Tab. 7 shows that SE(2)-CNNs with SIREN kernels outperform SE(()-CNNs with ReLU, LeakyReLU and Swish kernels by large margin all datasets considered. This result suggests that SIRENS are indeed better suited to represent continuous group convolutional kernels.
5.1 Experiments with deeper networks
In addition to the simple networks of the previous experiments, we also explore partial equivariance in group equivariant version of the 13-layer CNN of Laine and Aila [29]. Specifically, we construct partial group equivariant 13-layer CNNs using SF(() as base group, and vary the number of elements used in the Monte Carlo approximation of the group convolution. For each number of elements,
8

---

Table 4: Accuracy of G-CNNs with a MLP instead of global pooling, G-CNNs with final T(2)-conv. layer, and our proposed Partial G-CNNs.
CEAR-10 CPAR-100
BASE
No.
NET
CLASSIRIATTON ACCURACY (%)
GROUP
RoMMIIST
CIFARIO
CIFARI00
T(2)
99.04
82.76
52.51
SE(2)
99.00
86.25
PARTIAL
99.18
87.45
57.31
T(2)
97.98
86.68
57.61
B(2)
16
MLP
99.02
87 37.43
58.87
Figure Example group subsets learned
PARTIAL
98.58
90.12
by Partial G-CNNs.
we compare Partial 13-layer G-CNNs to their fully equivariant counterparts as well as equivalent 13-layer CNNs trained with Augerino. Our results are summarized in Tab. 5.
On partial equivariant settings (CIFARIO and CIFARI00), partial equivariant networks consistently outperform fully equivariant and Augerino networks for all number of elements used. Interestingly, translation equivariant CNNs outperform CNNs equivariant to SE(2) on CIFAR 10 and CIFAR 100. This illustrates that overly restricting equivariance constraints can degrade accuracy. In addition, partial equivariant CNNs retain the accuracy of full equivariant networks in fully equivariant settings. By looking at the group subsets learned by partial equivariant networks (Fig 6), we corroborate that partial equivariant networks learn to preserve full equivariance if full equivariance advantageous, and learn disrupt it otherwise.
6 Discussion
Memory consumption partial equivariant settings. G-CNNs fix the number of samples used to approximate the group convolution prior to training. In Partial G-CNNs we fix maximum number of samples and adjust the number used at every layer based on the group subset learned. Consequently, partial group convolution with a learned distribution p( (((x)2((( 1J) uses half of the elements used a corresponding group convolution. This reduction in memory and execution time leads to improvements in training and inference time for Partial G-CNNs on partial equivariant settings. We observe reductions up to 2.5x in execution time and memory usage on CIFAR-10 10 and C00.R-100.
Sampling per batch element. In our experiments, we sample once from the learned distribution p(u) at every layer, and use this sample for all elements in the batch. better estimation of p(u) can be obtained by drawing a sample per batch element. Though this method may lead to faster convergence and better estimations of the learned distributions, it comes at a prohibitive memory cost resulting from independent convolutional kernels that must be rendered for each batch element. Consequently, we use a single sample per batch at each layer in our experiments.
Better kernels with implicit neural representations. We replace LeakyReLU, ReLU, U ReLU and Swish kernels used so far for continuous group convolution kernels with SIREN [41]. Our results show that SIRENs are better at modelling group convolutional kernels than existing alternatives.
Going from small group subset to larger one. What does it mean and why is it advantageous? In Sec. 3.3 we described that partial group convolution can go from group subset (()) to larger group subset (22,, e.g., the whole group G. Nevertheless, once a layer becomes partially equivariant, subsequent layers cannot become fully equivariant even for 8 (2)==G. Interestingly, we observe that Partial G-CNNs often learn to disrupt equivariance halfway in the network, and return to the whole group afterwards (Fig. 5). As explained below, this behavior is actually advantageous.
Full equivariance restricts group convolutions to apply the same mapping on the entire group. As result, once the input is transformed, the output remains equal up to the same group transformations. In partial equivariance settings, Partial G-CNNs can output different feature representations for different input transformations. Consequently, Partial G-CNNs can use the group dimension to encode different feature mappings. Specifically, some kernel values are used for some input transformations and other ones are used for other input transformations. This means that when Partial G-CNNs go back to a larger group subset from a smaller one, they are able to use the group axis to encode mranformation-dependeee features, which in turn results in increased model expressivity.

---

Table 5: Accuracy vision benchmark datasets with (partial) group equivariant 13-layer CNNs [29].
GROUP
BASE
No. ELEMS
PARTIAL EQUIY.
AUGERINO
CLASSINCATION ACCURAUY (%)
ROTMNIST
CIFARIO
CIFAR100
T(2)
96,90
91.21
67.14
X
X
98.70
85.51
62.06
98.94
87.78
65.79
98.77
92.48
66.72
SE(2)
X
98.43
89.73
65.97
V
98.75 78
92.28
69.83
98.
67.70
8
X
/
99.28
89.96
69.00 66
Leareed subset
WIS CITAR-10
CIAAN-100
Figure 6: Group subsets learned by 13-layer Partial G-CNNs.
In Appx. we evaluate the effect of enforcing monotonically decreasing group subsets function of depth. That is, Partial G-CNNs whose subsets at deeper layers are equal or smaller than those at previous ones. Our results show that this monotonicity leads slightly worse results compared the unconstrained case, thus supporting the use of unconstrained learning of group subsets.
Limitations and future work
Partial equivariances for other group representations. The theory of learnable partial equivariances proposed here is only applicable architectures using regular group representations, e.g., [7, 37]. Nevertheless, other type of representations exist with which exact equívariance to continuous groups can be obtained: irreducible representations [50, 48, 46]. We consider extending the learning of partial equivariances to irreducible representations a valuable extension of our work.
model partial equivariance discrete groups with our proposed discrete probability distribution parameterization, we observed that these distributions can be unstable train. To cope with this, utilize 10x lower learning rate for the parameters of the probability distributions (See Appx. E.3 for details). Nevertheless, finding good ways of learning discrete distributions is active field of research [21, 22), and advances in this field could be used to further improve the learning of partial equivariances on discrete groups.
Unstable training discrete groups. Although
Scaling partial equivariance to large groups. Arguably the main limitation of G-CNNs with regular representations is their computational and memory complexity, which prevents the use of very large groups, e.g., simultaneous rotation, scaling, mirroring and translations. Partial equivariance particularly promising for large groups as the network is initialized with a prior towards being equivariant to the entire group, but is able focus those relevant to the task at hand. We consider learning partial equivariances on large groups an interesting direction for further research which orthogonal to other advances to scale group convolutions to large groups, e.g., via separable group convolutional kernels [32, 27].
Acknowledgments and Disclosure of Funding
David W. Romero and Suhas Lohit were supported by Mitsubishi Electric Research Laboratories. David W. Romero is also financed as part of the Efficient Deep Learning (EDL.) programme (grant number P16-25), partly funded by the Dutch Research Council (NWO). This work was partially carried out the Dutch national infrastructure with the support of SURF Cooperative.
10

---

References
[I] F. Alet, D. Doblar, A. Zhou, J. Tenenbaum, K. Kawaguchi, and C. Finn. Noether networks: meta-learning useful conserved quantities. Advances in Neural Information Processing Systems, 34, 2021.
[2] B. E. Bejnordi, M. Veta, Van Diest, B. Van Ginneken, N. Karssemeijer, Litjens, A. Van Der Laak, M. Hermsen, Q. F. Manson, M. Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases women with breast cancer. Jama,
3 (18(22)2199-22166 2017.
[3] G. Benton, M. Finzi, Izmailov, and A. G. Wilson. Learning invariances neural networks from training data. Advances in Neural Information Processing Systems, 33: 33:17605-17616 2020.
[4] A. Bogatskiy, B. Anderson, J. Offermann, M. Roussi, D. Miller, and R. Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992-1002, PMLR, 2020.
[5] E. Chatzipantazis, S. Pertigkiozoglou, E. Dobriban, and K. Daniilidis. Learning augmentation distributions using transformed risk minimization. arXiv preprint arXiv:2111.08190, 2021.
[6] S. Chen, E. Dobriban, and J. H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-71, 2020.
[7] T. Cohen and M. Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pages 2990-2999. PMLR, 2016.
[8] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling. Spherical cnns. In International Conference on Learning Representations, 2018.
[9] T. S. Cohen, M. Geiger, and M. Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[10] N. Dehmamy, R. Walters, Y. Liu, D. Wang, and R. Yu. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems, 34, 2021.
[11] S. Dieleman, J. De Fauw, and K. Kavukcuoglu. Exploiting cyclic symmetry convolutional neural networks. In International conference on machine learning, pages 1889-1898, PMLR, 2016.
[12] C. Esteves, A. Sud, Z. Luo, K. Daniilidis, and A. Makadia. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning, pages 1812-1822. PMLR, 2019.
[13] C. Esteves, Y. Xu, C. Allenn-Banchhete, and K. Daniilidis. Equivariant multi-view networks, In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1568-1577, 2019.
[14] C. Esteves, A. Makadia, and K. Daniilidis. Sein-weighted spherical cnns. Advances in Neural Information Processing Systems, 33:8614-8625 142555,, 2020.
[15] L. Falorsi, P. de Haan, T. R. Davidson, and P. Forré. Reparameterizing distributions lie groups. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3244-3253 PMLR, 2019.
[16] M. Finzi, S. Stanton, P. Izmailov, and A. G. Wilson. Generalizing convolutional neural networks for cquivariance to lie groups on arbitrary continuous data. International Conference on Machine Learning, pages 3165-3176. PMLR, 2020.
[17] M. Finzi, G. Benton, and G. Wilson. Residual pathway priors for soft equivariance constraints. Advances in Neural Information Processing Systems, 34, 2021.

---

[18] M. Finzi, M. Welling, and A. G. Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. International Conference on Machine Learning, pages 3318 8--388.. PMLR, 2021.
[19] F. Fuchs, D. Worrall, V. Fischer, and M. Welling, Se trantffofform::: 3d roto- antottranslation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970- 1981, 2020.
[20] R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama. Faster autoaugment: Learning augmen- tation strategies using backpropagation. In European Conference on Computer Vision, pages 1-16. Springer, 2020.
[21] E. Hoogeboom, J. Peters, R. Van Den Berg, and M. Welling. Integer discrete flows and lossless compression. Advances in Neural Information Processing Systems, 32, 2019.
[22] E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Towards non-autoregressive language models. arXiv preprint arXiv:2102.05372 2021.
[23] M. J. Hutchinson, C. Le Lan, Zaidi, E. Dupont, W. Teh, and H. Kim. Lietransformer: equivariant self-attention for lie groups. In International Conference on Machine Learning, pages 4533-4543. PMLR, 2021.
[24] S. Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. PMLR, 2015.
[25] E. Jang, S. Gu, and B. Poole. Categorical reparameterizztioon with gumbel-softmax arXiv preprint arXiv:1611.01144, 2016.
[26] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[27] D. M. Knigge, D. W. Romero, and E. J. Bekkers. Exploiting redundancy: Separable group convolutional networks on lie groups. In International Conference on Machine Learning, pages 11359-11386. PMLR, 2022.
[28] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv: 1610.02242, 2016.
[30] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference Muchine learning, pages 473-480, 2007.
[31] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
[32] A. Lengyel and van Gemert. Exploiting learned symmetries in group equivariant convolutions. In 202/ IEEE International Conference on Image Processing (ICIP), pages 759-763. IEEE, 2021.
[33] Y. Li, G. Hu, Y. Wang, T. M. Hospedales, N. M. Robertson, and Yang. DADA: differentiable automatic data augmentation. 2020.
[34] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
[35] C. J. Maddison, A. Mnih, and W. Teh. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint 1rXiv:1611.007122 2016.
[36] D. Romero, E. Bekkers, J. Tomczak, and M. Hoogendoom. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pages 3188-8199. PMLR, 2020.
12

---

[37] D. W. Romero and J.-B. Cordonnier. Group equivariant stand-alone self-attention for vision. In International Conference on Learning Representations, 2020.
[38] D. W. Romero, E. J. Bekkers, J. M. Tomczak, and M. Hoogendoom. Wavelet networks: Scale equivariant learning from raw waveforms, arXiv preprini arXiv:2006.05259 2020.
[39] D. W. Romero, A. Kuzina, E. Bekkers, J. M. Tomezak, and M. Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. 2020.
[40] K. TT Schütt, P.-J Kindermans, H. H. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller. Schnet: continuous- filter convolutional neural network for modeling quantum interactions. arXiv preprint arXiv: 1706, 170.00666,, 2017.
[41] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33, 2020.
neural
[42] I. Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. 2020.
[43] T. F. van der Ouderaa and M. van der Wilk. Learning invariant weights in neural networks. Workshop in Uncertainty Robustness in Deep Learning, ICML, 2021.
[44] M. van der Wilk, M. Bauer, S. John, and J. Hensman. Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems, 31, 2018.
[45] B. Veeling, Linmans, Winkens. T. Cohen, and M. Welling, Rotation equivariant cnns for digital pathology. In International Conference on Medical image computing and computer- assisted intervention, pages 210-218. Springer, 2018
[46] M. Weiler and G. Cesa. General (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
[47] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. arXiv preprint arXiv:1807.02544 2018.
[48] M. Weiler, F. A. Hamprecht, and M. Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 849-859 858, 2018.
[49] D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
[50] D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[51] A. Zhou, T. Knowles, and C. Finn. Meta-learning symmetries by reparameterization. In International Conference Learning Representations, 2020.
13


Reviews:
[{"review_id": "wDuyLQrPbkq", "reviewer": "Reviewer_5hXP", "summary": "The paper introduces a way to learn partial equivariances from data by using a Monte Carlo group convolution where the samples are taken from a 1D uniform distribution with its bounds learned via the reparametrization trick. The method is shown to work on some tasks where equivariant models fail, like when the task is to detect a specific transformation. It also seems to compare favorably against somewhat weak baselines on natural image classification tasks (CIFAR10/100).\n", "questions": "- The patterns shown in Fig 5 are intriguing and are the most interesting result of the paper in my opinion. I wouldn't expect the bounds to narrow down in middle layers and widen up at the final. I suggest investigating this further as it might provide useful insights. \n\n- There is a mismatch between the accuracy for SE(2), 16, Partial on CIFAR10 in tables 3 and 4.\n\n- L85- states \"The group convolution requires integration over a continuous domain. Hence, it cannot be computed in finite time and must be approximated.\". This is not true, there are many integrals over continuous domains that can be computed exactly in finite time, including group convolutions under certain conditions (for example, between bandlimited functions on compact groups).\n\n- L331- missing reference: \"Tab. ??\".\n", "limitations": "Adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nThe submission addresses an interesting and hard problem, which is how to learn which symmetries are present in the data and how to exploit them. It shows some intriguing results when allowing each layer of the model to be equivariant to a different subset of the euclidean group. \n\n# Weaknesses \n\n1) The major weakness I see is that the main ideas of the submission are similar to Augerino [1], which also uses the reparametrization trick to learn the bounds of a uniform distribution to which the model should be invariant. The main difference is that Augerino uses the learned distribution to augment the input, whereas the proposed method applies the idea to the samples of the Monte Carlo group convolution from Finzi et al [2].\n\n2) A second difference with respect to Benton et al [1] is that the submission introduces a way to learn discrete distributions; however, this seem only evaluated on the simplest 2-element group of reflections, and even in that case it seems unstable to train as stated in L373. I don't understand the motivation of the approach for the discrete case in L206-, wouldn't it make more sense to one  probability mass for each point and sample several points from the resulting distribution?\n\n3) The comparison in Table 2 doesn't seem fair to Augerino, as they report an accuracy of 93.81% on CIFAR10, versus <85% reported in the submission. I understand that the architectures and training scheme are different, but the results would be much more convincing if they followed the baseline's hyperparameters and showed improvements over those numbers. Moreover, Benton et al [1] also evaluate on more interesting tasks such as SE(3) invariance in molecular property prediction and color-space invariance on STL-10. The submission would be made stronger by including these and/or other tasks that involve groups other than E(2) subgroups.\n\n4) Some results in tables 2, 3, 4 are very close to one another, so it is unclear if the differences are statistically significant. I recommend running each training job several times and including confidence intervals.\n\n# References\n\n[1] Benton et al, \"Learning Invariances in Neural Networks\", NeurIPS'20.\n\n[2] Finzi et al, \"Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data\", ICML'20.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kNFEA2bA3DA", "reviewer": "Reviewer_9rGm", "summary": "The paper presents a strategy to learn partial equivariances from the data, in the setting of image classification. This approach allows to learn different degrees of equivariance, from the full E(2) to identity map. The idea is to parametrise the group elements via a probability distribution, which is learned via montecarlo sampling of convolution at different activation layers of a deep network. Results on toy and real data show good results, competitive with standard translation equivariant architectures.  ", "questions": "- It is unclear to me what is the base model structure used in T(2) settings. Authors mention the use of ResNets, but what are base models? De facto standard architectures are achieving lots better on CIFAR, but the current setting is a bit obscure to me. Not that I want to see numbers better than SOTA, of course, but I cannot understand the comparison. Maybe some details, a number of parameters to compare, could help. \n- In papers [1] numbers in Tab 4 for the CIFAR are competitive and the model is also simple. Models [1] and [2], which are similar, can be both extended to known partial equivariances (but not learned, which is the nugget here). Maybe those references could serve some comparisons or discussions, if time and space allow. \n- L220-229. Could it be that some group elements are not independent? e.g. vertical flips would prevent some specific rotations? Is there a way to cope with it? \n- Sec  5: Is not clear what \"mirror CNN\" are.\n- The number of elements vary always as 8 or 16. I am fine with that, but there does not seem to be a good explanation / motivation / justification. \n- There are some missing references (??) in the latex in the main text (L331) and supp mat (L53).\n\n\n\n\n\n[1] Zhou et al, oriented response network, CVPR 17\n[2] Marcos et al, rotation equivariant vector field networks, ICCV 2017", "limitations": "Limitations are addressed in Sec 7, which concludes the paper. I think those are enough and relevant. Some points raised above could be also discussed there, as probably not all my comments can be addressed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n- The paper makes an interesting point, and indeed I think that allowing models to learn partial equivariances to some sub-groups would improve or worst case match performance of standard models. At least, this seem what results are pointing at, even if datasets are maybe not so convincing. \n- The proposed solution, although maybe a bit hidden in the cumbersome formulations and description, is simple yet effective, and seem to be a very interesting solution. I find interesting the fact that the model can learn different levels of equivariance to predefined groups, at different layers. I would be interesting in future work potentially, to constrain specific layers to not have this level of flexibility, and to see if others compensate of if the learning breaks down. In Fig. 5 it seem that the model naturally learns to not be equivariant in layers 3/4 and it would be interesting to see why and what are the limits. \n\n### Weaknesses\n- The theoretical developments in the paper are complex, and as such, they should be followed by clear and exemplifying text, which is not always the case. I found these parts hard to follow, for messages that essentially can be simplified to high-level descriptions and reader sent to references. \n- This would leave some room to describe better the Monte Carlo estimation (sec. 4) and to move some details and results from the supplementary material to main text. I feel for some of that, the supp mat is not the right place (eg App A, with a clear description of what SE(2), T(2), E(2) are, App. D, some of E and F). \n- I feel CIFAR datasets are generally not so easy to use for further generalisation on realistic cases. MNIST even less. I unfortunately deeply miss some more realistic image classification dataset. ALthough is interesting to see results are improving by learning partial equivariances (some tilts, balancing horizontal flips in training set, pose for some classes, etc), I feel that those results alone are not enough to have general claims. \n- Figure 1 is impossible to read (resolution, colour), captions in general are not great. \n- Section 3.3 is unclear overall, maybe needs some reworking\n- In the experiments, I miss an illustration of a loss curve, or something that shows how the training converges. It is also underlined in Sec 7, that the training is unstable and tough, so I wonder, how the authors coped with it. If any trick, weird scheduling, or anything, were used, I think it would be fair to mention it. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kJuZqBtz0TZ", "reviewer": "Reviewer_BwFs", "summary": "This paper analyzes \"the extent of equivariance\"  when the input and output domains are the whole groups or the subsets of the group. Based on the analysis, the authors propose the partial G-CNN by learning the probability distribution $p(u)$ over the group in every neural network layer. Their method works for the discrete group, continuous group, and combinations. The experiments show that partial G-CNN can learn the different sets of transformations depending on the task.", "questions": "1.Line 237 replaces the [15] with a convolution, but I'm confused about what $v^{-1}u$ is, which is the same question in weakness 2.\n \n2.In the experiments, there are some tasks we already know; for example, MNIST6-180 and MNIST6-M, can we derive the fixed out domain but not learn the out domain layerwise to get a better result than conducting convolution over the whole group. I'm a little confused about how to leverage the analysis on partial equivariance quantitively.\n", "limitations": "The authors list the reasonable limitation in the paper and the broader social impact in the supplementary materials.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1.The problem, transformations in the data might be better represented by a subset of the group rather than the whole group, proposed in the paper is novel and is not discussed in previous work in equivariance. This problem deserves more exploration and attention.\n\n2.The analysis of group convolution for different input and output domains is sound, and the method to obtain the output domain by learning the probability distribution over the group is novel.\n\n 3.The paper is well organized, and the images in the paper help audience understand the paper easily.\n\nWeakness:\n1.The paper proposes a method of learning the probability distribution of discrete groups, continuous groups, and the combination. Could the authors show more experiments on the group other than SE(2), E(2), or the flipping group?\n\n2.I'm a little confused about the equation (3); what is $v^{-1}u$ since v is in the domain that $G$ acting on and $u$ is in G? How to get $v^{-1}$ and how does it act on the group element? Correct me if I'm wrong.\n\n3.There is a strong baseline that I think the authors should compare with; that is, the equivariant network over the whole group by replacing the max-pooling layer with the flatting operation and the MLP layer.  The conventional equivariant networks are equivariant to the whole group, and if we don't do pooling over the group dimension, then the transformation information is kept; for example, here, the permutation of features for \"6\" and \"9\" are different, which also can help the network to classify 6 and 9.\n\n4.There are several typos in the paper, especially the reference to the figure and the table. I suggest the authors could check the misalignment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept