Relaxing Equivariance Constraints with Non-stationary Continuous Filters
Authors: Tycho F.A. van der Ouderaa
David W. Romero
Mark van der Wilk
Venue: NeurIPS 2022 Accept (2022)
Keywords: relaxed
equivariance
constraints
non-stationary
generalized
convolution
filters
kernel
soft equivariance
partial equivariance

Abstract: Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterisation can be thought of as a building block to allow adjustable symmetry structure in neural networks. In addition, we demonstrate that the amount of equivariance can be learned from the training data using backpropagation. Gradient-based learning of equivariance achieves similar or improved performance compared to the best value found by cross-validation and outperforms baselines with partial or strict equivariance on CIFAR-10 and CIFAR-100 image classification tasks.

Relaxing Equivariance Constraints with Non-stationary Continuous Filters
Tycho F.A. van der Ouderaa Imperial College London
David W. Romero Vrije Universiteit Amsterdam
Mark van der Wilk Imperial College London United Kingdom
United Kingdom
The Netherlands
Abstract
Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that effectively interpolate between (i) non-equivariant linear product, (ii) strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterisation can be thought of a building block allow adjustable symmetry structure in neural networks. In addition, we demonstrate that the amount of equivariance can be learned from the training data using packpropagation. Gradient- based learning of equivariance achieves similar or improved performance compared the best value found by cross- vallddanln and outperforms baselines with partial strict equivariance CIFAR-10 and CIFAR-100 image classification tasks.
Introduction
Symmetric properties, such as equivariances and invariances, can be embedded into neural network architectures to provide inductive biases that leads to better data-efficiency and improved generalisa- tion. Convolutional layers known to provide translation equivariance in simple Euclidean spaces, and recent works have allowed various extensions to more complex groups and domains. However, symmetries are typically fixed, must be specified in advance, and can not be adjusted.
Symmetries embedded in network architectures enforce hard constraint on the functions neural network can represent. This can be an effective way to encode prior knowledge for problems that are inherently symmetric. However, hard symmetry constraints can become prohibitive if problem does not strictly follow the symmetries. For example, convolutional layers can not encode potentially relevant absolute positional information, and '6's and a '9's become difficult to distinguish under rotation invariance. Relaxed symmetry constraints misspecification without losing the useful inductive biases that symmetries provide.
mitigate such potential symmetry
We propose to relax equivariance constraints by generalising the convolution operator with non- stationary filters that can also depend on absolute group elements. This results in a layer that can efficiently interpolate between (i) con-invariaan linear products akin to cally-connected layer, (ii) strict group equivariant convolutions, and (iii) strict group invariant mappings (Fig. 1).
The importance of this work is twofold. First of all, relaxable symmetry constraints can directly improve the performance in cases where strict symmetries are misspecified and result in an overly restrictive model class. Secondly, automatically learning symmetry structure from data is an interest- ing problem. Work in this field often focuses on invariances [34, 4, 33. 29, 12]. which are easier to parameterise than equivariances, We show a way to effectively parameterise learnable equivariance constraints and demonstrate gradient-baser learning of layer-by-layer equivariance constraints.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Non- Von-stationary Continuous Filter
Constant kernel strict inverance Ap -
Stationary kemel str equivariance kute 0, Ratx (v)
Fully connected Incar Apdy luu)) (67")1609)
Non stan--tattonayyyyo kernel soft oquivariace Ryfor"
Eample comaain
*
2
2
Nofo "), (o)d)p))
My 1 - 14- 0
My 2- (fler Ingpanng
By Mg > (Ster feeuancy)
4,
fequongd
The non-stationary integral operator. Striet invariance, strict equivariance, non-equivariance, and relaxed equivariance are all special cases that depend on the frequency parameters w and w'. The case of the translation group G=T(2) is illustrated: unlike the regular convolution with stationary kernels, the proposed non-stationary filter can depend on the position where it is applied.
Figure I:
2 Related Work
Group equivariance. Equivariance constraints applied layers provide strong inductive bias that enforce transformations in the input to result in equivalent transformations the output. For compact groups it can be shown that this constraint naturally leads to (group) convolutions [15]. Many works have allowed for equivariances to groups other than translation, including continuous roto-translations [38, 36, 16], discrete roto-translations [5, 3], scale [37], and permutations [39] and non-Euclidean domains, such as spheres [6], points clouds [10] and graphs [27].
Symmetry misspecification and approximate equivariance. Although symmetry constraints can be very effective in machine learning, they can become prohibitive if data does not exactly follow the enforced symmetry. For instance, Liu et al. [17] raised an 'intriguing failing' of convolutional neural networks by showing that they can not encode absolute coordinate information. Adding explicit coordinate information was proposed as an ad-hoc solution to the problem, effectively breaking strict equivariance constraints. To achieve the same result, Finzi et al. [8] proposed to break equivariance of a convolutional layer by summing it with a non-equivariant fully-connected layer, requiring many additional parameters. Alternatively, Romero and Lohit [24] consider sparse local support around the group identity element to break equivariance. This is effective for sparsely sampled subgroups, such rotation, but becomes less practical for more densely sampled groups such as translation where the local support could lead to very sparse feature maps. Lastly, Wang et al. [35] propose relaxed convolutional operator similar our non-stationary approach, but require a low-rank factorisation which potentially limits the expressivity of the feature maps. The same work briefly discusses mathematical description without such factorisation similar to our proposal which is deemed "too large a trainable weight space to be practical". This is true for discrete convolutional kernels. Instead we define our kernels as continuous kernels [26] parameterised with a finite number of parameters. Consequently, our weight space becomes tractable and does not pose a problem at all.
Physics and filtering. Symmetries play a central role in physics. In [35] approximate equivariances used to allow more robust models for dynamical systems. In Noether networks [1] conservation laws are inferred by learning symmetries from data. Our work could offer new insights in how symmetry constraints could be relaxed in such applications. In geophysics and seismology. there some interesting parallels between non-stationary filtering methods and approximate equivariances in machine learning. In particular, [19] discusses generalisations of the convolution in the context of non-stationary filtering very similar to how we propose to relax equivariances in neural networks.
Automatic symmetry discovery. Automatically inferring symmetry structure of neural network architectures from data is an interesting open problem. For instance, Lorraine et al. [18] learn data augmentations by differentiating the validation loss using the implicit function theorem and Zhou et al. [40] showed how symmetry structure could be learned through meta- meta-lcarning outer loop. In

---

[4] and [8] symmetries are selected using training loss directly, using an additional regulariser.
Symmetry discovery methods often focus on learning global invariances [34, 4, 29, 33, 12]. which are easier to parameterise compared to layer-wise equivariances. We propose way to continuously relax equivariances allowing efficient parameterisation of learnable equivariance for symmetry discovery methods. Inspired by the regularisation used in [4] to learn augmented inputs, we demonstrate that our parameterisation can be used for gradient-base learning of equivariance in each layer from data.
3 Background
3.1 Groups
A Lic group provides a natural way to describe continuous symmetry, as it forms a continuous manifold of which underlying elements are equipped with a group structure. Lie groups do not necessarily form a vector space. However, to every Lie group G we can associate an underlying vector space called the Lie algebra g. The Lie algebra corresponds the tangent space of the group at the identity, has the same dimensionality as the group, and captures the local structure of the group. g can be expanded in basis in the Lie algebra Because the Lie algebra is vector space, elements
0; A; with coefficients a E Rddim()), The exponential map exp g - G maps elements from the Lie algebra the Lie group. We can also define logarithm log G map from the Lie group to the Lie algebra. Such a choice always exists, but is not necessarily smooth or unique.
a= Ei (
3.2 Equivariance and invariance
Equivariance is the property of mapping such that transformations to the input result in equivalent transformations to the output. If changes are invertible, they can be described as the action of group G on some space X. Formally, Mg x) = h(x) for all E G, A.
say that function h X X' is equivariant the group G if
If the output of the function is completely independent the action of the group G on the input, say that the function invariant to the group G. Formally, a function h - X' is invariant to the group G if hig- x) h(x) for all g G, E X.
3.3 The group convolution
Let f G be an input signal and kg G - be convolutional kernel parameterised by The group convolution is defined as:
huu) = (ke * f)( = / ())
(1)
where we integrate with respect the Haar measure of the group μ.. Group convolutional structure is not just sufficient, but also a necessary condition for equivariance to the action of a compact group [15]. The regular convolution for translation equivariance is the special case where the group the translation group G = T(n) = R", and the group action is given by addition, i.e., V -'u
The convolutional kernel Ko (v (2) which will sometimes refer to as 'filter' for stylistic purposes, called stationary because it can only depend on u and v through -- This invariance of the kernel is important constraint and in fact requirement for be equivariant. Breaking the stationarity of the kernel allows us to relax the equivariant symmetry constraints, which is one of the central ideas of this paper.
3.4
Group lifting
The input and output signals of a neural network are typically not defined on the group G, but rather on an input and output space X' and y. Using group lifting and projection procedure [15, 71. can define our model on the group and still apply it on the input and output space. We define a lifting operator G to map input signals f X' - R to functions on the group cc; G - R. Similarly,
define a projection operator that maps function on the group f G > IR to functions on the output space I' Ly: y IR.
3

---

4 Method
4.1
Non-stationary Integral Operator
The group convolution operation of Eq. (1) is strictly cquivariant due to the stationarity of the kernel. That is, the kernel & only depends on relative group element uu For translation G=T(2), the kernel only depends on relative coordinates (u - v) and not on the absolute position in the image. To relax equivariance constraints, we let the kernel also depend on the absolute group element e:
h(u) = (ke f)(u) = [/c ke(w ke(w='uy (((())(u(())
(2)
The main difference with the group convolution of Eq. (1) is that the kernel Kg G G R now has two input group elements. We will refer to the first input argument e uu E G as the stationary component and the second input argument v E as the non-stationary component. translation, this change lets the kernels depend also on the absolute coordinate of the input image on which it applied. Consequently, I(()) does no longer describe the regular convolution, but rather a non-stationary integral operator, for which the convolution remains limiting case.
case of
For the purpose of this work, there is no meaningful difference between expressing the statinaaionary component in the input domain Ke (u -u, v) or the output domain ke(v "u u). For consistency, we will stick with the input domain throughout this work. One could also think of writing the kernel in an even more general form: (u, v)). This, as such, does not lead to more expressive kernel in the sense that both forms can represent the same class of functions (for a proof, see App. C.I). Further, in this more general way of writing the non-stationary kernel it will become harder to formulate parameterisation that allows for controllable symmetry constraints, so we do not consider
The non-stationary integral operator of Section 4. 1 can represent interesting special cases through an independence of kg with respect to the first and second input arguments.
Remark 1 (Linear product / Fully- connece.... If the kernel equals an impulse response in the stationary component multiplied by some function kig in the non- stationtton component ke Pu v) = 8(v ( then the operator h(u) = (ke {)( corresponds to a linear product with Ky-
If we have kernel (o (o---u, = uu) (v), then we have that:
(u) = [ u,
[ (kk (o) = (o(
which is the linear product between K/g (u) and input f(u). This is akin to an element-wiss product continuously defined on G. Optionally, this could be followed by an invariant layer to average pool activations. If the operation is performed over channels and we regard these as output features, this corresponds to a fully-connected layer in the discrete case G=Z", with weights given by all kgg
Remark 2 (Group equivariance Convolution). If the kernel does not depend the non- non-slationary component, then the operator hiu)) = (kg f)( is equivalent to a strict G-equivariant convolution.
If the kernel only depends the first stationary component, then ke(u kelvv'u, v) kg(u -lo), u), and:
hu = 1 (u,( - - Ko (i) = (K/ (u),
(4)
which equals the group convolution which is known be strictly group equivariant. Remark 3 (Group Invariance/ / Pooling). If the kernel is independent of both stationary and non- stationary components, the operator h((u) = (Ki fu( is strictly G-invariant.
If the kernel does not depend on both inputs, it must be constant (o "u,u) u) = C, and obtain:
koow--u (u)) = ko(w -u, ()f(v)d)()) = C" s(o)du())
(5)
which is a scaled average 'global-pooling' over group actions and leads to strict group invariance. We obtain partial invariance or 'local-pooling' if the kernel is only non-zero constant some local support of the stationary domain.

---

4.2 Parameterising the kernel
Lie algebra basis. In Eq. (2), we have proposed a non-stationary integral operator with a kernel kg G G - I that takes two input arguments: the stationary group element v v='u 'u and non- stationary group element v. We take the approach of parameterising the kernel kg in terms of real vector spaces, through an explicit Lie algebra basis, which are practical to work with. To do so, we first define a logarithm function log and express the kernel
((lg 9, log h) in terms of kernel in the Lie algebra k= : x g - R. which always form vector spaces. For rotation G=SO(2), the exponential map exp sending Lie algebra elements (on line) to Liee group elements (on circle) is not left invertible, but is right invertible, as a function. As the right inverse is not unique, we choose to use the principal log sending Lie group elements to Lie algebra elements closest to the identity. In general, such a choice for the logarithm always exists but is not necessarily unique or continuous. We choose vector space basis {A, dim(() express Lie algehra elements g in terms of coefficients a E (elim(() that relate Lie group elements g E G by:
Gim(G)
g cxp((a)= exp
Σ o,
a E ddim(G)
Through this construction, each group element y corresponds through choice of log to unique Lie algebra element a which can be implemented as a real vector a.
Functions in the Lie algebra. We parameterised the non- sontstationary kernel kg G x G - R. taking two group elements as input, in the Lie algebra k : g x + IR acting on a product space of two vector spaces. We refer to the first vector space related to the stationary component as the filter space, as it corresponds to space in which filters of conventional convolutions are defined. And refer to the second vector space of the non-stationary component as the domain space, because elements directly relate to absolute group elements in the input domain.
In Section 4.1 we showed that an independence of the kernel with respect to the first and second input argument correspond to strict equivariant and invariant symmetry constraints. We would, therefore, like to choose k such that we can control the dependence on the stationary and non- stationary components, as this will allow for explicit control over symmetry constraints. To do so, we parameterise à using a set of random Fourier features [23] with a frequency parameter w for each dimension. The frrquency parameter gives control over the spectral properties of the function in filter space and domain space. But crucially, a frequency of w=0 results in a constant infinite lengthscale and therefore a function that is independent to the associated input dimension. This mechanic allows to select the special cases discussed in Section 4.1 and interpolate between them! Together, we have frequency parameter w E [d)m(()) that controls spectral properties in filter space and frequency parameter w' E [Rdim to control spectral properties in domain space. We propose a weight-space parameterisation that allows for explicit control over the symmetry properties of the layer through frequency parameters w and w'
Weight-space implementation. We choose to parameterise our kernel in a finite D-dimensional random Fourier features (RFF) [23, 31] basis Yo + 1222 to embed Lic algebra elements a E as real vectors using their coefficients a E Rdim(G) where the frequency can be explicitly controlled by frequency parameter w E Rdim(G), Values of W E Dxdim(()) are randomly initialised and can be kept fixed. We parameterise the Lie algebra kernel i: g x + R with a neural network NNg RD + IR taking concatenated Fourier features input:
(a)a) = Vb Cos (2 W @))) sin (W w))
w))
ko(a, Ius av) = NNo ( ([7w(a ((ru(aa-1a)))
(7)
Ywo
where Yo (a, 1n)) are the Fourier features on the filter space and You (a.)) are the Fourier features on the domain space, respectively parameterised by w and w' frequency parameter vectors.
Directly representing the kernel with an MLP on a without Fourier features will likely hamper performance as MI.Ps are known to suffer from 'spectral bias' [2, 22], making it difficult to encode high-frequency functions. Sinusoidal activations in random Fourier features [23, 31] and SIRENs [30] alleviate this issue and have been found suitable encodings of positional information in neural networks [32], including (place-coded) continuous kernels in neural networks [26. 25].
5

---

Input group elements
Fourier features sin))
Filter network NN,
a O
G
ay O
Tv
O
Relaxed Lift Gonnv
Relaxed Conv
Next layees
a' O O
R
veC
G
a', O
2 1
( fl
Continuous Filter kg : G x G + R
f c G
9 C G
Groups
T(2) S0(2) }, SE(2)
T(2) soft
50(2) soft ). SE(2) soft
Figure 2: Parameterisation of continuous filter ko in soft-SE(2) equivariant model. Fourier features "Yu and Ywr represent the stationary and non-stationary input components v v u E G and e E G. Frequency parameters and w' respectively control the spectra of the filter space and domain space, generalising the convolution to an operator with continuously adjustable symmetry constraints.
Intuition behind frequency parameters. The frequency parameters and w' directly control the bases of the respective 'filter space' and 'domain space' bases. Specific values of w and w' correspond to layer h that exactly or interpolates between special cases discussed in Section 4.1. Most notably, for w'=0 the layer becomes a strictly equivariant convolution and for both w= the layer performs invariant pooling. To understand why this is, notice that a feature with zero frequency parameter is constant Yo. by definition. For ='=0 we therefore have a constant feature for the non-stationary component Your (=.)=% and kernel that is solely defined in filter space. If both and the entire kernel must therefore be constant, resulting in invariant pooling. Positive values for w' introduce non- non-stationarity that effectively relaxes the strict equivariance constraints of the convolution at w'=0. Lastly, note that individual scalars w' correspond to particular subgroups and we can relax equivariance constraints of specific subgroups by letting w' be greater than zero in the associated dimension.
w=w'=0, both Fourier features become constant =% (q.-in)= (a.-t-
Example for translation group As example, we consider relaxing translation equivariance G-T(2). Since ((2) = 2, we have two dimensional filter and domain spaces and therefore also two-dimensional filter space frequencies w = [Wa: E 2R22 and two domain space frequencies w' = =I "/, E TR?. If (0= 0J", the layer reduces to translationally equivariant convolutional layer. In other words, for any value of or weights 0 the kernel remains stationary (as we have w'=0). In this case the layer equals a convolution where the same filter is applied to all locations in the input. Higher values of w, i.e. Was Why O, correspond to more rapid changes in x- and y-direction in filter space, but the overall operation remains a convolution with a filter that is applied to all locaitons in the image. If, instead, we let the components of the domain space frequencies w' become non- zero, i.e. W/r w/ > 0, then the kernel values can also depend on the absolute coordinate in the input image. Higher w/ result in a more rapidly changing kernel relative to different absolute locations in the input image. This non-stationarity breaks equivariance constraints in continuous way. For sufficiently high w', the function can become akin to a linear product that has enough variance to place an independent weight on each pixel location. In practice, this would also require sufficiently flexible NNg. Different symmetry modes for translation G2T(2) are illustrated in Fig. 1.
V. D
- 35
72"
Y- 200
- 144
180
- 216
v= 252"
286
324
360*
Sereet sndhaatorce
Selt eapheetiiee
samphd
Figure 3: Equivariant and soft-equivariant SO(2) filter banks. The top row shows a strict rotation equivariant filter bank for =0 The middle row shows a filter bank with positive non-stationary frequencies w/,00, without strict equivariance constraints. Sampled group elements at the bottom.

---

Table 1: Image classification test accuracies. Comparison between strict and relaxed equivariant models, Mean and standard error
v3 are reported over three different random seeds.
samples
accunicy
Group
in SO(2)
T(2)
x SO(2)
Model
CIFAR-10
CIFAR-100
suginentt
T(2)
Strict
CNN
$1.80 =0.45
85.87
#0.22
46.20
20.20
53.97 #0.35
Soft
Ours
8182 20.35
87.74
20.33
46.84
20.39
58,12 30.31
Striet
Strict
G-CNN
20.25
85.76
:0.21
20.38
49.81
Striet
Partial
Romero and Lobit [24)
83.04 10.25
84,26
10.35
46,69 11:08
52.67 10.00
SE(2)
Striet
Soft
Ours
83.21
87.02
49.47
54.48
T(2) 80(2)
Striet
Striet
G-CNN
82. 20.20
87,
+:0 00.00
46.43
20.14
54.07 =0
8
Strict
Partial
Romero and Lohit [24]
$4.82 20.20
87.37 #0.01
51.04 20.51
58,22 20.25
Striet
Soft
Ours
$6.39
88.38
54,68
60.83
Strict
Strict
G-CNN
83.33 _10.33
87.15 10:02
47.68 10.15
54,20 10.21
16
Striet
Partial
Romero and Lobit [24]
85.92 20
89.48 20.41
50.58
20
59.63 10.37
Strict
89.65 #0.22
=0.30
60.27
Experiments
5.1 Verifying soft-equivariance using symmetry misspecification toy example
To verify that relaxed equivariance allows for less restrictive functions compared to strict equivariance, we repeat the MNIST6- 180 problem of [24] and include our model. This toy problem is designed such that it can not be solved under strict symnetry constraints. It consists of the 6's in the MNIST dataset with half of the dataset randomly rotated by 180 degrees. The task is to determine whether sample has been rotated, i.e., a binary classification problem between '6's and '9's. In Table 2, we compare a rotation invariant G-CNN model that consists of multiple strictly equivariant layers followed by group pooling and compare with [24] and our model. In line with our expectations, we find that in this artificial set-up the relaxed ResNet model quickly converges to 100% test accuracy. The model with strict rotation symmetry is unable to distinguish between '6's and '9's s and can thus not improve over a simple coin-flip flip with 50% accuracy on average.
Table 2:
Toy problem. Models that obey strict rotation symmetry can not distinguish between 6's and 9's whereas the task can be solved under relaxed symmetry constraints.
Test accuracy
SO(2)
Model
MNIST6- 180
Striet
G-CNN
50.0
Sparse [24]
100.0
Relaxed
Soft (ours)
100.0
5.2 Evaluating different fixed levels of relaxed equivariance
We assess the effectivity of relaxing equivariance constraints for translation and rotation groups T(2). SO(2) and SE(2) CIFAR- 10 and CIFAR-100 image classification tasks. As strict symmetries can lead to misspecification, we hypothesise that relaxed equivariance can improve the model performance on these tasks. As baselines, we use strictly equivariant G-CNNs and Partial G-CNNs [24]. The difference in model sizes arising from different kernel input dimensions are neglectable compared to the total number of model parameters for the used architecture. In Fig. 4. we plot performance on the validation set for different values of frequency parameter w' and list final test accuracies in Table From Table we can see that our method outperforms the baseline on both datasets across all settings. This holds both with and without data augmentation, showing that the benefit of relaxed equivariance does not disappear when using augmentations. On T(2), we find that the improvement is negligible small and it would be interesting to see whether this gap becomes larger with increased number of basis functions or other choices of NNg. We observe particularly large improvements in test accuracy for (2) on CIFAR-10 and CIFAR-100. On CIFAR- 100, relaxing equivariance improves test accuracy of the non-invariant baseline by percentage points when using augmentation and improves by 8 percentage, without augmentation.
7

---

soft-50(2) (4 samples)
soft-SO(2) (8 samples)
5oft=S0:2) (16 samples)
5oft-T(2)
84
CIFAR-10
Accuracy
85
81
CIFAR-10
Accuracy
25
Accuracy (%)
CIFAR-100
50
teat: 49.47
CIFAR-100
Accuracy
45
valid 40 20
wilddttion accursty
validation accuracy
validation
validation
16
strict
relased
relezed
strict
releaed
Figure 4: Relaxing equivariance constraints. Test accuracy on CIFAR-IO and CIFAR-100 is shown for (E(()--quivarrint models with soft-SO(2) equivariances for different values of w/. and soft-T(2) equivariance varying W/ =wy, Some relaxation increases test accuracy in all cases, but the benefits deminish when relaxed too much. We use cross-validation to find the optimal relaxation and find that this outperforms strict equivariance constraints in all cases.
To further assess the effect of equivariance relaxation has on model performance, we can examine the impact different values for of non-stationary frequency parameter w' on model performance. In Fig. 4, we show validation accuracy for different w' averaged over three seeds with error bars corresponding to standard error (33) and show final test accuracy for model with best validation score. The scale of the x-axis should not directly be interpret as this depends on the chosen basis, parameterisation and initialisation of NNg. We observe that some relaxation of equivariance with frequency parameter w' * 0 leads to better test accuracy compared to the strict equivariant model at w'=0. The benefits of the relaxation diminishes when equivariance is further relaxed. This is to be expected, as this then corresponds to very high frequencies in domain space which are unlikely to provide useful features anymore. We find that there exists optimal amount of relaxation, which can find with cross-validation. In all cases, we find that sufficiently relaxing equivariance results in higher test accuracy compared strict equivariance. This indicates that strict equivariance might be slightly misspecified in this setting, to which our approach offers solution.
5.3 Gradient-based learning of relaxed equivariance
Instead of using cross-validation, also explore whether the right amount of equivariance can be learned using gradients. This is desirable, since cross-validation is an expensive procedure because model need be retrained for different relaxations and requires additional hold-out validation data, Furthermore, finding the amount of equivariance using cross-validation may require sharing w' across layers in practice limit the search space single dimension, causing each layer always have the same amount of relaxation. Training w' with backpropagation, alongside the model parameters, far from trivial as equivariance acts as a constraint on the functions a neural network can represent. Therefore, we can not expect that increased equivariance will lead to an improvement in terms of the regular (cross-entropy) training loss and be learned by directly optimising it with such loss. To learn equivariance constraints, which we expect will improve generalisation performance on test data, we propose to add a regularising term A| \||w>||3 132 to the objective, to encourage symmetry. The loss is inspired by the regularisation proposed in Augerino [4] to learn augmented inputs, and similarly requires an additional hyperparameter À E R. From a Baycsian point of view, the new objective can be interpret finding maximum posteriori probability (MAP) estimate after placing a Gaussian prior on the relaxation frequency parameter N (ww( 0, x I)
(see App. C.2 for details).
In Table 3, we compare models with the partial and strict equivariance baselines, well as the best fixed amount of equivariance found by cross- vooos-vaiiittion. We report the mean and standard error of the test accuracy on CIFAR-10 and CIFAR-100 over 3 seeds, trained with and without augmentation. For most settings of X, the model achieves similar or improved performance over the best fixed settings found by cross-validated settings, outperforming the baselines with partial and strict equivariance. This is a promising result as indicates that we can learn the amount of layer-wise equivariance as part of single training procedure, without requiring expensive validaaaio or validation data.
8

---

Table 3: Learning the amount of equivariance with gradients. Comparison of soft SE(2) relaxations where the amount of equivariance w' is learned for different regularisation strengths X. Equivariance learning achieves similar improved test accuracy on CIFAR-I/CCIFAL 1-00 tasks compared to the best value found by cross-validation outperforming baselines with striet and partial equivariance.
sumples
Test accurasy
Group
T(2)
80
Model
relaxatio of
CIFAR-IO
CIEAR-
T(4)
Sulet
CNN
$1.80 Lecas
11:22
11/0
53.97 43.33
Exed
vuliiued)
$1.82
87.74
46,84
58.12
Ours
Icemal a/
A-0 001
79.99
87.04
46.21 21
55.46
Soft
Ours
leamed a
A- 01
$2.04 10.15
87.28
15.17
40.05
10:10
M6.32 11:05
Soft
Ours
Jeamet af
AmdD 1
82.10 20:97
87:33
25.16
66.55
21.07
M6. 40 2316
Solt
Ours
Jeamed of
A=0.5 5.5
82.20 40.02
87.55
46.92
42:12
56.32 43:13
Soft
Ours
Jeamed of
2-1 1.0
$1.89 10.21
87.25
16.00
66,82
1E:10
56.90 11.0
Ours
lormal
A-5.0
81.00 20
11
47.00
57.02
Strict
Strie
G-CNN
81 21-33
85, 85.26.25 21-01
64.00
09.81
Strict
Partol
Romeno and Lobi (24)
$3.04 21-33
84.26
46.
60
5:14
52.67
23:40
(8(3) -S(())
Smrict
Soft
Ours
food
83.21 2n.a5
87.02 01
49.47
51:41
54.48
Stnst
X-n MMl
05:33
10:07
50.51
51:30
Suidt
Jeamel
53.70
55.70
Suict
82.65
85,94
55.37
Siriet
Soft
Ours
Iommal
A-100
82.65
85.
51.34
55.97
Siriet
Suft
Ours
Icemall
A-5.0
82:47
85.71
52.19
56.34
Stiit
Striet
CNN
82.16
54.07
Striet
Partal
Romero and Lobit [24]
84.82
Stiet
Soft
Ours
Sxed //
(cross validared)
$6.39 10.35
85.35
15:34
54.08
15.00
09.53
13.43
Soiat
Soft
Ours
Scamee a/
A-- 80L
$3.54 44004
86.09 44:00
50.14
41.16
56.20
+3:35
Smitt
Soft
Ours
lommod
Amd
85.27 25:35
81:10
21:00
50.48
23:00
Seict
Soft
Ouns
lormal
$6.36 23:09
89.26
21:00
00:94
Stidt
Jeamed
0.5
35.05 10..0
10:07
11:20
00:57
11:29
Jeamed
10.34
15.00
11:11
13.00
85.01
Sciil Stras
G-CNN
54.20 =1:3
16
Stnce Pirtal
Komero
[24]
$5.92 20.32
35.00
59.83
13:27
Stict
Ssed
validuted)
$6.29
$4,90
08.27
Strict
Soft
Ours
lommel =
Amm UV1
83,
87.87
49,75
57.22
Stiet
Soft
Ours
Jeamed u
A-0 2.01
$3.59 L.0.0
$7.96 1..11
49.63
15.08
57,57 11.25
Stict Soft
Ours
Jeamed of
2-0.1 1.1
85.81 10.05
89.00
15.10
55.11 18.15
02.22 11:23
Seici
Sofl
Ours
Icenal af
0.5
85.85 24.10
89.35
$3.80
61.87 =3.15
Seiel
Soft
Ours
Scemal af
A-1.0 1.0
85.74 04:.3
89.47
53.21
65.15
61.08
=1:3
Senat
Soft
Ouns
Jeamed o
A-5.0
85.05
20.19
89.30
20.22
$6.12
11:16
50.87
23:38
6 Conclusion
In this work, we have proposed a generalisation of the group convolution that allows for a smooth parameter-efficient relaxation of otherwise strict symmetry constraints. The main idea is to use non- stationary kernel that also depends on the absolute input group element, breaking strict equivariance constraints. Morcover, we show that we can obtain explicit control over the symmetry constraints through tunable frequency parameters, by representing group elements in a Fourier feature space.
We demonstrate relaxed equivariance in neural networks with roto- troo-srannlttion equivariance, relaxing the rotation and translation subgroups. We find that some relaxation of equivariance yield higher test accuracies on CIFAR-10 and CIF image classification tasks, with and without augmentation. Furthermore, we show that the w' parameter controlling the amount of equivariance can be learned with gradients from training data. Learning the amount of equivariance achieves similar or improved performance compared to the best value found by cross-validation and outperforms baselines with partial strict equivariance. To perform gradient-based equivariance learning, we encourage symmetry using a similar regularisation as proposed in Augerino [4] to learn augmented inputs. Some limitations of directly regularising symmetry have been discussed in [12], such as the need for an additional hyperparameter that needs tuning and dependence on the used parameterisation of symmetry. It would be interesting investigate whether alternative objectives [29, 33, 12] could resolve such issues. Symmetry discovery methods in literature often focus on invariances, which are easier to parameterise, whereas this work offers way parameterise learnable equivariance. This paves the way for automatic layer-by-layer symmetry discovery as part of a single training procedure.
By relaxing equivariance properties we can leverage useful inductive bias that symmetries can pro- vide, while preventing possible symmetry misspecification if data does not fully obey an embedded symmetries. Experimentally, we have demonstrated that neural network layers with relaxed equiv- ariance constraints can improve test accuracy on CIFAR- 10 and CIFAR- 100 image classification tasks, outperforming strict-equivariant models with up to 10 percentage points in test accuracy on CIFAR- 100. We hope that the proposed non-stationary kernel as a general building block can be useful in machine learning applications that require smooth relaxations of symmetry constraints.

---

References
[1] Ferran Alet, Dylan Doblar, Allan Zhou, Josh Tenenbaum, Kenji Kawaguchi, and Chelsea Finn. Noether networks: meta-learning useful conserved quantities. Advances in Neural Information Processing Systems, 34, 2021.
[2] Ronen Basri, Mcirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pages 685 694.. PMLR, 2020.
[3] Erik Bekkers, Maxime W Lafarge. Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In International conference medical image computing and computer- aassiste intervention, pages 440-448. Springer, 2018.
[4] Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010. 1882, 2020.
[5] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016.
[6] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint
1801 10130, 2018.
[7] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning, pages 3165-3176. PMLR, 2020.
[8] Mare Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivari- ance constraints. Advances in Neural Information Processing Systems, 34, 2021.
[9] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing cquivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning. pages 3318-3341 -328.. PMLR, 2021.
[10] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-111 10000199111111 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Alexander Immer, Tycho F. A. van der Ouderaa, Vincent Fortuin, Gunnar Rätsch, and Mark der Wilk. Invariance learning in deep neural networks with differentiable laplace approxi- mations, 2022.
[13] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456, PMLR, 2015.
[14] David M Knigge, David W Romero, and Erik J Bekkers. Exploiting redundancy: Separable group convolutional networks on lie groups. In International Conference on Machine Learning, pages 11359-1 1136. PMLR, 2022.
[15] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pages 2747-2755. PMLR, 2018.
[16] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. teebsch-gordan nets: fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems, 11:10117-10126, 2018
[17] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordcony solution. arXiv preprint 180iv:1807.08: 03247, 2018.
[18] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 1540-1552. PMLR, 26-28 Aug 2020.
10

---

[19] Gary F Margrave. Theory of nonstationary linear filtering in the fourier domain with application to timevariam filtering. Geophysics, 63(1):244-255 1998.
[20] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[21] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[22] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[23] Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS. Citeseer, 2007.
[24] David W Romero and Suhas Lohit. Learning equivariances and partial equivariances from data. arXiv preprint arXív:21 10.22010.11111111 10211, 2021.
[25] David W Romero, Robert-Jan Bruintjes, Jakub M Tomezak, Erik J Bekkers, Mark Hoogendoom, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.00000
2021.
[26] David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2 1eXiv:2102.022 1, 2021.
[27] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. International Conference on Machine Learning, pages 9323-9332. PMLR, 2021.
[28] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A contimuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
[29] Pola Schwöbel, Martin Jørgensen, Sebastian W Ober, and Mark van der Wilk. Last layer marginal likelihood for invariance learning. arXiv preprint (0.::2/06.07//2 2021.
[30] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[31] Danica J Sutherland and Jeff Schneider. On the error of random fourier features. arXiv preprint 1rXiv:1506.02785 2015.
[32] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Ragha- van, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 10739, 2020.
[33] Tycho FA van der Ouderaa and Mark van der Wilk. Learning invariant weights in neural networks. In Workshop in Uncertainty & Robustness in Deep Learning, ICML, 2021.
[34] Mark van der Wilk, Matthias Bauer, ST John, and James Hensman. Learning invariances using the marginal likelihood. arXiv preprint arXiv: arXiv:1808.00 11888800563. 2018.
[35] Rui Wang. Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. arXiv preprint arXiv:2201.11965 2022.
[36] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steer- able enns: Learning rotationally equivariant features in volumetric data. arXiv preprint 1rX1v:1807.02547, 2018
[37] Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint 1rXiv:195..11697 2019.
[38] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Har- monic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
11

---

[39] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets, arXiv preprint arXiv:1703.06114 2017.
[40] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprini arXiv:2007.02933, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Used datasets public and instructions can be found in the paper. However, the code is not yet publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hpbP3I-oI12", "reviewer": "Reviewer_bGts", "summary": "Strict equivariance constraints in deep learning may sometimes be overly restrictive for some tasks. This paper proposes a relaxation of equivariance by allowing the convolutional kernel to be non-stationary (thus depending on the absolute input location). They achieve so by introducing frequency parameters, which can interpolate between strict equivariance and element-wise linear product, thus adjusting the \"softness\" of equivariance. They empirically demonstrate that their proposed soft equivariance leads to better performance on CIFAR10 and CIFAR100 in terms of classification accuracy.", "questions": "1. How much performance gain comes from the usage of random Fourier features rather than the soft equivariance? Can the authors consider running an ablation study for this?\n2. I am very confused by the concept of \"kernel\" in this paper. On the one hand, it is only the convolutional filters, so it does not need to satisfy kernel properties such as positive semi-definite (See https://people.eecs.berkeley.edu/~jordan/kernels/0521813972c03_p47-84.pdf). However, on the other hand, the authors mention GPs and lengthscales, and use the random Fourier features. All of these are for the \"kernels\" in kernel methods literature. Furthermore, even though they are using random Fourier features, the neural networks take in the concatenation of features and directly produce a scalar as output. In this case, the kernel function is merely an arbitrary function on a pair of inputs.\n\nI am ready to raise my score if the authors could clarify my confusion and give a convincing justification.", "limitations": "I have no concern over negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. This paper studies an important problem: strict equivariance constraints can be overly restrictive and is misspecified for some problems. It would be nice to have soft equivariance and automatically adjust symmetries on data. This paper provides a simple, practical and parameter-effective solution.\n2. This paper shows clear performance improvements using the proposed soft-equivariance on CIFAR10 and CIFAR100. It would have a great impact if this improvement is widely applicable to lots of tasks.\n3. The proposed solution is new and it seems to solve the over-strict equivariance problems. (However, I think more careful ablation study and potentially more experiments on new datasets are needed in the future)\n\n**Weaknesses**\n1. Although this paper shows clear performance improvements for the proposed soft equivariance compared to strict equivariance on CIFAR10 and CIFAR100, it is not convincing enough that the performance gain actually comes from soft equivariance. It would be ideal to have an ablation study showing that, the performance gain does not come from using the random Fourier features. It would be more convincing if the authors could show performance improvements on more tasks and datasets. But I understand that this is not possible during the rebuttal period. So I am not asking the authors to run experiments on extra datasets, just a suggestion for future versions.\n2. This paper is not careful enough with mathematical terminology: For example, \n    * (line 166) \"cross-product of two vector spaces\". I believe they mean the product space of these two spaces, rather than the vector product here (https://en.wikipedia.org/wiki/Cross_product). \n    * (line 175) \"let $\\hat{k}$ be a Gaussian Processes (GPs)\". Firstly, $\\hat{k}$ is a kernel function while GP is a stochastic process with a mean function and kernel function. It does not make sense to me by saying a kernel function is a stochastic process. Secondly, the kernel function here corresponds to the convolutional filter, which is different from the kernel functions in GP literature. In GPs, the kernel function needs to be positive semi-definite. Thirdly, I think the authors only use the kernel function here and it has nothing to do with GPs (the probabilistic model)\n    * (eq 3) I believe it is only an element-wise product. For me, it is a bit contrived to say this corresponds to the fully-connected layers. Even if you apply the averaging pooling layers, you still have only one output unit? \n\n(Please correct me if I misunderstand)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ky7JzLdoV96", "reviewer": "Reviewer_qnnH", "summary": "The paper proposes a deep-neural network architecture for relaxing equivariance constraints. The main idea is to use a non-stationary kernel in a group convolution. They propose to model the kernels via continuous kernels parameterized via a neural network. Experiments on CIFAR10 and CIFAR100 demonstrate improvements over baseline models which are strictly equivariant.", "questions": "Please address the concerns in the weaknesses section above. Particularly, I am most interested in a comparison with data augmentation method, Augerino, and knowing whether the baselines use data augmentation in table 1. ", "limitations": "The paper does not have a limitations section. It would be great to discuss some shortcomings of the approach. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe topic of modeling parietal equivariance or learning symmetries from data is relevant and of interest to the NeurIPS community. Overall the writing of this paper is clear and the approach is clearly described. While similar to prior works in continuous kernels, the proposed non-stationary kernel is sufficiently novel.\n# Weaknesses\n1. The main weaknesses of this work is the empirical validation. While the proposed method is capable of modeling soft equivariances, it is unclear whether the proposed method is better than alternative approaches. For example, one could choose a model that has very few built in equivariance assumptions, e.g. transformer, and perform data augmentation. The model will learn to exhibit soft equiviarance behavior. For example, a Vision Transformer with positional encoding can be trained to be partially shift-equivariant. The current experiment lacks comparison and only conducts experiments on CIFAR10 and CIFAR 100 dataset, which is not very convincing. \n2. The paper could be strengthened if additional comparisons with other baselines were made. For example, a comparison with parameter sharing method [33] or augmentation based method [A] (Augerino also has an equivariant setting (Sec. 3.1)). \n[A] Benton, Gregory, et al. \"Learning invariances in neural networks from training data.\" Advances in neural information processing systems 33 (2020): 17605-17616.\n3. The authors propose to use Fourier features when parametrizing the kernel (Line. 193), it would be great to conduct an ablation study on this.\n4. Some experimental and training details are missing. It might be difficult to reproduce the results. Is data augmentation being used for the baseline?\n5. The authors should report the number of trainable parameters introduced and the model’s inference speed. It seems like the proposed approach would be slower than a standard ResNet. \n\n# Miscellaneous\n* The checklist should have been attached with the main paper and not in the supplemental materials.\n* Some references still cites the arXiv version instead of the conference version. For example, Deep-Sets [32] is published at NeursIPS.\n* Line 18. \"euclidean\" --> Euclidean\n* Line 75. \"continuosly\" --> continuously\n* Line 186. \"fourier\" --> \"Fourier\"  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HquGjY1HmWo", "reviewer": "Reviewer_hLit", "summary": "This paper proposes a parameter-efficient neural layer for relaxed equivariance. The main idea is to use a non-stationary integral operator whose kernel depends on absolute group elements as well as relative group elements.  This allows to interpolate between non-equivariant, strictly-equivariant, and strictly-invarint mappings in learning. The proposed method is validated on CIPAR-10 and CIFAR-100 image classification benchmarks, outperforming other strictly and partially equivariant methods. \n", "questions": "I think this is a good paper, but addressing the weaknesses I described above would strengthen the contribution. \nPlease respond to my points of the weaknesses. \n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Clear motivation for relaxed equivariance\n- Simple yet effective idea of non-stationary filter\n- Enabling targeted relaxation of a specific subgroup\n- Convincing results on MNIST and CIFAR image classification\n\nWeaknesses\n- Missing comparison to related work. The proposed method is compared only with G-CNN and Romero and Lohit [18] while there exist other recent related work, e.g., E(2)-CNN, Implicit Equivariance in Convolutional Networks. \n- Limited experimental setup. The experiments are done using a small network on small datasets. It is unclear whether the performance gain is retained when bigger networks (ResNet18, 50) and larger datasets (ImageNet) are used. Furthermore, the benefit of equivariance relaxation is not demonstrated beyond image classification. \n- Missing analyses. Comparison of FLOPs and # of parameters and ablation studies on design choices (activation function..) are not provided. \n- How to set non-frequency parameter w? While the advantage of the proposed method is the controllability of equivarience levels, the performance of a learned model looks significantly affected by the control parameter w as shown in Fig. 4. The sensitivity of performance to w and the determination of w in practice need to be discussed. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FSVWwgDW8", "reviewer": "Reviewer_WiTS", "summary": "The paper proposes soft equivariance parameterization of neural networks that can interpolate between non-equivariant linear product thru strict-equivariant convolution to strict-invariant mapping. The contributions of the paper are twofold: 1) improving applications where strict equivariance is suboptimal; 2) providing a way to learn equivariances from data. The main idea is to use a non-stationary kernel for the group convolution. The contribution of this paper is how to parameterize that kernel. While the paper proposes a general approach, which might work for any group, in practice it only shows the idea for T(2) and SO(2). The paper obtains marginal improvements over [18]. \n\nWhile I like the idea, I am afraid that the paper is missing experimental and implementation details. Furthermore, can the method be extended to larger datasets and tasks?\n\n\n", "questions": "Related work: please, consider citing works on data augmentation with self-supervised training objectives as an effective way of baking in approximate equivariance into neural networks, e.g.: https://arxiv.org/abs/2111.00899.\n\nLine 84: please, specify that $A_i$ are the basis vectors of the Lie algebra.\n\nLine 97: please, specify that *the layers of the neural network being a group convolution* (Eq. 1) is actually the necessary and sufficient condition (following Theorem 1 in [11]). Currently, it's not clear what exactly that condition is.\n\nLine 128: \"see ??\"; there is a broken link; please, fix it.\n\nLine 161: you mean Lie group elements instead of *Lie algebra*? The exponential map maps from the group to the algebra. Actually, isn't this supposed to be a \"log\" map instead of \"exp\"?\n\n", "limitations": "I am not sure if the authors have addressed the limitations of the approach properly. In particular, I would like to see implementation challenges and the limitations stemming from the implementation constraints, discussed in Section 5. Ablation studies would be helpful.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n\n1) Natural idea to interpolate between invariance, equivariance and non-equivariance.\n2) Effective implementation, improving reasonable baselines.\n3) Mostly well-written, with a few question (see below).\n\nCons: \n\n1) Computational metrics, such as training and inference time, memory constraints and implementation details are missing.\n2) The method provides only marginal gains over small datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept