Empowering Networks With Scale and Rotation Equivariance Using A Similarity Convolution
Authors: Zikai Sun
Thierry Blu
Venue: ICLR 2023 poster (2023)
Keywords: Representation Learning

Abstract: The translational equivariant nature of Convolutional Neural Networks (CNNs) is a reason for its great success in computer vision. However, networks do not enjoy more general equivariance properties such as rotation or scaling, ultimately limiting their generalization performance. To address this limitation, we devise a method that endows CNNs with simultaneous equivariance with respect to translation, rotation, and scaling.  Our approach defines a convolution-like operation and ensures equivariance based on our proposed scalable Fourier-Argand representation. The method maintains similar efficiency as a traditional network and hardly introduces any additional learnable parameters, since it does not face the computational issue that often occurs in group-convolution operators. We validate the efficacy of our approach in the image classification task, demonstrating its robustness and the generalization ability to both scaled and rotated inputs.

Published as a conference paper at ICLR 2023
EMPOWERING NETWORKS WITH SCALE AND ROTA- TION EQUIVARIANCE USING A SIMILARITY CONVOLU-
TION
Zikai Sun and Thierry Blu
Department of Electronic Engineering, The Chinese University of Hong Kong zksunë 1 ink. cuhk. edu.hk, thierry. wwwlmmu..u.ommcommm org
ABSTRACT
The translational equivariant nature of Convolutional Neural Networks (CNNs) is a reason for its great success in computer vision. However, networks do not enjoy more general equivariance properties such as rotation or scaling, ultimately limiting their generalization performance. To address this limitation, we devise method that endows CNNs with simultancous equivariance with respect to translation, rotation, and scaling. Our approach defines a convolution- like operation and ensures equivariance based on our proposed scalable Fourier-Argand representation. The method maintains similar efficiency traditional network and hardly introduces any additional learnable parameters, since it does not face the computational issue that often occurs in group-convolution operators. We validate the efficacy of our approach in the image classification task, demonstrating its robustness and the generalization ability to both scaled and rotated inputs.
INTRODUCTION
The remarkable success of network architectures can largely attributed the availability of large datasets and a large number of parameters, enabling them to "remember" vast amounts of information. On the contrary, humans can learn new concepts with very little data and are able generalize this knowledge. This disparity is due, in part, to the current limitations in modeling geometric deformations in network architectures. Networks are inclined to "remember" data through filter parameters rather than "learning" full generalization ability. For instance, in classification tasks, networks trained on datasets with specific object sizes often fail when tested on objects with different sizes that were present in the training set. The ability to factor out transformations, such as rotation or scaling, the learning process remains to be addressed. It is indeed quite frequent to deal with images in which objects have a different orientation and scale than in the training set, for instance, as a result of distance and orientation changes of the camera.
To mitigate this issue, it is common practice to perform data augmentation (Krizhevsky al. 2012) prior to training. However, this leads to substantially larger dataset and makes training more complicated. Moreover, this strategy tends learn a group of duplicates of almost the same filters, which often requires more learnable parameters to achieve competitive performance. A visualization of the weights of the first layer (Zeiler Fergus 2014) highlights that many filters are similar but rotated and scaled versions of common prototype, which results in significantly more redundancy.
The concept of equivariance emerged as a potential solution to this issue. Simply put, equivariance requires that if given input undergoes specific geometric transformation, the resulting output feature from the network (with weights randomly initialized) should exhibit a similarly predictable geometric transformation. Should network satisfy equivariance scalings and rotations, training with only one size and orientation would naturally generalize its performance to sizes and orientations.
To achieve this property, group convolution methods have been widely used in this field. An oversimplified interpretation of a typical group convolution method is as follows: Features convolved with dilated filters of the same template to obtain multi-channel features, where the distortion of input features corresponds to the cycle shift between channels. For example, equivariant

---

Published as a conference paper at ICLR 2023
CNNs truncated directions (Cohen & Welling 2016 Zhou al. 2017) leverage several directional filters to obtain equivariance within a discrete group. Further works extended rotation equivariance
to continuous group, using techniques such as stcerable filters (Weiler et al., 2018 Cohen et al.
2019), B-spline interpolation (Bekkers) 2020) or Lie Group Theory (Bekkers 2020: Finzi et al.
2020). A similar path scaling equivariance has been explored, although scaling
longer intrinsically periodic. The deep scale space (Worrall & Welling 2019) defined semi-symmetry group to approximately achieve scale equivariance, while Sosnovik al. (2020) applied steerable CNNs to scaling. However, integrating equivariance to both rotations and scalings simultaneously leads larger group (e.g.. a rotation group with M points and scaling group with N points results in M x N points for the joint rotation and scaling group), making the task more challenging. Additionally, certain "weight-sharing" techniques based on group convolution can be computationally and memory-intensive.
Despite the difficulties, empowering the model with equivariance of rotation and scaling together can be highly advantageous. For instance, in object detec- tion, the distance changes between the camera and the object, or the random rotations of the object, can significantly impact the accuracy of the method.
3REN
}cr
The aim of this paper is to propose CNN archi- tecture that achieves continuous equivariance with respect to both rotation and scaling, thereby filling void in the field of equivariant methods. To ac- complish this, we provide a theoretical framework and analysis that guarantees the network preserves this inherent property. Based on this, we propose a new architecture, the Scale and Rotation Equivariant Network (SREN), that avoids the abovementioned limitations and does not sizably increase computa- first designed a
b
SaIN
Figure 1: The visualization the Sim cquivariance property: Our SREN method inherently retains the structure information of the input, enabling handle distorted ob- jects (rotation, scaling, and translation) with- out additional training.
tional complexity. Specifically, scalable Fourier-Argand representation. The expres- sion of the basis makes it possible operate the angle and scale in one shot. Further, we propose a new convolution-lik operator that is functionally similar traditional convolution but with slight differences. We show that this new method has similar computational complexity to convolution and can easily replace the typical network structures. Our approach can model both rotation and scaling, enabling consistently achieve accurate results when tested with datasets that have undergone different transformations, such as rotation and scaling.
The main contributions in this paper are summarized follows:
We introduce the scalable Fourier-Argand representation, which enables achieving equivariance on rotation and scaling.
We propose the SimConv operator, along with the scalable Fourier-Argand filter, forming the Scale and Rotation Equivariant Network (SREN) architecture.
SREN is an equivariant network for rotation and scaling that is distinct from group- conoouoovoll neural networks, offering a new possible path to solving this problem for the community.
RELATED WORK
Group convolution To accomplish equivariance property. possible direction is the application of group theory to achieve equivariance. [Cohen & Welling (2016) introduced group convolution, which enforces equivariance to a small and discrete group of transformations, i.e., rotations by multiples of 90 degrees. Subsequent efforts have aimed to generalize equivariance (Zhou et al. (017) and also focus on continuous groups coupled with the idea of steerable filters (Cohen & Welling 2017). To achieve this purpose, Lie group theory has also been utilized, presented in works such as LieConv (Finz et al. 2020). albeit only for compact groups. Unfortunately, the scaling group is non-compact, methods typically treat it equivariance. TridentNet (Li et al. 2019) gets scale invariance by sharing weights among kernels with different dilation rates. Another approach is to apply scale-space theory (Lindeberg 2013). which
cemi-group and approximations to achieve truncated scaling
2

---

Published as a conference paper at ICLR 2023
considers the moving band-limits caused by rescaling, proposed by Worrall & Welling (2019) to attain semi-group equivariance. Bekkers (2020) generate the G-CNNs for arbitrary Lie groups by B-spline basis functions, and achieve scale rotation equivariance when using different settings. SiamSE (Sosnovik et al., 2021c) equip the Siamese network with additional built-in scale equivariance. Although [Sosovikk al. (202Tb a) replacing weight sharing scheme with dilated filters can parallel the process, and thus O(1) in terms of time, the overall computational load is still increased concerning the group size. Beyond all these methods, expanding the group to a larger space can increase the overhead computation.
Steerable filters Steerable filters have been used in previous works to achieve equivariance. H-Net (Worrall et al. a (017) and SESN (Sosnovik al. 2020) are examples of methods that utilized steerable filters to achieve equivariance. Specifically, H-Net utilized complex circular harmonics filter bases while SESN used Hermite polynomials. The underlying idea behind these methods is to represent filters of different sizes rotation angles as línear combinations of fixed basis functions. Although this idea is related to our method, achieving rotation scaling equivariance using this approach is difficult due to the lack of suitable basis in image processing. Other approaches have also improved equivariance in different aspects. For example, polar transformer networks (Esteves et al.] 2018) generalized group-equivariance rotation and dilation, while attentive group convolutions (Romero et al. 2020) utilized the attention mechanism to generalize group convolution. Some other techniques, such Shen al. (2020); Jenner & Weiler (2022), proposed using partial differential operators to maintain equivariance. Additionally, Gao et al. (2022) presented roto- translatsioon equivariant CNN, but expanded the filters of the G-CNN the scale dimension with a truncated interval.
Differences between related works and ours. Unlike previous methods that achieve equivariance only in one aspect, our goal is to ensure continuous rotation and scaling equivariance in a single network. Methods that modify the filters used the neural network to achieve equivariance require more learnable parameters and are computationally expensive. In contrast, we propose scalable and steerable filter representation (scalable Fourier-Arge Argand) to modify the convolution operator (SimConv) so that it embodies scale, rotation, and shift equivariance. This approach does not introduce new learnable parameters and allows us to achieve rotation and scale equivariance more efficiently than other methods.
PRELIMINARIES AND NOTATION
This section clarifies the notations about the sim(2) transformation and the convolution that is often mentioned later. The property of equivariant and invariant also formulated explicitly, which is derived from our proposed method.
3.1 SIM(2) TRANSFORMATION
In Fuclidean geometry, two objects can be transformed into each other by similarity transformation they share the same shape. This concept of similarity critical in instance-level computer vision, as objects remain unchanged under scale, translation, and rotation transformations. Consequently, we are motivated investigate the notion of similarity equivariance. To this end, we introduce the Sim(2) group, which is defined by an invertible translation matrix, as follows:
R O
R 8
R O
T
and T s-=s-
E Sim(2) C R
(1)
8
where R =
cas o
E
2222 denotes the rotation matrix. The translation vector is denoted by
sin
toe
t E ?R?, and s E R is the scale factor. By composing rotation (by an angle 0), scaling (by factor s), and translation (by vector t) into single matrix, we obtain a shape-presrrving transformation that belongs to the Sim(2) group, given by T A, Y,R. R., Now let's consider spatial index IR2 on 2D image plane. We can extend this index to a 3D homogeneous coordinate by adding 1 as the third component, that is, [ē, 1] E R Then, for an input signal f. that transforms feature maps f E L2(X) on some space X according the Sim(2) transformation T, written as:
i= [2]]
define a linear transformation LT L2(X) + Lx(X)
[x(/((x= [[](x)>=(TT>>>>
3

---

Published as a conference paper at ICLR 2023
The equation above can be interpreted in the following way: The left-hand side represents linear transformation LT applied to a set of feature maps f. and the right-hand side represents the value of the feature map f evaluated the point T x. Moreover, LAs [ which means that LT can be decomposed into series of shape-preserving transformations. The formula for the transformed feature map becomes
have LTl = L L(AYYIR))H =
f(T-'x) f (RE it)", ,1!")
(3)
This corresponds rotation transformation followed by translation and scaling.
3.2 FORMULATION OF CONVOLUTION
Let's consider a stack of two-dimensional features function f(-) (f(())== R² x x1 RN where N the number of channels. Similarly, filter bank containing M elements in a convolutional layer be formalized d/ [1hk,e)m1 and 2/ IR? x + R. is vector vector-valued output filter. Then, for continuous input with channels, we regard the spatial cross- corosscoorelttion between the input and the continuous
M where each filter with N channels yp {+ 1 k=1=
be written as
filter bank V/ as operator $[-] [ 4p) R.V + R.M, written as follows:
N
1 fa(x (t)dtt
$1[(x) = xx()
Without loss of generality, we can set = = M = 1 and simplify the above formula with the following formula:
① ([f[()) - [f +)())) - l f(x-)) tt)
(5)
To explicate, despite the fact that t is three-dimensional vector, the integration is still double integral, along the first and second dimensions, akin to the conventional convolution. The last
dimension of acts only symbolic index, and serves mere placeholder. This simplified formulation is employed throughout the remainder of the paper to enhance the comprehensibility of the deduction.
3.3 EQUIVARIANCE AND INVARIANCE PROPERTY
Definition 3.1 (Equivariance). An operator & L2(N) + L2(M) is equivariant to the transform LT there exists a predictable transform LT, such that the following equation holds for any x E Id+11
+ $[LT[f][(x) = [r(x[f](xx)
(6)
If T.T = 7.r, we can also say that these operators are commutable. This equivariance property properties for the network.
provides cructurr-preseeve
Definition 3.2 (Invariance). An operator + :L2(N) + L2(M) is invariant to the transform Ler if the following equation holds for any x E gd+1
+ ([L[[f](x)) +(f((x)
This paper aims to design convolution-lik operation that similar in function convolution without lifting the intermediate variable size of the network (as is common in group methods) as well as satisfies the equivariance or invariance property for any similar transform.
4 METHOD
In this section, we explore the feasibility of achieving simultaneous equivariance in rotation, scale, and translation for traditional convolutional neural networks. The underlying intuition behind the method is: first obtain the local scale and orientation of the image. Then, this local information is used to adapt the scale and direction of the filter used for the convolution. Moreover, this (image-dependent) opatiall--varying convolution has efficient implementation.
4

---

Published as a conference paper at ICLR 2023
K times
SimBlock
with S.P.A.
SumBlook
SimBlock
SimBlock
MLP
Iaput
Ourput
stth SeCoue
|/ 8 Pun @PLM((2)
Figure 2: SREN Architecture Overview: The architecture comprises multiple SimBlocks, each utilizing the scalable Fourier-Ae Fourier-Argand filter (Section 4.1 to extract geometry information Mx)x).
The similarity convolution structure (Section 4.2) combines this indicator achieve similarity equivariance. A head layer is added to convert the equivariant output to invariant output (Section 4.3
4.1 SCALABLE FOURIER-ARGAN REPRESENTATION
We propose a representation called the Scalable Fourier-Argand Representation to retrieve local geometric information and use it as covariance indicator in Equation (T2). It is exact and steerable.
This representation can be defined as follows:
Definition 4.1 (Scalable Fourier-Argand representation). Let h be a square-integrable function expressed as h(r, 0) in polar coordinates. Its scalable Fourier-Argand representation defined as the following series form:
Mrr, 0)
Σ Is K2,E
(he.ke exp [(ki0 + k2
))
2x In In Inbia m
Σ Ac K2.EEE
Hky (r,0)
(8)
where Z denoles the set of all integers, and me can be chosen as a constant value. The function has limited support, i.e., 0) 2n bl, and ki, k2 can be truncated for subset of inlegers to approximate h.
To compute the coefficient of each item ,k, we can use the following proposition:
Proposition 4.1. Let hks Kz Z - C as the coefficient of each item. We can compute as follows,
lub 1
2T
2mp
122 In
h(e?,0) exp (-iki0 ik2 Inb/a
exp
pm
d0dp
(9)
Jla 2T
This formula represents the filter composition of elementary feature basis. The part about 0 can be seen as the Fourier series of filters on the Argand plane, inspired by Zhao & Blu (2020). The filter itself is related to the harmonic filter (Worrall et al. 2017) in sense. For the scaling part, the logarithmic method make the scaling of the size a linear shift. We also restrict the support of the function to plane where the radius ranges from b, where -> 0+. In practice, we set my = 1, which acts like window function and ensures that the function vanishes at infinity. This why we call it the Scalable Fourier-Argand Representation. The proof of the proposition can be found in Appendix A
One benefit of our proposed expression is that its basis functions steerable for rotation and scalable for scaling. Specifically, consider a transformation matrix T = A, YoR., which only involves rotation and scaling. Using this matrix, have the following equation:
2T Inr/s
L LT 1,, (r = hh -xp((ik a) ik.
:(s)))rs))))) (rs-1)
In b/a
(10)
2n 1n
= -r 0) 8): exp(-iii ikja - ikz
-
In b/a
This equation holds any [0, 2x) and E (a, b). The following proposition easily verified: Proposition 4.2. For a continuous filter E IR? that can be decomposed by a set of basis, let T = A, Y.R transformation matrix. Then the transformed filter of h, noted Tor [h), can
5

---

Published as a conference paper at ICLR 2023
still be represented by the same basis, with steerable linear combination:
(r)
E
k2
(11)
2T Iu s (exp(-ikie - ikk In b/a )8 m41
Σ Ho, ks
Using this property, we can keep the basis Hk. My fixed and estimate the filter h for different rotations and scales by using different linear combinations. Additionally, to convolve the input signal f with h and its various LT hh]. can pre-convolve the image with the basis Ha, of the original filter. Moreover, to achieve more robust results than traditional convolution, we use the normalized cross-correlation denoted This allows calculate the intermediate variable Ak using the following formula:
HRy, xx)x)= If * HKi,,, (x((X)H Av
(x) H
(12)
where denotes convolution, while y and present the mean and variance of the signal or basis filters. By using these bases, we can determine the optimal orientation and scale through the calculation of the argmax, expressed as follows:
[Ay(x), arg max
Σ k1.k2=-K
Kk (x) Cki,k2 (A, arg Cy.
(13)
(2,7)
TiA
Where Ak (A, = exp( ikiy
il12 2x A ) is a coefficient that only relys on À and 7, Cy,à lab/a
and F are vectors of all possible fki,ks and Ck,, T can be understood the projection of signal for the orientation and scale aspect. These two indicators meet the following properties, Lemma 4.1. Let T Y, as a similarity transformation. Then for a input image f and its distorted version (x),,, for any position following property,
have relationship of this pair of images by
ALTSS = -, T [ = [T +
(14)
This is the condition we achieved and applied Section 4.2 The proof can be found in Appendix
4.2 SIMILARITY CONVOLUTION
We propose SimConv, a new convolution-like operation that serves as an alternative to traditional convolution. Our design aims to meet the following criteria: 1). It should exhibit equivariance properties for rotation, scale, and translation theoretically. 2). It should incorporate learnable parameters and extract image features by "blending" one function with another. 3). It should have computational complexity similar to that of traditional convolution, avoiding the computational disaster problem faced by group convolution when extending the group size. These criteria drive us design the SimCony follows,
Definition 4.2 (Similarity convolution). The similarity convolution between the input signal f and the filter yp is defined as
[f + p((x) (()))st)t +If1(x)
(15)
where (x) a pixel-wise matrix defined as:
My(x) = ( E Sim(2)
(16)
The SimConv operator exhibits convolution-like structure between the feature map and the learnable filter sp. This becomes apparent when substitute the variable with t M (x)t. Moreover, it can be degenerate traditional convolution when replace M, to the identity matrix
for all Furthermore, we define the SimCony as operator Φ(.] [- @ so) RN IR where

---

Published as a conference paper at ICLR 2023
set M N 1 for readability. This operator is, in fact, equivariant for rotation, scaling, and translation, which we can verify below when considering My (x) in Equation (16) and combining with the condition in Equation (35).
M (x) A (AL (TT (x)) T
= (Ay A. (T xx)) R., T
(17)
(Ay(T R(T
M, x)
This leads to the following condition: T M M
.V E R² 1. If we apply a similarity transformation LT. which rotates the signal by a degrees centered at and scales it by s, to the signal ff following Equation (2). we obtain the following transformation of the SimConv output:
+[LT]f](x) = [LT[f] @ (xx) = /, + (zn](())))tt
f R
f(T M
x
(18)
Furthermore, we can deduce the commutator operator of the above as
[T[([s][xx) = [T[[ @ []](xx) = -TT 1// f(x (x(x)t)
-TT//
= =
x(T (x)))
(19)
By replacing the second in Equation 19 and comparing with Equation (18) we have,
+ ([T[ff](x)) [r(+ff](xx)
(20)
This implies that applying a similarity transform to the input signal f and then applying the SimConv equivalent to first applying the SimConv and then the transform, which means that the proposed SimConv satisfies the equivalence property when using the scalable Fourier Argand representation. Moreover, each layer in the network satisfies this property, the transformation can be propagated from the first layer the last layer. This can be expressed using the following formula:
fn (x) = [Tr @ Po +
pn](x) = [Lr[fo @ Po @ @ (x]](()
(21)
There are several ways to convert equivariant features into invariant features. One option is to add an adaptive max pooling layer at the end of the network. Let P[-] a function that takes the maximum response over the entire spatial domain. Since the maximum value is not affected by the position distortion of the feature, the output is invariant. Using Equation (20). we obtain:
be
[[][L()]](() P[Trr [f]]](()) = ([][f](xx
(22)
We can see that the transformation matrix T has no influence on the output, which can be useful in tasks such as classification.
4.3 DISCRETIZATION METHOD
Although the continuous formulation in the previous section is necessary to go, from the intuitive approach (find scale orientation, then filter accordingly). to the efficiently implementable formu- lation (Equation 335 through change of variables in integral, digital images feature maps are usually discrete data aligned on mesh grid. Therefore, in this section, we provide detailed description of the discretization of the integral and its approximation implementation. We rewrite Equation 15 in discrete form as follows:
V f(x+ (x) n
$1f(()) = xl(x)
(23)
TE
where R is the support of and is the number of elements in the set R. For example, in the case of a x 3 convolution, R {(t,, ty. 1)" it.,ty E 0,1,0,17}, We define yt. = x (x)x),, which is generally a fractional location index. We approximate f(y+) using bilinear interpolation, written as
7

---

Published as a conference paper at ICLR 2023
=m Cly mm), where (m1, m,, 1)" E z and G is the bilinear interpolation kernel defined as Gmm, n) g m1, m1)g(m2, n2),, where g(a, b) = max aa bl). Thus, the similarity convolution becomes:
V N1
([f[()) - x](x)
= ΣΣ
G(y,. (m))((((t)
(24)
tER m
With this implementation. we make the similarity convolution differentiable. The gradient of the input is:
aG m) f(m)p(()
(25)
0x
0x
tER m
Since G is differentiable function, and it is non-zero only when m aligns on the grid of yı. it does not require too much computation.
EXPERIMENTS
5.1 CHARACTER RECOGNITION TASK
Dataset. The MNIST-ROT- 12K DATASET (Larochelle et al.) 2007) is commonly used to evaluate rotation-zquivariant algorithms. However, it inadequate to test the algorithms' equivariance both scaling and rotation. To facilitate fair model comparisons. we construct the SRT-MNIST DATASET by modifying the original MNIST dataset in a similar way. Specifically, we pad the images to 56 56 pixels, keep the training set unchanged, and randomly apply rotations, scalings, and translations to each test image within the ranges of [0, 2x), [1, and t +00. This out-of-distiibution setting of the test images provides a sufficient evaluation of the model's generalization ability.
Experimental setup. We adopt ResNet-18 18 He al. (2016) the baseline architecture and replace ev- ery convolutional layer with pro- posed SimConv while retaining the same trainable parameters. The scal- able Fourier- filters are shared across all layers. We use Adam op- timizer (Kingma & Ba 2015) with a weight decay 0.01, initialize the weights with Xavier (Glorot & Ben-
Table 1: Generalization ability test on SRT-MNIST
Methods
Type the test set.
MNIST
R-MNIST
S-MNIST
SRT-MNIST
CNNs
99,46
44.41
73.21
33.56
SO(2)-Conv
99,23
97.18
72.85
70.72
-Conv
99.31
35.23
99.21
39.2
SREN
99.12
96.91
98.48
92.3
SREN+
99.42
98.3
99.28
95.1
gio 2010), and set the learning rate to 0.01, which decays by a factor of 0.1 every 50 epochs. We set the batch size to 128 and stop training after 200 epochs. The model has 11.68M parameters, and the FLOPS for an input image of 56 x 56 pixels is O.
Generalization ability study. We ablate our method's equivariance on rotation and scaling sep- arately and concurrently using SRT-MNIST, where R- R-MNIST and S- MNIST are datasets with only rotation or scaling applied to test images. We compare SREN with SO(2)-Conv and -CCC which have the same structure but lack equivariance partially by setting all T Xy to unitary. Addi- tionally, we include SREN+ with random image rotation and scaling by +:30 degrees and [0.8, 1.2), respectively. From Table I SREN algorithm achieves accuracy rate of over 95% every dataset, whereas the CNNS overfit the original dataset with limited generalization ability.
Equivariance error analysis. We conduct numerical validation of the method's equivariance using the equivariant error. This helps us assess the stability of the equivariance property and identify the key factor that affects its stability. The equivariant error is defined by measuring the normalized distance as follows,
|| [ L [f
+ LLLIIII
Error
(26)
where || || is the Frobenius norm. The formula is the relative percentage error of the two obtained features after the input is first convolved then transformed. and vice versa. We compare the k-th layer feature with the convolutional network. The average error, shown in Figure 3 below 0.01, indicating that network achieves high-quality equivariance. Furthermore, tested specific case
8

---

Published as a conference paper at ICLR 2023
where the image is rotated by 90 degrees, and the equivariance enror is negligible, with value of 1.73 x 10-6. This result demonstrates that our method can achieve extremely accurate equivariance when without discretization issues.
5.2 NATURAL IMAGE CLASSTFICATION
experimental setup. To evalu-
Table 2: Performance and equivariance comparison on the STL-10 dataset.
ate the generaliza- tion ability of our method, we con- duct experiments on the STL-10 dataset Coates itt al.] (201T). The labeled subset is
Methods
R-Equi
S- S-Equi
Conti
ID Accuracy (%)
OOD Accuracy (%)
ResNet-16
X
X
X
82.66 + 0.53
37.63 + 1.95
RDCF
/
X
X
83.66 + 0.57
51.12 +: 4.21
SESN
X
/
V
83.79 + 0.24
47.26 + 0.63
SDCF
X
/
83.83 + 0.41
43.60 -: 0.87
RST-CNN
/
84.08 0.11
58.31 3.62
SREN
V
/
85.25 + 0.61
63.42 4: 2.57
an excellent choice to evaluate how efficient the network can use these limited training samples and how much the network's s generalization ability when the training set is small. We evaluate
the methods using in-distribution testing (ID), which keeps the test dataset unchanged, and dittof-diitrrbuutonOOODD testing, which randomly rotates and scales the dataset. The OOD test measures the ability of method to handle the never-seen inputs. We use a ResNet (He et al. 2016) with 16 layers as the backbone and replace all convolutional layers with our SimConv layers. The network is trained for 1000 epochs with batch size of 128, using Adam as the optimizer. The initial learning rate set to 0.1 and adjusted using cosine annealing schedule during training. Following Zhu et al. (2019), data augmentation without scaling and rotation is also applied. We compare our method with several other methods, including the Rotation Decomposed Convolutional Filters network (RDCF) (Cheng et al. 2019). Scale- lnalv--guivaraantt Steerable Networks (SESN) (Sosnovik) jet al.] 2020). Scale Decomposed Convolutional Filters network (SDCF) (Zhu et al., 2019). and Koto-SSale-Trras Translaionn Equivariant CNNs (Gao al. 2022). We choose ResNet with 16 layers the backbone for all methods to make the model parameters comparable and ensure a fair comparison.
Results & Discussion. Table Salleppreeett our main results compared to recent baselines. The columns R- Equi and S-Equi indicate whether a method achieves equivariance in rotation or scaling, respectively. The column Conti indicates whether a method achieves equivariance at a continuous scale rotation. Our method achieved the highest accuracy among all other approaches, particularly for the out-of-disttibuttin test, demonstrating its superior generalization abil- ity, should be noted that we compare our method with those that only"partially" achieve equivariance, as there lacks like-for-like methods (achieving both rotation and scaling equivariant in the continuous group) to compare. While MacDonald et al. (2022) guarantees equivariance any finit-diimensionll Lie group, its memory efficiency limits its scalability to large networks and comparison with our method. Achieving equivariance efficiently is challenging, being able to achieve this property itself is already another state-of-the-an
vauivariinnn difference diflerent layer
1.5
Eauivariann errar
1.0-
ConvNet
angle
angle 2
0.5
10
9.3
Network depth
Figure 3: Our method exhibits high equiv- ariant quality in a multi-layer network, and achieving an error level of 10-6 when there is no discretization approximation.
6 CONCLUSION
Although there have been numerous studies on how to achieve rotation and scale equivariant, achieving continuous equivariant in rotation and scaling simultaneously is novel, to knowledge. In this paper, we propose to develop scalable steerable filters based on the Fourier-Araand representation and to use the local scale and orientation provided by these filters empower the convolution operator with local scale and rotation equivariant: SimConv. Mathematical and experimental analyses are detailed explain why it works and what extent it can achieve the desired property.

---

Published as a conference paper at ICLR 2023
REFERENCES
Erik J Bekkers. B-spline cnns on lie groups. In International Conference on Learning Representations, 2020. htw.ates//pppirreeeeerrr//aaaaa2
Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. Rotdef: Decomposition of convolutional filters for rotation-equivariant deep networks. In International Conference on Learning Representations 2019 (ICLR'I9) 19). 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 223.22232222 JMLR Workshop and Conference Proceedings, 2011.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 9999.9999. PMLR, 2016.
Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations, 2017. URL /ttLtttt/ppeeeeev//i/
net wtt.:eee//EEoommoo000.....
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homoge- neous spaces. Advances in neural information processing systems, 32, 2019.
Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer
networks. International Conference on Learning Representations, 2018. URL https://
openreview net forum? wtt.uummnn-HHttRRL.....
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. International Conference on Machine Learning, pp. 1165-3176. PMLR, 2020.
Liyao Gao, Guang Lin, and Wei Zhu. Deformation robust roto-scalettranslatin equivaaaaa CNNs. Transactions on Machine Learning Research, 2022. URL. htttpss///ppenreerleew...
Forum? dd=ymmdxs=svsssx c7cD.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural networks. In ICLR, 2022.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally ccale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5000 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 15: 2012.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pp. 480- 2007.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-aware trident networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6054-6063, 2019.
Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer Science & Business Media, 2013.
10

---

Published as a conference paper at ICLR 2023
Lachlan E MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbi- trary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1183-8192, 2022.
David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pp. 8188-8199, PMLR, 2020.
Zhengyang Shen, Lingshen He, Zhouchen Lin, and Jinwen Ma. Pdo-econvs: Partial differential operator based equivariant convolutions, In International Conference Machine Learning, pp. 8697-8706. PMLR, 2020.
Ivan Sosnovik, Michat Szmaja, and Amold Smeulders. Scale-equivariant steerable networks. In International Conference Learning Representations, 2020.
Ivan Sosnovik, Artem Moskalev, and Amold Smeulders. Disco: accurate discrete scale convolutions. arXiv preprint arXiv:2106.0224 02.3333,, 2021a.
Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. How to transform kernels for scale- convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1092-1097, 2021b.
Ivan Sosnovik, Artem Moskalev, and Amold WM Smeulders. Scale equivariance improves siamese tracking. In Proceedings of the JEEF/CVF Winter Conference on Applications of Computer Vision, pp. 2765-2774, 2021c.
Maurice Weiler, Fred Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018.
Daniel E Worrall and Max Welling. Deep scale-spaces: equivariance over scale. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pp. 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 0028-5037, 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Tianle Zhao and Thierry Blu. The fourier-argand representation: An optimal basis of steerable patterns. IEEE Transactions Image Processing, (3963377-677,20222
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 519-528, 2017.
Wei Zhu, Qiang Qiu, Robert Calderbank, Guillermo Sapiro, and Xiuyuan Cheng. Scaling-translation translation- equivariant networks with decomposed convolutional filters. arXiv preprint arXiv: 1909. 193, 2019.

---

Published as a conference paper at ICLR 2023
A PROOF OF SCALABLE FOURIER ARGAND REPRESENTATION
Definition A.1 (Scalable Fourier-Argand representation). Let be square square-integrable function expressed as h(r, 0) in polar coordinates. Its scalable Fourier-Argand representation is defined as the following series form:
2x ( ()) ( Th exp [ikk10 + 122 lnb/a
h(r,
Σ KL
Σ K
Hk 0)
(27)
where Z denotes the set of all integers, and my can be chosen as a constant value. The function has limited support, i.e., (r,0) 0) E [0, (0,27T/ [[m, and k1, k2 be truncated for subset of integers to approximate h. Let lis 2² - C as the coefficient of each item. We can compute it as follows,
In 1
2r
2mP hie", exp (-iki ikg In b/a
hkik22 82
pm
d0dp
(28)
In J
2T
Proof.
We begin by substituting the variable in Equation 27 with exponential term, eP, where p == lnr.
Me,,0)= = Σ
NELL exp(pm),) exp(i(kz + k/ Inb Ina"
2mp
(29)
Ass A0E7
Additionally, we introduce the quantity fb and define it accordingly,
1 Inb 1 2m
/
exp((xxpikk) i il. 2mp pm.)dddp
As In bnb/a Jina
2T
Jo
b/ba
(30)
We then substitute h(c", 0) in Equation Equation
with the scalable Fourier-Argand representation in 30
1 In b/a
2m h(ee, .00 exp(
2mp
k2
ik2
In bnb/a pmm)dddp
Ina
2T
2m
Σ
)xp(pro,,)xx((((h)) htit exp(pmas,) exp(i(l10 + 12
2mp In b/a"
In
bnb/a
2T J
Iua
(s 1222
2mp exp( 1 ik
pmm)dddp
In b/a
1
Inb
2m
2TP Σ he exp(pmi,) exp((ti0) exp(ils MEEZ
In
b/a
Ju
In b/a
2E
2mp ik
exp( iky dl exp(
pm)ddp
In b/a
lnb
2n
exxp(itt exp(i(ti ko k0) exp(itt 2mp In b/b/a
ht d2 2
exp(pm,,)
In
inb/a
Jna
E2
2xp
exp( ikz
pm))dp
In b/ba
d
2x
hentz
exp(i(t1
kz )
Inb/a
2TT
J
Iaa
2mp
2mp ddp
exp(itz
expp - ik3
In inb/a
In b/a
(31)
12

---

Published as a conference paper at ICLR 2023
2
2mp
12 0411 h
ki] expp(p2
exp( ikk
ddp
Inb/a
Jnn
In b/b
In b/a
LEZHEZ
1 lab
2 )dp In lbba
do exp(i(t2 1z)
In b/a
Ina
22E2
Ind
2 Inb/a
(32)
hks 12
Inb/a Ina
exp(i(t2 - k2)
)dp
22
12022 N22l
E2
The validity of the last equation is demonstrated through two cases: first, when &2 k22
lub
2 b/aa
)dp /
In b/a Ina exp(i(l2 k2)
exp(i0)dr =
(33)
In b/a
and second, when 1o + li2.
1
2T
1 lnb
exp(in 2TP
exp(i(l2 - k2)
)dp
dd
0
(34)
In bb/a Ina Finally, the assertion in Equation
In b/b
In b/a Ina
In b/a
31
is proven by the aforementioned arguments.
B PROOF OF DISTORTION PROPERTY
= A, as similarity transformation. Then for input image f and its distorted version [x) for any position we can have relationship of this pair of images by following property,
Lemma B.1. =
AxT(s(x) = =A,(T (x) = (Ty(T- xx + a
(35)
Proof. Consider the original and distorted image f(x) and Lr(f((x), respectively, and use the scalable Fourier Argand basis. We obtain:
[x, arg (2,7)
Σ
Lr Lrl ck., (o
K
2x In
arg
(A-1)
Σ
[LT[[] Hx),)y((x) expl-ikir ikk
In b/ba
) -m
ln Ro K
(36)
Substituting x with T-1x, we have:
[A L'Lin
2x In X
(37)
Σ
[f * L (H (I(T)x)) exp((ikkr ik
arg max (A,7)
In b/
Besides, from Equation 10 we have:
2T In In b/a
L Hy, Aa (r, () = Ha, (r, (exp(-iks - iik
(38)
13

---

Published as a conference paper at ICLR 2023
Hence, can rewrite the previous equation as:
[Arxis (T)
(1) (T)
K
2x
= arg max (A,7)
Σ
If Lo [H [I 1x). copp(-iki - iks
) m
In b/a
kn k2=KK
arg max (X,7)
Σ
I Hkg,
(explitiaa ik2
2x In mey 1(T
In b/a
kn 12=K
2T In X In Inb/a
(39)
- ik1
}mm
K
2T In (n(ss))
Σ
[f HAT (T x exp(-iik(9 - -a) - -ik2
In b/a )(1/5)
arg max (7,7)
K
K
2x In(A)
arg max (As,++a)
Σ
If HA d( -expp--ikkk ikk
In b/a ( (A)
Finally, we can conclude that,
ALTI1 (x) -A
(40)
T (x) T (T- + a
C DETAILS, ANALYSIS, AND VISUALIZATION
C.I ADDITIONAL STUDIES
Stability We first quantify the deformation stability, and seek to answer in what degree of distortion can the method persist the equivariant (or the generalization ability) compared with original convolu- tion. To achieve this, evalue method the test dataset that rotated and scaled entirely by certain amount. The MNIST dataset is used in this experiment. The results of this experiment are presented in Figure 4 which shows the accuracy decay with different settings. From the figure, we observe that when the image is rotated significantly beyond what was seen during training, CNNs ex- perience large drop in accuracy, whereas our method maintains consistently excellent performance. For scale changing tests, our method retains relatively good capability over a wide range of scale changes. However, for extremely large scale changes, our method experiences performance drop due to the limited filter spot and sampling approximation.
(e) The senkc inpiverient compatration
(a) The cntation enaivaiion asinnd compar
109
Zeeunay (56)
10
89
Con SREN
ConoNer SNEN
1.4
1.2
1.6
3.6
25
50
75
196
125
158
375
The scaling factoe (simes)
The rotation angio (doggo))
Figure 4: Equivariance Stability: Our network demonstrates high equivariance stability when the entire test set undergoes translation and rotation by certain scale or angle. In contrast, ConvNet's performance drops significantly under such transformations.
C.2 FEATURE VISUALIZATION
In addition to numerical experiments. we employ feature visualization to intuitively verify the achieved equivariance of our network. The visualization results presented in Figure 5 This
14

---

Published as a conference paper at ICLR 2023
Image
Rotated Cauage
outpute
distected and staling
staling.
plsto CNNS
Congereuted view
Figure 5: Feature Visualization: We perform a feature visualization of both Convolutional Neu- ral Networks (CNNs) and our Scale-equivariant Residual Equilibrium Network (SREN) using compensating view. The visualization allows us to observe how the features vary with different transformations of the object. Our results indicate that the features extracted by our method remain stable under transformations, whereas those of CNNs vary.
Table 3: Methods comparation on STL-10 dataset, all methods use the WideResNet as the backbone.
Method
WRN
SiCNN
SI-ConvNet
DSS
SS-CNN
SESN
DISCO
SREN
11.48
11.62
12.48
11.28
25.47
8.51
8.07
8,23
approach examines whether the hidden features the network exhibit visual equivariance with respect rotation and scaling. The parameters randomly initialized for both the CNN and SREN models. Through compensation view, we can clearly observe that our network successfully achieves equivariance in both rotation and scaling.
C.3 ARCHITECTURE AND PIPELINE
The algorithmic framework is illustrated in Figure Our method can be extended to multi-layer network structure. Assuming that the network comprises blocks (which we named as SimBlock), each of which contains several convolutional layers. Let fa denote the input signal of the i-th block. We first compute the scalable Fourier Argand feature, which is introduced Section 4.1 to obtain spatial-wise matrix M,, (x). Then, all the SimConv layers, proposed in Section SimBlock share the same scalable Fourier Argand features. Assuming there are N SimConv layers in each SimBlock, according to the property of Equation (2T), the output feature fi+1 of the block is guaranteed to be an equivariant feature corresponding to the input image. In the classification task, the invariance property is desired. Therefore, perform pooling for each channel the last layer, followed by the MLP layer. Finally, the output is obtained and guaranteed to be invariant for any similar transform. The whole process presented in Algorithm
within
C.4 STL-10 DATASET COMPARATION
This section presents comparison of our method with other related approaches using the same backbone, namely WideResNet with 16 layers and a widen factor 8. Our network was trained for 1000 epochs using SGD the optimizer, with batch size of 128. The initial learning rate was set to 0.1 and decreased by factor of 0.2 after 300 epochs, while the drop rate probability was set to 0.3. We also utilized data augmentation techniques, described in Sosnovik et al. (002a).
15

---

Published as a conference paper at ICLR 2023
Algorithm 1: Pipeline of our SREN Algorithm
Input: Batch of data {fi) Output: predicted output value
for 1 K levels do
Initialize filter ht calculate its basis Hky Kag
Take fi as input;
Get far Jkz t f Hkz
Get f, As, 4- arg maxy, Cy,aF;;
Calculate M f- Ai
for 1 M layers in specific level do
Apply My to SimConv;
fi,m+1
Pi,m.
Jit1 + S1,M
Get the output with head layer: v MLP(())
Table 3 presents our main results compared to recent baseline methods, such as SiCNN Kanazawa
al. 2014), DSS (Worrall & Welling 2019), SESN (Sosnovik al.. 2020), and DISCO Sosnovik et al. 2a2 Our method achieved competitive results that are close to statc-of-the-at performance. Additionally, we would like to emphasize that our method's computational cost and parameter count are similar to those of the classic convolutional neural network with the same backbone. This is because the two quantities A and I can be shared across different filters, resulting compared to the abundance of the convolutional (or our SimConv) layer. Moreover, the complexity of our SimConv operator is similar to that of the convolutional layer. For comparison, as reported in Sosnovik et al. (2021a). the computational cost of the DISCO method takes more than five times longer and SESN takes more than 16.5 times longer than classic CNNs.
small cost
16


Reviews:
[{"review_id": "uHp0zo0jR3I", "reviewer": "Reviewer_MrDN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors highlight the fact that CNNs, which are capable of tackling translation, scale and rotation transformations, would be useful for computer vision, but are less studied than less general counterparts. Then the authors propose scalable Fourier-Argand representation as the key for building such models. In their experiments they demonstrate that the proposed SREN model is more accurate than the previous methods.", "summary_of_the_review": "The paper contains a novel idea and highlights an important issue: combining translation, scale and rotation in one networks is still challenging from the implementation point of view.", "strength_and_weaknesses": "The paper is well-written. Although, I am going to propose several adjustments, the paper is already clear enough to be understood. The paper has a good structure which makes it easy to follow the  idea of the authors. Additionally, the mathematical language is high level which allows to stick to very accurate formulations.\n\nThe idea discussed in the paper is novel. And it is interesting to read it. Although it may be rough, and may be at an early stage of its development, it made me start thinking about the area of the field, the problem and the solutions, which authors highlighted. It is a big plus.\n\nThe main weakness of the paper is that the novelty sounds very important, it seems to be of a large scale, while the experiments just slightly demonstrate its advantage over the previous methods.\n\n**Issues, which affect my rating**\n1. If I understood correctly, the rotation part of the proposed filters is very close to the idea of Harmonic Nets by Worral et al. [1] and E(2)-steerable CNNs with irreducible representations by Cesa & Weiler [2]. If so, then I think that a more accurate comparison to [1, 2] should be presented. If not, please add some remarks to the sections 4.1 to highlight the difference.\n2. The authors mention several times that the method is not based on group theory. I think such a distinction is misleading, artificial and somewhat incorrect mathematically. Moreover, the exact roots or each cited paper go far beyond than just \"group theory\". If it is not based on group theory, could you demonstrate how this fact may be useful to a reader. For example, can it be generalized to a non-group transformation?\n3. Introduction, the \"To achieve this property,\" paragraph. I find several phrases here a bit misleading and are of less respect to the previous approaches. I suppose it is just wording. For instance, the phrase \"these “copy-paste” approach with group theory\" may give a reader an incorrect understanding of the previous appoaches. What the authors call here as \"copy-paste\" is better to be referred to as \"weight-sharing\". It is a) more convenient and b) less vulgar. Finally, for some methods, such as [1, 2] it is not necessary to reuse filters to obtain their rotated versions.\n4. One of the arguments for the proposed method is that it allows for equivariant models with lower complexity than quadratic. In Worrall & Welling [3] and Sosnovik et al. [4, 5] it is demonstrated that scaling can be implemented as dilation (integer and fractional). Moreover, it is possible to implement it by utilizing multiple cores of GPUs to make it O(1) in terms of time, which is also discussed in the papers, as the weight sharing scheme is replaced with dilation in these papers. I think a more accurate comparison of computational complexities would help.\n5. I suppose that the results in Table 5 are copied from the paper [6]. The main issue with equivariant models when it comes to comparing a new model to the previous approaches is the hyperparameter tuning. In the original papers the authors spent some time to perform such a process. For example in [2, 4, 5, 7] the authors train and test by using exactly the same protocol and their reported results are more accurate than what is present in Table 2. One of the main reasons is because they used a WideResNet. If it is possible to additionally compare a WideResNet with the proposed blocks to the previous models. It would significantly increase the contribution of section 5.2\n\n**Issues, which do not affect my rating**\n1. The paper contains typos and writing issues. For instance, the word \"trannels\" (p1) seems rather misleading. The word \"convolute\" (p5) is incorrectly used (I suppose it must be \"convolve\").\n2. In Eq. 1 it makes sense to write the transformation as a 3x3 Matrix from the very start by using the block notation. It will improve the clarity of the equation. Because then, you introduce the 3rd coordinate to $[x_1, x_2, w]$. It also makes sense to write what the $w$ is.\n3. A more detailed illustration of how the equivariance of the SimBlock works would help. \n4. Very bottom, page 2. I am not sure if \"Worrall and Welling\" is the right reference for \"Scale-space theory\". I suppose the placing of the reference is just incorrect.\n\n- [1] Worrall D. E. et al. Harmonic networks: Deep translation and rotation equivariance. CVPR – 2017\n- [2] Weiler M., Cesa G. General E(2)-equivariant steerable CNNs. NeurIPS - 2019. \n- [3] Worrall D., Welling M. Deep scale-spaces: Equivariance over scale. NeurIPS – 2019.\n- [4] Sosnovik I., Moskalev A., Smeulders A. Disco: accurate discrete scale convolutions. BMVC - 2021\n- [5] Sosnovik I., Moskalev A., Smeulders A. How to Transform Kernels for Scale-Convolutions. CVPR – 2021, VIPriors Workshop\n- [6] Gao L., Lin G., Zhu W. Deformation robust roto-scale-translation equivariant cnns. TMLR - 2022\n- [7] Sosnovik I., Szmaja M., Smeulders A. Scale-equivariant steerable networks. ICLR 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t7E7TQGIGJF", "reviewer": "Reviewer_Ke2s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a new convolution operation for neural networks that is simultaneously equivariant to rotations, translations and scaling. The crux of the method is representing the images and filters using in what the authors call the scalable Fourier-Argand representation. With this operation defined, equivariance can be shown theoretically for continuous signals, which also holds empirically. Implementation of the idea via discretization is also discussed.  Experiments on multiple benchmark datasets clearly show that the proposed ideas are better at classification than other equivariant methods, especially when unknown scale, rotation and translations are applied simultaneously at the input. ", "summary_of_the_review": "I think the novel scalable Fourier-Argand representation and how it enables equivariance is very interesting. Experimental results provide validation. There are some weaknesses which I have listed above which I hope the authors can address. Overall, I recommend acceptance.\n\n\nUPDATE AFTER AUTHOR RESPONSE:\n\nI thank the authors for their response. I think the contributions are clearer now and the authors should have this discussion in the main paper. Having also read the other reviews, I maintain my original score of accepting the paper.", "strength_and_weaknesses": "Strengths:\n\n1. The ideas in the paper are novel. The scalable Fourier-Argand representation is clever and allows for enabling the required equivariance, given that the convolution operation is defined as shown. \n\n2. Equivariance has also been experimentally verified. \n\n3. Experiments with SRT-MNIST clearly show that the proposed method outperforms methods that do not take into the right equivariance into account.\n\nWeaknesses:\n\n1. I am confused by the authors stating at multiple points that they are describing the first method that achieves rotation and scaling equivariance simultaneously. But the authors themselves describe several works in the related work section that do it. The most recent example I know is from CVPR 2022: Enabling Equivariance for Arbitrary Lie Groups which shows a framework for equivariance to any finite-dimensional Lie group: https://openaccess.thecvf.com/content/CVPR2022/papers/MacDonald_Enabling_Equivariance_for_Arbitrary_Lie_Groups_CVPR_2022_paper.pdf\n\nThe authors should definitely make this point clearer in the paper and discuss why these could not be included as baselines for their experiments. I also don't see what the authors mean by their method not being based on group theory.\n\n2. There are some minor very minor issues in the formulation section 3.2. In many places, the equations use dimension $d$, while the text says two-dimensions and that the convolution integral is a double integral.\n\n 3. While the experiment with STL-10 is appreciated, it would be nice if the authors can have another experiment to clearly show that it is because of the joint scale and rotation equivariance that the experiments improve i.e. are the OOD samples OOD because of unseen scale and rotation transformations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LFCLIirhqI", "reviewer": "Reviewer_TG34", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new way to construct convolution-like neural networks that are equivariant to continuous translation, rotation, and scaling without relying on group theory. More specifically, they use the local scale/orientation of the input signal to adapt the scale/orientation of the filters (with the proposed Fourier-Argand representation) and implement spatially-varying convolutions. The proposed method does not introduce any significant computational overhead. They empirically verify the equivariance and generalization ability of their proposed models on scaled and rotated versions of MNIST and STL-20 datasets.", "summary_of_the_review": "I recommend acceptance of this paper based on the following:\nThis paper provides a new way to construct convolution-like neural networks that is equivariant to translation/rotation/scaling, with appealing computational efficiency, and can potentially be generalized to other transformations without relying on the group theory framework. The technical parts are solid and the empirical study is sufficient. Even though I have some technical concerns outlined in the question section, they do not stop this paper from being valuable research work.", "strength_and_weaknesses": "**Strength:**\n\n1. The proposed construction of equivariant networks does not rely on group theory, so it can potentially generalize to transformations that are hard to describe with transformation groups. Furthermore, their implementation is efficient and does not have computational overhead for larger groups.\n2. The technical parts are solid, and their empirical study sufficiently supports their claim.\n\n**Questions (potential weaknesses):**\n\n1. The modulation of filters depends on the optimal scale/orientation (computed in equation 13) which has the form of argmax function. I think it can lead to computational instability because: Firstly there might be multiple maximum values, how can you choose under this scenario? Secondly, even though this extreme case does not happen in practice, there may be cases where a little bit of noise can lead to dramatic change, e.g. orientation $0^{\\circ}$ is only slightly better than $180^{\\circ}$, a small amount of noise could lead to the filters rotate by $180^{\\circ}$. How are you planning to fix this issue?\n2. Still on equation 13, it seems to me that the scale/orientation of the filters depends on the input signal over the entire domain. That is to say, to compute local features, a change far away could change the orientation/scale of the filters. Would that break the locality of the convolutional layers? \n\nHowever, even with all the potential drawbacks mentioned above, I still think this is an interesting work that can inspire valuable research.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8l_yXsv_TT", "reviewer": "Reviewer_RpmD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to generate scale-rotation equivariant feature maps in a novel cnn architecture. This is achieved by constraining the filters to be steerable with respect to a scale-rotation equivariant basis. The authors define this basis to be image dependent and validate the approach on STL-10 and variants of MNIST.\n", "summary_of_the_review": "I think the proposed work is interesting and of value to the community; however, there are several typos which should be resolved.\n", "strength_and_weaknesses": "*Strengths*: The authors address the challenging and important problem of designing cnns to produce representations equivariant to larger transformation groups. \n\n*Weaknesses:*\n- I found the paper somewhat difficult to read:\n  - I believe the paper would benefit from proofreading by an editor.\n  - Some additional language in the appendices could make the proofs easier to follow\n- I think it is incorrect to say that previous methods can only achieve equivariance to rotation or scaling; the authors themselves cite Esteves et al. 2018 as an example\n\n*Questions:*\n- Does the size of the filters impact performance? I ask since (I imagine) the size of the filter determines the scale range.\n- I think there may be a typo in Appendix A, I don’t see 4→5 in equation 33, specifically, what happened to exp(\\rho m_{m_{t_1}})? Is it possible it should cancel with exp(-\\rho m) in 5→6?\n- I don’t see 3→4 in Appendix B, should the last quantity be \\lambda^{-m}?\n- What is meant by a “covariance indicator” (Sec 4.1 para 1)?\n- The last step in (19) seems inconsistent with (17) (i.e., pre- vs. post- multiplication with T^{-1}), am I missing something here? Is this because the group is Abelian?\n- Is P \\circ L_T = P proved somewhere?\n- Are \\Lambda_f (x), and \\Gamma_f (x) computed for every layer at every iteration? Have the authors compared the computational costs of the proposed method to the baseline methods?\n\n*Possible typos:*\n- “Specifically, We” → Specifically, we\n- ““equivariance in rotation, scale” → equivariant in rotation, scale\n- ““if we want to convolute…” → if we want to convolve\n- ““We can pre-convolve the filter” → we can pre-convolve the image(?)\n- ““implementable formulation (Equation (41))” → implementable formulation (Equation (14))\n- ““following summarized condition: T” → following summarized condition: T^{-1}\n- ““the communicated operator” → the commutator operator\n- ““satisfies the equivalent property” → satisfies the equivariance property\n- “Eq 23: “f (x + Mf (x)” → f (x + M_f (x)t)\n- ““y_t = x + M_f (x” →y_t = x + M_f (x)t\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster