Truly Scale-Equivariant Deep Nets with Fourier Layers
Authors: Md Ashiqur Rahman
Raymond A. Yeh
Venue: NeurIPS 2023 poster (2023)
Keywords: Scale Equivariance
Fourier Neural Network

Abstract: In computer vision, models must be able to adapt to changes in image resolution to effectively carry out tasks such as image segmentation; This is known as scale-equivariance. Recent works have made progress in developing scale-equivariant convolutional neural networks, e.g., through weight-sharing and kernel resizing. However, these networks are not truly scale-equivariant in practice. Specifically, they do not consider anti-aliasing as they formulate the down-scaling operation in the continuous domain. To address this shortcoming, we directly formulate down-scaling in the discrete domain with consideration of anti-aliasing. We then propose a novel architecture based on Fourier layers to achieve truly scale-equivariant deep nets, i.e., absolute zero equivariance-error. Following prior works, we test this model on MNIST-scale and STL-10 datasets. Our proposed model achieves competitive classification performance while maintaining zero equivariance-error.

Truly Scale-Equivariant Deep Nets with Fourier Layers
Md Ashiqur Rahman Raymond A. Yeh Department of Computer Science, Purdue University (rahman79, rayyeh] (otnnuepuuduecccmm edu
Abstract
In computer vision, models must be able to adapt to changes in image resolution to effectively earry out tasks such image segmentation; This is known as scale- equivariance. Recent works have made progress in developing scale-equivariant convolutional neural networks, e.g., through weight-sharing and kernel resizing. However, these networks are not truly scale-equivariant in practice. Specifically, they do not consider anti-aliasing as they formulate the down-scaling operation the continuous domain. To address this shortcoming, we directly formulate down- scaling in the discrete domain with consideration of anti-aliasing. We then propose novel architecture based on Fourier layers to achieve truly scale-cquivariant deep nets, i.c., absolute zero equivariance-error Following prior works, we test this model MNIST-scan and STL-10 10 datasets. Our proposed model achieves competitive classification performance while maintaining zero equivariance-erros
Introduction
Consider the task of image classification; an object in the image is scaled (resized), then its corresponding object label should remain the same, i.e., scale- ccale-invariant. Similarly, for semantic segmentation, if an object is scaled, then its corresponding mask should also be scaled accordingly, i.e., scale-equivariant. Similarly, for illustration. These invariant and equívariant properties are important to many computer vision tasks due the nature of images. A photo of the same scenery be taken from different distances, and objects the scenes may come in different sizes. Developing representations that effectively capture this multi-resolution aspect of images has been long-standing quest [1, 9, 11, 17, 53].
would expect the features extracted to be scale-equivariant;
Fig. 1
Recently, there has been a line of work on developing scale-equivariant convolutions networks [8, 13, 41, 42, 46] more effectively learn multi resolutionll features. At high level, these works achieve scule-equivariant convolution layers through weight-sharing and kernel resizing, i.e., use the "same" " but resized kernel across all scales [5]. The innovation of these works is how to properly resize the kernel. For example, Bekkers [2] and Sosnovik et al. [41] formulate kernel resizing as a continuous operation and then discretize the kernel when implemented in practice. However, this discretization leads to non-negligible equivariance error. On the other hand, Worrall and Welling [46) and Sosnovik al. [42] directly formulate kernel resizing in the discrete domain, e.g., using dilation solving for the best kernel given fixed scale set, and achieve low equivariance-error.
Despite these successes, we point out that the aforementioned works not truly scale-equivariant in practice. Specifically, these works are derived using a continuous domain down-scaling operation, i..,, there is need to consider anti-aliasing. However, when performing down-scaling space, the Nyquist theorem [23, 30] tells us that an anti-alias filter is necessary to avoid high-freeg frequency content to alias into lower frequencies. The canonical example of aliasing is the "wagon- -aggel effect", where a wheel in a video appears to be rotating slower or even in reverse from its true rotation. To
discrete
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

5
5
Regular
Ours (Scale- Equivariant)
Ideal Down- sampling
Ideal Down- sampling
Ideal sampling
Ideal sampling
Regular CNN
5
Ours (Scale-
Equivariant)
(a) Illustration that regular CNNs not -cule-equivariant. (b) Illustration that model caalee-quivariit.
Figure Comparison scale-cquivariance on CNN vs. our model. For a regular CNN, the features extracted from the corresponding high/low image resolution look very different. On the other hand, downsampling the high-res feature is guaranteed to achieve the same feature obtained from the low-resolution image.
address this gap from prior work, consider the down-scaling operation directly in the discrete domain, taking he anti-aliasing into account.
In this work, we formulate down-scaling as the ideal downsampling from signal processing [23]. We then propose family of deep nets that are truly saaeeequuvariint based on this ideal downsampling. This involves rethinking all the components in the deep net, including convolution layers, non- linearities, and pooling layers.
With the developed deep net, we focus on the task of image classification. We further point out that truly scale-invariant classifiers are not desirable. A truly scale-invariant model's 's performance is limited by the lowest-resolution image. Instead, the more desirable property is that high-resolution image should achieve better performance than its corresponding low-resolution image. This motivated us to design a classifier architecture suitable for this property.
Following prior works, conduct experiments the MNIST-scale [40] and STL [4] dataset. By design, our method achieves zeros scale equivariance-era cerror both in theory and practice. In terms of accuracy, compare to recent scale-equivarian CNNs. We found approach to be competitive in classification accuracy and exhibit better data efficiency low-resource settings.
Our contributions are as follows:
We formulate down-scaling in the discrete domain with considerations anti-aliasing.
We propose family of deep nets that truly caale-qquivariant by designing novel scale-equivariant modules based on Fourier layers.
We conduct extensive experiments validating the proposed approach. On MNIST and STL datasets, the proposed model achieves absolute zero end-to-end scale-equivariance error while maintain- ing competitive classification accuracy.
Related Work
Scale-equivariance and invariance. The notation of scale-equivariance is deeply rooted in image processing and computer vision. For example, classic band-designed scale-invariant features such as SIFT [21, 22] have made tremendous contributions to the field of computer vision. Earlier works propose to use an image or spatial pyramid to capture the multi-rcsolution aspect of an image [1, 9, 17] by extracting features at several scales in efficient manner.
More recently, there have been interests in developing scale- equivariant CNN [2, 8, 13, 41, 42, 46, 54]. Based on Group-Con Conv [5], these works achieve scale- equee-equvariancce convolution layers through weight- sharing and kernel resizing. Different from these works, we consider the down-scaling in the discrete domain formulated as ideal downsampling from signal processing. We then develop modules that are truly eqalveequivariant enable a deep net that achieves zero equivariance-erron measured from end to end. Finally, note that there is rich literature of equivariant deep nets [3, 5, 33, 36-38, 44, 45, 50]

---

with numerous applications applied to various domains, e.g., sets [10, 26, 32. 34, 58,1,, 51], graphs [6, 15, 19, 20, 25, 28, 39, 49], etc. Moreover, several recent studies have also identified and tackled the issue of aliasing generated from the pooling layer to attain finer translation equivariance [35, 47, 52] and image generation [14].
Fourier transform in neural networks. Fourier transforms have been previously used in deep learning. For example, Mathieu et [27] proposes to use Fast Fourier Transform (FFT) to speed up CNN training. Fourier transform has also been used to develop network architectures, including various convolutional neural networks operating in the Fourier space [16, 31]. Recently, Fourier layers are capable of handling inputs of varying resolution, which have been employed in neural operators, facilitating applications in partial differential equations and in state space models [18, 29]. Fourier convolutions have also found success in low-level image processing tasks, e.g.. inpainting [43], deblurring [24]. Different from these works, we focus on developing truly scale-equivariant deep nets and leverage Fourier layers to achieve this goal.
3 Preliminaries
We briefly introduce and review the definition of Fourier transform, ideal downsampling, and scale- equivariance. For readability, we use ID data to define these concepts. These ideas extended to 2D data with multiple channels when implemented in practice.
Discrete Fourier Transform (DFT), Given an input vector R.N, we consider F: F:R.. - C.N be the discrete Fourier Transform (DFT) which has the form
X F(x) such that X(K)
Σ x x kp
(1)
where j denotes the unit imaginary number, i.e., 1 -1. The index in Eq. (1) is commonly within the domain of [0, .j. Note that as Eq. (1) is N N-periodic, for readability, we will use k from N-1 where k 0 corresponds the lowest frequency.
The corresponding inverse DFT (IDFT) F-1 CN + R defined as
N-1
x = F (X) such that x[n] = Σ Xikle'
k=
By the convolution property of DFT, the circular convolution between x and a kernel k E R& can be represented the element-wist wise multiplication in the Fourier domain, iee.
F(x + k) F(x) o F(k) = X K,
where @ denotes the circular convolution and denotes element-wise multiplication. Unless explicitly mentioned, we will represent the input vector with lowercase letters (c.g., x) and its corresponding DFT with uppercase letters (e.g.. X).
Down-scaling operation. To reduce the scale (or resolution) of a signal x E RRN, one could perform subsampling Subę g by factor of R
(ubb(()(m) x[Rnn..
However, naively subsampling leads aliasing. Hence, anti-aliasing is performed in a multi-rate system. In signal processing, the analysis commonly uses the ideal anti-aliasing filter h, which zeros out all the fighffreueenc content, i.e., its DFT H A F(h) is defined as:
N
H k if | S
and 0 otherwise.
(5)
2R
See Fig. 2a for illustration of the ideal anti-aliasing filter.
In this work, we define the overall down-scaling operation to be the ideal downsampling DR by factor of R, which performs anti-aliasing followed by a subsampling operation:
D Dx(x) e SubR (h @ x) VR < N,
(6)

---

X
H
H X
-3 - -1
0 1
Frequency
-3 - -1 0
Frequency
-3 -2 -1 0
Frequency
(a) Illustration of ideal anti-aliasing (low pass-pass) filter.
(b) Illustration of Claim 1.
Figure (a), we illustrate an idcal low-pass filter showing that it zeros out the high frequencies. In (b), we illustrate the structure deseribed in Claim for linear G. The gray regions correspond to the value being zero.
where their DFT related by
N/2R).
(7)
Scale-equivariance.
With the down-scaling operation defined, a deep net (IR R2, {IR RR2 RN) scale-equivariant if:
+ RN)
g 8(Dx(x)) = Dx(g(x)) Vx E RR... J and R < dim(x),
(8)
where {R² R2 ,.... RN) represents the space of input/output signals different scales. In this paper, we are interested in designing a family of deep nets that satisfies the equality in Eq. (8). Scale-invariance can be defined in a similar manner as
g g(Dx(x)) = g(x) Vx {R', R2,
JN and < dim(x).
Fourier layer. Given multi-channel input vector, x E &CNN and kernel k E CC XN where Cin/out is the number of input/output channels, the circular convolution layer is defined
F((x k))[c]
Σ x Xxc KidokKl,
(10)
where X and denotes the DFT and k applied independently for each channel.
4 Approach
Our goal is to design truly scale-equivariant deep nets. To accomplish this goal, propose scale- equivariant versions of CNN modules, including, the convolution layer, non- linearities, and pooling layers. In Sec. 4.1, we detail the operation for each of the proposed modules. In Sec. 4.2, demonstrate how to build a classifier that is suitable for image classification with scale- equivaraui features. We now explain our overarching design principle for the scale-equivariant modules.
From a frequency perspective, as reviewed in Eq. (6), the ideal downsampling operation results the loss of higher frequency terms of the signal. In other words, if feature's 's frequency terms depend on any higher frequency terms the input, then it is not scale-equivariant, as the information will be lost after downsampling. We now formally state this observation in Claim 1.
Claim 1.
Let g denote deep net such that y g(x). If this deep net g can be equivalently represented set of functions G₁. C2k+1 - C such that
Ylk] = -
k)) Vk
(11)
then g is scale- equeveeauiaaiann defined in Eq. (8). In other words. an output's 's frequency terms
only have dependencies on the terms in X that are even lower in frequencies. We illustrate this structure with linear function in Fig. 2b.

---

Proof. We denote the deep net's input and output as and y with corresponding DFT X and Y. We denote the deep net's down-scaled input and output as x' [x) and y' (xx)) with corresponding DFT X' and Y'. Now assume that R + R Vn E . .N is deep net that satisfies Claim then
N
Y| = - Ak & R
(12)
-x((X' k]) ('K Following the property of Dn in Eq. (7)
(13)
Therefore, Yk: S H Y(M) = k.. By the definition of ideal downsampling Y Dn YY
concluding that (PR((x)) ( (x)), i.e.. g is scale-equivariant.
For ease of understanding, here assume that the deep net's input and output are of the same size. A version with a more relaxed assumption is provided in Appendix Sec. Al.
4.1 Scale Equivariant Fourier Networks.
We now describe the proposed modules and show that they are truly scale-equivariant
Spatially local Fourier layer. For computer vision, learning local features is crucial. The Fourier layer in Eq. (10) is global in nature. To efficiently learn local features, propose localized Fourier layer where we constrain the degree of freedom in the kernel K such that the respective spatial kernel k is spatially localized.
Let E Id and k' R' to be and I dimensional kernel such that k[i] = k [i] if < otherwise 0, i.e. k is spatially local to have receptive field of size l. We denote K and K' be the DFT of the kernel k and k' respectively. We claim that K can be written
L- e
K
pl
Σ m==
K
[m
2)))(B-7))
(14)
d
From Eq. (14), instead of modeling all the degrees of freedom in K, will directly parameterize K to enforce the learned kernel to be localized spatially. We defer the proof to the Appendix Sec. A2.
Claim The spatially local Fourier layer is scale-equivariant
Proof. The kernel k has a corresponding DFT K. As reviewed, a circular convolution between k and input can be expressed
Y K[K] O X[K] Vk.
(15)
Observe that X [k] is subset of X k: k)), iee., Claim is satisfied.
Scale-equivariant non-linearity (o.)) Element- wise non-linearities, e.g., ReLU, the spatial domain generally not scale-equivariant under the ideal downsampling operation DR. While applying element-wise non-linearity in the frequency domain is scale-equivariant, this strategy empirically leads to degraded performance classification tasks. To address this, propose scale-cquivariant non-lincarity ag in the spatial domain.
Given a non-linearity e.g., ReLU, we construct corresponding scale-equivariant version that satisfies Claim 1. Let E R. and y N to denote the input and output of a.. We define ecale-equivariant non- non-linearity (x(x)= ---(--() where Y takes the following form:
F (xxoF((((0)))
F(OoF-1(X 1: 1)))
h
Y[k]
(16)
F F |k| (8)) 11))
k=
5

---

A/k] S and X denotes the DFT of the input, i.e., F(x). In practice, we choose a to be ReLU in our implementation.
Next, it is generally computationally expensive to achieve equivariance over all scales. In practice, we only enforce set of scales for which of corresponding resolutions as R = (m.... N) with R[i] < Rli + 1]. To achieve scalc-equivariant
achieve equivariance, which can be denoted in terms
non-linearity over the scales of R, 0a FY) can be efficiently computed as
= - [ X
R
R 1) [k:] for the s.t.
R'i
k
R'i 2
(17)
Here, the ordered set R' = R U (0). By Eq. (17), all the Fourier coefficients k between any two consecutive resolutions in R, i.e. Rli 11/2 |k| S Rli]/2 can be computed by a single Fourier transform pair.
Scale-equivariant pooling, Pooling operation is crucial for deep nets' scalability larger images and datasets as they make the network more memory and computationally efficient. Commonly used pooling operations are max/average pooling, which reduces the input size by the factor of its window size and is not scale-equivariant To address this, we propose scale-equivariant pooling Pool
Let Pool denote max/average pooling operation with window size and Pool Rd - R
We define scale-equivariant pooling operation Pool Rd R mapping from to y where
y = (Y) follows
A = F FPPoo
w/k| : [(+l))))
Vk S
(18)
2w
Observe that this pooling layer satisfies Claim I by construction. Similar to non- linearity we can enforce the equivariance over the set following the same formulation in Eq. (17).
Note that as pooling reduces the size of the output by factor w, the operation is only scale-equivariant every th resolution. When the input size not multiple of there is truncation of the input.
Time Complexity. We now provide the time complexity of our equivariuu Fourier layer and compare it with standard group convolutions. Let's consider ID signal of length N and kernel of length K. Our proposed model involves:
A transformation of local filter to global with time complexity O(KN)
A convolution using Fourier transform with time complexity O(Nlog(N))
Our scale equivariant non-linearity depends the size of the group. Let A be the of group actions. The time complexity of the proposed scale-equivariant non-linearity is O[AA]N log(N)), where |A| denotes the cardinality of the set A.
So, the time complexity for each layer becomes
O(|4 log(NN + KN).
As comparison, the time complexity of regular group convolution is O(KN))) in the first layer and O(K for all intermediate layers, assuming the cost of group action is negligible constant [12].
Considering the time complexity of the intermediate layers of group convolutions, our proposed method is more efficient when
K
KN KAPKKN = log(N)
<A
A
So, when K << |A| and (Ng(N) < |A|K, i.e., assuming the set of group actions of moderate size, then our method is faster than group convolutions.
Modern GPUs are specifically optimized for regular convolution operations that can be performed in place. In contrast, the FFT algorithm does not fully capitalize on GPUs' advantages, primarily due to unique memory access patterns and moderate arithmetic intensities. Consequently, our approach unable to harness the full potential of GPUs. When executed on a GPU, regular group convolutions implemented as standard convolutions might exhibit comparable or even shorter running times than approach.

---

4.2 Classifier for equivariant features
truly invarainvariant, defined in Eq. (9), the model's performance is limited by the lowest resolution as the prediction needs to be the same. In the extreme, the prediction can only depend on a single mean pixel. Instead of invariance, we believe that it more desirable to ensure that a high-resolution image achieves better performance than its down-scaled version, i.e,, the performance is scale "consistent". To achieve this property, we propose a suitable classifier architecture and training scheme.
Classifier. In order to enforce scale-consistency, we need a classifier that outputs a prediction per scale. This motivated the following proposed architecture. Let be a classifier with M classes where gxx) I [ R(x) is defined as the set of resolutions smaller than the input resolution in the considered scales R. i... R(x) = {k : l S dim(x) and & E R]. Here, 9 is scale- eeale-equivariant deep-net that extracts features Φ (x) with corresponding DFT of Φ. Our proposed classifier has the form:
Ik| 2
k 1)
k MLP o Pool (Padu
Ak E R(x)
(19)
where Pady is Fourier padding operation that symmetrically pads zero to either side of the DFT to fixed size N, Pool is spatial pooling operation and MLP maps the pooled feature to the predicted logits ÿ kk] for each scale; Note the MLP shared across all scales. As we are sharing the MLP, we need to ensure that the input sizes are identical. Hence, we padded the features & to a fixed size. Finally, at test-time, we use the output from [diimxx)] to make prediction.
Training. Given dataset T = {(x, y)). we train our model using the sum of two losses. The first term is standard sum of cross entropy loss L over the scales:
Σ
[()
(20)
(E)(x)
The second term is consistency loss to encourage the performance of high-resolution to be better than the low-resolution:
Σ max -(jl -1,)))))))
(21)
(ER(x)
This is hinge loss that penalizes the model when the cross entropy loss L on high-resolution features (larger k) is greater than that of the low-resolution features (smaller k)).
5 Experiments
To study the effectiveness of our model, we conduct experiments on two benchmark datasets, MNIST- scale [40] and STL10 [4], following our theoretical setup using ideal downsampling, In this case, the theory exactly matches practice, and our approach achieves perfect scale-equivariance. We also conduct experiments comparing the models' generalization to unseen scales and data efficiency. Finally, we conduct experiments using a non-ideal anti-al anti-aliasing filter in down-scaling. Under this setting, our model longer achieves zero scale equivariance-error. However, we interested in how the models behave under this mismatch in theory and practice.
Evaluation metrics. To evaluate task performance, introduce a metric to measure the ccale-consistency. Given sample from the test set, we check whether the cross entropy loss is less than or equal to the classification loss of its down-scaled version. We compute this as a percentage over the dataset and report the scalc-consistent rate defined as:
report classification accuracy. Next, we
Σ E, ( [I[[(x:))
(
Scale-Con.
(
5 ((())())))
(22)
IT
(x.W)ET
where r is uniformly sampled over the set of scales for which we want to achieve equivariance and 1 denotes the indicator function.
Finally, we quantify the equivariance-err over the final feature map given by a fully trained model on the dataset. The equivariance-error (Fqui-Err.) is defined as
(l)(D,(x)) D, (g(x))11 (x)))1
Equi-Err.
(23)
S
XET PER
7

---

Table 1. Accuracy of different models MNIST-scale
Table 2. Accuracy of different models MNIST-scale (ideal downsampling) with missing scales.
(ideal downsampling) with scales.
Models
Acc.
Scale-Con.1
Equi-Emr.
Models
Acc.1
Scale-Con.1
Equi-Err..
CNN
0.9737
0,6621
CNN
0.9842
0.7617
Per Res. CNN
0,9388
0.0527
Per Res. CNN
0.9763
0.3594
SESN
0,9791
0.6640
SESN
0,9892
0,8339
DSS
0,9731
0,6503
DSS
0,9884
0,8105
SI-CovNet
0,9797
0.6425
SI-CovNet
0.9878
0.6621
SS-CNN
0.9613
0.3105
SS-CNN
0.9870
0.3593
DISCO
0.9856
0,5585
0.44
DISCO
0.9914
0,5371
0.35
Fourier CNN
0.9713
0.2421
0.28
Fourier CNN
0.9820
0.1250
0.23
Ours
0.9889
0,9716
0.00
Ours
0.9888
0.9366
0.00
Here, R is the of all scales over which enforce equivariance. We report the average equivariance error over the samples of the test set TT. We note that this equivariance-erro differs from the one reported by Sosnovik et al. [42] where they measured the error for the "scale-convolution with weights initialized randomly." Contrarily, we measure the equivariance error from end-to-end over trained models, which more closely matches how the models are used in practice.
Baselines. Following prior works in equivaraui neural networks [41, 42] we compare to baselines: DISCO [42], SI-ConvNet [13], SS-CNN [8], DSS [46], and SESN [41]. For the baseline, we follow the architecture and training scheme provided by Sosnovik et al. [42]. We also prepared three additional baseline models: (a) standard CNN, (b) Per Res, CNN where train a separate
CNN for each resolution in the training set, and (c) Fourier CNN [18] which utilizes Fourier layers.
5.1 MNIST-scale (Ideal downsampling)
Experiment setup. We create the MNIST-scale dataset following the procedure in prior works [13, 42]. Each image in the original MNIST dataset is randomly downsampled with a factor of ~M [ such that every resolution from 8 8 to 28 28 contains an equal number of samples. As the baseline models (except the Fourier CNN) can not handle images of different resolutions, following prior works, lower-resolution images are zero-padded the original resolution. We do not need to pad the input for our model and Fourier CNN. We used 10k, 2k, and 50k for training, validation, and test set samples. For this experiment, we enforce equivariance over scales that correspond the discrete resolutions of R = {8...., 28).
l0.3 1].
Implementation details. For the baselines and CNN, we follow the implementation, hyper- parameters, and architecture provided in prior works [41, 42]. For Per Res. CNN, we train separate CNN for each resolution. Each of these CNNs uses the architecture of baseline CNN. For Fourier CNN, we use the Fourier block introduced in the Fourier Neural operator [18]. Inspired by their design, we use 1 complex convolution in the Fourier domain along with the scale-equivariant convolution. We follow the baseline for all training hyper-parametert except we included a weight decay of 0.01.
Results. In Tab. 1, we report the accuracy of the MNIST-scale dataset. We observe that our approach achieved zero equivariance error and the highest accuracy. While all models achieve similar accuracy, there is more notable difference in the scale consistency rate. This means that our model properly captures the additional information that comes with increased resolution.
Generalization to unseen scales. We study the generalization capabilitics of the scale-cquivariant modes to unseen scales; we train them on a dataset with 10k full resolution (28 x 28) MNIST images and test on 50k samples of MNIST-scale, i.e., containing different scales. For the baselines, we added random scaling argumentation during training. In Tab. 2, we observe that our model can guarantee zero equivariance error even for the unseen scales and achieves comparable performance to baselines trained with data augmentation.
Data efficiency. We also conduct experiments studying the data efficiency of the different models. Following the same setup MNIST-scale, we train the models on limited training examples, 5k, 2.5k, and 1k, of different resolutions and test on 50k samples across all resolutions. Tab. 3, we observe that model is more data efficient than the baselines. DISCO achieves the second-best
8

---

Table 3. MNIST-scale accuracy with different numbers
Table 4. The classification accuracy of different models on STL1O-scale dataset.
training samples.
Models/ / # Samples
5000
2500
1000
Models
Acc.f
Scale-Con.
Equi-Ere E
CNN
0,9432
0.9389
0.8577
Wide ResNet
0.5596
0.2916
0.16
Per Res. CNN
0.9118
0.8392
0.5815
SESN
0,5525
0.4166
0.04
DISCO
0,9794
0.9665
0.9457
DSS
0.5347
0.1979
0.02
SESN
0,9638
0,9402
0,9207
SI-CovNet
0.5588
0.2187
0.03
SI-CovNet
0,9641
0.9437
0,9280
SS-CNN
0,4788
0.1979
1.82
SS-CNN
0.9477
0.9259
0.9176
DISCO
0.4768
0.3541
0.06
DSS
0.9654
0.9401
0,9281
Fourier CNN
0.9567
0.9419
0.8910
Fourier CNN
0.5844
0.2812
0.19
Ours
0.9835
0.9767
0.9606
Ours
0.7332
0.6770
0.00
Table 5. Ablation on consistency loss.
w/ consistency
w/o consistency
# Samples
Acc.t Scalc-Con.1
Acc.t Scale-Con.1
5000
0.9835
0.9296
0.9831
0.9150
2500
0.9767
0.8906
0.9755
0.8633
1000
0.9606
0,8183
0.9599
0.8144
performance. We also see that Per Res. CNN suffers the most when trained with fewer data points, as it trains separate CNN for each scale and does not share parameters across different scales.
Ablation, We perform an ablation on the consistency loss in Eq. (21) over different training set sizes. From Tab. we can observe that the consistency loss improves the accuracy of our model as well as the scale-consistency. This result validates the effectiveness of the proposed consistency loss.
5.2 STL1O-scale (Ideal downsampling)
Experiment setup. Following the same procedure the MNIST-scale dataset, create the STL100 scale dataset. Each image of the dataset is randomly scaled with a randomly chosen downsampling factor between [1 such that every resolution from 48 97 contains an equal number of samples. We use 7k, Ik, and 5k samples in our training, validation, and test set. For the baseline models, we again zero-pad the downsampled images to the original size.
Implementation details. For the baseline models, we use the Wide ResNet as the CNN baseline following prior work [41, 42]. For Fourier CNN, use six Fourier blocks followed by a two-layered MLP. For our model, we use six seale-equivariant Fourier blocks followed by a two-layer MLP. All of the models of 0.01. The learning rate is reduced by a factor of 0.1 after every 100 epoch. For scalability, we consider achieving equivariance over scales that correspond the discrete resolutions in the set R = {48 < 48 + i x 8 < 97} Vi E 10,1,2,...
trained for 250 epochs with Adam optimizer with initial learning rate
Results. In Tab. 4, we observe that our model achieves zero equivariance error with higher accuracy and scale consistency over the baselines. As the baseline models accept fixed-sized input, the downsampled images
vero-padded following prior work's preprocessing MNIST-scale. Note, MNIST images have uniform black background, and zero- prro-paddin does not create artifacts. However, for colored images with diverse backgrounds, such as STL-10, any padding scheme resize the image will cause artifacts. We believe this artifact hurts the performance of baseline models on the 1TL.10-scale dataset. However, it is unclear whether there is more suitable padding strategy.
5.3 MNIST-scale (Non-ideal downsampling)
Ideal interpolation suffers from artifacts known as ringing effect caused by Gibbs phe- nomenon [23]; see the down-scaled image in Fig. 3a. In practice, non-ideal low-pass filter will be used instead. Taking this into consideration, we conduct the experiments using more commonly used anti-alia anti-aliasing scheme with Gaussian blur instead of the ideal low-pass filter.

---

9
9
9
9
23
2
2
2
20
2
(a) Learned feature from ideal downsampling.
(b) Learned feature for non-ideal downsampling.
Figure 3. Feature visualization for ideal and non-ideal downsampling settings. In both settings, our model seems to learn spatially local features such as digit contour and edges.
Table 6. The accuracy of different models MNIST-scale (non-ideal downsampling).
Models
Acc.f
Scale Con.t
Equi Err.
CNN
0.9642
0.1033
Per Res. CNN
0,9450
0.0742
SESN
0,9710
0.6666
DSS
0.9772
0,5716
SI-CowNet
0.9694
0.4453
SS-CNN
0.9670
0.3144
DISCO
0.9830
0,4500
0.63
Fourier CNN
0.9745
0.1716
0.29
Ours
0.9880
0,9760
0.05
Experiment details. We follow the same experimental setup and training scheme as in MNIST-scale with the ideal downsampling experiment. The only difference is that we use Gaussian kernel to perform anti-aliasing.
Results. From Tab. 6, we observe that our model achieves higher classification accuracy and Scale consistency. Importantly, our model achieves lower equivariance error than the baseline despite the gap in the theory of non-ideal downsampling.
6 Conclusion
We propose a family of scale-equivariant deep nets that achieve zero equivariance error measured from end to end. We formulate down-scaling in the discrete domain with proper consideration of anti- aliasing. To achieve scale-equivariance, we design novel modules based on Fourier layers, enforcing that the lower frequency content of output does not depend on the higher frequency content of the input. Furthermore, we motivated the ssule-consistence property that the performance of higher- resolution input should be better than that of the lower resolution and designed suitable classifier architecture. Empirically, our approach achieves competitive accuracy on image classification tasks, with improved scale consistency and lower equivariance-error compared to baselines. Similar to other equivariant methodologies, defining consistent scales or group actions to achieve equivalence before constructing the model is crucial. Moreover, common challenge all equivariant and invariant techniques face is the significant demands on memory and computational resources. In our upcoming research, plan enhance our approach by applying high-resolution image datasets and dense prediction tasks, such as instance segmentation.
10

---

References
[1] E. H. Adelson, C. Anderson, J. R. Bergen, J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer, 1984. 1,2
[2] E. J. Bekkers. B-spline CNNs on Lie groups. In Proc. ICLR, 2020. 1,2
[3] M. M. Bronstein, Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE SPM, 2017. 2
[4] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proc. AISTATS, 2011.
[5] T. Cohen and M. Welling. Group equivariant convolutional networks. In Proc. ICML, 2016. 1,
2
[6] de Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions geometric graphs. In Proc. ICLR, 2021. 3
[7] M. Defferrard, X. Bresson, and Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering, In Proc. NeurIPS, 2016. 3
[8] R. Ghosh and A. K. Gupta. Scale steerable filters for locally scale-invarion convolutional neural networks. arXiv preprint arXiv:1906.0 038661 2019. 1, 2, 8
[9] K. Grauman and T. Darrell. The pyramid match kernel: discriminative classification with sets of image features. In Proc. ICCV, 2005. 1.2
[10] Hartford, D. Graham, K. Seyton-Brownn and S. Ravanbakhsh. Deep models of interactions across sets. In Proc. ICML, 2018.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE TPAMI, 2015.
[12] I. He, Y. Chen, Y. Dong, Y. Wang, Z. Lin, el. Efficient equivariant network. In Proc. NeurIPS, 2021.
[13] A. Kanazawa, A. Sharma, and D. Jacobs. Locally ocale-invariant convolutional neural networks. arXiv preprint arXi::1412.5000 2014. 2,
[14] T. Karras, M. Aittala, Laine, E. Härkönen, J. Hellsten, Lehtinen, and T. Aila. Alias-free generative adversarial networks. In Proc. NeurlPS, 2021. 3
[15] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Proc. ICLR, 2017.
[16] R. Kondor, Z. Lin, and S. Trivedi. Clebsch-Gordan Nets: a fully Fourier space spherical convolutional neural network. In Proc. NeurIPS, 2018.
[17] Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. CVPR, 2006. 1,2
[18] Z. Li, N. B. Kovachki, K. Arizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandku- mar. Fourier neural operator for parametric partial differential equations. In Proc. ICLR, 2021. 3
[19] I.JJ Liu, R. Yeh, and A. Schwing. Pic: permutation invariant critic for agett deep reinforcement learning. In Proc. CORL, 2020. 3
[20] L.J. Liu, Z. Ren, R. A. Yeh, and A. G. Schwing. Semantic tracklets: An object-centric represen- tation for visual multi-agent reinforcement learning. In Proc. IROS, 2021.
[21] D. G. Lowe. Object recognition from local scale-invariant features. In Proc. ICCV, 1999.
[22] D. G. Lowe. Distinctive image features from scale-invan iale--ivvariant keypoints. IJCV, 2004. 2
11

---

[23] D. G. Manolakis and V. K. Ingle. Applied digital signal processing: theory and practice. Cambridge university press, 2011. 1,2,
[24] X. Mao, Y. Liu, F. Liu, Q. Li, W. Shen, and Y. Wang. Intriguing findings of frequency selection for image deblurring. In Proc. AAAI, 2023. 3
[25] H. Maron, H. Ben- Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks. In Proc. ICLR, 2019. 3
[26] H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In Proc. ICML, 2020.
[27] M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through FFTss. arXiv preprint arXiv: 1312.5851, 2013.
[28] C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SpeqNets: Sparsity-aware permutation cquivariant graph networks. In Proc. ICML, 2022. 3
[29] E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. Ré. S4nd: Modeling images and videos multidimensional signals with state spaces. In Proc. NeurlPS, 2022. 3
[30] H. Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 1928.
[31] H. Pratt, B. Williams, F. Coenen, and Y. Zheng. FCNN: Fourier convolutional neural networks. In ECML PKDD, 2017. 3
[32] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR, 2017.
[33] S. Ravanbakhsh, J. Schneider, and B. Póczos. Equivariance through parameter-sharing. In Proc. ICML, 2017. 2
[34] Ravanbakhsh, J. Schneider, and B. Poezos. Deep learning with sets and point clouds. In Proc. ICLR workshop, 2017.
[35] R. A. sampling for shift invariant and equivariant convolutional networks. In Proc. NeurIPS, 2022.
T.-Y. Lim, A. Schwing, M. Do, and R. A. Yeh. Learnable polyphase
[36] R. A. Dojas-Gomez, T.-Y. Lim, M. N. Do. and R. A. Yeh. Making vision transformers truly shift-equivariant. arXiv preprint arXiv:2305. 16316, 2023. 2
[37] D. Romero, E. Bekkers, J. Tomczak, and M. Hoogendoorn. Attentive group equivariant convolutional networks. In Proc. ICML, 2020.
[38] M. Shakerinava and S. Ravanbakhsh. Equivariant networks for pixelized spheres. Proc. ICML, 2021.
[39] D. I. Shuman, S. K. Narang, Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending nigh-dimensional data analysis to networks and other irregular domains. IEEE SPM, 2013.
[40] K. Sohn and H. Lee. Learning invariant representations with local transformations. In Proc. ICML, 2012. 2,7
[41] Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. Proc. ICLR, 2020. 2, 8,
[42] I. Sosnovik, A. Moskalev, and A. Smeulders. DISCO: accurate discrete scale convolutions, In Proc. BMVC, 2021. 1, 8,
[43] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong,
H. Goka, K. Park, and Lempitsky. Resolution-robust large mask inpainting with Fourier convolutions. In Proc. WACV, 2022. 3
12

---

[44] S. R. Venkataraman, S. Balasubramanian, and R. R. Sarma. Building deep equivariant capsule networks. In Proc. ICLR, 2020. 2
[45] M. Weiler and G. Cesa. General E(2)-equivariant steerable CNNs. In Proc. NeurIPS, 2019. 2
[46] D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. In Proc. NeurIPS, 2019. 2, 8
[47] Xu, H. Kim, T. Rainforth, and Y. Teh. Group equivariant subsampling. In Proc. NeurIPS, 2021.
[48] R. A. Yeh, Y.-T. Hu, and A. Schwing. Chirality nets for human pose regression. In Proc. NeurIPS, 2019. 3
[49] R. A. Yeh, A. G. Schwing, J. Huang, and K. Murphy, Diverse generation for multi-agent sports games. In Proc. CVPR, 2019.
[50] R. A. Yeh, Y.-T. Hu, M. Hasegawa-Johnson, and A. Schwing. Equivariance discovery by learned parameter-sharing. In Proc. AISTATS, 2022.
[51] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. Smola. Deep sets. In Proc. NeurlPS, 3
[52] R. Zhang. Making convolutional networks shift-invariant again. In Proc. JCML, 2019.
[53] H. Zhao, Shi, X. Qi, X. Wang, and Jia. Pyramid scene parsing network. Proc. CVPR, 2017.
[54] W. Zhu, Q. Qiu, R. Calderbank, G. Sapiro, and X. Cheng. traning-ttanlaattooequvviaaattt networks with decomposed convolutional filters. JMLR, 2022.
13


Reviews:
[{"review_id": "m0Y8wxlRkg", "reviewer": "Reviewer_qJeg", "summary": "This paper proposes a formulation of scale equivariant networks, which takes anti-aliasing into account. This is in contrast to previous works, in which the issue of aliasing is often disregarded. The proposed method achieves competitive classification accuracy on MNIST-scale and STL-10, while maintaining zero equivariance error.", "questions": "* I find the definition of the convolutional kernels (Eq. 14) too convoluted. I would simply say that you define a kernel of size k in the spatial domain and send it to the Fourier domain next.\n\n* There are several works that tackle the issue of downsampling breaking equivariance (even in a general group setting) as well as tackling aliasing in many other settings, e.g.,[1,2,3,4,5,6,7] among many more. I believe that discussing these works would be beneficial to scope the paper better.\n\n* In line 213, you mention that you do the analysis of Sosnovik et al on trained models. Although I understand your reasoning, I actually believe that treating non-trained networks directly, i.e., random nets, makes more sense in order to evaluate the real capability of the networks to remove aliasing and preserve equivariance. Have you done this analysis as well? What were the results? Did you still get zero equivariance error?\n\n* I believe line 165 has an error.\n\n* The paper has some small typos. Please correct them.\n", "limitations": "As mentioned previously, the authors have not listed any limitations of their work in the paper.\n\n### Conclusion\n\nIn conclusion, I think this is an interesting paper with important contributions to the field. However, I am not sure if this paper should be framed as a scale-equivariant paper or rather as a resolution-agnostic neural architecture. I believe –please do tell me if you don't agree– that the latter option is more accurate. Although it requires some rewriting, I believe it would be possible to still do this during the rebuttal period.\n\n#### References\n\n[1] Romero, D. W., Bruintjes, R. J., Tomczak, J. M., Bekkers, E. J., Hoogendoorn, M., & van Gemert, J. C. (2021). Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059.\n\n[2]  Nguyen, Eric, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. \"S4nd: Modeling images and videos as multidimensional signals with state spaces.\" Advances in neural information processing systems 35 (2022): 2846-2861.\n\n[3] Bond-Taylor, S., & Willcocks, C. G. (2023). $\\infty $-Diff: Infinite Resolution Diffusion with Subsampled Mollified States. arXiv preprint arXiv:2303.18242.\n\n[4] Karras, Tero, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. \"Alias-free generative adversarial networks.\" Advances in Neural Information Processing Systems 34 (2021): 852-863.\n\n[5] Zhang, R., 2019, May. Making convolutional networks shift-invariant again. In International conference on machine learning (pp. 7324-7334). PMLR.\n\n[6] Xu, J., Kim, H., Rainforth, T. and Teh, Y., 2021. Group equivariant subsampling. Advances in Neural Information Processing Systems, 34, pp.5934-5946.\n\n[7] Rojas-Gomez, Renan A., Teck-Yian Lim, Alex Schwing, Minh Do, and Raymond A. Yeh. \"Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks.\" Advances in Neural Information Processing Systems 35 (2022): 35755-35768.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work addresses a problem that has been thus far disregarded when talking about scale equivariance: the aliasing issue. The proposed methods and solutions are sound and are empirically evaluated with very good results.", "weaknesses": "* My main concern is that, to the best of my understanding, the proposed framework does not consider inter-scale relationships. Instead, it considers multiple scales in parallel which are processed independently of one another. This brings into question whether this work should be defined as a work in group equivariance or rather a work on models aiming to be resolution-agnostic, i.e., able to work regardless of the resolution of the input, e.g., FlexConv [1], S4ND[2], \\inf-Diffusion [3].\n\n* This method relies heavily on using the Fourier and inverse Fourier transform for each of its layers. In addition, it is well-known that performing convolutions on the spatial domain is faster and requires less memory than performing them in the Fourier domain when convolutional kernels are small O(Nk) vs O(NlogN). However, the proposed method actually relies on small convolutional kernels. These two factors call into question the computational and memory complexity of the proposed method. I believe that this is the main practical limitation of the method, and I would like the authors to discuss this in the paper –in case it is indeed an issue–.\n\n* Connecting to the previous point, I believe that having a section in which the limitations of the work are clearly stated and discussed is vital.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Group equivariance vs. resolution agnostic.\n\nWe believe our work covers both aspects, where we construct equivariant features to build a classifier that is resolution agnostic. \n\nFirst, we view our proposed method (Sec. 4.1) to be a method for learning equivariant features. Following the definition of equivariance (Line 103 ), a deep-net $g$ is equivariant with the group action $D_r$ if it commutes with the group action $D_r$, i.e., $D_r(g(x)) = g(D_r(x))$ is satisfied. We proved in the paper that our proposed model is equivariant when $D_r$ is an ideal down_scaling operation (Eq. 6). \n\nNext, in Sec. 4.2, we propose a classifier for using these proposed equivariant features with the goal that the model works well across different input resolutions. Hence, this work also covers the resolution agnostic aspect.  We note that the agnostic approaches like S4ND, and \\inf-Diff use a continuous kernel for convolution which we showed that such methods are not truly scaled equivariant when considering anti-aliasing and non-linearity (See “Fourier CNN” in Tab. 1 and Tab. 2).\n\nFinally, we want to clarify that while our approach models the scales in parallel they are not independent of one another. The interactions between the scales happen in two ways:  (a) as stated in Claim 1, the “output’s frequency terms can only have dependencies on the terms in $X$ that are even lower in frequencies.” The lower frequencies can be interpreted as a smaller scaler; (b) the model parameters are shared between images across all scales. \n\nThanks for this clarifying question, we incorporate our responses here into the paper.\n\n> Q2. Computational and memory complexity of the method\n\nWe agree with the reviewer that, when compared with regular CNN, convolution using FFT is not as efficient. However, we think that our work should be compared to group convolution instead of regular convolutions. Please see the general response for a detailed discussion.\n\n> Q3. A limitation section?\n\nPlease see the general response.\n\n> Q4. Definition of the convolutional kernels in Eq. 14\n\nThanks for the suggestion. We derived the kernel analytically to have a better understanding of its form. In hindsight, it indeed seems convoluted. We will address this.\n\n> Q5. Related works on downsampling breaking equivariance\n\nThanks for the list of related works regarding anti-aliasing breaking other equivariance. In the paper, we focused on reviewing related works on scale-equivariance. We will include and discuss all the mentioned papers.\n\n> Q6. Evaluation of equivariance with random initialization\n\nWe also understand the perspective of evaluating with random initialization. Yes, our model achieves zero equivariance error even when tested with randomly initialized parameters. We will add this result to the paper.\n\n> Q7. Small typos.\n\nThanks for the detailed review. We will address this.\n\n> Q8. Rewriting during the rebuttal period.\n\nIt seems like NeurIPS does not allow updates to the paper during the rebuttal period. We find the suggestions to be very helpful and will incorporate all the recommended changes.\n"}, {"review_id": "HrmHH48bGg", "reviewer": "Reviewer_JFCH", "summary": "The submitted paper explores the problem of scale-equivariant neural networks.\nTo get the desired property by design, the authors propose new building blocks: a spatially local Fourier layer, a novel scale-equivariant pooling layer, and scale-equivariant way of applying activation functions.\nAll the introduced modifications rely heavily on the Fourier transform of the input feature map. \nAdditionally, a new loss function is proposed which aims to promote the natural consistency of predictions made at different scales. \n\nThe effect of the new approach is demonstrated on the problem of image classification on two datasets: MNIST-scale and STL-10-scale. ", "questions": "Please, address the aforementioned weaknesses.", "limitations": "No additional inquiries.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the proposed modifications well explained and motivated. \nTo the best of my knowledge, the presented approach is original enough, and it may be interesting for the broad community. \nQuality of the results seems very satisfactory and I hope they can serve as a reliable baseline for further research on this topic. \n", "weaknesses": "1. While the proposed results seem strong enough for the task of object recognition, the authors have not reported the computational or memory overhead introduced by their approach. Without this knowledge, it is hard to understand if their method can be directly scaled to more high-resolution data without any significant changes.\n\n1. I admit that most prior papers also mostly report their performance for image classification, it is quite clear that the property of scale equivariance may be even more desirable for the dense prediction tasks, e.g. semantic segmentation. Are there any barriers to applying the same method to the common segmentation architectures?\n\n1. Regarding the low performance of baseline models on STL-10-scale data (line 274), have the authors tried other padding schemes than zero-padding, e.g. so-called reflect/replicate/circular strategies (see the [link](https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html))? I find it suspicious that, e.g. the reported DISCO performance of 91.93% accuracy on STL-10 [1] drops to 47.68% on STL-10-scale (Tab. 4) in the provided evaluation results. I think, more discussion is needed on that matter.\n\n[1] Sosnovik et al. DISCO: accurate Discrete Scale Convolutions. In BMVC, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Computational or memory overhead of the approach\n\nPlease see the general response for a detailed discussion.\n\n> Q2. Evaluation of dense prediction tasks, e.g., semantic segmentation?\n\nFor a consistent evaluation, we followed prior works on scale-equivariance to evaluate using the MNIST and STL10 datasets, e.g., [36]. We note that extension to semantic segmentation requires finding a suitable architecture. Existing spatial domain segmentation architectures are not directly applicable to scale equivariant operations with Fourier layers.\n\nFor example, the popular semantic segmentation architecture U-Net [i] or its neural operator variant [ii] contains downsampling, skip-connections, and upsampling, which first reduce the size of the input image and then expand the output to the original resolution. In other words, a scale-equivariant, with consideration of anti-aliasing, upsampling layer is needed to undo the effect of the downsampling layers. However, as the scaling operation on discrete vectors forms a semi-group, it does not necessarily contain inverse operations. It remains an open challenge to design truly scale equivariant Fourier layers that support upsampling. We plan to investigate this problem in the future.\n\n\n- [i] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. \"U-net: Convolutional networks for biomedical image segmentation.\" In Proc. MICCAI, 2015\n- [ii] Rahman, Md Ashiqur, Zachary E. Ross, and Kamyar Azizzadenesheli. \"U-no: U-shaped neural operators.\" arXiv preprint arXiv:2204.11127 (2022).\n- [36] Sosnovik, Ivan, Michał Szmaja, and Arnold Smeulders. \"Scale-equivariant steerable networks.\" In Proc. ICLR, 2022\n\n\n> Q3. Low performance of baseline models on STL-10-scale data (line 274). Have the authors tried other padding schemes than zero-padding, e.g. so-called reflect/replicate/circular strategies?\n\nWe experimented with STL10-Scale (following the same setup as MNIST-Scale) which contains training images of resolutions 48x48-97x97. We note that the baselines, such as DISCO, do not consider anti-aliasing in their formulation. At large downsampling rate anti-aliasing matters. \n\nNext, for the zero-padding processing step, we followed the prior work’s choice from MNIST-Scale. As suggested by the reviewer, we conduct additional experiments (see table below) using different padding techniques (reflect, replicate, and circular). We observed that the padding schemes do not significantly change the model performance. There is no specific padding scheme that helps all the methods.\n\n| Method \\\\ Padding | Replicate | Circular | Reflect |\n|----------------|-----------|----------|---------|\n| DISCO          | 0.60      | 0.52     | 0.44    |\n| SESN           | 0.64      | 0.65     | 0.50    |\n| DSS            | 0.61      | 0.48     | 0.49    |\n| SS-CNN         | 0.47      | 0.50     | 0.50    |\n| SI-ConvNet     | 0.54      | 0.63     | 0.63    |\n\n"}, {"review_id": "RD6b07m24R", "reviewer": "Reviewer_5utz", "summary": "This paper introduces Truly Scale-Equivariant Deep Nets with Fourier Layers, which directly formulate down-scaling in the discrete domain to address the anti-aliasing problem. The proposed methods are validated on the MNIST-scale and STL-10 datasets, demonstrating good classification performance while preserving zero equivariance error.", "questions": "Please refer to the weakness part.", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses how to learn scale-equivariant representations in deep neural networks, overcoming the limitation of previous methods that did not account for anti-aliasing.\n2. The paper proposes a method that can consider the down-scaling operation directly in the discrete domain with anti-aliasing and achieves good performance.\n", "weaknesses": "1. This paper does not provide sufficient experiments to demonstrate the generality of the proposed method. It only tests on two small datasets and a single high-level vision task (image classification), which limits the impact of the proposed method. This paper should also evaluate the method on more larger-scale datasets, more high-level vision tasks (e.g., image segmentation), and more CNN architectures.\n2. Why does the baseline use zero-padding to the original resolution as the input, instead of resizing to the original resolution? Does this affect baseline's representation learning and performance?\n3. The paper lacks an analysis of the computational cost of the proposed method and other compared baseline during training and inference.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Evaluation of large-scale datasets and dense prediction tasks.\n\nThanks for the suggestion. For a consistent evaluation, we followed prior works on scale-equivariance to evaluate using the MNIST and STL10 datasets, e.g., [36]. \n\nWe agree that large-scale datasets and dense prediction tasks would be interesting. However, computation and memory bottlenecks remain challenging for our method **and** the compared scale-equivariant methods. We will add a limitation section to discuss this open challenge.\n\nNext, extension to semantic segmentation requires finding a suitable architecture.  The existing spatial domain segmentation architectures are not directly applicable to perfect scale equivariant operations and Fourier layers.\n\nFor example, the popular semantic segmentation architecture U-Net [i] or its neural operator variant [ii] contains downsampling, skip-connections, and upsampling, which first reduce the size of the input image and then expand the output to the original resolution. In other words, a scale-equivariant, with consideration of anti-aliasing, upsampling layer is needed to undo the effect of downsampling layers. However, as the scaling operation on discrete vectors forms a semi-group, it does not necessarily contain inverse operations. It remains an open challenge to design truly scale equivariant Fourier layers that support upsampling operations. We plan to investigate this problem in the future.\n\nFinally, our experiments aim to validate the effectiveness of our model on the benchmark datasets of other scale-equivariant techniques. We also conducted ablations to justify each of the proposed components. We believe the empirical evaluation is sufficient to validate our contribution. In future work, we aim to extend the proposed technique to larger datasets and dense prediction tasks.\n\n- [i] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. \"U-net: Convolutional networks for biomedical image segmentation.\" In Proc. MICCAI, 2015\n- [ii] Rahman, Md Ashiqur, Zachary E. Ross, and Kamyar Azizzadenesheli. \"U-no: U-shaped neural operators.\" arXiv preprint arXiv:2204.11127 (2022).\n- [36] Sosnovik, Ivan, Michał Szmaja, and Arnold Smeulders. \"Scale-equivariant steerable networks.\" In Proc. ICLR, 2022\n\n> Q2. Why does the baseline use zero-padding to the original resolution as the input, instead of resizing to the original resolution? Does this affect the baseline's representation of learning and performance?\n\nThe use of zero-padding follows strictly from prior works [8, 12, 36, 37]. \nIn the supplemental, we include results on ablating zero-padding vs. resizing the image (See Tab. A2 and Tab. A4 in the appendix). Baselines show little improvement in their classification accuracy, whereas the equivariance error does not show any improvement.\n\n- [8] R. Ghosh and A. K. Gupta. Scale steerable filters for locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1906.03861, 2019.\n- [12] A. Kanazawa, A. Sharma, and D. Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104, 2014.\n- [36] I. Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. In Proc. ICLR, 2020\n- [37] I. Sosnovik, A. Moskalev, and A. Smeulders. DISCO: accurate discrete scale convolutions. In Proc. BMVC, 2021\n\n> Q3. Comparison of computation or memory usage\n\nPlease see the general response for a detailed discussion."}, {"review_id": "hvFE7ZQ7nM", "reviewer": "Reviewer_JmYY", "summary": "The authors of this study introduce architectural modifications to the conventional CNN framework by incorporating Fourier layers. Their aim is to achieve scale-equivariance, addressing the anti-aliasing problem that occurs when images are downscaled in the continuous domain. To tackle this issue, the authors propose a discrete domain downsampling formulation and propose corresponding changes to the network architecture. Through experiments on a few datasets, the authors demonstrate that their method achieves zero scale-equivariance error.", "questions": "I encourage the authors to respond to the points I raised on weaknesses and fix minor text errors. I am willing to raise my score if the authors adequately respond to my concerns.", "limitations": "The authors have not included a Broader Impact section and did not discuss the potential negative societal impact of their work. They also do not have a discussion section where some limitations of their work could have been discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, precise, and easy to follow. In particular, their motivation is stated clearly, they introduce the preliminaries succinctly, and the approach stated is coherent and structured.\n\n- To the best of my knowledge the proposed method is novel and the claims that the module modifications achieve scale-equivariance is correct.\n\n- I am unaware of any method but this that solves the chosen problem completely, i.e achieves zero scale-equivariance error when downsampling is done via anti-aliasing and subsampling ('ideal' subsampling). The results are also competent when they do 'non-ideal' subsampling when compared to standard baselines. \n\n- The motivation that images of higher-scale should achieve better classification accuracy than the ones of lower scales is reasonable and the corresponding proposed constraint is sound.\n\nOverall I think the ideas presented in this paper maybe of interest to the broad ML community.", "weaknesses": "- Due to the complexity involved in the several DFT/Inverse-DFT related operations, the baselines they compare to which involve kernel resizing are certainly not as costly as the proposed method. I suggest the authors to discuss how their method compares to the baselines in terms of computational/time complexity.\n\n- For the same aforementioned reasons, I suspect that the method might not scale to high dimensional datasets like ImageNet or standard architectures like ResNet50. This I think would be a fundamental limitation, therefore I encourage the authors to discuss the challenges in scaling their method to the mentioned settings. I would be interested to know which modules will be particularly harder to scale.\n\n- Some minor text changes:\n\n     - Line 119, Claim 1: 'even lower in frequencies' needs to be 'even in lower frequencies.'\n\n     - Line 138, Claim 2: 'equivaraint' needs to be 'equivariant.'\n\n- Figure 1, Figtitle: “On the other hand, downsampling the high-res feature is guaranteed to achieve the same low-resolution feature.” -> Unclear formulation. It is not clear how the downsampled version of the features was achieved. The text suggests that it has merely been downsampled from the high-resolution version, but then it is not clear which role the new proposed architecture plays. Please rephrase for clarity.\n\n\n- Line 32: “Specifically, these works are derived using a continuous domain down-scaling operation, i.e., there is no need to consider anti-aliasing. However, when performing a down-scaling, the Nyquist theorem [21, 27] tells us that an anti-alias filter is necessary to avoid high-frequency content to alias into lower frequencies. (...) To address this gap, from prior work, we consider the down-scaling operation directly in the discrete domain taking into account of the anti-aliasing.” Several grammar errors and the paragraph is unclear to me. In the first sentence, the authors write that previous works “do not need to consider anti-aliasing”. But then, they aim to bridge the gap and take anti-aliasing into account. Please fix the formulation as it is currently very unclear whether previous works had an issue with anti-aliasing or not.\n\n\n- Line 103: The notation for the definition of g is unclear. It is unclear what is meant by that x is \\in {R^1, R^2 … R^N}, which makes it hard to understand the following definitions. The notation of {..} suggests that x can be from R^1, but then how is the Fourier transform of one number even defined? The Fourier transform is needed for D_R(x). I am confused by this definition, please rewrite for clarity. \n\n\n- It would be helpful if the authors would state that small x refers to the spatial domain and big X always refers to the Fourier domain. While this is the common practice, it would still be nice to state it explicitly.\n\n\n- I don’t understand Fig. 2b. Are the yellow regions ones and the gray ones zero?\n\n\n- Line 142: “Element-wise non-linearities, e.g., ReLU, in the spatial domain are generally not scale-equivariant under the ideal downsampling operation DR.” The reasoning of this claim is not obvious to me, please explain / give some intuition.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Computational/time complexity.\n\nPlease see the general response for a detailed discussion. \n\n> Q2. Discussion on limitation\n\nPlease see the general response.\n\n> Q3. Typos\n\nThanks for the careful review. We will address the typos.\n\n> Q4. Caption clarification for Fig. 1\n\nFig. 1 illustrates the exact scale equivariance property of the neural network $g$, i.e., $D_R(g(x)) = g(D_R(x))$. Where $D_R$ is the ideal downsampling, representing scaling operation, $g$ is the neural network, and  $g(x)$ is the features of the image $x$. In the caption, the 'high-res features' refers to the extracted feature maps from our approach using a high-resolution image, while the 'low-res features' refers to those from a downscaled, low-resolution image. As our model achieves exact equivariance, the 'low-res' feature precisely matches the downscaled version of the 'high-res' features. We will revise this caption to enhance clarity.\n\n> Q5. Clarification Line 32\n\nThe down-scaling operation of a signal $f$ can be written as $D_s\\[f\\](x) = f(sx)$ with $s>1$. When $f$ is in its continuous form, such an operation does not produce any artifact. On the other hand, when $f$ is sampled and represented as a finite-sized vector, such a down-scaling operation refers to the discarding of values. For example, down-scaling by $s=2$ can be performed by discarding every other sampled point. This naive discarding of sampled values might reduce the sampling rate below the Nyquist Frequencies and introduces the aliasing effect.\n\nIn short, aliasing happens only when the signal is sampled, not in its continuous form.  Previous work considers signals in their continuous form thus avoiding the issue of aliasing, which is not practical. As for most practical purposes, we always work with sampled signals. Our proposed method considers the aliasing effect and works directly on sampled signals. We will rewrite it for clarity.\n\n\n> Q5. Clarification Line 103\n\nIn Line 103, we define a neural network, $g$,  acting directly on the discretized/sampled signal. When we sample any signal, depending on the sampling rate we get vectors of different sizes. In other words, the domain of the neural network is vectors of different sizes which is defined as the set $\\\\{ R^1, R^2, … R^N \\\\}$. Each $R^n$ represents all possible real vectors of size $N$.\n\n\nThe output of the neural network will be at the same sampling rate as the input. So, the range of the neural network $g$ is also $\\\\{ R^1, R^2, … R^N \\\\}$, i.e., it is composed of spaces of vectors of different sizes. It also can be equivalently described as $\\cup_{j=1}^n R^j$.\n\nSide note, the DFT of a single number is just defined to be itself. We will also rewrite the section for clarity.\n\n> Q7. The notation for small $x$ and big $X$.\n\nYes, we followed the common practice of using small $x$ for the spatial domain and big $X$ for the Fourier domain. As suggested, we will explicitly write this out. \n\n> Q9. The explanation for Fig. 2b. Are the yellow regions ones and the gray ones zero?\n\nThe gray regions correspond to zeros (See Fig. 2b’s caption) and the yellow regions correspond to non-zero values. The figure illustrates the intuition that an output's higher frequencies can only depend on even lower frequencies in the input.\n\n> Q10. Intuition for Element-wise non-linearities, e.g., ReLU, being not scale-equivariant under the ideal downsampling operation.\n\nIn short, the low-pass filter (smoothing) of the value of the signal at a point during the ideal-downsampling operation depends on the neighboring values. Specifically, smoothing is not commutative with ReLU. \n\nTo see this, consider a signal that alternates between +2 and -4, and we use a box filter (a simple average) of size 2 with circular padding for smoothing. \n\n[+2,-4,+2,-4,+2,-4] → Smooth → [-1,-1,-1,-1,-1,-1] → ReLU → [0,0,0,0,0,0] \n\n[+2,-4,+2,-4,+2,-4] → ReLU → [2,0,2,0,2,0] → Smooth → [1,1,1,1,1,1] \n\nObserve that the two outputs are different, which proves that ReLU and smoothing are not commutative."}]

Author rebuttal:
We thank the reviewers for their constructive feedback and appreciation of our contribution. We first address the common concerns. Individual questions are answered separately in each reviewer's thread.

> Q1. Computational and memory complexity of the method

We now provide further analysis and empirical results regarding memory and computational complexity.

Consider a 1D signal of length $N$ and a kernel size of $K$. Our proposed model involves: (a) a transformation from local filter to global filter with a time complexity of $O(KN)$, (b) a convolution using Fourier transform with a time complexity of $O(N \log N)$, and (c) our scale equivariant non-linearity where the complexity depends on the group size. Let $A$ be the set of group actions, then the time complexity of the proposed scale-equivariant non-linearity is $O(|A| N \log N)$, where $|A|$ denotes the cardinality of the set $A$. The time complexity for each layer is then $O(|A| N \log N + KN)$. 

As a comparison, the time complexity of regular a group convolution layer [i] is $O(KN|A|)$ in the first layer and $O(KN|A|^2)$ for all intermediate layers [ii], assuming the cost of group action is a negligible constant. 

Considering the time complexity of the intermediate layers of group convolutions, our proposed method is more efficient when 
$|A| N \log N + KN < |A|^2KN$
$\implies \log N + K/|A| < |A|K$. 

In other words, if $K << |A|$ and $\log N < |A|K$, i.e., assuming the set of group actions of moderate size, then our method is faster than group convolutions. 

- [i]. Cohen, Taco, and Max Welling. "Group equivariant convolutional networks." Proc. ICML, 2016.
- [ii]. He, Lingshen, et al. "Efficient equivariant network." Proc. NeurIPS, 2021


Next, we present the measured run time and memory usage of a single-scale equivariant convolution layer for different methods (See the table below). These results are measured using a batch of 200 images of resolution $28 \times 28$. Please note that runtime and memory usage for our approach are measured including the scale-equivariant non-linearity. Other approaches use regular pointwise non-linearities that are done in place with no memory overhead.

**Number of Parameters, run time, and memory requirements:**
| model      | #parameter | Exe. Time on CPU (sec) | Memory (MB) | Exe. Time on GPU (sec) |
|------------|------------|------------------------|-------------|-----------------------|
| Disco      | 50272      | 0.857                  | 0.77        | 0.0008                |
| SS-CNN     | 56384      | 0.220                  | 0.05        | 0.0070                |
| SI-ConvNet | 50272      | 0.200                  | 0.24        | 0.0010                |
| SESN       | 50272      | 0.304                  | 0.96        | 0.0004                |
| DSS        | 50240      | 0.162                  | 0.96        | 0.0012                |
| Ours       | 50496      | 0.071                  | 0.59        | 0.0360                |

We observe that our approach requires less memory than all other scale-equivariant approaches except for SS-CNN. Our approach’s runtime on the CPU is lower than other approaches, however, this reduction does not transfer to GPU. From our understanding, modern GPUs are heavily optimized for regular convolution operation which can be done in place. On the other hand, the FFT algorithm does not get the full advantage of GPUs due to reasons like unique memory access patterns, moderate arithmetic intensities, etc. As there are many efforts to accelerate FFT for GPUS [iii, iv], we believe this performance gap will be reduced in the future. We will include all these results in the paper and add a limitation section.

- [iii]. Durrani, Sultan, et al. "FFT blitz: the tensor cores strike back." In Proc. ACM SIGPLAN, 2021.
- [iv]. Li, Binrui, Shenggan Cheng, and James Lin. "tcfft: Accelerating half-precision FFT through tensor cores." arXiv, 2021.
  
>Q2. Limitation & Broader Impact Section

Thanks for the suggestion. Yes, we agree that a limitation section will strengthen the paper. We will include a discussion on the limitation of the computation and the practical implications. Next, we do not foresee any negative societal impact caused by our work. We conducted experiments on image classification using standard benchmarks. These benchmarks are widely studied by the machine learning and computer vision community.


Meta-review: This work closely studies a particular aspect of signal processing for achieving scale equivariance in convolutional nets and demonstrates improvements in task accuracy, consistency across scales, and the equivariance of feature maps across scales. These results rely on the the proposed Fourier filtering layer and accompanying scale-equivariant nonlinearity and pooling. Four expert reviewers with backgrounds in filtering, equivariance, and generalization agree on acceptance (qJeg: 7, JmYY : 7, JFCH: 6, 5utz: 6). While the experiments are limited to toy datasets (MNIST, STL) for the analysis of spatial scale, these datasets are inherited from prior work, and the comparison with existing works is comprehensive and fair. The experiments in ideal and non-ideal filtering conditions, reflecting realistic computation, demonstrate the reduction of scale equivariance error to zero and least w.r.t. baselines respectively. Therefore, while this work could be more broadly impactful if larger models and data were considered, it is still an improvement within its scope. Its signal processing contribution to deep convolutional nets can inform the community for the pursuit of spatial scale equivariance on more tasks and larger scales of models and data.

For clarity, the AC advises the authors to 1. mention the limitation—shared by this work and others—that equivariance is restricted to a chosen set of discrete and bounded resolutions and 2. reference the supplement from the text for the results on resizing vs. padding (see comment by 5utz) to verify the fairness of these comparisons.
Decision: Accept (poster)