Title: Truly Scale-Equivariant Deep Nets with Fourier Layers
Venue: NeurIPS 2023
Accepted: yes
Keywords: Scale Equivariance

Fourier Neural Network
Abstract: In computer vision, models must be able to adapt to changes in image resolution to effectively carry out tasks such as image segmentation; This is known as scale-equivariance. Recent works have made progress in developing scale-equivariant convolutional neural networks, e.g., through weight-sharing and kernel resizing. However, these networks are not truly scale-equivariant in practice. Specifically, they do not consider anti-aliasing as they formulate the down-scaling operation in the continuous domain. To address this shortcoming, we directly formulate down-scaling in the discrete domain with consideration of anti-aliasing. We then propose a novel architecture based on Fourier layers to achieve truly scale-equivariant deep nets, i.e., absolute zero equivariance-error. Following prior works, we test this model on MNIST-scale and STL-10 datasets. Our proposed model achieves competitive classification performance while maintaining zero equivariance-error.

Truly Scale-Equivariant Deep Nets with Fourier Layers

Abstract

In computer vision, models must be able to adapt to changes in image resolution to effectively carry out tasks such as image segmentation; This is known as scaleequivariance. Recent works have made progress in developing scale-equivariant convolutional neural networks, e.g., through weight-sharing and kernel resizing. However, these networks are not truly scale-equivariant in practice. Specifically, they do not consider anti-aliasing as they formulate the down-scaling operation in the continuous domain. To address this shortcoming, we directly formulate downscaling in the discrete domain with consideration of anti-aliasing. We then propose a novel architecture based on Fourier layers to achieve truly scale-equivariant deep nets, i.e., absolute zero equivariance-error. Following prior works, we test this model on MNIST-scale and STL-10 datasets. Our proposed model achieves competitive classification performance while maintaining zero equivariance-error.

1 Introduction

Consider the task of image classification; if an object in the image is scaled (resized), then its corresponding object label should remain the same, i.e ., scale-invariant. Similarly, for semantic segmentation, if an object is scaled, then its corresponding mask should also be scaled accordingly, i.e ., scale-equivariant. Similarly, one would expect the features extracted to be scale-equivariant; see Fig. 1 for illustration. These invariant and equivariant properties are important to many computer vision tasks due to the nature of images. A photo of the same scenery can be taken from different distances, and objects in the scenes may come in different sizes. Developing representations that effectively capture this multi-resolution aspect of images has been a long-standing quest [1, 9, 11, 17, 53].

Recently, there has been a line of work on developing scale-equivariant convolutions networks [8, 13, 41, 42, 46] to more effectively learn multi-resolution features. At a high level, these works achieve scale-equivariant convolution layers through weight-sharing and kernel resizing, i.e ., use the 'same' but resized kernel across all scales [5]. The innovation of these works is how to properly resize the kernel. For example, Bekkers [2] and Sosnovik et al. [41] formulate kernel resizing as a continuous operation and then discretize the kernel when implemented in practice. However, this discretization leads to non-negligible equivariance error. On the other hand, Worrall and Welling [46] and Sosnovik et al. [42] directly formulate kernel resizing in the discrete domain, e.g ., using dilation or solving for the best kernel given a fixed scale set, and achieve low equivariance-error.

Despite these successes, we point out that the aforementioned works are not truly scale-equivariant in practice. Specifically, these works are derived using a continuous domain down-scaling operation, i.e ., there is no need to consider anti-aliasing. However, when performing a down-scaling on discrete space, the Nyquist theorem [23, 30] tells us that an anti-alias filter is necessary to avoid high-frequency content to alias into lower frequencies. The canonical example of aliasing is the 'wagon-wheel effect', where a wheel in a video appears to be rotating slower or even in reverse from its true rotation. To

Regular CNN

Regular CNN

Ideal Down-

sampling

Ideal Down-

sampling

Ours (Scale-

Equivariant)

Ours (Scale-

Equivariant)

Ideal Down-

sampling

Ideal Down-

sampling

5

5

G

address this gap from prior work, we consider the down-scaling operation directly in the discrete domain, taking he anti-aliasing into account.

In this work, we formulate down-scaling as the ideal downsampling from signal processing [23]. We then propose a family of deep nets that are truly scale-equivariant based on this ideal downsampling. This involves rethinking all the components in the deep net, including convolution layers, nonlinearities, and pooling layers.

With the developed deep net, we focus on the task of image classification. We further point out that truly scale-invariant classifiers are not desirable. A truly scale-invariant model's performance is limited by the lowest-resolution image. Instead, the more desirable property is that a high-resolution image should achieve a better performance than its corresponding low-resolution image. This motivated us to design a classifier architecture suitable for this property.

Following prior works, we conduct our experiments on the MNIST-scale [40] and STL [4] dataset. By design, our method achieves zeros scale equivariance-error both in theory and in practice. In terms of accuracy, we compare to recent scale-equivariant CNNs. We found our approach to be competitive in classification accuracy and exhibit better data efficiency in low-resource settings.

Our contributions are as follows:

We formulate down-scaling in the discrete domain with considerations of anti-aliasing.

Wepropose a family of deep nets that is truly scale-equivariant by designing novel scale-equivariant modules based on Fourier layers.

We conduct extensive experiments validating the proposed approach. On MNIST and STL datasets, the proposed model achieves an absolute zero end-to-end scale-equivariance error while maintaining competitive classification accuracy.

2 Related Work

Scale-equivariance and invariance. The notation of scale-equivariance is deeply rooted in image processing and computer vision. For example, classic hand-designed scale-invariant features such as SIFT [21, 22] have made tremendous contributions to the field of computer vision. Earlier works propose to use an image or spatial pyramid to capture the multi-resolution aspect of an image [1, 9, 17] by extracting features at several scales in an efficient manner.

More recently, there have been interests in developing scale-equivariant CNN [2, 8, 13, 41, 42, 46, 54]. Based on Group-Conv [5], these works achieve scale-equivariance convolution layers through weightsharing and kernel resizing. Different from these works, we consider the down-scaling in the discrete domain formulated as ideal downsampling from signal processing. We then develop modules that are truly scale-equivariant to enable a deep net that achieves zero equivariance-error measured from end to end. Finally, we note that there is a rich literature of equivariant deep nets [3, 5, 33, 36-38, 44, 45, 50]

with numerous applications applied to various domains, e.g ., sets [10, 26, 32, 34, 48, 51], graphs [6, 7, 15, 19, 20, 25, 28, 39, 49], etc. Moreover, several recent studies have also identified and tackled the issue of aliasing generated from the pooling layer to attain finer translation equivariance [35, 47, 52] and image generation [14].

Fourier transform in neural networks. Fourier transforms have been previously used in deep learning. For example, Mathieu et al. [27] proposes to use Fast Fourier Transform (FFT) to speed up CNN training. Fourier transform has also been used to develop network architectures, including various convolutional neural networks operating in the Fourier space [16, 31]. Recently, Fourier layers are capable of handling inputs of varying resolution, which have been employed in neural operators, facilitating applications in partial differential equations and in state space models [18, 29]. Fourier convolutions have also found success in low-level image processing tasks, e.g ., inpainting [43], deblurring [24]. Different from these works, we focus on developing truly scale-equivariant deep nets and leverage Fourier layers to achieve this goal.

3 Preliminaries

We briefly introduce and review the definition of Fourier transform, ideal downsampling, and scaleequivariance. For readability, we use 1D data to define these concepts. These ideas are extended to 2D data with multiple channels when implemented in practice.

Discrete Fourier Transform (DFT). Given an input vector x ∈ R N , we consider F : R N → C N be the discrete Fourier Transform (DFT) which has the form

where j denotes the unit imaginary number, i.e ., j 2 = -1 . The index k in Eq. (1) is commonly within the domain of [0 , N ] . Note that as Eq. (1) is N -periodic, for readability, we will use k from [ -N -1 2 , N -1 2 ] where k = 0 corresponds the lowest frequency.

The corresponding inverse DFT (IDFT) F -1 : C N → R N is defined as

By the convolution property of DFT, the circular convolution between x and a kernel k ∈ R N can be represented as the element-wise multiplication in the Fourier domain, i.e .,

where /circleasterisk denotes the circular convolution and /circledot denotes element-wise multiplication. Unless explicitly mentioned, we will represent the input vector with lowercase letters (e.g., x ) and its corresponding DFT with uppercase letters (e.g., X ).

Down-scaling operation. To reduce the scale (or resolution) of a signal x ∈ R N , one could perform a subsampling Sub R by a factor of R

However, naively subsampling leads to aliasing. Hence, anti-aliasing is performed in a multi-rate system. In signal processing, the analysis commonly uses the ideal anti-aliasing filter h , which zeros out all the high-frequency content, i.e ., its DFT H /defines F ( h ) is defined as:

See Fig. 2a for an illustration of the ideal anti-aliasing filter.

In this work, we define the overall down-scaling operation to be the ideal downsampling D R by a factor of R , which performs anti-aliasing followed by a subsampling operation:

X

H

H

/circledot

X

/circledot

=

Frequency

0

-1

1

-2

-3

1

Frequency

0

-1

1

-2

-3

Frequency

0

-1

1

-2

-3

(a) Illustration of ideal anti-aliasing (low-pass) filter.

-3

-2

-1

0

1

2

3

X

-3

-2

-1

0

1

2

3

Y

˜

G

=

(b)

Illustration of Claim 1.

where their DFT are related by

Scale-equivariance. With the down-scaling operation defined, a deep net g : { R 1 , R 2 , . . . , R N } ↦→ { R 1 , R 2 , . . . , R N } is scale-equivariant if:

where { R 1 , R 2 , . . . , R N } represents the space of input/output signals at different scales. In this paper, we are interested in designing a family of deep nets that satisfies the equality in Eq. (8). Scale-invariance can be defined in a similar manner as

Fourier layer. Given a multi-channel input vector, x ∈ R C in × N and kernel k ∈ R C out × C in × N , where C in / out is the number of input/output channels, the circular convolution layer is defined as

where X and K denotes the DFT of x and k applied independently for each channel.

4 Approach

Our goal is to design truly scale-equivariant deep nets. To accomplish this goal, we propose scaleequivariant versions of CNN modules, including, the convolution layer, non-linearities, and pooling layers. In Sec. 4.1, we detail the operation for each of the proposed modules. In Sec. 4.2, we demonstrate how to build a classifier that is suitable for image classification with scale-equivariant features. We now explain our overarching design principle for the scale-equivariant modules.

From a frequency perspective, as reviewed in Eq. (6), the ideal downsampling operation results in the loss of higher frequency terms of the signal. In other words, if a feature's frequency terms depend on any higher frequency terms of the input, then it is not scale-equivariant, as the information will be lost after downsampling. We now formally state this observation in Claim 1.

Claim 1. Let g denote a deep net such that y = g ( x ) . If this deep net g can be equivalently represented as a set of functions ˜ G k : C 2 k +1 → C such that

then g is scale-equivariant as defined in Eq. (8) . In other words, an output's frequency terms can only have dependencies on the terms in X that are even lower in frequencies. We illustrate this structure with a linear function in Fig. 2b.

Proof. We denote the deep net's input and output as x and y with corresponding DFT X and Y . We denote the deep net's down-scaled input and output as x ′ = D R ( x ) and y ′ = g ( x ′ ) with corresponding DFT X ′ and Y ′ . Now assume that g : R n → R n ∀ n ∈ { 1 , 2 , . . . N } is a deep net that satisfies Claim 1 then

Therefore, ∀ k ≤ N R Y [ k ] = Y ′ [ k ] . By the definition of ideal downsampling Y ′ = D R ( Y ) concluding that g ( D R ( x )) = D R ( g ( x )) , i.e ., g is scale-equivariant.

For ease of understanding, here we assume that the deep net's input and output are of the same size. A version with a more relaxed assumption is provided in Appendix Sec. A1.

4.1 Scale Equivariant Fourier Networks.

We now describe the proposed modules and show that they are truly scale-equivariant.

Spatially local Fourier layer. For computer vision, learning local features is crucial. The Fourier layer in Eq. (10) is global in nature. To efficiently learn local features, we propose a localized Fourier layer where we constrain the degree of freedom in the kernel K such that the respective spatial kernel k is spatially localized.

Let k ∈ R d and k l ∈ R l to be d and l dimensional kernel such that k [ i ] = k l [ i ] if i < l otherwise 0 , i.e ., k is spatially local to have a receptive field of size l . We denote K and K l be the DFT of the kernel k and k l respectively. We claim that K can be written as

From Eq. (14), instead of modeling all the degrees of freedom in K , we will directly parameterize K l to enforce the learned kernel to be localized spatially. We defer the proof to the Appendix Sec. A2.

Claim 2. The spatially local Fourier layer is scale-equivariant.

Proof. The kernel k has a corresponding DFT K . As reviewed, a circular convolution between k and input x can be expressed as

Observe that X [ k ] is a subset of X [ -k : k ] , i.e ., Claim 1 is satisfied.

Scale-equivariant non-linearity ( σ s ). Element-wise non-linearities, e.g ., ReLU, in the spatial domain are generally not scale-equivariant under the ideal downsampling operation D R . While applying element-wise non-linearity in the frequency domain is scale-equivariant, this strategy empirically leads to degraded performance on classification tasks. To address this, we propose a scale-equivariant non-linearity σ s in the spatial domain.

Given a non-linearity σ , e.g ., ReLU, we construct a corresponding scale-equivariant version σ s that satisfies Claim 1. Let x ∈ R N and y ∈ R N to denote the input and output of σ s . We define scale-equivariant non-linearity σ s ( x ) = F -1 ( Y ) where Y takes the following form:

∀ | k | ≤ N 2 and X denotes the DFT of the input, i.e ., F ( x ) . In practice, we choose σ to be ReLU in our implementation.

Next, it is generally computationally expensive to achieve equivariance over all scales. In practice, we only enforce a set of scales for which we want to achieve equivariance, which can be denoted in terms of corresponding resolutions as R = ( m,... N ) with R [ i ] < R [ i +1] . To achieve scale-equivariant non-linearity over the scales of R , σ s = F -1 ( Y ) can be efficiently computed as

Here, the ordered set R ′ = R ∪ { 0 } . By Eq. (17), all the Fourier coefficients k between any two consecutive resolutions in R , i.e ., R [ i -1] / 2 < | k | ≤ R [ i ] / 2 can be computed by a single Fourier transform pair.

Scale-equivariant pooling. Pooling operation is crucial for deep nets' scalability to larger images and datasets as they make the network more memory and computationally efficient. Commonly used pooling operations are max/average pooling, which reduces the input size by the factor of its window size w and is not scale-equivariant. To address this, we propose scale-equivariant pooling Pool w s .

Let Pool w denote a max/average pooling operation with a window size w and Pool w : R d → R d w . We define scale-equivariant pooling operation Pool w s : R d → R d w mapping from x to y where y = F -1 ( Y ) follows

Observe that this pooling layer satisfies Claim 1 by construction. Similar to non-linearity, we can enforce the equivariance over the set R following the same formulation in Eq. (17).

Note that as pooling reduces the size of the output by a factor w , the operation is only scale-equivariant at every w th resolution. When the input size is not a multiple of w there is a truncation of the input.

Time Complexity. We now provide the time complexity of our scale-equivariant Fourier layer and compare it with standard group convolutions. Let's consider a 1D signal of length N and a kernel of length K . Our proposed model involves:

A transformation of local filter to global with time complexity O ( KN )

A convolution using Fourier transform with time complexity O ( Nlog ( N ))

Our scale equivariant non-linearity depends on the size of the group. Let A be the set of group actions. The time complexity of the proposed scale-equivariant non-linearity is O ( | A | N log( N )) , where | A | denotes the cardinality of the set A .

So, the time complexity for each layer becomes

As a comparison, the time complexity of regular group convolution is O ( KN | A | ) in the first layer and O ( KN | A | 2 ) for all intermediate layers, assuming the cost of group action is a negligible constant [12].

Considering the time complexity of the intermediate layers of group convolutions, our proposed method is more efficient when

So, when K << | A | and log ( N ) < | A | K , i.e., assuming the set of group actions of moderate size, then our method is faster than group convolutions.

Modern GPUs are specifically optimized for regular convolution operations that can be performed in place. In contrast, the FFT algorithm does not fully capitalize on GPUs' advantages, primarily due to unique memory access patterns and moderate arithmetic intensities. Consequently, our approach is unable to harness the full potential of GPUs. When executed on a GPU, regular group convolutions implemented as standard convolutions might exhibit comparable or even shorter running times than our approach.

4.2 Classifier for equivariant features

A truly scale-invariant, defined in Eq. (9), the model's performance is limited by the lowest resolution as the prediction needs to be the same. In the extreme, the prediction can only depend on a single mean pixel. Instead of invariance, we believe that it is more desirable to ensure that a high-resolution image achieves a better performance than its down-scaled version, i.e ., the performance is scale 'consistent'. To achieve this property, we propose a suitable classifier architecture and training scheme.

Classifier. In order to enforce scale-consistency, we need a classifier that outputs a prediction per scale. This motivated the following proposed architecture. Let c be a classifier with M classes where ˆ y = c ◦ g ( x ) ∈ R |R ( x ) | × M . R ( x ) is defined as the set of resolutions smaller than the input resolution in the considered scales R . i.e ., R ( x ) = { k : k ≤ dim ( x ) and k ∈ R} . Here, g is a scale-equivariant deep-net that extracts features φ = g ( x ) with corresponding DFT of Φ . Our proposed classifier has the form:

where Pad N is a Fourier padding operation that symmetrically pads zero to either side of the DFT to a fixed size N , Pool is a spatial pooling operation and MLP maps the pooled feature to the predicted logits ˆ y [ k ] for each scale; Note the MLP is shared across all scales. As we are sharing the MLP, we need to ensure that the input sizes are identical. Hence, we padded the features Φ to a fixed size. Finally, at test-time, we use the output from ˆ y [ dim ( x )] to make a prediction.

Training. Given a dataset T = { ( x , y ) } , we train our model using the sum of two losses. The first term is a standard sum of cross entropy loss L over the scales:

The second term is a consistency loss to encourage the performance of high-resolution to be better than the low-resolution:

This is a hinge loss that penalizes the model when the cross entropy loss L on high-resolution features (larger k ) is greater than that of the low-resolution features (smaller k ).

5 Experiments

To study the effectiveness of our model, we conduct experiments on two benchmark datasets, MNISTscale [40] and STL10 [4], following our theoretical setup using ideal downsampling. In this case, the theory exactly matches practice, and our approach achieves perfect scale-equivariance. We also conduct experiments comparing the models' generalization to unseen scales and data efficiency. Finally, we conduct experiments using a non-ideal anti-aliasing filter in down-scaling. Under this setting, our model no longer achieves zero scale equivariance-error. However, we are interested in how the models behave under this mismatch in theory and practice.

Evaluation metrics. To evaluate task performance, we report classification accuracy. Next, we introduce a metric to measure the scale-consistency. Given a sample from the test set, we check whether the cross entropy loss is less than or equal to the classification loss of its down-scaled version. We compute this as a percentage over the dataset and report the scale-consistent rate defined as:

where r is uniformly sampled over the set of scales for which we want to achieve equivariance and 1 denotes the indicator function.

Finally, we quantify the equivariance-error over the final feature map given by a fully trained model on the dataset. The equivariance-error (Equi-Err.) is defined as

Here, R is the set of all scales over which we enforce equivariance. We report the average equivariance error over the samples of the test set T . We note that this equivariance-error differs from the one reported by Sosnovik et al. [42] where they measured the error for the 'scale-convolution with weights initialized randomly." Contrarily, we measure the equivariance error from end-to-end over trained models, which more closely matches how the models are used in practice.

Baselines. Following prior works in scale-equivariant neural networks [41, 42] we compare to baselines: DISCO [42], SI-ConvNet [13], SS-CNN [8], DSS [46], and SESN [41]. For the baseline, we follow the architecture and training scheme provided by Sosnovik et al. [42]. We also prepared three additional baseline models: (a) standard CNN, (b) Per Res, CNN where we train a separate CNN for each resolution in the training set, and (c) Fourier CNN [18] which utilizes Fourier layers.

5.1 MNIST-scale (Ideal downsampling)

Experiment setup. We create the MNIST-scale dataset following the procedure in prior works [13, 42]. Each image in the original MNIST dataset is randomly downsampled with a factor of ∼ [ 1 0 . 3 -1] , such that every resolution from 8 × 8 to 28 × 28 contains an equal number of samples. As the baseline models (except the Fourier CNN) can not handle images of different resolutions, following prior works, lower-resolution images are zero-padded to the original resolution. We do not need to pad the input for our model and Fourier CNN. We used 10k, 2k, and 50k for training, validation, and test set samples. For this experiment, we enforce equivariance over scales that correspond to the discrete resolutions of R = { 8 , . . . , 28 } .

Implementation details. For the baselines and CNN, we follow the implementation, hyperparameters, and architecture provided in prior works [41, 42]. For Per Res. CNN, we train a separate CNN for each resolution. Each of these CNNs uses the architecture of baseline CNN. For Fourier CNN, we use the Fourier block introduced in the Fourier Neural operator [18]. Inspired by their design, we use 1 × 1 complex convolution in the Fourier domain along with the scale-equivariant convolution. We follow the baseline for all training hyper-parameters, except we included a weight decay of 0 . 01 .

Results. In Tab. 1, we report the accuracy of the MNIST-scale dataset. We observe that our approach achieved zero equivariance error and the highest accuracy. While all models achieve similar accuracy, there is a more notable difference in the scale consistency rate. This means that our model properly captures the additional information that comes with increased resolution.

Generalization to unseen scales. We study the generalization capabilities of the scale-equivariant modes to unseen scales; we train them on a dataset with 10k full resolution (28 × 28) MNIST images and test on 50k samples of MNIST-scale, i.e ., containing different scales. For the baselines, we added random scaling argumentation during training. In Tab. 2, we observe that our model can guarantee zero equivariance error even for the unseen scales and achieves comparable performance to baselines trained with data augmentation.

Data efficiency. We also conduct experiments studying the data efficiency of the different models. Following the same setup as MNIST-scale, we train the models on limited training examples, 5k, 2.5k, and 1k, of different resolutions and test on 50k samples across all resolutions. In Tab. 3, we observe that our model is more data efficient than the baselines. DISCO achieves the second-best

performance. We also see that Per Res. CNN suffers the most when trained with fewer data points, as it trains a separate CNN for each scale and does not share parameters across different scales.

Ablation. We perform an ablation on the consistency loss in Eq. (21) over different training set sizes. From Tab. 5, we can observe that the consistency loss improves the accuracy of our model as well as the scale-consistency. This result validates the effectiveness of the proposed consistency loss.

5.2 STL10-scale (Ideal downsampling)

Experiment setup. Following the same procedure as the MNIST-scale dataset, we create the STL10scale dataset. Each image of the dataset is randomly scaled with a randomly chosen downsampling factor between [1 -2] such that every resolution from 48 to 97 contains an equal number of samples. We use 7k, 1k, and 5k samples in our training, validation, and test set. For the baseline models, we again zero-pad the downsampled images to the original size.

Implementation details. For the baseline models, we use the Wide ResNet as the CNN baseline following prior work [41, 42]. For Fourier CNN, we use six Fourier blocks followed by a two-layered MLP. For our model, we use six scale-equivariant Fourier blocks followed by a two-layer MLP. All of the models are trained for 250 epochs with Adam optimizer with an initial learning rate of 0 . 01 . The learning rate is reduced by a factor of 0 . 1 after every 100 epoch. For scalability, we consider achieving equivariance over scales that correspond to the discrete resolutions in the set R = { 48 ≤ 48 + i × 8 ≤ 97 } ∀ i ∈ { 0 , 1 , 2 , . . . } .

Results. In Tab. 4, we observe that our model achieves zero equivariance error with higher accuracy and scale consistency over the baselines. As the baseline models accept a fixed-sized input, the downsampled images are zero-padded following prior work's preprocessing on MNIST-scale. Note, MNISTimages have a uniform black background, and zero-padding does not create artifacts. However, for colored images with diverse backgrounds, such as STL-10, any padding scheme to resize the image will cause artifacts. We believe this artifact hurts the performance of baseline models on the STL10-scale dataset. However, it is unclear whether there is a more suitable padding strategy.

5.3 MNIST-scale (Non-ideal downsampling)

Ideal interpolation suffers from artifacts known as the ringing effect caused by Gibbs phenomenon [23]; see the down-scaled image in Fig. 3a. In practice, a non-ideal low-pass filter will be used instead. Taking this into consideration, we conduct the experiments using a more commonly used anti-aliasing scheme with a Gaussian blur instead of the ideal low-pass filter.

7

7

2

2

(a) Learned feature from ideal downsampling.

(b)

Learned feature for non-ideal downsampling.

Experiment details. We follow the same experimental setup and training scheme as in MNIST-scale with the ideal downsampling experiment. The only difference is that we use a Gaussian kernel to perform anti-aliasing.

Results. From Tab. 6, we observe that our model achieves higher classification accuracy and Scale consistency. Importantly, our model achieves lower equivariance error than the baseline despite the gap in the theory of non-ideal downsampling.

6 Conclusion

We propose a family of scale-equivariant deep nets that achieve zero equivariance error measured from end to end. We formulate down-scaling in the discrete domain with proper consideration of antialiasing. To achieve scale-equivariance, we design novel modules based on Fourier layers, enforcing that the lower frequency content of output does not depend on the higher frequency content of the input. Furthermore, we motivated the scale-consistency property that the performance of higherresolution input should be better than that of the lower resolution and designed a suitable classifier architecture. Empirically, our approach achieves competitive accuracy on image classification tasks, with improved scale consistency and lower equivariance-error compared to baselines. Similar to other equivariant methodologies, defining consistent scales or group actions to achieve equivalence before constructing the model is crucial. Moreover, a common challenge all equivariant and invariant techniques face is the significant demands on memory and computational resources. In our upcoming research, we plan to enhance our approach by applying it to high-resolution image datasets and dense prediction tasks, such as instance segmentation.

References

E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer , 1984. 1, 2

E. J. Bekkers. B-spline CNNs on Lie groups. In Proc. ICLR , 2020. 1, 2

M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE SPM , 2017. 2

A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proc. AISTATS , 2011. 2, 7

T. Cohen and M. Welling. Group equivariant convolutional networks. In Proc. ICML , 2016. 1, 2

P. de Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In Proc. ICLR , 2021. 3

M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Proc. NeurIPS , 2016. 3

R. Ghosh and A. K. Gupta. Scale steerable filters for locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1906.03861 , 2019. 1, 2, 8

K. Grauman and T. Darrell. The pyramid match kernel: discriminative classification with sets of image features. In Proc. ICCV , 2005. 1, 2

J. Hartford, D. Graham, K. Leyton-Brown, and S. Ravanbakhsh. Deep models of interactions across sets. In Proc. ICML , 2018. 3

K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE TPAMI , 2015. 1

L. He, Y. Chen, Y. Dong, Y. Wang, Z. Lin, et al. Efficient equivariant network. In Proc. NeurIPS , 2021. 6

A. Kanazawa, A. Sharma, and D. Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104 , 2014. 1, 2, 8

T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. In Proc. NeurIPS , 2021. 3

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Proc. ICLR , 2017. 3

R. Kondor, Z. Lin, and S. Trivedi. Clebsch-Gordan Nets: a fully Fourier space spherical convolutional neural network. In Proc. NeurIPS , 2018. 3

S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. CVPR , 2006. 1, 2

Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In Proc. ICLR , 2021. 3, 8

I.-J. Liu, R. A. Yeh, and A. G. Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Proc. CORL , 2020. 3

I.-J. Liu, Z. Ren, R. A. Yeh, and A. G. Schwing. Semantic tracklets: An object-centric representation for visual multi-agent reinforcement learning. In Proc. IROS , 2021. 3

D. G. Lowe. Object recognition from local scale-invariant features. In Proc. ICCV , 1999. 2

D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV , 2004. 2

D. G. Manolakis and V. K. Ingle. Applied digital signal processing: theory and practice . Cambridge university press, 2011. 1, 2, 9

X. Mao, Y. Liu, F. Liu, Q. Li, W. Shen, and Y. Wang. Intriguing findings of frequency selection for image deblurring. In Proc. AAAI , 2023. 3

H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks. In Proc. ICLR , 2019. 3

H. Maron, O. Litany, G. Chechik, and E. Fetaya. On learning sets of symmetric elements. In Proc. ICML , 2020. 3

M. Mathieu, M. Henaff, and Y. LeCun. Fast training of convolutional networks through FFTs. arXiv preprint arXiv:1312.5851 , 2013. 3

C. Morris, G. Rattan, S. Kiefer, and S. Ravanbakhsh. SpeqNets: Sparsity-aware permutationequivariant graph networks. In Proc. ICML , 2022. 3

E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. In Proc. NeurIPS , 2022. 3

H. Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers , 1928. 1

H. Pratt, B. Williams, F. Coenen, and Y. Zheng. FCNN: Fourier convolutional neural networks. In ECML PKDD , 2017. 3

C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR , 2017. 3

S. Ravanbakhsh, J. Schneider, and B. Póczos. Equivariance through parameter-sharing. In Proc. ICML , 2017. 2

S. Ravanbakhsh, J. Schneider, and B. Poczos. Deep learning with sets and point clouds. In Proc. ICLR workshop , 2017. 3

R. A. Rojas-Gomez, T.-Y. Lim, A. Schwing, M. Do, and R. A. Yeh. Learnable polyphase sampling for shift invariant and equivariant convolutional networks. In Proc. NeurIPS , 2022. 3

R. A. Rojas-Gomez, T.-Y. Lim, M. N. Do, and R. A. Yeh. Making vision transformers truly shift-equivariant. arXiv preprint arXiv:2305.16316 , 2023. 2

D. Romero, E. Bekkers, J. Tomczak, and M. Hoogendoorn. Attentive group equivariant convolutional networks. In Proc. ICML , 2020.

M. Shakerinava and S. Ravanbakhsh. Equivariant networks for pixelized spheres. In Proc. ICML , 2021. 2

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE SPM , 2013. 3

K. Sohn and H. Lee. Learning invariant representations with local transformations. In Proc. ICML , 2012. 2, 7

I. Sosnovik, M. Szmaja, and A. Smeulders. Scale-equivariant steerable networks. In Proc. ICLR , 2020. 1, 2, 8, 9

I. Sosnovik, A. Moskalev, and A. Smeulders. DISCO: accurate discrete scale convolutions. In Proc. BMVC , 2021. 1, 2, 8, 9

R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky. Resolution-robust large mask inpainting with Fourier convolutions. In Proc. WACV , 2022. 3

S. R. Venkataraman, S. Balasubramanian, and R. R. Sarma. Building deep equivariant capsule networks. In Proc. ICLR , 2020. 2

M. Weiler and G. Cesa. General E(2)-equivariant steerable CNNs. In Proc. NeurIPS , 2019. 2

D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. In Proc. NeurIPS , 2019. 1, 2, 8

J. Xu, H. Kim, T. Rainforth, and Y. Teh. Group equivariant subsampling. In Proc. NeurIPS , 2021. 3

R. A. Yeh, Y.-T. Hu, and A. Schwing. Chirality nets for human pose regression. In Proc. NeurIPS , 2019. 3

R. A. Yeh, A. G. Schwing, J. Huang, and K. Murphy. Diverse generation for multi-agent sports games. In Proc. CVPR , 2019. 3

R. A. Yeh, Y.-T. Hu, M. Hasegawa-Johnson, and A. Schwing. Equivariance discovery by learned parameter-sharing. In Proc. AISTATS , 2022. 2

M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In Proc. NeurIPS , 2017. 3

R. Zhang. Making convolutional networks shift-invariant again. In Proc. ICML , 2019. 3

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proc. CVPR , 2017. 1

W. Zhu, Q. Qiu, R. Calderbank, G. Sapiro, and X. Cheng. Scaling-translation-equivariant networks with decomposed convolutional filters. JMLR , 2022. 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this study introduce architectural modifications to the conventional CNN framework by incorporating Fourier layers. Their aim is to achieve scale-equivariance, addressing the anti-aliasing problem that occurs when images are downscaled in the continuous domain. To tackle this issue, the authors propose a discrete domain downsampling formulation and propose corresponding changes to the network architecture. Through experiments on a few datasets, the authors demonstrate that their method achieves zero scale-equivariance error.\n\nSTRENGTHS:\n- The paper is well-written, precise, and easy to follow. In particular, their motivation is stated clearly, they introduce the preliminaries succinctly, and the approach stated is coherent and structured.\n\n- To the best of my knowledge the proposed method is novel and the claims that the module modifications achieve scale-equivariance is correct.\n\n- I am unaware of any method but this that solves the chosen problem completely, i.e achieves zero scale-equivariance error when downsampling is done via anti-aliasing and subsampling ('ideal' subsampling). The results are also competent when they do 'non-ideal' subsampling when compared to standard baselines. \n\n- The motivation that images of higher-scale should achieve better classification accuracy than the ones of lower scales is reasonable and the corresponding proposed constraint is sound.\n\nOverall I think the ideas presented in this paper maybe of interest to the broad ML community.\n\nWEAKNESSES:\n- Due to the complexity involved in the several DFT/Inverse-DFT related operations, the baselines they compare to which involve kernel resizing are certainly not as costly as the proposed method. I suggest the authors to discuss how their method compares to the baselines in terms of computational/time complexity.\n\n- For the same aforementioned reasons, I suspect that the method might not scale to high dimensional datasets like ImageNet or standard architectures like ResNet50. This I think would be a fundamental limitation, therefore I encourage the authors to discuss the challenges in scaling their method to the mentioned settings. I would be interested to know which modules will be particularly harder to scale.\n\n- Some minor text changes:\n\n     - Line 119, Claim 1: 'even lower in frequencies' needs to be 'even in lower frequencies.'\n\n     - Line 138, Claim 2: 'equivaraint' needs to be 'equivariant.'\n\n- Figure 1, Figtitle: “On the other hand, downsampling the high-res feature is guaranteed to achieve the same low-resolution feature.” -> Unclear formulation. It is not clear how the downsampled version of the features was achieved. The text suggests that it has merely been downsampled from the high-resolution version, but then it is not clear which role the new proposed architecture plays. Please rephrase for clarity.\n\n\n- Line 32: “Specifically, these works are derived using a continuous domain down-scaling operation, i.e., there is no need to consider anti-aliasing. However, when performing a down-scaling, the Nyquist theorem [21, 27] tells us that an anti-alias filter is necessary to avoid high-frequency content to alias into lower frequencies. (...) To address this gap, from prior work, we consider the down-scaling operation directly in the discrete domain taking into account of the anti-aliasing.” Several grammar errors and the paragraph is unclear to me. In the first sentence, the authors write that previous works “do not need to consider anti-aliasing”. But then, they aim to bridge the gap and take anti-aliasing into account. Please fix the formulation as it is currently very unclear whether previous works had an issue with anti-aliasing or not.\n\n\n- Line 103: The notation for the definition of g is unclear. It is unclear what is meant by that x is \\in {R^1, R^2 … R^N}, which makes it hard to understand the following definitions. The notation of {..} suggests that x can be from R^1, but then how is the Fourier transform of one number even defined? The Fourier transform is needed for D_R(x). I am confused by this definition, please rewrite for clarity. \n\n\n- It would be helpful if the authors would state that small x refers to the spatial domain and big X always refers to the Fourier domain. While this is the common practice, it would still be nice to state it explicitly.\n\n\n- I don’t understand Fig. 2b. Are the yellow regions ones and the gray ones zero?\n\n\n- Line 142: “Element-wise non-linearities, e.g., ReLU, in the spatial domain are generally not scale-equivariant under the ideal downsampling operation DR.” The reasoning of this claim is not obvious to me, please explain / give some intuition.\n\nQUESTIONS:\nI encourage the authors to respond to the points I raised on weaknesses and fix minor text errors. I am willing to raise my score if the authors adequately respond to my concerns.\n\nLIMITATIONS:\nThe authors have not included a Broader Impact section and did not discuss the potential negative societal impact of their work. They also do not have a discussion section where some limitations of their work could have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper introduces Truly Scale-Equivariant Deep Nets with Fourier Layers, which directly formulate down-scaling in the discrete domain to address the anti-aliasing problem. The proposed methods are validated on the MNIST-scale and STL-10 datasets, demonstrating good classification performance while preserving zero equivariance error.

STRENGTHS:
1. This paper addresses how to learn scale-equivariant representations in deep neural networks, overcoming the limitation of previous methods that did not account for anti-aliasing.
2. The paper proposes a method that can consider the down-scaling operation directly in the discrete domain with anti-aliasing and achieves good performance.

WEAKNESSES:
1. This paper does not provide sufficient experiments to demonstrate the generality of the proposed method. It only tests on two small datasets and a single high-level vision task (image classification), which limits the impact of the proposed method. This paper should also evaluate the method on more larger-scale datasets, more high-level vision tasks (e.g., image segmentation), and more CNN architectures.
2. Why does the baseline use zero-padding to the original resolution as the input, instead of resizing to the original resolution? Does this affect baseline's representation learning and performance?
3. The paper lacks an analysis of the computational cost of the proposed method and other compared baseline during training and inference.

QUESTIONS:
Please refer to the weakness part.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submitted paper explores the problem of scale-equivariant neural networks.\nTo get the desired property by design, the authors propose new building blocks: a spatially local Fourier layer, a novel scale-equivariant pooling layer, and scale-equivariant way of applying activation functions.\nAll the introduced modifications rely heavily on the Fourier transform of the input feature map. \nAdditionally, a new loss function is proposed which aims to promote the natural consistency of predictions made at different scales. \n\nThe effect of the new approach is demonstrated on the problem of image classification on two datasets: MNIST-scale and STL-10-scale.\n\nSTRENGTHS:\nI find the proposed modifications well explained and motivated. \nTo the best of my knowledge, the presented approach is original enough, and it may be interesting for the broad community. \nQuality of the results seems very satisfactory and I hope they can serve as a reliable baseline for further research on this topic.\n\nWEAKNESSES:\n1. While the proposed results seem strong enough for the task of object recognition, the authors have not reported the computational or memory overhead introduced by their approach. Without this knowledge, it is hard to understand if their method can be directly scaled to more high-resolution data without any significant changes.\n\n1. I admit that most prior papers also mostly report their performance for image classification, it is quite clear that the property of scale equivariance may be even more desirable for the dense prediction tasks, e.g. semantic segmentation. Are there any barriers to applying the same method to the common segmentation architectures?\n\n1. Regarding the low performance of baseline models on STL-10-scale data (line 274), have the authors tried other padding schemes than zero-padding, e.g. so-called reflect/replicate/circular strategies (see the [link](https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html))? I find it suspicious that, e.g. the reported DISCO performance of 91.93% accuracy on STL-10 [1] drops to 47.68% on STL-10-scale (Tab. 4) in the provided evaluation results. I think, more discussion is needed on that matter.\n\n[1] Sosnovik et al. DISCO: accurate Discrete Scale Convolutions. In BMVC, 2021.\n\nQUESTIONS:\nPlease, address the aforementioned weaknesses.\n\nLIMITATIONS:\nNo additional inquiries.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a formulation of scale equivariant networks, which takes anti-aliasing into account. This is in contrast to previous works, in which the issue of aliasing is often disregarded. The proposed method achieves competitive classification accuracy on MNIST-scale and STL-10, while maintaining zero equivariance error.\n\nSTRENGTHS:\nThis work addresses a problem that has been thus far disregarded when talking about scale equivariance: the aliasing issue. The proposed methods and solutions are sound and are empirically evaluated with very good results.\n\nWEAKNESSES:\n* My main concern is that, to the best of my understanding, the proposed framework does not consider inter-scale relationships. Instead, it considers multiple scales in parallel which are processed independently of one another. This brings into question whether this work should be defined as a work in group equivariance or rather a work on models aiming to be resolution-agnostic, i.e., able to work regardless of the resolution of the input, e.g., FlexConv [1], S4ND[2], \\inf-Diffusion [3].\n\n* This method relies heavily on using the Fourier and inverse Fourier transform for each of its layers. In addition, it is well-known that performing convolutions on the spatial domain is faster and requires less memory than performing them in the Fourier domain when convolutional kernels are small O(Nk) vs O(NlogN). However, the proposed method actually relies on small convolutional kernels. These two factors call into question the computational and memory complexity of the proposed method. I believe that this is the main practical limitation of the method, and I would like the authors to discuss this in the paper –in case it is indeed an issue–.\n\n* Connecting to the previous point, I believe that having a section in which the limitations of the work are clearly stated and discussed is vital.\n\nQUESTIONS:\n* I find the definition of the convolutional kernels (Eq. 14) too convoluted. I would simply say that you define a kernel of size k in the spatial domain and send it to the Fourier domain next.\n\n* There are several works that tackle the issue of downsampling breaking equivariance (even in a general group setting) as well as tackling aliasing in many other settings, e.g.,[1,2,3,4,5,6,7] among many more. I believe that discussing these works would be beneficial to scope the paper better.\n\n* In line 213, you mention that you do the analysis of Sosnovik et al on trained models. Although I understand your reasoning, I actually believe that treating non-trained networks directly, i.e., random nets, makes more sense in order to evaluate the real capability of the networks to remove aliasing and preserve equivariance. Have you done this analysis as well? What were the results? Did you still get zero equivariance error?\n\n* I believe line 165 has an error.\n\n* The paper has some small typos. Please correct them.\n\nLIMITATIONS:\nAs mentioned previously, the authors have not listed any limitations of their work in the paper.\n\n### Conclusion\n\nIn conclusion, I think this is an interesting paper with important contributions to the field. However, I am not sure if this paper should be framed as a scale-equivariant paper or rather as a resolution-agnostic neural architecture. I believe –please do tell me if you don't agree– that the latter option is more accurate. Although it requires some rewriting, I believe it would be possible to still do this during the rebuttal period.\n\n#### References\n\n[1] Romero, D. W., Bruintjes, R. J., Tomczak, J. M., Bekkers, E. J., Hoogendoorn, M., & van Gemert, J. C. (2021). Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059.\n\n[2]  Nguyen, Eric, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. \"S4nd: Modeling images and videos as multidimensional signals with state spaces.\" Advances in neural information processing systems 35 (2022): 2846-2861.\n\n[3] Bond-Taylor, S., & Willcocks, C. G. (2023). $\\infty $-Diff: Infinite Resolution Diffusion with Subsampled Mollified States. arXiv preprint arXiv:2303.18242.\n\n[4] Karras, Tero, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. \"Alias-free generative adversarial networks.\" Advances in Neural Information Processing Systems 34 (2021): 852-863.\n\n[5] Zhang, R., 2019, May. Making convolutional networks shift-invariant again. In International conference on machine learning (pp. 7324-7334). PMLR.\n\n[6] Xu, J., Kim, H., Rainforth, T. and Teh, Y., 2021. Group equivariant subsampling. Advances in Neural Information Processing Systems, 34, pp.5934-5946.\n\n[7] Rojas-Gomez, Renan A., Teck-Yian Lim, Alex Schwing, Minh Do, and Raymond A. Yeh. \"Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks.\" Advances in Neural Information Processing Systems 35 (2022): 35755-35768.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}