Global Filter Networks for Image Classification
Authors: Yongming Rao
Wenliang Zhao
Zheng Zhu
Jiwen Lu
Jie Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: image classification
network architecture

Abstract: Recent advances in self-attention and pure multi-layer perceptrons (MLP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard to scale up when high-resolution features are required. In this paper, we present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: a 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and a 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-offs of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available at https://github.com/raoyongming/GFNet

Global Filter Networks for Image Classification
Yongming Rao"
Wenliang Zhao' Zheng Zhu
Jiwen Lu Jie Zhou
Department Automation, Tsinghua University State Key Lab of Intelligent Technologies and Systems Beijing National Research Center for Information Science and Technology
Abstract
Recent advances self-attention and pure multi-layer perceptrons (MIP) models for vision have shown great potential in achieving promising performance with fewer inductive biases. These models are generally based on learning interaction among spatial locations from raw data. The complexity of self-attention and MLP grows quadratically as the image size increases, which makes these models hard scale up when high-resolution features are required. In this paper, present the Global Filter Network (GFNet), a conceptually simple yet computationally efficient architecture, that learns long-term spatial dependencies in the frequency domain with log-linear complexity. Our architecture replaces the self-attention layer in vision transformers with three key operations: 2D discrete Fourier transform, an element-wise multiplication between frequency-domain features and learnable global filters, and 2D inverse Fourier transform. We exhibit favorable accuracy/complexity trade-off's of our models on both ImageNet and downstream tasks. Our results demonstrate that GFNet can be very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness. Code is available https: thub. www.coo///////e comm/raoongmmnn// heapangginng///ee..... /GFNet.
Introduction
The transformer architecture, originally designed for the natural language processing (NLP) tasks [42]. has shown promising performance on various vision problems recently [10, 40, 27, 49, 4, 47, 35, 5]. Different from convolutional neural networks (CNNs), vision transformer models self-attention layers to capture long-term dependencies, which are able learn more diverse interactions between spatial locations. The pure multi-layer perceptrons (MLP) models [38, 39] further simplify the vision transformers by replacing the self- aeff-attention layers with MLPs that are applied across spatial locations. Since fewer inductive biases introduced, these two kinds of models have the potential to learn more generic and ficxible interactions among spatial locations from raw data.
One primary challenge of applying self-attention and pure MIP models to vision tasks is the considerable computational complexity that grows quadratically as the number of tokens increases. Therefore, typical vision transformer style models usually consider relatively small resolution for the intermediate features (e.g. 14 x 14 tokens are extracted from the input images both ViT [10] and MLP-Mixer [38]). This design may limit the applications of downstream dense prediction tasks like detection and segmentation.
possible solution to replace the global self-attention with several local self-attention like Swin transformer [27]. Despite the effectiveness in practice, local self-attention brings quite few aane-made choices (e.g., window size, padding strategy, etc.) and limits the receptive field of each layer.
EEqual contribution.
CCoreespoddigg author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Linear
Nx
Global Average Pooling
Layer Norm
FFN
Feed Forward Network (FFN)
2D IFFT
Global Filter Layer
frequency domain features
learnable global filters
Patch Embedding
2D FFT
Layer Norm
Global Filter Layer
Figure The overall architecture of the Global Filter Network. Our architecture based on Vision Transformer (ViT) models with some minimal modifications. We replace the self-attention sub-layer with the proposed global filter layer, which consists of three key operations: 2D discrete Fourier transform to convert the input spatial features to the frequency domain, element-wise multiplication between frequency-domain features and the global filters, and 2D inverse Fourier transform to map the features back to the spatial domain. The efficient fast Fourier transform (FFT) enables us to learn arbitrary interactions among spatial locations with log-lineat complexity.
In this paper, we present new conceptually simple yet computationally efficient architecture called Global Filter Network (GFNer), which follows the trend of removing inductive biases from vision models while enjoying the log-linear complexity computation. The basic idea behind our architecture is to learn the interactions among spatial locations in the frequency domain. Different from the self-attention mechanism in vision transformers and the fully connected layers in MLP models, the interactions among tokens are modeled to the spectrum of the input features. Since the global filters are able to cover all the frequencies, model can capture both long-term and short-term interactions. The filters are directly learned from the raw data without introducing human priors. Our architecture is largely based on the vision transformers only with some minimal modifications. We replace the self-attention sub-layer in vision transformers with three key operations: a 2D discrete Fourier transform to convert the input spatial features the frequency domain, element-wiss multiplication between frequency-don features and the global filters, and a 2D inverse Fourier transform to map the features back to the spatial domain. Since the Fourier transform used mix the information of different tokens, the global filter is much more efficient compared to the self- attention and MLP thanks to the OL log L) complexity of the fast Fourier transform algorithm (FFT) [7]. Benefiting from this, the proposed global filter layer is less sensitive to the token length L and thus is compatible with larger feature maps and CNN-style hierarchical architectures without modifications. The overall architecture of GFNet is illustrated in Figure 1. We also compare our global filter with prevalent operations in deep vision models in Table
set of learnable global filters that are applied
Our experiments on ImageNet verify the effectiveness of GFNet. With a similar architecture, our model outperform the recent vision transformer and MLP models including DeiT' [40], ResMLP [39] and gMLP [26]. When using the hierarchical architecture, GFNet can further enlarge the gap. GFNet also works well downstream transfer learning and semantic segmentation tasks. Our results demonstrate that GFNet can be very competitive alternative to transformer-style models and CNNs in efficiency, generalization ability and robustness.
Related works
Vision transformers. Since Dosovitskiy al. [10] introduce transformers to the image classification and achieve competitive performance compared to CNNs, transformers begin to exhibit their

---

Complexity (FLOPs)
# Parameters
Depthwise Convolution
O HWD) D) O(HWD² + H) O(H2W
k2
Self-Attention
4D²
Spatial MLP
H²
Global Filter
O [[og, + HWD)
HWD
Table I:
Comparisons of the proposed Global Filter with prevalent operations in deep vision models. H, W and D are the height, width and the number of channels of the feature maps. l is the kernel size of the convolution operation. The proposed global filter is much more efficient than self-attention and spatial MLP.
potential in various vision tasks [3. 4, 49]. Recently, there are a large number of works which aim to improve the transformers [40, 41, 27, 44, 18, 11, 48]. These works either seek for better training strategies [40, 11] or design better architectures [27, 44, 48] or both [41, 11]. However, most of the architecture modification of the transformers [44, 18, 27, 48] introduces additional inductive biases similar to CNNs. In this work, we only focus on the standard transformer architecture [10, 40) and our goal is to replace the heavy self-attention layer (O()2) an more efficient operation which can still model the interactions among different spatial locations without introducing the inductive biases associated with CNNs.
MLP-like models. More recently, there are several works that question the importance of self- attention in the vision transformers and propose to use MLP to replace the self-attention layer in the transformers [38, 39, 26]. The MLP-Mixer [38] employs MLPs to perform token mixing and channel mixing alternatively in each block. ResMLP [39] adopts similar idea but substitutes the Layer Normalization with Affine transformation for acceleration. The recently proposed gMLP [26] uses a spatial gating unit to re-weight tokens in the spatial dimension. However, all of the above models include MLPs to mix the tokens spatially, which brings two drawbacks: (1) like the self-attention in the transformers, the spatial MLP still requires computational complexity quadratic to the length of tokens. (2) unlike transformers, MLP models are hard to scale up to higher resolution since the weights of the spatial MLPs have fixed sizes. Our work follows this trend and successfully resolves the above issues in MLP-like models. The proposed GFNet enjoys log- linear complexity and can be easily scaled up to any resolution.
Applications of Fourier transform in vision. Fourier transform has been an important tool in digital image processing for decades [32, 1J. With the breakthroughs of CNNs in vision [14, 13], there are a variety of works that start to incorporate Fourier transform in some deep learning method [24, 46, 9, 22, 6] for vision tasks. Some of these works employ discrete Fourier transform to convert the images to the frequency domain and leverage the frequency information to improve the performance in certain tasks [22, 46), while others utilize the convolution theorem to accelerate the CNNs via fast Fourier transform (FFT) [24, 9]. FFC [6] replaces the convolution in CNNs with an Local Fourier Unit and perform convolutions in the frequency domain. Very recent works also try to leverage Fourier transform to develop deep learning models to solve partial differential equations [25] and NLP tasks [23]. In this work, we propose to use learnable filters to interchange information globally among the tokens in the Fourier domain, inspired by the frequency filters in the digital image processing [32]. We also take advantage of some properties of FFT to reduce the computational costs and the number of parameters.
3 Method
3.1 Preliminaries: discrete Fourier transform
We start by introducing the discrete Fourier transform (DFT), which plays an important role in the area of digital signal processing and is crucial component in our GFNet. For clarity, We first consider the ID DFT. Given sequence of N complex numbers æ[n],00 0 S < N - 1, the ID DFT converts the sequence into the frequency domain by:
N-1
N --1
X = Σ (n2e-=(2=/NN)a
Σ
x x|n] W
(3.1)
n=0
3

---

Algorithm Pseudocode of Global Filter Layer.
# x: the token features, B H x x (chare X H W) # K: the Trequency-dosaia rilter, B M_hat D (where W_bat // see Section 3.2 for detaile)
x rfft2(x, dim=(1, 2))
x_tilde x
irffe2(K_tidde, dim- (1, 2))
xrfc2/intre2: FFTMIFFT for real signal
where j the imaginary unit and WN = - The formulation of DFT in Equation (3.1) can be derived from the Fourier transform for continuous signal by sampling in both the time domain and the frequency domain (see Appendix for details). Since [[k] repeats on intervals of length N,
is suffice to take the value of [[k at N consecutive points k = 0, 1, .......... N - 1. Specifically, [[k
represents the spectrum of the sequence æ|n] at the frequency wi 2rk/N. /N
It is also worth noting that DFT is a one-to-one transformation. Given the DFT X we can recover the original signal s/n] by the inverse DFT (IDFT):
N
[n]
Σ k=
x [ (27/))k)
(3.2)
N
For real input æ(n].. it can be proved that (see Appendix A) its DFT is conjugate symmetric, i.e., X[N [*]].. The reverse is true as well: we perform IDFT to X(k) which conjugate symmetric, real discrete signal can be recovered. This property implies that the half of the DFT {X[A] : 0 < k < [N/2]] contains the full information about the frequency characteristics of x x[n).
DFT is widely used in modern signal processing algorithms for mainly two reasons: (I) the input and output of DFT are both discrete thus can be easily processed by computers; (2) there exist efficient algorithms for computing the DFT. The fast Fourier transform (FFT) algorithms take advantage of the symmetry and periodicity properties of W and reduce the complexity to compute DFT from O(N²) to O(N log N). The inverse DFT (3.2), which has a similar form to the DFT, can also be computed efficiently using the inverse fast Fourier transform (IFFT).
The DFT described above can be extend to 2D signals. Given the 2D signal X|m, n).0 0 & m S M 1,0 < I N the 2D DFT of x: x/m, n] is given by:
M- 1 M-IN-1
X (iu v) = Σ Σ
x|m, nle -j2n 47 #)
(3.3)
TPL O m
The 2D DFT can be viewed as performing ID DFT on the two dimensions alternatively. Similar to ID DFT, 2D DFT of real input x æ[m, ni satisfied the conjugate symmetry property X[M u, N - -]
X [u, v). The FFT algorithms can also be applied to 2D DFT to improve computational efficiency.
3.2 Global Filter Networks
Overall architecture. Recent advances in vision transformers [10, 40] demonstrate that models based on self-attention can achieve competitive performance even without the inductive biases associated with the convolutions. Henceforth, there are several works [39, 38] that exploit approaches (e.g., MLPs) other than self-attention to mix the information among the tokens. The proposed Global Filter Networks (GFNet) follows this line of work and aims to replace the heavy self-attention layer (N2)) with a simpler and more efficient one.
The overall architecture of our model is depicted in Figure 1. Our model takes as an input / x W non-overlapping patches and projects the flattened patches into L H tokens with dimension D. The basic building block of GFNet consists of: I) a global filter layer that can exchange spatial information efficiently (O(L L)); 2) feedforward network (FFN) as in [10, 40]. The output tokens of the last block are fed into global average pooling layer followed by a linear classifier.
Global filter layer. We propose global filter layer as an alternative the self-attention layer which can mix tokens representing different spatial locations. Given the tokens E H x
first

---

perform 2D FFT (see Section 3.1) along the spatial dimensions to convert æ to the frequency domain:
X == F(a] E CHx
(3.4)
where F[|] denotes the 2D FFT. Note that X is complex tensor and represents the spectrum of We can then modulate the spectrum by multiplying a learnable filter K E CHxWxD to the X:
X K X,
(3.5)
where @ is the element-wise multiplication (also known as the Hadamard product). The filter K is called the global filter since it has the same dimension with X, which can represent an arbitrary filter in the frequency domain. Finally, we adopt the inverse FFT to transform the modulated spectrum X back to the spatial domain and update the tokens:
x
[X].
(3.6)
The formulation of the global filter layer is motivated by the frequency filters in the digital image processing [32], where the global filter K can be regarded as a set of learnable frequency filters for different hidden dimensions. can proved (see Appendix A) that the global filter layer is equivalent to a depthwise global circular convolution with the filter size H x W. Therefore, the global filter layer is different from the standard convolutional layer which adopts a relatively small filter size to enforce the inductive biases of the locality. We also find although the proposed global filter can also be interpreted as a spatial domain operation. the filters learned in our networks exhibit more clear patterns in the frequency domain than the spatial domain, which indicates our models tend to capture relation in the frequency domain instead of spatial domain (see Figure 4). Note that the global filter implemented in the frequency domain is also much more efficient compared the spatial domain, which enjoys a complexity of O(DL log L) while the vanilla depthwise global circular convolution in the spatial domain has ODL2²) complexity. We will also show that the global filter layer is better than its local convolution counterparts in the experiments.
It is also worth noting that in the implementation, we make use of the property of DFT to reduce the redundant computation. Since x is a real tensor, its DFT X is conjugate symmetrie, i.e. X[H - u, W - v, : = X [H, W. Therefore, we can take only the half of the values in the X but preserve the full information at the same time:
x, X(: == F1 W= = [W/2],
(3.7)
where F, denotes the 2D FFT for real input. In this way, we can implement the global filter as K2 E (C#xWe DD which can reduce half the parameters. This can also ensure F, [K, @ X,] is real tensor, thus it can be added directly to the input æ. The global filter layer can be easily in modern deep learning frameworks (e.g.. Py Torch [31]), as is shown in Algorithm 1. The FFT and IFFT are well supported by GPU and CPU thanks to the acceleration libraries like cuFFT and mkl-fft, which makes our models perform well on hardware.
Relationship to other transformer-style models. The GFNet follows the line of research about the exploration of approaches to mix the tokens. Compared to existing architectures like vision transformers and pure MLP models, we exhibit that GFNet has several favorable properties: I) GFNet is more efficient. The complexity of both the vision transformers [10, 40, 41] and the MLP models [38, 39] is O(L²). Different from them, global filter layer only consists an FFT (O L)), element-wise multiplication (OL)) and an IFFT (O log L)), which means the total computational complexity is OLL log L). 2) Although pure MLP models are simpler compared to transformers, it is hard to fine-tune them on higher resolution (e.g., from 224 224 resolution to 384 x 384 resolution) since they can only process a fixed number of tokens. As opposed to pure MLP models, we will show that our GFNet can be easily scaled up to higher resolution. Our model is more flexible since both the FFT and the IFFT have no learnable parameters and can process sequences with arbitrary length. We can simply interpolate the global filter K to K' E CA' xW' x D for different inputs, where H' x W' is the target size. The interpolation is reasonable due to the property of DFT. Each element of the global filter K(u, w] corresponds to the spectrum of the filter at W. = 2Tu/ H, Wp = 2Tv/ W and thus, the global filter K can be viewed as a sampling of a continuous spectrum K ((w,, wr),, where wn. W, E [0, 2nr), Hence, changing the resolution is equivalent to changing the sampling interval of K (ww,wo)) Therefore, we only need to perform interpolation to shift from one resolution to another.
5

---

Table Detailed configurations of different variants of GFNet. For hierarchical models, we provide the number of channels and blocks in 4 stages. The FLOPs are calculated with 224 x 224 input.
Model
#Blocks
#Channels
Params (M)
FLOPs (G)
GFNet-Ti
12
256
1.3
GFNet-XS
12
384
16
2.9
GFNet-S
19
384
4.5
GFNet-B
19
512
43
7.9
GFNet-H-Ti
[3, 10,
[64, 128, 256, 512]
15
2.1
GFNet-H-S
[3, 10, 3]
[96. 192, 384, 768]
32
4.6
GFNet-H-B
[3, 3, 27, 3]
[96, 192, 384, 768]
54
8.6
We also notice recently concurrent work FNet [23] leverages Fourier transform mix tokens. Our work is distinct from FNet in three aspects: (1) FNet performs FFT to the input and directly adds the real part of the spectrum the input tokens, which blends the information from different domains (spatial/frequency)) together. On the other hand, GFNet draws motivation from the frequency filters, which is more reasonable. (2) FNet only keeps the real part of the spectrum. Note that the spectrum of real input is conjugate symmetric, which means the real part is exactly symmetric and thus contains redundant information. Our GFNet, however, utilizes this property simplify the computation. (3) FNet is designed for NLP tasks, while our GFNet focuses on vision tasks. In our experiments, we also implement the FNet and show that our model outperforms
Architecture variants. Due to the limitation from the quadratic complexity in the self-attention vision transformers [10, 40] are usually designed to process a relatively small feature map (e.g., 14 x 14). However, our GFNet, which enjoys log-linean complexity, avoids that problem. Since in our GFNet the computational costs do not grow such significantly when the feature map size increases, we adopt hierarchical architecture inspired by the success of CNNs [21, 14]. Generally speaking, we can start from a large feature map (e.g.. 56 x 56) and gradually perform downsampling after few blocks. In this paper, we mainly investigate two kinds of variants of GFNet: transformer-style models with a fixed number of tokens in each block and CNN-style hierarchical models with gradually downsampled tokens. For transformer-style models, we begin with 12-layer model (GFNei-XS) with a similar architecture with DeïT-S and ResMLP-12. Then, we obtain 3 variants of the model (GFNet-Ti, GFNet-S and GFNer-B) by simply adjusting the depth and embedding dimension, which have similar computational costs with ResNet- 18, 50 and 101 [14]. For hierarchical models, we also design three models (GFNet-H-Tr, GFNer-H-S and GFNer-H-BD that have these three levels of complexity following the design of PVT [43]. We use 4 x 4 patch embedding to form the input tokens and use non-overlapping convolution layer to downsample tokens following [43, 27]. Unlike PVT [43] and Swin [27], we directly apply our building block on different stages without any modifications. The detailed architectures are summarized in Table
4 Experiments
We conduct extensive experiments to verify the effectiveness of our GFNet. We present the main results on ImageNet [8] and compare them with various architectures. We also test our models on the downstream transfer learning datasets including (IFAR-10/100 [20], Stanford Cars [19] and Flowers- 102 [30]. Lastly, we investigate the efficiency and robustness of the proposed models and provide visualization to have an intuitive understanding of our method.
4.1 ImageNet Classification
Setups. We conduct our main experiments on ImageNet [8], which is widely used large-scale scale benchmark for image classification. ImageNet contains roughly 1.2M images from 1,000 categories. Following common practice [14, 40], we train our models on the training set of ImageNet and report the single-crop top-1 accuracy 50,000 validation images. To fairly compare with previous works [40, 39], we follow the most training details for our models and not add extra regularization methods like [18]. Different from [40], we does not use EMA model [33], RandomEarse [50] and

---

Table Comparisons with transformer-style architectures on ImageNet. We compare different transformer-style architectures for image classification including vision transformers [40], MLP-like models [39, 26] and our models that have comparable FLOPS and the number parameters. We report the top-1 accuracy on the validation set of ImageNet as well as the number of parameters and FL.Oss. All of our models are trained with 224 224 images. We use "1384" to represent models
finetuned on 384 x 384 images for 30 epochs.
Model
Params (M)
FLOPs (G)
Resolution
Top-1 Acc. (%)
Top-5 Acc. (%)
DeiT-Ti [40]
1.2
224
72.2
91.1
gMLP-Ti [26]
1.4
224
72.0
GFNet-Ti
1.3
224
74.6
92.2
ResMLP-12 [39]
15
3.0
224
76.6
GFNet-XS
16
2.9
224
78.6
94.2
DeïT-S [40]
22
4.6
224
79.8
95.0
gMLP-S [26]
20
4.5
224
79.4
GFNet-S
25
4.5
224
80.0
94.9
ResMLP-36 [39]
45
8.9
224
79.7
GFNet-B
43
7.9
224
80.7
95.1
GFNe1-XS1384
18
8.4
384
80.6
95,4
DeïT-B [40]
86
17.5
224
81.8
95.6
gMLP-B [26]
73
15.8
224
81.6
GFNet-S1384
28
13.2
384
81.7
95.8
3FN
47
23.3
384
82.1
95.8
Table 4: Comparisons with hierarchical architectures on ImageNet. We compare different hi- erarchical architectures for image classification including convolutional neural networks [14, 34], hierarchical vision transformers [43, 27] and our hierarchical models that have comparable FLOPs and number of parameters. We report the top-1 accuracy on the validation set of ImageNet as well as the number of parameters and FLOPs. All models are trained and tested with 224 x 224 images.
Model
Params (M)
FLOPs (G)
Top-1 Acc. (%)
Top-5 Acc. (%)
ResNet-18 [14]
1.8
69.8
89.1
RegNetY-1.6GF [34]
11
1.6
78.0
PVT-Ti [26]
13
1.9
75.1
GFNet-H-Ti
15
2.1
80.1
95.1
ResNet-50 [40]
26
4.1
76.1
92.9
RegNetY-4.0GF [34]
21
4.0
80.0
PVT-S [26]
25
3.8
79.8
Swin-Ti [27]
29
4.5
81.3
GFNet-H-S
32
4.6
81.5
95.6
ResNet-101 [40]
45
7.9
77.4
93.5
RegNetY-8.0GF [34]
39
8.0
81.7
PVT-M [26]
44
6.7
81.2
Swin-S [27]
50
8.7
83.0
GFNet-H-B
54
8.6
82.9
96.2
repeated augmentation [17], which are important to train DeiT while sightly hurting the performance of our models. We the gradient clipping norm for all of our models. During finetuning the higher resolution, we use the hyper-parameters suggested by the implementation of [40] and train the model for 30 epochs. All of our models are trained on single machine with 8 GPUs. More details can be found in Appendix B.
Comparisons with transformer-style architectures. The results are presented in Table 3. We compare method with different transformer-style architectures for image classification including

---

vision transformers (DeiT [40]) and MLP-like models (ResMLP [39] and gMLP [26]) that have similar complexity and number of parameters. We see that our method can clearly outperform recent MI.P-like models such ResMLP [39] and gMLPP [26], and show similar performance to DeiT. Specifically, GFNet-XS outperforms ResMLP-12 by 2.0% while having slightly fewer FLOPs. GFNet-S also achieves better top-1 accuracy compared to gMLPP- and DeiT-S. Our tiny model is significantly better compared to both DeïT-Ti (+2.4%) and gMLP-Ti (+2.6%) with the similar level of complexity.
Comparisons with hierarchical architectures. We compare different kinds of hierarchical models in Figure 4. ResNet [14] is the most widely used convolutional model while RegNet [34] is family of carefully designed CNN models. We also compare with recent hierarchical vision transformers PVT [43] and Swin [27]. Benefiting from the log-linear complexity, GFNet-H models show signifi- cantly better performance than ResNet, RegNet and PVT and achieve similar performance with Swin while having much simpler and more generic design.
Fine-tuning at higher resolution. One prominent problem of MLP-like models is that the feature resolution is not adjustable. On the contrary, the proposed global filter more flexible. We demonstrate the advantage of GFNet by finetuning the model trained at 224 x 224 resolution to higher resolution following the practice in vision transformers [40]. shown in Table 3, our model can easily adapt to higher resolution with only 30 epoch finetuning and achieve better performance.
Table 5: Results on transfer learning datasets. We report the top-1 accuracy on the four datasets
well the number of parameters and FLOPs.
Model
FLOPs
Params
CIFAR-10
CIFAR 100
Flowers- 102
Cars-
ResNet50 [14]
4.1G
26M
96.2
90.0
EfficientNet-B7 [37]
37G
66M
98.9
91.7
98.8
94.7
VIT-B/16 [10]
55.4G
86M
87.1
89.5
ViT-L/16 [10]
190.7G
307M
97.9
86.4
89.7
Deit-B/16 [40]
17.5G
86M
99.1
90.8
98.4
92.1
ResMLP-12 [39]
3.0G
15M
98.1
87.0
97.4
84.6
ResMLP-24 [39]
6.0G
30M
98.7
89.5
97.9
89.5
GFNet-XS
2.9G
16M
98.6
89.1
98.1
92.8
G-Net-H-B
8.6G
54M
99.0
90.3
98.8
93.2
4.2 Transfer learning
To test the generality of our architecture and the learned representation, we evaluate GFNet on set of commonly used transfer learning benchmark datasets including CIFAR-10 [20], CIFAR- 1-0 [20], Stanford Cars [19] and Flowers- 102 [30]. We follow the setting of previous works [37, 10, 40, 39], where the model is initialized by the ImageNet pre-trained weights and finetuned on the new datasets. We evaluate the transfer learning performance of our basic model and best model. The results presented in Table 5. The proposed models generally work well on downstream datasets. GFNet models outperform ResMLP models by large margin and achieve very competitive performance with state-of-the-art EfficientNet-B7. Our models also show competitive performance compared to state-of-the-arr CNNs and vision transformers.
4.3 Analysis and visualization
Efficiency of GFNet. We demonstrate the efficiency of our GFNet in Figure where the models compared in theoretical FLOPs, actual latency and peak memory usage on GPU. We test a single building block of each model (including one token mixing layer and one FFN) with respect the different numbers of tokens and set the feature dimension and batch size to 384 and 32 respectively. The self-attention model quickly runs out of memory when feature resolution exceeds 562, which is also the feature resolution of our hierarchical model. The advantage of the proposed architecture becomes larger as the resolution increases, which strongly shows the potential of our model vision tasks requiring high-resolution feature maps.

---

VIT
VIT
VIT
803
TeafLP
ResNLLP GFNet
ReaRLL GPNe:
Vet
GFLOPS
Latency (5
6.8
Memory (GI)
62-
207
327
247
1324
3427
188
287
1407
Number of tokiens
Number tokens
(b)
(c)
Figure 2: Comparisons among GFNet, VIT [10] and ResMLP [39] in (a) FLOPs (b) latency and (c) GPU memory with respect to the number of tokens (feature resolution). The dotted lines indicate the estimated values when the GPU memory has run out. The latency and GPU memory is measured using a single NVIDIA RTX 3090 GPU with batch size 32 and feature dimension 384.
Table 6: Comparisons among the GFNet and other variants based on the transformer-lis architecture ImageNet. We show that GFNet outperforms the ResMLP [39], FNet [23] and models with local depth- wise convolutions. We also report the number of pa-
ImageNet Acc (9%)
rameters and theoretical complexity in FLOPs.
18.
Acc (%)
Param (M)
FLOPs (G)
Model
DeiT-S [40]
79.8
22
4.6
Local Cony (3 x 3)
77.7
15
2.8
m
Local Conv (5 x 5)
78.1
15
2.9
ouro
Dell
Local Conv (7 x 7)
78.2
15
2.9
ResMLP
qMLP
ResMLP [39]
76.6
15
3,0
0.0
25
5.0
75
10.0
12.5
15.0
11s
PNet [23]
71.2
15
2.9
GFLOPs
Figure 3: ImageNet acc. vs model complexity.
GFNet-XS
78.6
16
2.9
Complexity/accuracy trade-offs. We show the computational complexity and accuracy trade-off's of various transformer-style architectures in Figure 3. It is clear that GFNet achieves the best trade-off among all kinds of models.
Ablation study on the global filter. To more clearly show the effectiveness of the proposed global filters, we compare GFNet-XS with several baseline models that are equipped with different token mixing operations. The results are presented in Table 6. All models have similar building block ( token mixing layer + FFN and the same feature dimension of = 384. We also implement the recent FNet [23] for comparison, where ID FFT on feature dimension and 2D FFT spatial
dimensions are used to mix tokens. As shown in Table 6, our method outperforms all baseline methods except DeiT-S that has 64% higher FLOPs.
Robustness & generalization ability. Inspired by the [29], we further conduct experiments to evaluate the robustness and the generalization ability of the GFNet. For robustness, consider ImageNet-A, ImageNet-C, FGSM and PGD. ImageNet-A [16] (IN-A) a challenging dataset that contains natural adversarial examples. ImageNet-C [15] (IN-C) is used validate the robustness of the model under various types of corruption. We use the mean corruption error (mCE, lower is better) on ImageNet-C
the evaluation metric. FGSM [12] and PGD [28] are two widely used algorithms that are targeted to evaluate the adversarial robustness of the model by single-step attack and multi- step attack, respectively. For generalization ability, we adopt two variants of ImageNet validation set: ImageNet- V2 [36] (IN-V2) and ImageNet-Reall [2] (IN-Real), ImageNet- V2 is re-collected version of ImageNet validation set following the same data collection procedure of ImageNet, while ImageNet-Reat contains the same images as ImageNet validation set but has reassessed labels. We compare GFNet-S with various baselines in Table 7 including CNNs, Transformers and MI.P-lik architectures and find the GFNet enjoys both favorable robustness and generalization ability.

---

Table 7: Evaluation of robustness and generalization ability. We measure the robustness from different aspects, including the adversarial robustness by adopting adversarial attack algorithms including FGSM and PGD and the performance on popppte/oouuoo-ddstibbbuttt ImageNet-A [16] (top-1 accuracy) and ImageNet-C [15] (mCF, lower is better). The generalization
datasets including
ability evaluated on ImageNet-Ve V2 [36] and ImageNet- [2].
FLOPs
Params
ImtgeNet
Generalization
Robustness
Model
(G)
(M)
Top-1f
Top-S1
IN-V2t
IN-Real"
FGSM
PGD
IN-CL
IN-At
ResNet-50 [14]
4.1
26
76.1
92.9
67.4
85.8
12.2
0.9
76.7
0.0
ResNeX150-32x4l [45]
4.3
25
79.8
94.6
68.2
85.2
34.7
13.5
64.7
10.7
DeiT-S [40]
4.6
22
79.8
95.0
68.4
85.6
40.7
16.7
54.6
18.9
ResMLP--12 [39]
3.0
76.6
93.2
83.3
23.9
GFNet-S
4.5
25
80.1
14.3
layer
layer
layar
layer
layer
layar
layer
layer
layer 10
layer 11
layer
(a) Frequeney domain
(b) Spatial domain
Figure 4: Visualization of the learned global filters in GFNet-XS. We visualize the original frequency domain global filters in (a) and show the corresponding spatial domain filters for the first 6 columns in (b). There are more clear patterns in the frequency domain than the spatial domain.
Visualization. The core operation GFNet the element- weise multiplication between frequency- domain features and the global filter. Therefore, it is easy to visualize and interpret. We visualize the frequency domain filters as well as their corresponding spatial domain filters in Figure 4. The learned global filters have more clear patterns in the frequency domain, where different layers have different characteristics. Interestingly, the filters in the last layer particularly focus on the low- ewwffreqennyy component. The corresponding filters in the spatial domain are less interpretable for humans.
Conclusion
We have presented the Global Filter Network (GFNer), which is a conceptually simple yet computa- tionally efficient architecture for image classification. Our model replaces the self-attention sub-layer in vision transformer with 2D FFT/IFFT and set of learnable global filters in the frequency domain. Benefiting from the token mixing operation with log-linear complexity, our architecture is highly efficient. Our experimental results demonstrated that GFNet be a very competitive alternative to vision transformers, MLP-like models and CNNs in accuracy/complexity trade-offs.
Acknowledgment
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802. in part by the National Natural Science Foundation China under Grant 621 25603, Grant 61 8226003, Grant U1813218, Grant U1713214, in part by Beijing Academy of Artificial Intelligence (BAAT), and in part by grant from the Institute for Guo Qiang, Tsinghua University.
10

---

References
[1] Gregory A Baxes. Digital image processing: principles and applications. John Wiley & Sons, Inc., 1994. 3
[2] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint urXiv:2006.07/59, 2020. 9, 10
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020. 3
[4] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR, 2021. 1,3
[5] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021.
[6] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. NeurIPS, 33, 2020.
[7] James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematies of computation, 19(90):977-301, 1965. 2
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages
2009.
[9] Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Youwei Zhuo, Chao Wang, Xuehai Qian, Yu Bai, Geng Yuan, et al. Cirenn: accelerating and compressing deep neural networks using block-cireulant weight matrices. In MICRO, pages 395-408, 2017. 3
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999, 2020. 1, 2, 3, 4, 8,
[11] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Improve vision transformers training by suppressing over-smoothing. arXiv preprint 104.
3
[12] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint
2014.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016. 3, 6, 7,8, 10
[15] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
2019. 9, 10
[16] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262-12277 2021. 9, 10
[17] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, pages 8129-8138, 2020.
[18] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint wwXi::210411085: 2021. 3,6
[19] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, pages 554-561, 2013, 6.
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009. 6. 8
11

---

[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 25:1097-1105, 2012.
[22] Jae-Han Lee, Minhyeok Heo, Kyung-Ras Rae Kim, and Chang-Su Kim. Single-image depth estimation based on fourier domain analysis. In CVPR, pages 330-339, 2018. 3
[23] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint (eX.:::210500300 2021. 3, 6,
[24] Shaohua Li, Kaiping Xue, Bin Zhu, Chenkai Ding, Xindi Gao, David Wei, and Tao Wan. Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions. In CVPR, pages 8705-8714, 2020. 3
[25] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. ICLR, 30
[26] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. arXiv preprint arXiv:2105.08030 2021. 2, 7,8
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021. 1, 6, 8
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.00 1706.06066000
2017. 9
[29] Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye, Yuan He, and Hui Xue.
Rethinking the design principles of robust vision transformer. arXiv preprint arXiv:2105.07926, 2021. 9
[30] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number classes. In ICVGIP, pages 722-729, 2008. 6,
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high performeffmm deep learning library. In NeurIPS, 2019. 5
[32] loannis Pitas. Digital image processing algorithmss and applications. John Wiley & Sons, 2000.
3,5
[33] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992. 6
[34] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, 2020. 7,
[35] Yongming Rao, Wenliang Zhao, Benlin I. Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. In NeurlPS, 2021.
[36] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pages 5389-5400. PMLR, 2019. 9, 10
[37] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, pages 6105 PMLR, 2019. 8
[38] Ily Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.0 2021. 1, 3, 4,5
[39] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint 21X1v:210..03404, 2021. 2, 3, 4, 5, 6, 8, 9, 10
12

---

[40] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprini arXiv: /2877, 2020. 1, 2, 3, 4, 5, 6, 7, 8, 10
[41] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.172:3 2021. 3,
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008 2017.
[43] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021. 6, 7.8
[44] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang Cvt: Introducing convolutions vision transformers. arXiv preprint arXiv:2103.15808 2021.
3
[45] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pages 1492-1500, 2017. 10
[46] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, pages 4085-4095, 2020. 3
[47] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In ICCV, 2021. 1
[48] Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 2021.
[49] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang. Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequee sequence perspective with transformers. arXiv preprint arXiv:2012.15840 2020. 1,3
[50] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, volume 34, pages 13001-13008, 2020. 6
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We develop a general framework for image classification in this paper. Our model not for specific applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report enror bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] We follow the common practice used in our baseline methods, where no error bars are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See our implementation details in the experiment part.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "yNGLb4sRA1o", "reviewer": "Reviewer_6RZZ", "summary": "This paper presents global filter network, a simple and efficient architecture for visual recognition. Unlike transformer-based visual recognition models that utilize self-attention to encode spatial information, this paper proposes to build long-range dependencies among spatial locations in the frequency domain by taking advantage of 2D FFT and IFFT. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The originality of this paper is clear. Encoding spatial information in the frequency domain is interesting. The classification performance on ImageNet is also good.\n\n- This paper is also well presented. It is simple and easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- In the experiment section, the authors say that strong data augmentation methods are used for training. However, these are no experiments demonstrating how each data augmentation method contributes to the model performance. I think an ablation path on data augmentation should be given.\n\n- The authors use token-wise distillation strategy while training. Have you attempted to use the original token labeling while training? Does the token-wise distillation strategy perform better than token labeling?\n\n- It is great to see the visualization of the learned filters. It would be better if the authors could provide more explanations on what the proposed approach learns in terms of the filters.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "aCGfQtqzlpk", "reviewer": "Reviewer_niRJ", "summary": "The work proposes a new attention-based architecture for computer vision (classification). The proposed architecture replaces the self-attention layer in ViT with computational blocks involving the Fourier transform. Due to the efficiency of FFT and IFFT the quadratic complexity of self-attention can be reduced to a log-linear computational cost. In order to boost performance distillation of taken features from a pre-trained vision transformer is used. Results are presented on ImageNet as well as on some transfer learning experiments. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Conceptually, the proposed model seems to be very close to a ConvNet with large filters. L145 states that the global filter layer is equivalent to a circular convolution with a large filter. Furthermore because these layers are stacked (L195-196), it because to look very close to a ConvNet. Furthermore, a very similar model (FNet, 18) was recently proposed for NLP. The motivation of this work is to remove inductive biases from modern architectures (largely based on ConvNets), however it seems to me that in attempting to do so we arrived back at stacked convolutions (computed in the frequency domain). This model doesn't seem to offer a great performance advantage, considering that the SoTA top-1 performance on ImageNet is >90% (large vision transformer). For example, without distillation, GFNet-12 achieves 78.5% while EfficientNet-B3 achieves 81.6% with fewer trainable parameters. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not particularly convinced by the motivation of \"removing inductive biases\" -- why? Certainly with modern accelerators and advanced training tricks we can match or sometimes exceed the performance of ConvNets, but it seems to me that the inductive biases embodied by ConvNets are still relevant and useful. Not only that, but these inductive biases are often based on domain knowledge and give insight to the problem they solve. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "MK4jBU-w6ya", "reviewer": "Reviewer_wrgX", "summary": "This paper proposes Global Filter Networks (GFNet), which make use of discrete Fourier transform (DFT) and a global filter layer to mix the input tokens. According to the convolution theorem, the combination of the two operations can be regarded as a depthwise global circular convolution, and thus this work can be viewed as a kind of convolutional network with global convolutional layers. By taking advantage of FFT, the proposed method is faster than previous methods such as self-attention and spatial MLP with comparable performance on ImageNet.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ The work is well-written and easy to understand.\n+ According to the experiments, the proposed method is advantageous. Compared to Vit and ResMLP, the method has less latency and memory footprint, as shown in Fig. 2. It also achieves comparable or better image classification performance than these works, as shown in Table 4.\n+ Nice tricks are used to speed up the computation by using the conjugate symmetry of DFT.\n\nCons:\n- The novelty is limited. It is well-known that in the frequency domain, the convolution operation can be transformed into an element-wise multiplication according to the convolution theorem. The paper can be viewed as an application of the fact, but the fact has actually been widely used in CNNs.\n- The proposed method does not have obvious advantages over traditional CNNs. As shown in Table 4, compared with \"Local Conv (7x7)\", GFNet-12 achieves 78.5 vs 78.2 with similar FLOPS. Given the well-optimized implementation of CNNs, I am pretty sure that the actual latency of GFNet will be longer than CNNs, which makes GFNet less practical.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors said the limitations can be found in Section 5, but I did not find them. The authors should include them.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "7fl8sJj1Lhn", "reviewer": "Reviewer_yCXs", "summary": "This paper proposes to replace self-attention in vision transformer by a *Global Filter Layer* consisting of a 2D fast Fourier transform (FFT), a point-wise multiplication with learned weights and an 2D inverse FFT. The proposed layer compares favourably with (a) self-attention as it replaces the quadratic dependency on compute cost and memory footprint with a log-linear rate and (b) variants of MLP-mixer as the number of weights scales linearly with the number of pixels instead of quadratically. The performances of the proposed model are convincing both for supervised classification on ImageNet and fine-tuning to downstream task on smaller datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI would like to thank the authors for this very interesting piece of work. This is a timely contribution to the field and a very interesting read.\n\nThe introduced Global Filter Layer is simple and well motivated. The idea is novel and allows to avoid the quadratic dependency in attention models while reaching similar performance. It also solves two problems inherent to MLP-Mixer: (1) it requires only a linear number of parameters in term of input size and (2) moving to the Fourier domain gives a principled approach to upscale a trained model to process images at a different resolution. \nI find the contribution solid and I would tend to accept the paper.\n\n### Questions and concerns\n\nI am not sure what is the common way to account for complex multiplications but if we consider it requires 4 FLOPS, should there be a factor 2 in the second term of the number of FLOPS in Table 1? (accounting that you halve the size of the signal with line 161).\n\nConcerning Figure 4 (a) frequency domain, can you clarify if you are plotting the module of the weights or their real part maybe?\n\nI found in the code that you use a `num_heads` parameter in `GlobalFilter` (file `code/gfnet.py` line 58) which is not mentioned in the paper. Could you clarify its usage? If I understand the code correctly, it seems that some weights in $\\mathbf{K}$ (eq. 3.5) are shared.\nThis decreases the number of parameters of the layers by a factor $D / N_{heads}$ but constraints the diversity of frequency filters that can be learned. Could you comment if the model proposed in the paper ($N_{heads} = D$) performs well in practice or if you would advise for your alternative with heads?\n\nBeyond proposing a novel *Global Filter Layer*, the authors further push the performance of their model with some interesting techniques that are often orthogonal to the main contribution:\n\n* The best performing models are the hierarchical version H18 and H24 and they are only introduced in the Appendix. Please detail how you chose the hyper-parameters for this architecture. Even though the performance is great and the architecture interesting, this goes against the general idea of MLP-mixers to introduce ``fewer inductive biases'' and requires more hyper-parameter tuning.\n\n* Token-wise distillation could be applied to other baselines to be fair (except ConvNets). Same goes for upscaling to 384 for vision transformers (which is standard) but not for MLP-like model where the proposed model introduces a principled approach for upscaling.\n\n\n### Checklist\n\n3. (d). Please provide the total amount of compute needed to train your model on your 8 GPUs machine. Can you also provide in the Appendix a version of Figure 3 with the training time as x-axis?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "-h7FXAtRZ33", "reviewer": "Reviewer_oVSg", "summary": "This paper replaces the self-attention layer with a global filter layer:  a sequence of a 2D FFT, einsum, and then 2D inverse FFT. A benefit of the approach is that the attention computation is reduced from O(n^2) to O(nlogn) time-complexity. This demonstrates that GFNet is Pareto optimal versus other Transformer-like models on ImageNet on an accuracy-FLOPs basis.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am positively inclined towards this paper. Their approach appears to improve over other concurrent efforts to use Fourier Transforms like FNet (Lee-Thorp, 2021).\n\nQuestions:\n* Putting aside FLOPs, how fast are these on modern accelerators (e.g GPUs, TPUs)? For instance, certain operations, like depthwise convolutions, while FLOP-efficient, are relatively slow on a wall-clock basis. Slow performance doesn’t disqualify the idea of using FFT, but it should be noted here.\n* For many larger models, it seems that the O(n^2) time-complexity of self-attention operations are usually not the bottleneck. Could the authors comment on how this improvement scales with model size? \n* Does the visualization of the learned features in the frequency domain suggest further improvements to the model? Is diversity of features important to model performance?\n* Did the authors consider the high resolution applications discussed in the paper? Given the scalability benefit of the technique, a positive result here might increase the impact of this work.\n* Can the paper dedicate more details to the token-wise distillation in the main body? Given the +1% boost to performance, it might be useful to present more details to the reader earlier.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes replacing self-attention with a global filter layer and applies this method to image classification problems. The resulting method replaces O(n^2) attention operation with a O(nlogn) time-complexity while achieving favorable performance in computation time vs accuracy trade-off curves. The reviewers commented positively on the motivation, clarity of exposition, and the implementation. Although the reviewers noted some concerns about novelty with respect to previous architectures, overall the reviewers were favorable to the acceptance of this work. Despite these minor concerns, this paper is accepted to the conference.

Decision: Accept (Poster)