Global Filter Networks for Image Classification

1 Introduction . The transformer architecture , originally designed for the natural language processing ( NLP ) tasks [ 42 ] , has shown promising performance on various vision problems recently [ 10 , 40 , 27 , 49 , 4 , 47 , 35 , 5 ] . Different from convolutional neural networks ( CNNs ) , vision transformer models use self-attention layers to capture long-term dependencies , which are able to learn more diverse interactions between spatial locations . The pure multi-layer perceptrons ( MLP ) models [ 38 , 39 ] further simplify the vision transformers by replacing the self-attention layers with MLPs that are applied across spatial locations . Since fewer inductive biases are introduced , these two kinds of models have the potential to learn more generic and flexible interactions among spatial locations from raw data . One primary challenge of applying self-attention and pure MLP models to vision tasks is the considerable computational complexity that grows quadratically as the number of tokens increases . Therefore , typical vision transformer style models usually consider a relatively small resolution for the intermediate features ( e.g . 14× 14 tokens are extracted from the input images in both ViT [ 10 ] and MLP-Mixer [ 38 ] ) . This design may limit the applications of downstream dense prediction tasks like detection and segmentation . A possible solution is to replace the global self-attention with several local self-attention like Swin transformer [ 27 ] . Despite the effectiveness in practice , local self-attention brings quite a few hand-made choices ( e.g. , window size , padding strategy , etc . ) and limits the receptive field of each layer . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we present a new conceptually simple yet computationally efficient architecture called Global Filter Network ( GFNet ) , which follows the trend of removing inductive biases from vision models while enjoying the log-linear complexity in computation . The basic idea behind our architecture is to learn the interactions among spatial locations in the frequency domain . Different from the self-attention mechanism in vision transformers and the fully connected layers in MLP models , the interactions among tokens are modeled as a set of learnable global filters that are applied to the spectrum of the input features . Since the global filters are able to cover all the frequencies , our model can capture both long-term and short-term interactions . The filters are directly learned from the raw data without introducing human priors . Our architecture is largely based on the vision transformers only with some minimal modifications . We replace the self-attention sub-layer in vision transformers with three key operations : a 2D discrete Fourier transform to convert the input spatial features to the frequency domain , an element-wise multiplication between frequency-domain features and the global filters , and a 2D inverse Fourier transform to map the features back to the spatial domain . Since the Fourier transform is used to mix the information of different tokens , the global filter is much more efficient compared to the self-attention and MLP thanks to the O ( L logL ) complexity of the fast Fourier transform algorithm ( FFT ) [ 7 ] . Benefiting from this , the proposed global filter layer is less sensitive to the token length L and thus is compatible with larger feature maps and CNN-style hierarchical architectures without modifications . The overall architecture of GFNet is illustrated in Figure 1 . We also compare our global filter with prevalent operations in deep vision models in Table 1 . Our experiments on ImageNet verify the effectiveness of GFNet . With a similar architecture , our model outperform the recent vision transformer and MLP models including DeiT [ 40 ] , ResMLP [ 39 ] and gMLP [ 26 ] . When using the hierarchical architecture , GFNet can further enlarge the gap . GFNet also works well on downstream transfer learning and semantic segmentation tasks . Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency , generalization ability and robustness . 2 Related works . Vision transformers . Since Dosovitskiy et al . [ 10 ] introduce transformers to the image classification and achieve a competitive performance compared to CNNs , transformers begin to exhibit their potential in various vision tasks [ 3 , 4 , 49 ] . Recently , there are a large number of works which aim to improve the transformers [ 40 , 41 , 27 , 44 , 18 , 11 , 48 ] . These works either seek for better training strategies [ 40 , 11 ] or design better architectures [ 27 , 44 , 48 ] or both [ 41 , 11 ] . However , most of the architecture modification of the transformers [ 44 , 18 , 27 , 48 ] introduces additional inductive biases similar to CNNs . In this work , we only focus on the standard transformer architecture [ 10 , 40 ] and our goal is to replace the heavy self-attention layer ( O ( L2 ) ) to an more efficient operation which can still model the interactions among different spatial locations without introducing the inductive biases associated with CNNs . MLP-like models . More recently , there are several works that question the importance of selfattention in the vision transformers and propose to use MLP to replace the self-attention layer in the transformers [ 38 , 39 , 26 ] . The MLP-Mixer [ 38 ] employs MLPs to perform token mixing and channel mixing alternatively in each block . ResMLP [ 39 ] adopts a similar idea but substitutes the Layer Normalization with an Affine transformation for acceleration . The recently proposed gMLP [ 26 ] uses a spatial gating unit to re-weight tokens in the spatial dimension . However , all of the above models include MLPs to mix the tokens spatially , which brings two drawbacks : ( 1 ) like the self-attention in the transformers , the spatial MLP still requires computational complexity quadratic to the length of tokens . ( 2 ) unlike transformers , MLP models are hard to scale up to higher resolution since the weights of the spatial MLPs have fixed sizes . Our work follows this trend and successfully resolves the above issues in MLP-like models . The proposed GFNet enjoys log-linear complexity and can be easily scaled up to any resolution . Applications of Fourier transform in vision . Fourier transform has been an important tool in digital image processing for decades [ 32 , 1 ] . With the breakthroughs of CNNs in vision [ 14 , 13 ] , there are a variety of works that start to incorporate Fourier transform in some deep learning method [ 24 , 46 , 9 , 22 , 6 ] for vision tasks . Some of these works employ discrete Fourier transform to convert the images to the frequency domain and leverage the frequency information to improve the performance in certain tasks [ 22 , 46 ] , while others utilize the convolution theorem to accelerate the CNNs via fast Fourier transform ( FFT ) [ 24 , 9 ] . FFC [ 6 ] replaces the convolution in CNNs with an Local Fourier Unit and perform convolutions in the frequency domain . Very recent works also try to leverage Fourier transform to develop deep learning models to solve partial differential equations [ 25 ] and NLP tasks [ 23 ] . In this work , we propose to use learnable filters to interchange information globally among the tokens in the Fourier domain , inspired by the frequency filters in the digital image processing [ 32 ] . We also take advantage of some properties of FFT to reduce the computational costs and the number of parameters . 3 Method . 3.1 Preliminaries : discrete Fourier transform . We start by introducing the discrete Fourier transform ( DFT ) , which plays an important role in the area of digital signal processing and is a crucial component in our GFNet . For clarity , We first consider the 1D DFT . Given a sequence of N complex numbers x [ n ] , 0 ≤ n ≤ N − 1 , the 1D DFT converts the sequence into the frequency domain by : X [ k ] = N−1∑ n=0 x [ n ] e−j ( 2π/N ) kn : = N−1∑ n=0 x [ n ] W knN ( 3.1 ) Algorithm 1 Pseudocode of Global Filter Layer . # x : the token features , B x H x W x D ( where N = H * W ) # K : the frequency-domain filter , H x W_hat x D ( where W_hat = W // 2 + 1 , see Section 3.2 for details ) X = rfft2 ( x , dim= ( 1 , 2 ) ) X_tilde = X * K x = irfft2 ( X_tilde , dim= ( 1 , 2 ) ) rfft2/irfft2 : 2D FFT/IFFT for real signal where j is the imaginary unit and WN = e−j ( 2π/N ) . The formulation of DFT in Equation ( 3.1 ) can be derived from the Fourier transform for continuous signal by sampling in both the time domain and the frequency domain ( see Appendix A for details ) . Since X [ k ] repeats on intervals of length N , it is suffice to take the value of X [ k ] at N consecutive points k = 0 , 1 , . . . , N − 1 . Specifically , X [ k ] represents to the spectrum of the sequence x [ n ] at the frequency ωk = 2πk/N . It is also worth noting that DFT is a one-to-one transformation . Given the DFT X [ k ] , we can recover the original signal x [ n ] by the inverse DFT ( IDFT ) : x [ n ] = 1 N N−1∑ k=0 X [ k ] ej ( 2π/N ) kn . ( 3.2 ) For real input x [ n ] , it can be proved that ( see Appendix A ) its DFT is conjugate symmetric , i.e. , X [ N − k ] = X∗ [ k ] . The reverse is true as well : if we perform IDFT to X [ k ] which is conjugate symmetric , a real discrete signal can be recovered . This property implies that the half of the DFT { X [ k ] : 0 ≤ k ≤ dN/2e } contains the full information about the frequency characteristics of x [ n ] . DFT is widely used in modern signal processing algorithms for mainly two reasons : ( 1 ) the input and output of DFT are both discrete thus can be easily processed by computers ; ( 2 ) there exist efficient algorithms for computing the DFT . The fast Fourier transform ( FFT ) algorithms take advantage of the symmetry and periodicity properties of W knN and reduce the complexity to compute DFT from O ( N2 ) to O ( N logN ) . The inverse DFT ( 3.2 ) , which has a similar form to the DFT , can also be computed efficiently using the inverse fast Fourier transform ( IFFT ) . The DFT described above can be extend to 2D signals . Given the 2D signal X [ m , n ] , 0 ≤ m ≤ M − 1 , 0 ≤ n ≤ N − 1 , the 2D DFT of x [ m , n ] is given by : X [ u , v ] = M−1∑ m=0 N−1∑ n=0 x [ m , n ] e−j2π ( um M + vn N ) . ( 3.3 ) The 2D DFT can be viewed as performing 1D DFT on the two dimensions alternatively . Similar to 1D DFT , 2D DFT of real input x [ m , n ] satisfied the conjugate symmetry property X [ M − u , N − v ] = X∗ [ u , v ] . The FFT algorithms can also be applied to 2D DFT to improve computational efficiency .

Summary: This paper proposes to replace self-attention in vision transformer by a *Global Filter Layer* consisting of a 2D fast Fourier transform (FFT), a point-wise multiplication with learned weights and an 2D inverse FFT. The proposed layer compares favourably with (a) self-attention as it replaces the quadratic dependency on compute cost and memory footprint with a log-linear rate and (b) variants of MLP-mixer as the number of weights scales linearly with the number of pixels instead of quadratically. The performances of the proposed model are convincing both for supervised classification on ImageNet and fine-tuning to downstream task on smaller datasets.