Global Filter Networks for Image Classification

1 Introduction . The transformer architecture , originally designed for the natural language processing ( NLP ) tasks [ 42 ] , has shown promising performance on various vision problems recently [ 10 , 40 , 27 , 49 , 4 , 47 , 35 , 5 ] . Different from convolutional neural networks ( CNNs ) , vision transformer models use self-attention layers to capture long-term dependencies , which are able to learn more diverse interactions between spatial locations . The pure multi-layer perceptrons ( MLP ) models [ 38 , 39 ] further simplify the vision transformers by replacing the self-attention layers with MLPs that are applied across spatial locations . Since fewer inductive biases are introduced , these two kinds of models have the potential to learn more generic and flexible interactions among spatial locations from raw data . One primary challenge of applying self-attention and pure MLP models to vision tasks is the considerable computational complexity that grows quadratically as the number of tokens increases . Therefore , typical vision transformer style models usually consider a relatively small resolution for the intermediate features ( e.g . 14× 14 tokens are extracted from the input images in both ViT [ 10 ] and MLP-Mixer [ 38 ] ) . This design may limit the applications of downstream dense prediction tasks like detection and segmentation . A possible solution is to replace the global self-attention with several local self-attention like Swin transformer [ 27 ] . Despite the effectiveness in practice , local self-attention brings quite a few hand-made choices ( e.g. , window size , padding strategy , etc . ) and limits the receptive field of each layer . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we present a new conceptually simple yet computationally efficient architecture called Global Filter Network ( GFNet ) , which follows the trend of removing inductive biases from vision models while enjoying the log-linear complexity in computation . The basic idea behind our architecture is to learn the interactions among spatial locations in the frequency domain . Different from the self-attention mechanism in vision transformers and the fully connected layers in MLP models , the interactions among tokens are modeled as a set of learnable global filters that are applied to the spectrum of the input features . Since the global filters are able to cover all the frequencies , our model can capture both long-term and short-term interactions . The filters are directly learned from the raw data without introducing human priors . Our architecture is largely based on the vision transformers only with some minimal modifications . We replace the self-attention sub-layer in vision transformers with three key operations : a 2D discrete Fourier transform to convert the input spatial features to the frequency domain , an element-wise multiplication between frequency-domain features and the global filters , and a 2D inverse Fourier transform to map the features back to the spatial domain . Since the Fourier transform is used to mix the information of different tokens , the global filter is much more efficient compared to the self-attention and MLP thanks to the O ( L logL ) complexity of the fast Fourier transform algorithm ( FFT ) [ 7 ] . Benefiting from this , the proposed global filter layer is less sensitive to the token length L and thus is compatible with larger feature maps and CNN-style hierarchical architectures without modifications . The overall architecture of GFNet is illustrated in Figure 1 . We also compare our global filter with prevalent operations in deep vision models in Table 1 . Our experiments on ImageNet verify the effectiveness of GFNet . With a similar architecture , our model outperform the recent vision transformer and MLP models including DeiT [ 40 ] , ResMLP [ 39 ] and gMLP [ 26 ] . When using the hierarchical architecture , GFNet can further enlarge the gap . GFNet also works well on downstream transfer learning and semantic segmentation tasks . Our results demonstrate that GFNet can be a very competitive alternative to transformer-style models and CNNs in efficiency , generalization ability and robustness . 2 Related works . Vision transformers . Since Dosovitskiy et al . [ 10 ] introduce transformers to the image classification and achieve a competitive performance compared to CNNs , transformers begin to exhibit their potential in various vision tasks [ 3 , 4 , 49 ] . Recently , there are a large number of works which aim to improve the transformers [ 40 , 41 , 27 , 44 , 18 , 11 , 48 ] . These works either seek for better training strategies [ 40 , 11 ] or design better architectures [ 27 , 44 , 48 ] or both [ 41 , 11 ] . However , most of the architecture modification of the transformers [ 44 , 18 , 27 , 48 ] introduces additional inductive biases similar to CNNs . In this work , we only focus on the standard transformer architecture [ 10 , 40 ] and our goal is to replace the heavy self-attention layer ( O ( L2 ) ) to an more efficient operation which can still model the interactions among different spatial locations without introducing the inductive biases associated with CNNs . MLP-like models . More recently , there are several works that question the importance of selfattention in the vision transformers and propose to use MLP to replace the self-attention layer in the transformers [ 38 , 39 , 26 ] . The MLP-Mixer [ 38 ] employs MLPs to perform token mixing and channel mixing alternatively in each block . ResMLP [ 39 ] adopts a similar idea but substitutes the Layer Normalization with an Affine transformation for acceleration . The recently proposed gMLP [ 26 ] uses a spatial gating unit to re-weight tokens in the spatial dimension . However , all of the above models include MLPs to mix the tokens spatially , which brings two drawbacks : ( 1 ) like the self-attention in the transformers , the spatial MLP still requires computational complexity quadratic to the length of tokens . ( 2 ) unlike transformers , MLP models are hard to scale up to higher resolution since the weights of the spatial MLPs have fixed sizes . Our work follows this trend and successfully resolves the above issues in MLP-like models . The proposed GFNet enjoys log-linear complexity and can be easily scaled up to any resolution . Applications of Fourier transform in vision . Fourier transform has been an important tool in digital image processing for decades [ 32 , 1 ] . With the breakthroughs of CNNs in vision [ 14 , 13 ] , there are a variety of works that start to incorporate Fourier transform in some deep learning method [ 24 , 46 , 9 , 22 , 6 ] for vision tasks . Some of these works employ discrete Fourier transform to convert the images to the frequency domain and leverage the frequency information to improve the performance in certain tasks [ 22 , 46 ] , while others utilize the convolution theorem to accelerate the CNNs via fast Fourier transform ( FFT ) [ 24 , 9 ] . FFC [ 6 ] replaces the convolution in CNNs with an Local Fourier Unit and perform convolutions in the frequency domain . Very recent works also try to leverage Fourier transform to develop deep learning models to solve partial differential equations [ 25 ] and NLP tasks [ 23 ] . In this work , we propose to use learnable filters to interchange information globally among the tokens in the Fourier domain , inspired by the frequency filters in the digital image processing [ 32 ] . We also take advantage of some properties of FFT to reduce the computational costs and the number of parameters . 3 Method . 3.1 Preliminaries : discrete Fourier transform . We start by introducing the discrete Fourier transform ( DFT ) , which plays an important role in the area of digital signal processing and is a crucial component in our GFNet . For clarity , We first consider the 1D DFT . Given a sequence of N complex numbers x [ n ] , 0 ≤ n ≤ N − 1 , the 1D DFT converts the sequence into the frequency domain by : X [ k ] = N−1∑ n=0 x [ n ] e−j ( 2π/N ) kn : = N−1∑ n=0 x [ n ] W knN ( 3.1 ) Algorithm 1 Pseudocode of Global Filter Layer . # x : the token features , B x H x W x D ( where N = H * W ) # K : the frequency-domain filter , H x W_hat x D ( where W_hat = W // 2 + 1 , see Section 3.2 for details ) X = rfft2 ( x , dim= ( 1 , 2 ) ) X_tilde = X * K x = irfft2 ( X_tilde , dim= ( 1 , 2 ) ) rfft2/irfft2 : 2D FFT/IFFT for real signal where j is the imaginary unit and WN = e−j ( 2π/N ) . The formulation of DFT in Equation ( 3.1 ) can be derived from the Fourier transform for continuous signal by sampling in both the time domain and the frequency domain ( see Appendix A for details ) . Since X [ k ] repeats on intervals of length N , it is suffice to take the value of X [ k ] at N consecutive points k = 0 , 1 , . . . , N − 1 . Specifically , X [ k ] represents to the spectrum of the sequence x [ n ] at the frequency ωk = 2πk/N . It is also worth noting that DFT is a one-to-one transformation . Given the DFT X [ k ] , we can recover the original signal x [ n ] by the inverse DFT ( IDFT ) : x [ n ] = 1 N N−1∑ k=0 X [ k ] ej ( 2π/N ) kn . ( 3.2 ) For real input x [ n ] , it can be proved that ( see Appendix A ) its DFT is conjugate symmetric , i.e. , X [ N − k ] = X∗ [ k ] . The reverse is true as well : if we perform IDFT to X [ k ] which is conjugate symmetric , a real discrete signal can be recovered . This property implies that the half of the DFT { X [ k ] : 0 ≤ k ≤ dN/2e } contains the full information about the frequency characteristics of x [ n ] . DFT is widely used in modern signal processing algorithms for mainly two reasons : ( 1 ) the input and output of DFT are both discrete thus can be easily processed by computers ; ( 2 ) there exist efficient algorithms for computing the DFT . The fast Fourier transform ( FFT ) algorithms take advantage of the symmetry and periodicity properties of W knN and reduce the complexity to compute DFT from O ( N2 ) to O ( N logN ) . The inverse DFT ( 3.2 ) , which has a similar form to the DFT , can also be computed efficiently using the inverse fast Fourier transform ( IFFT ) . The DFT described above can be extend to 2D signals . Given the 2D signal X [ m , n ] , 0 ≤ m ≤ M − 1 , 0 ≤ n ≤ N − 1 , the 2D DFT of x [ m , n ] is given by : X [ u , v ] = M−1∑ m=0 N−1∑ n=0 x [ m , n ] e−j2π ( um M + vn N ) . ( 3.3 ) The 2D DFT can be viewed as performing 1D DFT on the two dimensions alternatively . Similar to 1D DFT , 2D DFT of real input x [ m , n ] satisfied the conjugate symmetry property X [ M − u , N − v ] = X∗ [ u , v ] . The FFT algorithms can also be applied to 2D DFT to improve computational efficiency .

Summary: This paper replaces the self-attention layer with a global filter layer:  a sequence of a 2D FFT, einsum, and then 2D inverse FFT. A benefit of the approach is that the attention computation is reduced from O(n^2) to O(nlogn) time-complexity. This demonstrates that GFNet is Pareto optimal versus other Transformer-like models on ImageNet on an accuracy-FLOPs basis.