Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper replaces the self-attention layer with a global filter layer:  a sequence of a 2D FFT, einsum, and then 2D inverse FFT. A benefit of the approach is that the attention computation is reduced from O(n^2) to O(nlogn) time-complexity. This demonstrates that GFNet is Pareto optimal versus other Transformer-like models on ImageNet on an accuracy-FLOPs basis.
main_review: I am positively inclined towards this paper. Their approach appears to improve over other concurrent efforts to use Fourier Transforms like FNet (Lee-Thorp, 2021).

Questions:
* Putting aside FLOPs, how fast are these on modern accelerators (e.g GPUs, TPUs)? For instance, certain operations, like depthwise convolutions, while FLOP-efficient, are relatively slow on a wall-clock basis. Slow performance doesn’t disqualify the idea of using FFT, but it should be noted here.
* For many larger models, it seems that the O(n^2) time-complexity of self-attention operations are usually not the bottleneck. Could the authors comment on how this improvement scales with model size? 
* Does the visualization of the learned features in the frequency domain suggest further improvements to the model? Is diversity of features important to model performance?
* Did the authors consider the high resolution applications discussed in the paper? Given the scalability benefit of the technique, a positive result here might increase the impact of this work.
* Can the paper dedicate more details to the token-wise distillation in the main body? Given the +1% boost to performance, it might be useful to present more details to the reader earlier.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes Global Filter Networks (GFNet), which make use of discrete Fourier transform (DFT) and a global filter layer to mix the input tokens. According to the convolution theorem, the combination of the two operations can be regarded as a depthwise global circular convolution, and thus this work can be viewed as a kind of convolutional network with global convolutional layers. By taking advantage of FFT, the proposed method is faster than previous methods such as self-attention and spatial MLP with comparable performance on ImageNet.
main_review: Pros:
+ The work is well-written and easy to understand.
+ According to the experiments, the proposed method is advantageous. Compared to Vit and ResMLP, the method has less latency and memory footprint, as shown in Fig. 2. It also achieves comparable or better image classification performance than these works, as shown in Table 4.
+ Nice tricks are used to speed up the computation by using the conjugate symmetry of DFT.

Cons:
- The novelty is limited. It is well-known that in the frequency domain, the convolution operation can be transformed into an element-wise multiplication according to the convolution theorem. The paper can be viewed as an application of the fact, but the fact has actually been widely used in CNNs.
- The proposed method does not have obvious advantages over traditional CNNs. As shown in Table 4, compared with "Local Conv (7x7)", GFNet-12 achieves 78.5 vs 78.2 with similar FLOPS. Given the well-optimized implementation of CNNs, I am pretty sure that the actual latency of GFNet will be longer than CNNs, which makes GFNet less practical.

limitations_and_societal_impact: The authors said the limitations can be found in Section 5, but I did not find them. The authors should include them.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to replace self-attention in vision transformer by a *Global Filter Layer* consisting of a 2D fast Fourier transform (FFT), a point-wise multiplication with learned weights and an 2D inverse FFT. The proposed layer compares favourably with (a) self-attention as it replaces the quadratic dependency on compute cost and memory footprint with a log-linear rate and (b) variants of MLP-mixer as the number of weights scales linearly with the number of pixels instead of quadratically. The performances of the proposed model are convincing both for supervised classification on ImageNet and fine-tuning to downstream task on smaller datasets.
main_review: 
I would like to thank the authors for this very interesting piece of work. This is a timely contribution to the field and a very interesting read.

The introduced Global Filter Layer is simple and well motivated. The idea is novel and allows to avoid the quadratic dependency in attention models while reaching similar performance. It also solves two problems inherent to MLP-Mixer: (1) it requires only a linear number of parameters in term of input size and (2) moving to the Fourier domain gives a principled approach to upscale a trained model to process images at a different resolution. 
I find the contribution solid and I would tend to accept the paper.

### Questions and concerns

I am not sure what is the common way to account for complex multiplications but if we consider it requires 4 FLOPS, should there be a factor 2 in the second term of the number of FLOPS in Table 1? (accounting that you halve the size of the signal with line 161).

Concerning Figure 4 (a) frequency domain, can you clarify if you are plotting the module of the weights or their real part maybe?

I found in the code that you use a `num_heads` parameter in `GlobalFilter` (file `code/gfnet.py` line 58) which is not mentioned in the paper. Could you clarify its usage? If I understand the code correctly, it seems that some weights in $\mathbf{K}$ (eq. 3.5) are shared.
This decreases the number of parameters of the layers by a factor $D / N_{heads}$ but constraints the diversity of frequency filters that can be learned. Could you comment if the model proposed in the paper ($N_{heads} = D$) performs well in practice or if you would advise for your alternative with heads?

Beyond proposing a novel *Global Filter Layer*, the authors further push the performance of their model with some interesting techniques that are often orthogonal to the main contribution:

* The best performing models are the hierarchical version H18 and H24 and they are only introduced in the Appendix. Please detail how you chose the hyper-parameters for this architecture. Even though the performance is great and the architecture interesting, this goes against the general idea of MLP-mixers to introduce ``fewer inductive biases'' and requires more hyper-parameter tuning.

* Token-wise distillation could be applied to other baselines to be fair (except ConvNets). Same goes for upscaling to 384 for vision transformers (which is standard) but not for MLP-like model where the proposed model introduces a principled approach for upscaling.


### Checklist

3. (d). Please provide the total amount of compute needed to train your model on your 8 GPUs machine. Can you also provide in the Appendix a version of Figure 3 with the training time as x-axis?

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents global filter network, a simple and efficient architecture for visual recognition. Unlike transformer-based visual recognition models that utilize self-attention to encode spatial information, this paper proposes to build long-range dependencies among spatial locations in the frequency domain by taking advantage of 2D FFT and IFFT. 
main_review: - The originality of this paper is clear. Encoding spatial information in the frequency domain is interesting. The classification performance on ImageNet is also good.

- This paper is also well presented. It is simple and easy to follow.
limitations_and_societal_impact: - In the experiment section, the authors say that strong data augmentation methods are used for training. However, these are no experiments demonstrating how each data augmentation method contributes to the model performance. I think an ablation path on data augmentation should be given.

- The authors use token-wise distillation strategy while training. Have you attempted to use the original token labeling while training? Does the token-wise distillation strategy perform better than token labeling?

- It is great to see the visualization of the learned filters. It would be better if the authors could provide more explanations on what the proposed approach learns in terms of the filters.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work proposes a new attention-based architecture for computer vision (classification). The proposed architecture replaces the self-attention layer in ViT with computational blocks involving the Fourier transform. Due to the efficiency of FFT and IFFT the quadratic complexity of self-attention can be reduced to a log-linear computational cost. In order to boost performance distillation of taken features from a pre-trained vision transformer is used. Results are presented on ImageNet as well as on some transfer learning experiments. 
main_review: Conceptually, the proposed model seems to be very close to a ConvNet with large filters. L145 states that the global filter layer is equivalent to a circular convolution with a large filter. Furthermore because these layers are stacked (L195-196), it because to look very close to a ConvNet. Furthermore, a very similar model (FNet, 18) was recently proposed for NLP. The motivation of this work is to remove inductive biases from modern architectures (largely based on ConvNets), however it seems to me that in attempting to do so we arrived back at stacked convolutions (computed in the frequency domain). This model doesn't seem to offer a great performance advantage, considering that the SoTA top-1 performance on ImageNet is >90% (large vision transformer). For example, without distillation, GFNet-12 achieves 78.5% while EfficientNet-B3 achieves 81.6% with fewer trainable parameters. 
limitations_and_societal_impact: I am not particularly convinced by the motivation of "removing inductive biases" -- why? Certainly with modern accelerators and advanced training tricks we can match or sometimes exceed the performance of ConvNets, but it seems to me that the inductive biases embodied by ConvNets are still relevant and useful. Not only that, but these inductive biases are often based on domain knowledge and give insight to the problem they solve.
review_title: null

Meta-review: This work proposes replacing self-attention with a global filter layer and applies this method to image classification problems. The resulting method replaces O(n^2) attention operation with a O(nlogn) time-complexity while achieving favorable performance in computation time vs accuracy trade-off curves. The reviewers commented positively on the motivation, clarity of exposition, and the implementation. Although the reviewers noted some concerns about novelty with respect to previous architectures, overall the reviewers were favorable to the acceptance of this work. Despite these minor concerns, this paper is accepted to the conference.

Decision: Accept (Poster)