Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a MLP-based model that achieves comparable performance with transformers for vision tasks and language modeling. While this MLP-based model performs slighly worse on natural language understanding tasks (e.g. MNLI), introducing a small single-head self-attention module can compenate for most of the performance loss. Compared to other MLP-based models introduced in concurrent works, the proposed model uses a gating module before the spatial projection. Performance wise, the proposed method is stronger than the concurrent MLP variant MLP-mixer on vision and language modeling tasks, showing the advantage of the proposed gating methods

Overall I find the proposed method to be interesting and empirically strong. I am leaning toward accept but also wish the authors had done more work in understanding the proposed models. I offer some suggestions and have some questions below.
main_review: Pros

- The empirical performance is strong
- The proposed model is novel in the sense that it combines a gating mechanism with MLP-only model.
- The submission is well-written and clear.

Cons/Questions

- The submission very much focuses on the empirical performance of the proposed method. This is understandable but I also hope the authors had done more analysis work to understand the proposed methods. Here are some suggestions:
    - Learning dynamics: Are the proposed method noticeably different from transformers in terms of convergence behavior?
    - Data efficiency: Is there trade-off between transformers and gMLP in terms of data efficiency? For example, does gMLP perform better or worse relative to transformer as we add in more training data?
- Can the authors document the computational advantage of gMLP? Is the proposed method faster to inference or does gMLP save GPU memory?
- The fact that attention mechanism is especially important for cross-sentence understanding tasks is very interesting. I wonder if the authors can identify any patterns in the learned single head attention. Or, can the authors offer more discussion on how the attention mechanism is helping?
limitations_and_societal_impact: I don't see any negative societal impact from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a simple architecture based on MLPs, which only adopts channel projections and spatial projections with the spatial gating units. On image classification, the presented gMLP could achieve on par performance with DeiT and ViT, and better than previous MLP-based approaches, such as MLP Mixer. On masked language modeling tasks, gMLP could also achieve on par performance with Transformer on Perplexity, and gMLP could perform slightly worse than Transformer on MNLI.
main_review: For originality, this paper presents a new MLP-based approach, e.g. gMLP. But compared to MLP-Mixer, the contribution of this gMLP seems marginal. Only the spatial gating unit is newly designed and verified to be effective. But the experimental study of this paper is sufficient and valuable to the community.

The quality of this paper is good. From an experimental report perspective, this paper contains sufficient experimental results, verifies important claims, and achieves relatively good performance on both ImageNet classification and masked language modeling in NLP, comparing to the previous approaches. The diversity of the experiments is sufficient.

The clarity of this paper is good. This paper is well-organized and easy to follow.

My major concern is about spatial projection. This is a simple design, but the application scenario is also limited due to the fixed size of the input spatial resolution. For example, for some vision downstream tasks of object detection and semantic segmentation, the input sizes are different from that of the image classification. This architecture could not be directly applied to these tasks.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes and studies an interesting research problem whether we can remove the self-attention in Transformer based models. The authors devise a novel architecture called gMLPs by removing the self-attention layer and introducing a new Spatial Gating Unit into the FFN module to well model the interactions between independent tokens. Comprehensive experiments on CV or NLP tasks confirmed that the proposed gMLPs can achieve comparable performances with the standard ViT or BERT models.    
main_review: I enjoyed reading this paper, the whole paper is very clear and well-written, and I believe this paper is going to be useful for the community and inspire other researchers’ work on this topic. 

Below are some questions or confusions I had: 

1. The proposed architecture are evaluated on encoder-style pre-trained models, e.g. ViT or BERT, will it be possible to extend the method to encoder-decoder or decoder-style models, like T5 or GPT?

2. Transformer-based models tend to have better OOD generalization ability, the authors are suggested to add some OOD evaluation experiments as conducted in the paper “Pretrained Transformers Improve Out-of-Distribution Robustness”. 

3. In Tables 4 and 5, it is better to add inference time and flops as extra evaluation metrics. 

4. In Figure 4, it seems that the learned filters tend to have local correlations, will this limit the ability of gMLPs in modeling long range dependencies.  

5. In appendix D the learned spatial projection matrices seems to be Toeplitz-like, can you quantitatively evaluate how these matrices be Toeplitz-like?

6. Will the proposed gMLPs model be sensitive to the length of input sentence? The authors are suggested to add some analysis with considering the sequence length factor, e.g. accuracy vs length on downstream tasks.

7. In practice, before deploying to products, pre-trained models often need to be compressed, I wonder whether the gMLPs can be compressed into some tiny models for IoT devices.      

8. It is better to discuss some potential limitations of the proposed gMLPs. 

limitations_and_societal_impact: I see no potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a simple mechanism: gated MLP and questions the necessity of self attention.
gMLP achieves comparable performance as DeiT on the classification task. Experiments on NLP talk also support the validness of this method. 


main_review: Strength:
1. Experiments are designed across both vision classification and NLP tasks.
2. Overall, the experiment results are good.  It achieves similar performance as DeiT on ImageNet 1k classificaition and NLP baseline. gMLP achieves better performance than the MLP baseline MLPmixer.
3. The motivation is clear and good. However, the claim is too strong because of the weakness. 

Weakness:
1. One big concern  is that the gMLP cannot scale to down stream tasks such as object detection, where different resolution is required during the training.  The spatial_gating_unit function in the pseudo code can only handle fixed length.
 I believe this is one fatal and fundamental drawback and would greatly limit the application of this paper. Note that self attention can handle it easily, and there are a lot of new work [1,2,3] before NeurIPS, which can easily handle this case and achieve impressive results on segmentation and object detection.   In contrast, the gMLP framework cannot deal with it directly.
Please correct me if I'm wrong. 

2. Some important factors are neglected and maybe leads to unfair comparisons. One important baseline is DeiT, however, many improvements [1,2,3,4,5,6,7] are proposed based on DeiT and further boost the performance. The performance gap between transformer and gMLP are enlarged if they are included. For example,  gMLP utilizes global average pooling GAP in classification, while DeiT doesn't. This will boost transformer about at least 1-1.5% accuracy on ImageNet validation dataset [4].  And fine-tuning the drop path rate (gMLP also do this) [2, 3, 7] can marginally improve the performance of transformer.  I believe proper discussions and comparisons with these work would benefit this paper and community.

Minors:
1. The gated title is a bit confusing since it doesn't behave like  commonly used meaning. 
2. Inference speed (imgs/s) is not reported (Table 2)



Reference

[1] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions,
 https://arxiv.org/abs/2102.12122

[2]Twins: Revisiting the Design of Spatial Attention in Vision Transformers, https://arxiv.org/abs/2104.13840

[3] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows https://arxiv.org/abs/2103.14030

[4] Conditional Positional Encodings for Vision Transformers, https://arxiv.org/abs/2102.10882

[5]Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet https://arxiv.org/abs/2101.11986

[6] Transformer in Transformer, https://arxiv.org/abs/2103.00112.

[7] Going deeper with Image Transformers https://arxiv.org/abs/2103.17239


limitations_and_societal_impact: See weakness 1 of the main review.
review_title: null

Meta-review: The paper introduces an MLP-based model with gating (gMLP) that achieves comparable performance to Transformers, showing that self-attention is not critical for the success of these models. While multiple concerns were raised regarding the limitations of the proposed method, all four reviewers appreciate the strong empirical results reported in the paper and recommend acceptance. The AC agrees with this decision, and requests the authors to add to the final version the discussion and additional information provided in the rebuttal, as well as more clearly describe the limitations of the proposed approach.
Decision: Accept (Poster)