Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes to a new architecture for image classification, based on recently proposed Vision Transformers. Instead of employing self-attention as a global operation on patch embeddings, the authors use fully connected layers, followed by channel mixing done with fully connected layers too. The changes are motivated by improving performance on proprietary hardware (TPU), designed for matrix multiplication. Unlike ViT which has quadratic cost on the number of input patches, the proposed Mixer model has linear cost. However, it loses the ability to handle arbitrary image resolutions. The experimental evaluation shows that the proposed model is comparable performance to state-of-the-art convolutional and Transformer-based models on proprietary and public data.
main_review: Originality: the idea of splitting image classification networks into mixing spatially and channel-wise is not novel and was explored in Network-in-Network [Lin et al, 2014], in which convolutions were interleaved with MLPs too. This important citation and comparison are missing in the paper. The authors motivate Mixer with "a convolution is more complex than plain matrix multiplication in MLPs as it requires an additional costly reduction to matrix multiplication  and/or specialized implementation" (lines 36-37). I disagree with this statement. The concepts of matrix multiplication and convolution are both equally simple, and matrix multiplication requires a specialized implementation as well. The motivation to use matrix multiplication comes from the usage of proprietary hardware (TPU), which was originally designed for matrix multiplication, and is not well suited to convolution. The paper thus addresses an issue that only exists within one company which has access to the particular proprietary hardware and large proprietary datasets. This needs to be clearly stated in motivation. Compared to Network-In-Network, the authors show that spatial mixing can be done with MLP instead of convolution, or self-attention as in ViT, without the need of positional encoding, which is an interesting and novel finding. However, such reduction of inductive bias in architecture comes at a price:
- the model cannot be applied to images of different resolutions easily (as the authors state in section 3), as unlike CNN or ViT it cannot be applied in "fully-convolutional" manner. There are at least two very important cases for this, finetuning and object detection;
- the model becomes more data-hungry;
- the number of model parameters should increase: that would be expected, and important for deployment of such networks, yet never discussed in the paper.
The paper lacks related work on extremely simple and high performant CNN architectures, e.g. RepVGG [Ding et al. 2021], consisting only of convolutions and nonlinearities.

Quality: The authors provide an extensive evaluation and test the model for invariance to input permutations. Tables miss the indication of numbers of parameters for different models. I would be valuable to include throughput on other devices than TPUs, e.g. GPUs.

Clarity: the paper is well written and is easy to follow. 

Significance: the finding that image classification can be done with 1x1 convolutions interleaved with MLPs for spatial mixing is signficant, but will mostly be interesting for the users of the proprietary devices for which the model was designed for, as it also loses some desirable features of CNN and ViT.
limitations_and_societal_impact: The authors addressed the limitations and potential negative societal impact of their work adequately.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents an all-MLP based architecture for visual recognition tasks. The presented model is to replace the multi-head self-attention block with a spatial MLP in the original Transformer model. With a large-scale pre-trained dataset, this model with few inductive biases could achieve relatively good performance on image classification tasks.
main_review: For originality, the architecture proposed in this paper is simple and somewhat novel. A spatial MLP is an effective approach to perform spatial connections when a large-scale dataset is available.

The paper is well written and well organized.

With a large-scale pre-trained dataset, this model with few inductive biases could achieve relatively good performance on image classification tasks. This observation is interesting and valuable to the community.

My major concern is about the spatial MLP (MLP1). This is a simple design, but the application scenario is also limited due to the fixed size of the input spatial resolution. For example, for some downstream tasks of object detection and semantic segmentation, the input sizes are different to that of the image classification. This architecture could not be directly applied to these tasks.

Also, it is interesting to see the performance of the proposed approach on natural language processing tasks, such as in comparison to BERT on mask language modeling.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a neural network architecture of all MLP layers without any convolution or attention layers. Although it only has MLP layers, it still needs to exchange the information across spacial locations by token mixing strategy. By pretraining on large-scale private datasets, the proposed method achieve comparable results as the standard CNN and transformer networks.
main_review: -This paper is the first work show strong results by only using MLP layers. But this idea is not too difficult to think, given the recent great successes of transformer based vision (ViT) models. Most of techniques and experiences are from ViT, and the only changes is to replace the attention layers by the MLP layers. 

-The quality of the papers is good. Experiments are sufficient and most of the claims are supported. But this paper is a little bit obsessed with classification task, especially on ImageNet. I don't quite understand the point to push forward the ImageNet top-1 accuracy, e.g. over 90%, by pretraining a super large model on a super large private dataset, especially when ImageNet itself has some annotation errors. And Table 3 reports "Avg.5"? VTAB is quite common for this kind of Google papers. 

-The paper is quite clear and easy to read.

-I am not confident about the significance of this paper. We can say it is a reduced alternative to ViT, by replacing attention layer with token-mixing MLP layer. But we already have CNN and ViT, why do we still need MLP-Mixer? As claimed in the paper, it is an alternative to CNN and ViT, then what are the advantages and disadvantages of MLP-Mixer, for classification and beyond classification? I don't think MLP-Mixer can replace CNN or transformer and can do some tasks that are unavailable for CNN or transformer.

-I think this paper overclaim the significance of MLP-Mixer as alternative to CNN or transformer. As mentioned in the paper, it is a special case of CNN, with 1x1 and global depth-wise convolution. How could a special case be an alternative for a general case?

================after rebuttal================

I am disappointed by the rebuttal. The rebuttal doesn't have satisfactory responses to the two major concerns: 1) the application limit of this paper and 2) the overclaimed significance. What's worse is that the authors seem not to be willing to improve them. Given these, I will downgrade my rating.
limitations_and_societal_impact: No much about limitations. See above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use MLPs in Image Classification and obtains correct performances on ImageNet and VTAB .
Their architecture is similar to vision transformers except that it replaces the self attention with an MLP (similar to the FFN of transformers) applied on the tokens.
main_review: 
Strengths:
- The results obtained are good for an MLP architecture that is simple. This results are very interesting and very aligned with the recent work on vision transformers in computer vision.
- Results are reported on ImageNet, ImageNet-real and VTAB  with pre-training with different amount of date which is a good practice and very useful. Nevertheless it would be interesting to have also the results on ImageNet-v2 [1].
- The paper is well written and easy to follow.


Weakness:
- The comparison on ImageNet without extra-training data with other types of architecture is quite incomplete. There are only comparisons with BiT and ViT which are not state of the art in Image classification for convnets and transformers architecture. 
It would be interesting to have comparisons with EfficientNet-v2 [2], NFNet [3], CaiT [4], Swin [5] or CvT [6]. 
For the comparison on ImageNet with extra-data, MLP-mixer is compared with more architecture but it lacks some state-of-the-art approaches like EfficientNet-v2 [2], Swin [5] or CvT[6]. 
This may bias the conclusion: "MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models" L9
Nevertheless, the paper indicates that the MLP mixer is still below ViT: "When trained from scratch on ImageNet, Mixer-B/16 achieves a reasonable top-1 accuracy of 76.44%. This is 3% behind the ViT-B/16 model" L182

Minor comments:

- The ViT results pre-trained on ImageNet with extra regularization are quite low in comparison to the DeiT paper. One may wonder why the paper does not use the same training procedure.

- FLOPs of the models are not reported. It would be very useful for the comparison with other architecture of the literature since it is a fairly commonly used metric. This metric is not perfect but the throughput also has some flaws as it depends a lot on the implementation, the hardware, ....

[1] Recht et al.,  Do ImageNet Classifiers Generalize to ImageNet? 
[2] Tan et al., EfficientNetV2: Smaller Models and Faster Training
[3] Brock et al. High-Performance Large-Scale Image Recognition Without Normalization
[4] Touvron et al. Going deeper with Image Transformers
[5] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
[6] Wu et al. CvT: Introducing Convolutions to Vision Transformers

limitations_and_societal_impact: Yes the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: Initially, the paper received scores 7, 7, 6, and 5. All reviewers consider the idea is very interesting, experiments are comprehensive, the paper is well-written, and results are strong given the simplicity of the model. At the same time, multiple concerns were raised, specifically pointing out limitations of the model ( e.g., inability to handle multiple resolutions and tasks such as detection/segmentation; the model is data-hungry, parameter-inefficient, and tailored to specialized hardware/TPUs). During the discussion phase, two reviewers were disappointed by the author response, which did not address well questions such as reporting ImageNet results from other architectures, adopting the same training procedure as in DeiT, and over claimed significance of results. They downgraded their rating from 7 to 6, still recommending borderline acceptance. While the concerns raised by the reviewers are all legitimate, the AC considers that the strengths of the paper outweighs its weaknesses, and agrees with the majority that it passes the acceptance bar of NeurIPS. In particular, the proposed architecture has the potential to make an impact and inspire further research in the field, as it is conceptually simple yet achieves strong results. The authors should include in the camera-ready the discussion based on reviewer’s comments, properly convey the limitations of the architecture, and also address the two ethical concerns pointed out in the ethics review.
Decision: Accept (Poster)