Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes an MLP-based architecture for sequence-based DL tasks (vision, NLP). Basically, the idea is to remove the self-attention primitive from Transformers. Instead, it transposes the seq and channel dimension and applies linear projections. This is the same idea as the concurrent work MLP-Mixer from Google. The spatial mixing is now basically a DWise Conv with full receptive field shared across channels. Paper shows reasonable results on ImageNet benchmark (needs a lot more params and worse on throughput for the same top-1 accuracy wrt ViT, and also worse than pure ConvNet baselines). Paper has plenty of ablations on the training recipe and architecture. NLP tasks only include short sequence benchmarks like Translation, with promising signs.
main_review: Pros:
1. Timely paper - along with several concurrent works - on exploring MLPs for sequence models
2. Removing LayerNorm and CLS tokens
3. Benchmarking on multiple tasks and modalities
4. Good ablation studies

Cons:
1. Lukewarm results (not the fault of authors, but it is super clear self-attention is absolutely necessary to get better numbers)
2. Paper sells the idea of FCNs a lot - but it's pretty clear the architecture is a reparametrized ConvNet in disguise - locality and sparsity are what make it work well and scale well
3. Should show more results on a different vision task - ex detection/segmentation where pairwise interactions are known to help/matter more
4. No results on BERT benchmarks would be nice to have.
5. Studies on pre-training with more data.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces Residual Multi-Layer Perceptrons (ResMLP): a purely multi-layer perceptron (MLP) based architecture for image classification. ResMLP takes image patches as input, projects them with a linear layer, and sequentially updates them in turn with two residual operations: (i) a simple linear layer that provides interaction between the patches, which is applied to all channels independently; and (ii) an MLP with a single hidden layer, which is independently applied to all patches. At the end of the network, the patches are average pooled, and fed to a linear classifier. The main change from ViT to ResMLP is replacing the self-attention layer with a simple linear layer cross patches. There are some other small changes, like pooling layers and normalization layers.

The claimed contributions are:
1. despite their simplicity, Residual Multi-Layer Perceptrons reach surprisingly good accuracy/complexity trade-offs with ImageNet-1k training only. From this point of view, this "surprisingly good" is very subjective. At least, from the results, the accuracy/complexity trade-offs is not as good as Vision Transformers (DeiT models).
2. The ResMLP benefits from distillation methods and also works well with self-supervised method DINO.
3. The simple linear layer enables observations on the spatial interaction that the network learns across layers.
4. The authors also try ResMLP on machine translation. 


main_review: This paper provides a comprehensive study on ResMLP, which aims to replace self-attention layers in ViT with MLPs. There are lots of experiments (trained on ImageNet1K only) to show the effect of different designs and training recipes. It is a good literature for researchers who are interested in this area but do not have rich resources to conduct such extensive studies.

However, from my understanding, the results in fact seem to be negative for ResMLP, when compared with ViT. Therefore, I doubt that the main contribution of this paper, "ResMLP reaches surprisingly good accuracy/complexity trade-offs with ImageNet-1k training only", is valid or not in this paper. In Table 1, when we examine the trade-off between Top-1 acc and throughput, the Transformer networks clearly win over the feedforward networks. Moreover, because linear layers are highly optimized in Pytorch, there should be no excuse from the implementation of ResMLP. Compared with ResNets, both ViTs and ResMLPs removed the strong inductive bias, e.g., locality and translational invariance. Although ResMLP (with a linear layer) sounds simpler than ViT (with a self-attention layer), their cross patch interactions are equally general and I do not think that ResMLP use less inductive bias than ViT. From my understanding, "minimal inductive bias" and "good accuracy/complexity trade-offs" are the main claimed advantage of ResMLP. But I do not see its advantage over its exact baseline, ViT. Do we have a positive result for ResMLP compared with ViT counterparts?

Transformers has shown to be very successful in the claimed 2/3/4 contributions. We can only see that ResMLP can also do 2/3/4, but in fact, from the current results, seems that transformers are even doing better in terms of the 2/3/4 contributions. 

Detailed comments:
1. The comparison between "ResMLP-S12 with linear -> conv 3x3 depth-wise" and its baseline "ResMLP-S12" in Table 3. Why there is only little parameter saving compared with the baseline (15.4 ->14.9)? Is it because that the main number of parameters is in the channel-wise MLP layers? If so, it will be helpful for the authors to provide the comparison of number of parameters in patch-wise interaction layers and the channel-wise MLP layers.
2. Line 217: "differential" should be difference.
3. In Table 4, ResMLP performs clearly worse than ResNets on the last three datasets (cars/iNat18/iNat19). The first three sets (Cifar10, Cifar100, Flowers102) have very small domain gap with the pretrained ImageNet1k dataset, but the last three are expected to have larger domain gap. Does this table show that ResMLP is worse than ResNets in terms of generalization? 
Line 267-268: "We initialize the LayerScale parameter to 0.2" I guess that the authors do a hyperparameter search here. Then does this mean that the default small initialization of LayerScale does not work here? Why? 

In Appendix A:
Class-attention!class-MLP: "If we further replace the class-attention layer of CaiT by a MLP as
described in our paper, then we obtain an attention-free network whose top-1 accuracy on Imagenet-val is 79.2%". Which model is this 79.2% one in the expansion factor table? The x0.5 one, or the x1 one?

Ablation on the size of the communication MLP: "For the MLP that replaced the class-attention, we have explored different sizes of the latent layer, by adjusting the expansion factor e ... This has motivated the choice of adopting a simple linear layer of size N ×N: This subsequently improved performance to 79.5% in a
setting comparable to the table above". The ablation study of expansion factor e is on the class-attention layers here, but then it motivated the choice of adopting a simple linear layer in the layers that replacing self-attention with two-layer MLPs? Have the authors tried to search for a better expansion factor e for two-layer MLPs in the patch-wise interaction layers, i.e., the "Patch communication: linear->MLP" in Table 3? In Table 3, it seems that "Patch communication: linear->MLP" performs better than the final simple linear choice. This conflicts with the claim here that simple linear layer improves the performance... Can the authors provide a concrete comparison with the two-layer MLP and a simple linear layer for replacing self-attention layers? 

In Appendix D
Table D.1: This table is wrong? "ResNet-50 training 90 epochs" should be around 76.5 (in stead of the reported 69.2) and the "DeiT-style training" should be much higher than the reported number 76.6. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Recently, vision Transformers have been popular and achieve SOTA performance on various tasks. This paper proposes a more simple variant of the Transformer architecture, *i.e.*, connect image patches via simple linear/MLP layers. The proposed ResMLP shows promising performance under various setups, *e.g.*, supervised and self-supervised training, and on different datasets. It is a new exploration of the methodology of neural architectures.
main_review: **Originality:** 
Using MLP-like modules to construct the Transformer network, instead of self-attention, is novel. Though this idea is also proposed in concurrent works, this paper provides some new insights about MLP-like architectures, e.g., the affine transformation module to help training. 

**Quality:** 
The empirical studies are adequate. The experiments are performed on various datasets and setups. It is suggested to provide some theoretical insights if possible.

**Clarity:** 
The submission is written clearly and easy to follow.

**Significance:** 
The MLP-like architecture is a novel variant of Transformer architectures, which may bring some heuristic ideas for future explorations about this field. 

Here are some questions and issues to be answered or resolved.
1. Though ResMLP shows promising performance in various experiments, the MLP-like architecture still does not show an evident advantage over attention/convolution-based ones, *e.g.*, ResMLP-S24 shows a lower accuracy than DeiT-S with worse throughput and FLOPs. I admit that the MLP architecture is a novel attempt at the Transformer structure. However, I still want to hear about the real advantage/potential MLP carries.
2. The linear-based patch interacting manner first transposes the matrix and then applies the linear projection, which equals a depthwise convolution with a global receptive field and shared parameters over channels. So what is the intrinsic difference between the MLP design and convolution? And it is more reasonable to compare ResMLP with a global receptive field (14x14) depthwise convolution in Tab. 3. In addition, the depthwise convolution in Tab. 3 shows similar accuracy with a smaller budget, *e.g.*, Params, and FLOPs, which makes the advantage of the MLP architecture confusing.
3. Some typos need to be checked and resolved, *e.g.*, "a convolutional layers" in line 90.
limitations_and_societal_impact: As the authors indicate, ResMLP is only studied on image classification. Explorations on downstream tasks are expected if possible.
The main concern lies in the real value MLP carries, which is suggested to be demonstrated in rebuttal.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduce the ResMLP architecture. An alternative to self-attention layer in vision transformer using simply a $T \times T$ linear layer to allow long-range communication between tokens. They show that the performance of this model is comparable to the original vision transformer in supervised, self-supervised and transfer learning as well as knowledge distillation. 
main_review: 
I would like to thank the authors for this interesting piece of work.
The proposed model is simple and well motivated. The extended ablation studies give a good insight of what makes the success of different token mixing methods.
The empirical performance is competitive with Vision Transformers.
The main limitation of the proposed architecture with linear token communication is that it operates on fixed size input images, contrary to convolutional or attention networks combined with a global pooling strategy.
Also it cannot learn data-dependant long range dependancies, which is verified by the similar performance of conv 3x3.
I would expect ResMLP to fail on the Long Range Arena [1] but I am curious of the results, maybe you could mention it in future work.

**Concerning communication accross patches.**
The extensive ablation studies are very insightful. The comparison with concurent work using MLP mixing is interesting.
MLP communication improves performance compared to Linear layer with only a minor increase in number of parameter, but with a significant increase in FLOPS.
The most insightful discussion is on comparing with conv 3x3: in most aspects convolution is better than linear mixing.

**Concerning self-supervised learning with DINO.**
One of the limitation of the linear communication of ResMLP compared to multi-head self-attention in Vision Transformers is that there is no attention maps per head to extract segmentation maps. The data independent ````''single head'' linear mixing would not exhibit object-centric behavior similar to Figure 3 in DINO.

**Application to Neural Machine Translation.**
It seems that you did not apply ResMLP to cross-attention, is there a fundamental limitation that could not allow it?
How do you handle masking in the decoder? Are you simply masking the weights with a triangular matrix?



Have you tried to process upscaled images at resolution 384x384 as it is commonly done with vision transformer?
The proposed method of only having a linear mixing layer gives a principle way to interpolate the weights of the matrix compared to concurrent patch communication approaches such as MLP-mixer. It might not even require finetuning.

Remarks:
- Figure 3: can you show the sparsity metric at initialization to have an horizontal bar as a reference?
- Missing related work [2] on early application of vision transformer. They showed that the learned attention maps were sparse and localised similarly to convolution which complement your finding on ResMLP.

[1] Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler
abs/2011.04006

[2] On the Relationship between Self-Attention and Convolutional Layers
Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi
ICLR 2020
limitations_and_societal_impact: N/A
review_title: null

Meta-review: In summary, the reviewers see the paper as very close to the acceptance threshold with no reviewer opposing acceptance, but no reviewer strongly arguing for acceptance, either. Half of the reviewers are leaning towards accepting, half are leaning towards rejecting.

The paper discusses an MLP-based architecture for DL tasks, which results in surprisingly good performance on vision and NLP tasks.

Highlights of the paper mentioned by the reviewers include:
* This paper is concurrent (and partially complimentary) work with several other papers exploring a similar direction (as noted in footnote 1 and pointed out in reviews). This may indicate an interest in this line of work in the community.
* The paper is well-written and the model is well-motivated.

The main concerns expressed by the reviewers include the following:
* While the performance that is reached may be interesting, it is not clear whether the architecture can be relevant in practice when compared to ViTs or ResNets.
* Some related work could be discussed more appropriately.

Overall, the paper is very close to the acceptance threshold, but sufficient support for acceptance has not been reached.
Decision: Reject