ResMLP: Feedforward networks for image classification with data-efficient training
Authors: Hugo Touvron
Piotr Bojanowski
Mathilde Caron
Matthieu Cord
Alaaeldin El-Nouby
Edouard Grave
Gautier Izacard
Armand Joulin
Gabriel Synnaeve
Jakob Verbeek
Herve Jegou
Venue: NeurIPS 2021 Submitted (2021)
Keywords: deep learning
neural networks
image classification
machine translation
transformer

Abstract: We  present ResMLP an architecture built entirely upon multi-layer perceptrons  for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii)  a two-layer feed-forward network in which channels interact independently per patch. 
When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. 
We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. 
Finally, by adapting our model to machine translation we achieve  surprisingly good results. 

We will share our code based on the Timm library and  pre-trained models.

32 This architecture strongly inspired by the vision transformers (ViT) [16], yet much simpler in several ways: we do not use any form of attention, only linear layers along with the GELU ResMLP: Feedforward networks for image classification with data-efficient training
Anonymous Author(s)
Affiliation Address
email
Abstract
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) two-layer forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexit trade- offs on ImageNet. We also train ResMLP models in a self-supervised setup. to further remove priors from employing labelled dataset. Finally, by adapting model to machine translation we achieve surprisingly good results.
3
5
10
We will share our code based on the Timm library and pre-trained models.
Introduction
11
12 Recently, the transformer architecture [60], adapted from its original use natural language pro- cessing with only minor changes, has achieved performance competitive with the state of the art ImageNet-Ik [50] when pre-trained with sufficiently large amount of data [16]. Retrospectively, this achievement is yet another step towards less priors: convolutional neural networks had removed a lot of hand-made choices compared to hand-designed pre-CNN approaches, moving the paradigm of hard-wireh features to hand-designed architectural choices. Vision transformers avoid making assumptions inherent to convolutional architectures and noticeably the translation invariance.
14
16
17
18
What these recent transformer-basee works suggest is that longer training schedules, more parameters, more data [16] and/or more regularization [56], are sufficient to recover the important priors for tasks complex ImageNet classification. See also discussion of related work in Section 4. This 22 concurs with recent studies [2, 15] that better disentangle the benefits from the architectures from those of the training scheme.
20
23
24 In this paper, we push this trend further, and propose Residual Multi-Layer Perceptrons (ResMLP): purely multi-layer perceptron (MLP) based architecture for image classification. We outline our architecture in Figure and detail it further in Section It is intended to simple: it takes image patches input, projects them with a linear layer, and sequentially updates them in turn with two residual operations: (i) a simple linear layer that provides interaction between the patches, which is applied to all channels independently; and (ii) an MLP with single hidden layer, which is independently applied to all patches. At the end of the network, the patches are average pooled, and fed to linear classifier.
25
26
28
30
34 lin-aiiyy [25]. Since our architecture much more stable train than transformers, we do not
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

X8
A-T
Linear T
Linear
Linear
A
Poo
Linear
Class
Figure 1:
The ResMLP architecture: After linearly projecting the image patches, our network alternately processes them by (1) communication layer between vectors implemented as linear layer; (2) two-layer residual pereeptron. We denote by the Affine element-wist transformation, and by transposition.
35 need batch-specific or cross-channel normalizations such as BatchNorm, GroupNorm or LayerNorm.
36 Our training procedure mostly follows the one initially introduced for DeiT [56] and CaiT [57].
37 Due to its linear nature, the patch interactions in our model can be easily visualised and interpreted. While the interaction pattern learned in the first layer is very similar to a small convolutional filter,
39 we observe more subtle interactions across patches in deeper layers. These include some form of axial filters, and long-range interactions early in the network.
41 In summary, this paper, we show that
despite their simplicity, Residual Multi-Layer Perceptrons reach surprisingly good accuracy/com- plexity trade-offs with ImageNet-1K 1 training only', without requiring normalization based on batch channel statistics;
43
44
45
these models benefit significantly from distillation methods [56]; they are also compatible with modern self-supervised learning methods based on data augmentation, such DINO [6];
46
thank to its design where patch embeddings simply "communicate" through linear layer, can make observations on the spatial interaction that the network learns across layers;
48
49
we adapt ResMLP to machine translation, and again obtain surprisingly good results.
Method
50
51
Our model, depicted in Figure 1, is inspired by the VIT model, from which we adopt the columnar structure with fixed-resolution feature maps. We proceed two drastic simplifications. We refer the reader to Dosovitskiy et al. [16] for more details about the VIT architecture.
52
53
54
The overall ResMLP architecture, Our model, denoted by ResMLP, takes grid of N non- overlapping patches as input, where the patch size is typically equal to 16 x 16. The patches are then independently passed through linear layer form set of N2 d-dimensional embeddings.
55
56
57
The resulting set of 22 embeddings are fed to a sequence of Residual Multi- Layer- Perceptron layers to produce set of 2 d-dimensional output embeddings. These output embeddings then averaged 59 as a d-dimension vector to represent the image, which is fed to linear classifier to predict the label associated with the image. Training uses the cross-entropy loss.
61
The Residual Multi-Perceptron Layer. Our network is sequence of layers that all have the same structure: linear sublayer followed by a feedforward sublayer. Similar the Transformer layer, each sublayer is paralleled with skip-connection [23]. We do not apply Layer Normalization [i] because training is stable without it when using the following Affine transformation:
63
o = Diag(a)x +3.
(1)
where and & are learnable weight vectors. This operation simply rescales and shifts the input 88 element-wise. Moreover, it has no cost at inference time, as it can absorbed in the adjacent linear
87 layer. Note, when writing Af (X) the operation is applied independently
each column of X.
88 While similar to BatchNorm [30] and Layer Normalization [1], the Aff operator does not depend on
89 any batch statisties. Therefore, is closer the recent LayerScale method [57], which improves the
'Concurrent work by Tolstikhin performance with larger MLP models pre-trained on the larger public ImageNet-22k and even more data with the proprietary JFT-300M. In contrast, we focus faster models trained on ImageNet 1k. Other concurrent related work includes that of Melas Melas-Kyriazi [39] and the RepMLP [14] and gMLP [38] models.
[55] brings complementary insights to ours: they achieve interesting

---

70 optimization of deep transformers when initializing a to small value. Note, LayerScale does not
71 have a bias term.
72 We apply this transformation twice for each residual block. As as a pre-normalization Aff replaces
73 the LayerNormalization, and avoids using channel-wise statistics. Here, we initialize
and
74 3 = 0. As a post-processing of the residual block, Aff implements LayerScale and therefore we
75 follow the same small value initialization for a as in [57] for the post-normalization.
76 Finally, we follow the same structure for the feedforward sublayer as in the Transformer; we only
77 replace the ReLU non-lineati con-linearity by a GELU function [25].
78 Overall, our Multi-layer perceptron takes set of N2 d-dimensional input features stacked in a d x 22 matrix X, and outputs set of N2 d-dimension output features, stacked in matrix Y with
80 the following set of transformations:
Z = X+AEE (A AFE (x)
(2)
Y = + (C GELU( Aff(Z))),
(3)
81 where A, B and C are the main learnable weight matrices of the layer. The dimensions of the 82 parameter matrix A are N2 x x22, i.e., this sublayer exchanges information across all the locations, while the feedforward sublayer works per location. As consequence, the intermediate activation matrix Z has the same dimensions as the matrices and Y. Finally, the weight matrices B and C have the same dimensions as in Transformer layer, which are 4d d and d 4d, respectively.
84
The main difference compared to a Transformer layer is that we replace the self-attention by the linear interaction defined in Eq. (2). While self-attention computes convex combination of other features with coefficients that are data dependent, the linear interaction layer Eq. (2) computes general linear combination using learned coefficients that are not data dependent. As compared 90 to a convolutional layers which have local support and share weights across space, our linear patch interaction layer offers global support and does not share weights, moreover it is applied independently across channels.
86
87
86
92
93
Relationship to the Vision Transformer. Our model can be regarded as a drastic simplification of the ViT model by Dosovitskiy et al. [16]. We depart from this model as follows:
94
95
We do not include any self-attention block. Instead we have linear patch interaction layer without any non-linearity.
96
We do not have the extra "class" token that is typically used in these models to aggregate information via attention. Instead, we simply use average pooling. We do, however, also consider a specific aggregation layer variant, which we describe in the next paragraph.
98
99
100
We do not include any form of positional embedding: the linear communication module between patches implicitly takes into account the patch position.
102
Instead of mre-LayeNormmlizztion, we use a simple learnable affine transform, thus avoiding any form of batch and channel-wise statistics.
103
104
Class MLP.. As an alternative to average pooling, we also experimented with an adaptation of the class-attention introduced in CaiT [57]. In CaiT, this consists of two layers that have the same structure as the transformer, but in which only the class token is updated based on the frozen patch embeddings. We translate this method to our architecture, except that, after aggregating the patches with linear layer, we replace the attention- based interaction between the class and patch embeddings by simple linear layers, still keeping the patch embeddings frozen. This increases the performance, at
105
106
107
108
109
110
the expense of adding some parameters and computational cost. We refer to this pooling variant as Mlass-MLP* since the purpose of these few layers to replace average pooling.
111
3 Experiments
113
In this section, we present experimental results for our ResMLP architecture for image classification. 114 We also study the impact of the different components series of ablations. We consider three training paradigms in our experiments:
3

---

Table 1: Comparison between architectures ImageNet classification. We compare different architectures based on convolutional networks, Transformers feedforward networks with comparable FLOPs and number of parameters. We report Top-1 accuracy on the validation set of ImageNet-1K Ik with different measure of complexity: throughput, FLOPs, number of parameters and peak memory usage. All the models use 224 224 images input. By default the Transformers and feedforward networks uses 14x 14 patches of size 1 16, Table detailed specification our main models. The throughput is measured on a single V100-32GB GPU with batch size fixed to 32. For reference, we include the state of
with ImageNet training only.
Arch.
#params (x10%)
throughput (im/s)
FLOPS
Peak Mem
Top-1
(x10")
(MB)
Acc.
CaiT-M481448T [57]
356
5.4
329.6
5477.8
86.5
State of the art
NfNet-F6 SAM [5]
438
16.0
377.3
5519.3
86.5
RfficientNet-BB [53|
661.8
1.8
1174.0
81.1
EfficientNet-BB [53]
349.4
4.2
1898.9
82.6
EfficientNet-B. [53]
169.1
9.9
2734,9
83.3
Convolutional networks
RegNerY-4GF [47]
21
861.0
4.0
568.4
80.0
RegNetY-8GFF [47]]
534.4
8.0
841.6
81.7
RegNetY-16GF [47]
334.7
16.0
1329.6
82.9
DeiT-S [56]
22
940.4
4.6
217.2
79.8
Transformer networks
DeiT-B [56]
86
292.3
17.5
573.7
81.8
CaiT-XS24 [57]
27
447.6
5.4
245.5
81.8
ResMLP-S12
1415.1
3.0
179.5
Feedforward networks
RcsMLP-S24
715.4
6.0
235.3
79.4
ResMLP-B22
116
231.3
23.0
663.0
81.0
Supervised learning: We train ResMLP from labeled images with softmax classifier and cross- entropy loss. This paradigm is the main focus of our work.
117
118
Self-supervised learning: We train the ResMLP architecture without labels. We consider the DINO method of Caron et al. [6] that trains network by distilling knowledge from previous instances of the same network, leading to form of self-distillation without labels.
118
120
Knowledge distillation: We employ the knowledge distillation procedure proposed by Touvron et al. [56] to guide the supervised training of ResMLP with a convnet.
122
123
3.1
Experimental setting
124
Datasets. We train our models on the ImageNet-Ik lk dataset [50], that contains 1 1.2M images evenly spread over 1,000 object categories. In the absence of available follow the standard practice in the community by reporting performance on the validation set. This ideal since the validation methods on this set may not be conclusive enough because an improvement in performance may not be caused by better modeling, but by better selection of hyper-paramettrs. To mitigate this risk, we report additional results in transfer learning and on two alternative versions of ImageNet that have been built have distinct validation and test sets, namely the ImageNet-real [3] and ImageNet-v2 [49] datasets. We also report a few data-points when training ImageNet-21k. Our hyper-parametee are mostly adopted from Touvron et al. [56, 57].
125
for this benchmark,
126
was originally designed to select hyper-pas apper-parmmttrs... Comparing
127
128
129
131
135
134
Hapame-er settings. In the case of supervised learning, we train our network with the Lamb optimizer [63] with a learning rate of 5 x 10 3 and weight decay 0.2. We initialize the LayerScale parameters as a function of the depth by following CaiT [57]. The rest of the hyper ruper-parameters follow use the same RegNety- 16GF [48] as in Dei'T with the same training schedule. The majority of our models take two days train on eight V100-32GB GPUs.
135
136
137
the default setting used in DeiT [56]. For the knowledge distillation paradigm,
139
140
3.2
Main Results
141
In this section, we compare our architecture to models with more conventional network architectures of comparable size and throughput ImageNet.
142
143
Comparison with Transformers and convnets in supervised setting. In Table 1, we compare ResMLP with different convolutional and Transformer architectures. For completeness, report

---

Table 2: Self-supervised learning with DINO [6]. Classification accuracy on ImageNet-Ik Ik val. ResMLPs
evaluated with linear and I--N evaluation on ImageNet are comparable to convnets but inferior VIT.
Models
ResNet-50
ViT-S/16
ViT-S/8
ViT-B/16
ResMLP-S12
ResMLP--22
Params. (x109)
25
22
22
87
15
30
FLOPS (x10%)
4.1
4.6
22.4
17.5
3.0
6.0
Linear k-NN
75.3
77.0
79.7
78.2
67.5
72.8
67.5
74.5
78.3
76.1
62.6
69.4
145
the best-published numbers obtained with a model trained on ImageNet alone. As expected, in terms of the trade-off between accuracy, FLOPs, and throughput, ResMLP is not as efficient as convolutional networks or Transformers. However, their accuracy is encouraging: we compare them with architectures that have benefited from years of research and careful optimization towards these trade-offs. Overall, our results suggest that the structural constraints imposed by the layer design do not have a drastic influence on performance, especially when training models with enough data and recent advances in training and regularization.
148
147
148
149
150
152
Self-supervised pre-training of ResMLP. We explore the possibility of training ResMLP using DINO, a recent self-supervised learning approach [6]. We pre-train ResMLP-S12 models with this approach during 300 epochs. We report our results in Table 2. As expected given the supervised classification results, the accuracies obtained with ResMLP are less good than with ViT. Nevertheless, the performance is surprisingly high for a pure MLP architecture and competitive with Convnet in knn evaluation. We hope that these result will serve as a baseline for future work.
153
154
155
156
157
After self-super pre- pre-training, we also fine-tur mune the network ImageNet using ground truth labels. This pre-training substantially improves the accuracy, when comparing with the same model ResMLP-S24 solely trained with labels (top-1 acc. of 79.9% on ImageNet-val instead of 79.4% for ResML.--$24, with the same total number of epochs). Results on ImageNet-v2 suggest that it reduces overfitting (68.6% on ImageNet-v2. vs 67.9% with supervised training only).
159
160
161
162
163
Improving models with knowledge distillation. We study our model when training following the knowledge distillation approach of Touvron et al. [56]. In their work, the authors show the impact of training a ViT model by distilling from a RegNet. In this experiment, we explore if ResMLP also benefits from this procedure and summarize our results in Table 3 (Blocks "Baseline models" and "Training"). . We observe that similar to DeiT models, ResMLP greatly benefits from distilling from a convnet. This result concurs with the observations made by d' Ascoli el al. [13], who used convnets to initialize feedforward networks. Even though our setting differs from theirs in scale, the problem of overfitting for feedforward networks is still present on ImageNet. The additional regularization obtained from the distillation is possible explanation for this improvement.
165
167
168
169
170
171
Visualisation. Because they are linear, our patch interaction layers from Eq. (2) are casily inter- pretable. In Figure we visualise the rows of the interaction matrices A as X images, for our ResMLP-S24 model. The early layers show convolution-liks patterns: the learned weights resemble shifted versions of each other and have local support. Interestingly, in many layers, the support also extends along both axes, most prominently seen in layer seven. The last seven layers of the network are different: they consist of a spike for the patch itself and a diffuse response across other patches with larger smaller magnitude; see layer 20.
173
175
176
177
178
179
3.3
Visualization & analysis of the linear interaction between patches
180
Measuring sparsity of the weights The visualizations described above suggest that the linear communication layers are sparse. We analyze this quantitatively in more detail in Figure 3. We measure the sparsity of the matrix A, and compare it to the sparsity of B and C from the per-patch
182
183
MLP. Since there are no exact zeros, we measure the rate of components whose absolute value is lower than 5% of the maximum value. Note, discarding the small values is analogous to the case where we normalize the matrix by its maximum and use a finite-precision representation of weights.
184
186
For instance, with 4-bits representation of weight, one would typically round to zero all weights whose absolute value is below 6.25% of the maximum value.
5

---

Layer 1
Layer 7
Layer 10
Layer 20
Visualisation of the linear layers in ResMLP-S24, For each layer visualise the of the matrix A set of 14 pixel images, for sake of space we only show the rows corresponding to the 6x6 central patches. We observe patterss in the lincar layers that share similaritics with convolutions. appendix we provide comparable visualizations for all layers of a ResMLP-SI -12 model.
Figure 2:
95%
B24/8 1mnet21k a.
Fraction of "small" components
90%
B24/8 distil
80%
Imagenet- V2 top1 acc. (%)
72
60%
70
B224
68
S24
EfficientNet DeiT
40%
66
Linear: N-SN
RegNet
MLP: D->DD->
12
ResMLP
0%
64
1
16
24
77
78
79
80
81
82
83
84
85
Layer
Imagenet-val top1 acc. (%)
Figure 3: Sparsity of linear interaction layers. For each layer (linear and MLP), we show the rate of components whose absolute value is lower than 5% the maximum. Linear interaction layers are sparser than the matrices involved in the per cor-pacch MLP.
Figure 4: Top-1 accuracy on ImageNet-V2 vs. ImageNet-val. ResMLPs tend to overfit slightly more under identical training method. This is partially alle- viated with by introducing more regularization (more data or distillation, see e.g., ResMLP 1---B244---itte
186
The measurements Figure show that all three matrices are sparse, with the layers implementing the patch communication being significantly more so. This suggests that they may be compatible with parameter pruning, or better, with modern quantization techniques that induce sparsity at training time, such Quant-Noise [20] and DiffQ [19]. The sparsity structure, in particular in earlier layers, see Figure. 2, hints that we could implement the patch interaction linear layer with a convolution. We provide some results for convolutional variants in ablation study. Further research network compression beyond the scope of this paper, yet we believe worth investigating in the future.
190
192
194
Communication across patches if we remove the linear interaction layer (linear none), we obtain substantially lower accuracy (-20% top-1 acc.) for a "bag-of-patchhe" approach. We have tried several alternatives for the linear patch interaction layer, which are presented in Table 3 (block "patch communication"). Amongst them, using the same MLP structure as for patch processing (linear - MLP), which we analyze in more details in the supplementary material. The simpler choice of single linear square layer led to a better accuracy/performance trade-off considering that the MLP variant requires compute halfway between ResMLP-S12 and ResMLP-S24 and requires fewer parameters than a residual MLP block.
196
198
200
202
203
The visualization in Figure 2 indicates that many linear interaction layers look like convolutions. In our ablation, we replaced the linear layer with different types of 3 3 convolutions. The depth-wise convolution does not implement interaction across channels as our linear patch communication layer and yields similar performance at a comparable number of parameters and FL.OPs. While full 3 3 convolutions yield best results, they come with roughly double the number of parameters
204
206

---

Ablation
Model
Palch
Params x10"
FLOPS
x10
Variant
lop-1 acc. on ImageNet
real [3]
[49]
ReMMLP-122
16
15.4
layers, working dimension 384
76.6
83.3
64.4
Baseline models
R-s21
16
30.0
6.0
layers, working dimension 384
79.4
85.3
67.9
ResMLP-B24
115.7
23.0
layers, working dimension 768
81.0
86.1
69.0
Normalization
Re:MLP-812
16
15.4
3.0
Art Layemorn
77.7
84.1
65.7
Pooling
ResMLP-S12
16
17.7
3.0
average pooling Class-MLP
77.5
84.0
66.1
ResMLP $12
16
14.9
2.8
linear - nome
56.5
63.4
43.1
Patch
ResMI.P-S12
84.0
communication
ResMLP-S12
16
30.8
6.0
lincar conv 3x3
77.3
84.4
66.7
ResMLP-S12
16
14.9
2.8
linear - conv depth-wise
76.3
83.4
64.6
RaMLP-S12
16
16.7
3.2
linear conv 3x3 depth separable
77.0
84.0
65.5
RssM1P-$11214
14
15.6
4.0
patch size 16x 16x
76.9
83.7
65.0
Patch size
RsMMLP-S12/8
22.1
14.0
patch size 16x16-88x8
79.1
85.2
67.2
ResMLP-B24/8
129.1
100.2
patch size 16x16->8x8
81.0
85.7
68.6
ResMLP-S12
16
15.44
3.0
old old-fashioned (90 epochs)
69.2
76.0
56.
ResM1P-S11
pre-trained SSL (DINO)
83.6
ResMLP-S12
16
15.4
3.0
distillation
77.8
84.6
66.0
Training
ResMLP-S24
16
30.0
6.0
pre-trained SSL (DINO)
79.9
85.9
68.6
ResMLP-S22
16
30.0
6.0
distillation
80.8
86.6
69.8
ResMLP-B244
129.1
100.2
distillation
83.6
88.4
73.4
ResMIP BB24/8
129.1
100.2
trained ImageNet 21k (60 epochs)
84.4
88.9
74.2
Table 3: Ablation. Our default configurations presented in the three first rows. By default we train during 400 epochs. The "old-fashione"" is similar what was employed for ResNet [23]: SGD, 90-epochs waterfall schedule, same augmentations up to variations due to library used.
and FL.OPs. Interestingly, the depth--eparable convolutions combine accuracy close that of full 208
3 3 convolutions with number of parameters and FLOPS comparable to our linear layer. This suggests that convolutions on low-resolution feature maps at all layers is an interesting alternative to the common pyramidal design of convnets, where early layers operate at higher resolution and smaller feature dimension.
210
212
213
3.4
Ablation studies
214
Table 3 reports the ablation study of our base network and summary of our preliminary exploratory studies. We discuss the ablation below and give more detail about early experiments in Appendix A.
215
Control of overfitting. Since MLPs subject to overfitting, show in Fig. 4 control experiment to probe for problems with generalization. We explicitly analyze the differential performance between the ImageNet-val and the distinct ImageNet-V2 test The relative offsets between curves reflect to which extent models are overfitted to ImageNet-val w.r.t. hyper-paramett selection. The degree of overfitting of our MI.P-based model is overall neutral slightly higher to that of other Transformer-basee architectures or convnets with same training procedure.
217
218
219
220
221
Normalization & activation. Our network configuration does not contain any batch normalizations. Instead, we use the affine per-channel transform Aff. This is akin to Layer Normalization [1], typically used in transformers, except that avoid to collect any sort of statisties, since do no need it for convergence. preliminary experiments with pre-norm and post-norm [24], we observed that both choices converged. Pre-normalization in conjunction with Batch Normalization could provide an accuracy gain in some cases, see Appendix A.
223
225
227
We choose to use a GELU [25] function. In Appendix A we also analyze the activation function: ReL.U [22] also gives a good performance, but it was a bit more unstable some settings. We did not manage get good results with SiLU [25] and HardSwish [28].
229
Pooling, Replacing average pooling with Class-MLP, see Section 2, brings a significant gain for negligible computational cost. We do not include it by default to keep our models more simple.
232
Patch size. Smaller patches significantly increase the performance, but also increase the number of flops (see Block "Patch size" Table 3). Smaller patches benefit more to larger models, but only with improved optimization scheme involving more regularization (distillation) more data.
234
235
296
Training. Consider the Block "Training' in Table 3. ResMLP significantly benefits from modern training procedures such those used in DeiT. For instance, the DeiT training procedure improves
7

---

Architecture
FLOPs
Res.
CIFAR10
CI
Flowers103
Cars
iNat
iNat39
EfficientNet-B7 [53]
37.0B
600
98.9
91.7
98.8
94.7
ViT-B/16 [16]
55.5B
384
98.1
87.1
89.5
VIT-1/16 [16]
190.7B
384
97.9
86.4
89.7
Deit-B/16 [56]
17.5B
224
99.1
90.8
98.4
92.1
73.2
77.7
ResNet50 [58]
4.1B
224
96.2
90.0
68.4
73.7
Grafit/ResNet50 [58]
4.1B
224
97.6
92.7
68.5
74.6
ResMLP-S12
3.0B
224
87.0
97.4
84.6
60.2
ResMLP-S24
6.0B
224
98.7
89.5
97.9
89.5
64.3
72.5
Table 4: Evaluation transfer learning. Classification accuracy (top-1) of models trained on ImageNet- 1k for transfer datasets covering different domains. The ResMLP architecture takes 224 x 224 images during training and transfer, while ViTs and EfficientNet-BB work with higher resolutions, see "Res." column.
the performance of ResMLP-S12 by 7.4% compared the training employed for ResNet [23 This is in line with recent work pointing out the importance of the training strategy over the model choice [2, 48]. Pre-training on more data and distillation also improve the performance of ResMLP, especially for the bigger models, e.g., distillation improves the accuracy of ResMLP-B24/8 by 2.6%.
239
240
241
Other analysis. In our early exploration, evaluated several alternative design choices. As in transformers, we could use positional embeddings mixed with the input patches. In our experiments did not any benefit from using these features, see Appendix A. This observation suggests that our linear patch interaction layer provides sufficient spatial communication, and referencing absolute positions obviates the need for any form of positional encoding.
243
245
248
3.5
Transfer learning
248
We evaluate the quality features obtained from a ResMLP architecture when transferring them to other domains. The goal is to assess if the features generated from feedforward network are more prone to overfitting on the training data distribution. We adopt the typical setting where we pre-train model ImageNet- 1k and fine-tune it on the training set associated with specific domain. We
250
252
report the performance with different architectures on various image benchmarks in Table 4, namely CIFAR- 10 and CIFAR- [34], Flowers- 102 [42]. Stanford Cars [33] and iNaturalist [27]. We refer the reader to the corresponding references for more detailed description of the datasets. We observe that the performance of our ResMLP is competitive with the existing architectures, showing that pretraining feedforward models with enough data and regularization via data augmentation greatly reduces their tendency to overfit on the original distribution. Interestingly, this regularization also prevents them from overfitting on the training set of smaller dataset during the fine-tuning stage.
254
256
257
258
259
3.6
Machine translation
We also evaluate the ResMLP caunspose-mechaniie to replace the self-attention in the encoder and decoder of neural machine translation system. We train models on the WMT 2014 English- Gnglish-German and English-Frenel tasks, following the setup from Ott et al. [45]. We consider models of dimension 512, with a hidden MLP size of 2048, and with or 12 layers. Note that the current state of the art
281
263
employs much larger models: 6 layers model is more comparable the base transformer model
285
from Vaswani et al. [60], which serves as a baseline, along with pre-transformer architectures such as Recurrent and convolutional neural networks. We use Adagrad with learning rate 0.2, 32k steps of
286
257
linear warmup, label smoothing 0.1, dropout rate 0.15 for en-de and 0.1 for en-fr. We initialize the LayerScale parameter 0.2. We generate translations with the beam search algorithm, with beam of size 4. As shown in Table 5, the results are at least on par with other architectures:
268
289
Table 5: Machine translation on WMT 2014 translation tasks, We report tokenized BLEU on newstes12014.
Models
GNMT [61]
ConvS2S [21]
Transf. (base) [60]
ResMLP-6
ResMLP-12
EN-DE DE
24.6
25.2
27.3
26.4
26.8
EN-FR
39.9
40.5
38.1
40.3
40.6
Interestingly, if trained with this "old-fashion setting. ResMLP-S12 outperforms AlexNet [35] margin.
8

---

4
Related work
270
271
We review the research on applying Fully Connected Network (FCN) for computer vision problems as well as other architectures that shares common modules with our model.
273
Fully-connected network for images. Many studies have shown that FCNs are competitive with convnets for the tasks of digit recognition [11, 51], keyword spotting [7] and handwritting recogni- tion [4]. Several works [37, 40, 59] have questioned if FCNs are also competitive on natural image datasets, such as CIFAR- 10 [34]. More recently, d' Ascoll et al. [13] have shown that FCN initialized
274
275
278
277
with the weights of pretrained convnet achieves performance that are superior than the original convnet. Neyshabur [41] further extend this line of work by achieving competitive performance by
278
279
training an FCN from scratch but with a regularizer that constrains the models to be close to a convnet. These studies have been conducted on small scale datasets with the purpose of studying the impact of architectures on generalization in terms of sample complexity [18] and energy landscape [31]. In our work, we show that, in the larger scale setting of ImageNet, FCNs can attain surprising accuracy without any constraint or initialization inspired by convnets.
280
281
285
284
Finally, the application of FCN networks in computer vision have also emerged in the study of the properties of networks with infinite width [43], or for inverse scattering problems [32]. More interestingly, the Tensorizing Network [44] is an approximation of very large FCN that shares similarity with our model, in that they intend to remove prior by approximating even more general tensor operations, i.e., not arbitrarily marginalized along some pre-defined sharing dimensions. However, their method is designed to compress the MLP layers of a standard convnets.
285
286
287
288
289
280
Other architectures with similar components. Our FCN architecture shares several components with other architectures, such as convnets [35, 36] or transformers [60]. A fully connected layer is equivalent to a convolution layer with 1 x 1 receptive field, and several work have explored convnet architectures with small receptive fields. For instance, the VGG model [52] uses 3x3 3 convolutions. and later. other architectures such as the ResNext [62] or the Xception [10] mix 1 x 1 and 3 x 3 convolutions. In contrast to convnets, in our model interaction between patches is obtained via a linear layer that is shared across channels, and that relies on absolute rather than relative positions.
281
292
293
294
295
296
More recently, transformers have emerged promising architecture for computer vision [9, 17, 46, 56, 66]. In particular, our architecture takes inspiration from the structure used in the Vision Transformer (ViT) [17], and as consequence, shares many components. Our model takes a set of non-overlapping patches as input and passes them through a series of MLP layers that share the same structure as ViT, replacing the self-attention layer with linear patch interaction layer. Both layers have a global field-of-view, unlike convolutional layers. Whereas in self-attention the weights to aggregate information from other patches are data dependent through queries and keys, in ResMLP the weights are not data dependent and only based on absolute positions of patches. In our implementation we follow the improvements of DeiT [56] to train vision transformers, use the conecconess from ResNets [23] with pre-normaae normaaizzzaion of the layers [8, 24].
298
289
300
301
302
303
304
305
306
307
Finally, our work questions the importance of self-attention in existing architectures. Similar ob- servations have been made in natural language processing. Notably, Synthesizer [54] shows that dot-product self-attention can be replaced by feedforward network, with competitive performance on sentence representation benchmarks. As opposed to our work, Synthesizer does use data dependent weights, but in contrast transformers the weights are determined from the queries only.
308
309
310
311
Conclusion
312
313
In this paper we have shown that a simple residual architecture, whose residual blocks consist of a one-hidden layer feed-forward network and a linear patch interaction layer, achieves an unexpectedly high performance on ImageNet classification benchmarks, provided that we adopt modern training strategy such as those recently introduced for transformer- based architectures. Thanks to their simple structure, with linear layers as the main mean of communication between patches, we can vizualize
314
315
316
317
318
the filters learned by this simple MLP. While some of the layers are similar to convolutional filters,
319
we also observe sparse long-range interactions as early as the second layer of the network. We hope
that our model free of spatial priors will contribute to further understanding of what networks with
321
less priors learn, and potentially guide the design choices of future networks without the pyramidal
design prior adopted by most convolutional neural networks.
9

---

References
323
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.
arXiv preprint
325
arXiv: 160.006500. 2016. 2,7
326
[2] Irwan Bello, W. Fedus, Xianzhi Du, E. D. Cubuk, A. Srinivas, Tsung-Yi Yi Lin, Jonathon Shlens, and Barret
327
Zoph. Revisiting ResNets: Improved training and scaling strategies. arXiv preprint arXiv:2103.0755
328
2021. 1,8
329
[J Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we done with ImageNet? arXiv preprint arXiv:2006 0700000119, 2020. 4,7
330
331
[4] Théodore Bluche. Deep neural networks for large vocabulary handwritten text recognition. PhD thesis.
332
Université Paris-Sud, 2015. 9
333
[5] A. Brock, Soham De, S. Smith, and K. Simonyan. High- performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171 2021. 4
334
335
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal. Piotr Bojanowski, and Armand
336
Joulin. Emerging properties in superviee vision transformers. arXiv preprint arXiv:2104. 14294, 2021. 2.
337
338
77]
Clément Chatelain. Extraction de séquences nunériques dans des documents manuscrits quelconques. PhD thesis, Université de Rouen, 2006. 9
339
340
[8] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, ct al. The best of both worlds: Combining recent advances in neural machine translation. In Annual Meeting of the Association for Computational Linguistics, 2018. 9 Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskover. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10509, 2019. 9
341
343
19]
345
[00] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Conference on Computer Vision and Pattern Recognition, 2017. 9
347
[11]
Dan Claudiu Cireşan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuher. Deep big multilayer perceptrons for digit recognition. In Neural networks: tricks of the trade, pages 581-598. Springer, 2012. 9
349
[12] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with reduced search space. arXiv preprint
350
ptte:/999.11111 2019. VI
351
[13] Stéphane d' Ascoli, Levent Sagun, Joan Bruna, and Giulio Biroli. Finding the needle in the haystack with
352
convolutions: on the benefits of architectural bias. In NeurlPS, 2019. 5.9
353
[14] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. RepMLP: Re-parameterizing con- volutions into fully-connected layers for image recognition. arXiv preprint arXiv:2105.01885 2021.
354
356
[15] Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling, arXiv preprint
arXiv:2103.06877 2021. 1
358
[16] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning
360
361
Representations, 2021. 1, 2, 3, 8
362
[17] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NewrlPS, 2014.
363
364
[18] Simon S Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Aarti Singh.
365
How many samples are needed to estimate convolutional neural network? NeuriPS, 2018. 9
366
[19] Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo
367
quantization noise. arXiv preprint 2rXiv:2104.099)) (04.09987, 2021. 6
368
[20] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. In International Conference on Learning Representations, 2021.
369
370
371
[21]
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1243- 1252. PMLR, 06 11 Aug 2017. 8
372
373
375
[22] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Machine Learning, 2011.
377
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference Computer Vision and Pattern Recognition, 2016. 2, 7, 8, 9, V
379
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks.
arXiv preprint arxiv: 1603 2016. 7.9
381
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), arXiv preprint 180v::6600.08415.
2016. 3.7.7
10

---

383
[26]
Elad Hoffer, Tal Ben-Nun, Iay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Conference on Computer Vision and Pattern Recoonittion, 2020. VI
305
386
127]
Grant Van Hom, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The iNaturalist species classification and detection datasct. arXiv preprint arXiv:1707.066
388
2017. 8 [28] A. Howard, Mark Sandler, G. Chu, Liang-Chied Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Quoe V. Le, and H. Adam. Searching for MobileNerV3, International Conference on Computer
389
390
Vision, 2019. 7
392
[29]
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. VI
393
394
330 S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal
395
covariate shift. In International Conference on Machine Learning. 2015. 2
396
[31] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations. 2017. 9
396
389
[32] Yuehaw Khoo and Lexing Ying. Switchnet: neural network model for forward and inverse scattering problems. SIAM Journal on Scientific Computing, 41(5):A3182 A3201, 2019. 9
401
[33 Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13),
402
403
2013. 8
404
[34] A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of
405
Toronto, 2009. 8,
406
[35] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural
407
networks. NeurlPS, 2012. 8, 9
408
[36] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. 9
410
[37]
Zhouhan Lin, Roland Memisevic, and Kishore Konda. How can we go without convolution: Improving
411
fully-connected networks. arXiv preprint arXiv: 151 1.02500, 2015. 9
412
[38] Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention MLPs. arXiv preprint
413
1rXiv:2105.080:: 2021. 2
414
[39]
Luke Melas Syriazi. Do you even need attention? A stack of feed-forward layers does surprisingly well
ImageNet. arXiv preprint wwXiv2205500222) 2021. 2
416
[40]
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications,, 91)): 2018. 9
417
418
419
[41] Behnam Neyshabur. Towards learning convolutions from scratch. In NeurlPS, 2020. 9
420
[42]
M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 8
422
[43] Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang. Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Bayesian deep convolutional networks with many channels are Gaussian processes. In International Conference on Learning Representations, 2019. 9
423
424
425
[44] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Neurips, 2015. 9
426
[45] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In ACL, pages 1-9. Association for Computational Linguistics, 2018. 8
428
429
[46]
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, 2018. 9
431
[47]
Filip Radenović, Giorgos Tolias, and Ondrej Chum. Fine-tuning CNN image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7):1655 - 1668, 2018. 4 Tija Radosavovie, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. Conference on Computer Vision and Pattern Recognition, 2020. 4,
432
438
[48]
434
435
[49]
B. Recht, Rebecca Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet?
In International Conference Machine Learning, 2019. 4.7
437
50
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 1-2222,,, 2015. 1, 4 Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks
438
440
[51]
441
applied to visual document analysis. In ICDAR, 2003. 9
[22
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In
443
International Conference on Learning Representations. 2015. 9
444
[53]
Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks.
arXiv preprint Xrviv: I 1905. / /946, 2019. 4,8
11

---

448
[54] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng, Synthesizer: Rethinking self-attention in transfomeer models. arXiv preprini TrXi::2005.00745 2020.
448
[55] Ily Tolstikhin, Neil Houlsby, Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An
449
all-MLP architecture for vision. arXiv preprini (KKr:2005.0160) 2021. 2
451
[56]
Hugo Touvron, M. Cord, M. Douze, F. Massa, Alexandre Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint urXiv:2012. 12877. 2020. 1, 2, 4, S, 8,
452
9. I, V
454
[57]
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper
455
with image transformers. arXiv preprint 100: 17239, 2021. 2, 3, 4
456
588
Hugo Touvron, Alexandre Sablayrolles, M. Douze, M. Cord, and H. Jégou. Grafit: Learning fine- fine-grained
457
image representations with coarse labels. arXiv preprini arXiv:2011.12960 2020. 8
[59]
Gregor Urban, Krzysztof J Geras. Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana. Abdelrahman Mohamed, Matthai Philipose, and Mau Richardson. Do deep convolutional nets really need to be deep and convolutional? In International Conference on Learning Representations, 2017. 9
459
460
[60]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS. 2017. 1, 8, 9
462
463
[61]
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le. Mohammad Norouzi, Wolfgang Macherey. Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint ww0iv:/60.00///1 2016.
465
488
[62]
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor- mations for deep neural networks. In Conference Computer Vision and Pattern Recognition, 2017.
468
489
[63]
Yang You, Jing Li. Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. International Conference on Learning Representations, 2020. Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. CutMix: Regularization strategy train strong classifiers with localizable features. arXiv preprint arXiv:1905.04890 2019. VI
472
[64]
474
475
[65]
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.094/ 2017. VI
477
[66]
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Conference on Computer Vision and Pattern Recognition, 2020. 9
478
479
167|
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation.
480
In AAAI, 2020. VI
12

---

Checklist
481
482
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
484
contributions and scope? [Yes]
485
(b) Did you descrihe the limitations of your work? [Yes]
486
(c) Did you discuss any potential negative societal impacts of your work? [No] We only
487
use MLPs for image classification. There is already other system much more powerful therefore the impact is negligible
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
490
491
2. If you are including theoretical results...
492
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
493
(b) Did you include complete proofs of all theoretical results? [N/A]
494
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide code in appendix
496
498
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
499
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Because it's too expensive computational resources. But for some experiments we run multiple times and we have ~ 0.1 as standard deviation (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The majority of our experiments take days on GPUs V100-32GB
501
504
505
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
507
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] But license can be found with the references we mention
509
510
(c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [No] We only use datasets that are widely used in computer vision such as ImageNet or CIFAR
511
512
513
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We only use datasets that are widely used in computer vision such as ImageNet or CIFAR
515
517
5. If you used crowdsourcing conducted research with human subjects...
518
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
519
520
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
523
13


Reviews:
[{"review_id": "hHwHcGvLumN", "reviewer": "Reviewer_rpHJ", "summary": "Recently, vision Transformers have been popular and achieve SOTA performance on various tasks. This paper proposes a more simple variant of the Transformer architecture, *i.e.*, connect image patches via simple linear/MLP layers. The proposed ResMLP shows promising performance under various setups, *e.g.*, supervised and self-supervised training, and on different datasets. It is a new exploration of the methodology of neural architectures.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \nUsing MLP-like modules to construct the Transformer network, instead of self-attention, is novel. Though this idea is also proposed in concurrent works, this paper provides some new insights about MLP-like architectures, e.g., the affine transformation module to help training. \n\n**Quality:** \nThe empirical studies are adequate. The experiments are performed on various datasets and setups. It is suggested to provide some theoretical insights if possible.\n\n**Clarity:** \nThe submission is written clearly and easy to follow.\n\n**Significance:** \nThe MLP-like architecture is a novel variant of Transformer architectures, which may bring some heuristic ideas for future explorations about this field. \n\nHere are some questions and issues to be answered or resolved.\n1. Though ResMLP shows promising performance in various experiments, the MLP-like architecture still does not show an evident advantage over attention/convolution-based ones, *e.g.*, ResMLP-S24 shows a lower accuracy than DeiT-S with worse throughput and FLOPs. I admit that the MLP architecture is a novel attempt at the Transformer structure. However, I still want to hear about the real advantage/potential MLP carries.\n2. The linear-based patch interacting manner first transposes the matrix and then applies the linear projection, which equals a depthwise convolution with a global receptive field and shared parameters over channels. So what is the intrinsic difference between the MLP design and convolution? And it is more reasonable to compare ResMLP with a global receptive field (14x14) depthwise convolution in Tab. 3. In addition, the depthwise convolution in Tab. 3 shows similar accuracy with a smaller budget, *e.g.*, Params, and FLOPs, which makes the advantage of the MLP architecture confusing.\n3. Some typos need to be checked and resolved, *e.g.*, \"a convolutional layers\" in line 90.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the authors indicate, ResMLP is only studied on image classification. Explorations on downstream tasks are expected if possible.\nThe main concern lies in the real value MLP carries, which is suggested to be demonstrated in rebuttal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "QdL_K_Xuwu_", "reviewer": "Reviewer_NGeM", "summary": "This paper introduces Residual Multi-Layer Perceptrons (ResMLP): a purely multi-layer perceptron (MLP) based architecture for image classification. ResMLP takes image patches as input, projects them with a linear layer, and sequentially updates them in turn with two residual operations: (i) a simple linear layer that provides interaction between the patches, which is applied to all channels independently; and (ii) an MLP with a single hidden layer, which is independently applied to all patches. At the end of the network, the patches are average pooled, and fed to a linear classifier. The main change from ViT to ResMLP is replacing the self-attention layer with a simple linear layer cross patches. There are some other small changes, like pooling layers and normalization layers.\n\nThe claimed contributions are:\n1. despite their simplicity, Residual Multi-Layer Perceptrons reach surprisingly good accuracy/complexity trade-offs with ImageNet-1k training only. From this point of view, this \"surprisingly good\" is very subjective. At least, from the results, the accuracy/complexity trade-offs is not as good as Vision Transformers (DeiT models).\n2. The ResMLP benefits from distillation methods and also works well with self-supervised method DINO.\n3. The simple linear layer enables observations on the spatial interaction that the network learns across layers.\n4. The authors also try ResMLP on machine translation. \n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a comprehensive study on ResMLP, which aims to replace self-attention layers in ViT with MLPs. There are lots of experiments (trained on ImageNet1K only) to show the effect of different designs and training recipes. It is a good literature for researchers who are interested in this area but do not have rich resources to conduct such extensive studies.\n\nHowever, from my understanding, the results in fact seem to be negative for ResMLP, when compared with ViT. Therefore, I doubt that the main contribution of this paper, \"ResMLP reaches surprisingly good accuracy/complexity trade-offs with ImageNet-1k training only\", is valid or not in this paper. In Table 1, when we examine the trade-off between Top-1 acc and throughput, the Transformer networks clearly win over the feedforward networks. Moreover, because linear layers are highly optimized in Pytorch, there should be no excuse from the implementation of ResMLP. Compared with ResNets, both ViTs and ResMLPs removed the strong inductive bias, e.g., locality and translational invariance. Although ResMLP (with a linear layer) sounds simpler than ViT (with a self-attention layer), their cross patch interactions are equally general and I do not think that ResMLP use less inductive bias than ViT. From my understanding, \"minimal inductive bias\" and \"good accuracy/complexity trade-offs\" are the main claimed advantage of ResMLP. But I do not see its advantage over its exact baseline, ViT. Do we have a positive result for ResMLP compared with ViT counterparts?\n\nTransformers has shown to be very successful in the claimed 2/3/4 contributions. We can only see that ResMLP can also do 2/3/4, but in fact, from the current results, seems that transformers are even doing better in terms of the 2/3/4 contributions. \n\nDetailed comments:\n1. The comparison between \"ResMLP-S12 with linear -> conv 3x3 depth-wise\" and its baseline \"ResMLP-S12\" in Table 3. Why there is only little parameter saving compared with the baseline (15.4 ->14.9)? Is it because that the main number of parameters is in the channel-wise MLP layers? If so, it will be helpful for the authors to provide the comparison of number of parameters in patch-wise interaction layers and the channel-wise MLP layers.\n2. Line 217: \"differential\" should be difference.\n3. In Table 4, ResMLP performs clearly worse than ResNets on the last three datasets (cars/iNat18/iNat19). The first three sets (Cifar10, Cifar100, Flowers102) have very small domain gap with the pretrained ImageNet1k dataset, but the last three are expected to have larger domain gap. Does this table show that ResMLP is worse than ResNets in terms of generalization? \nLine 267-268: \"We initialize the LayerScale parameter to 0.2\" I guess that the authors do a hyperparameter search here. Then does this mean that the default small initialization of LayerScale does not work here? Why? \n\nIn Appendix A:\nClass-attention!class-MLP: \"If we further replace the class-attention layer of CaiT by a MLP as\ndescribed in our paper, then we obtain an attention-free network whose top-1 accuracy on Imagenet-val is 79.2%\". Which model is this 79.2% one in the expansion factor table? The x0.5 one, or the x1 one?\n\nAblation on the size of the communication MLP: \"For the MLP that replaced the class-attention, we have explored different sizes of the latent layer, by adjusting the expansion factor e ... This has motivated the choice of adopting a simple linear layer of size N ×N: This subsequently improved performance to 79.5% in a\nsetting comparable to the table above\". The ablation study of expansion factor e is on the class-attention layers here, but then it motivated the choice of adopting a simple linear layer in the layers that replacing self-attention with two-layer MLPs? Have the authors tried to search for a better expansion factor e for two-layer MLPs in the patch-wise interaction layers, i.e., the \"Patch communication: linear->MLP\" in Table 3? In Table 3, it seems that \"Patch communication: linear->MLP\" performs better than the final simple linear choice. This conflicts with the claim here that simple linear layer improves the performance... Can the authors provide a concrete comparison with the two-layer MLP and a simple linear layer for replacing self-attention layers? \n\nIn Appendix D\nTable D.1: This table is wrong? \"ResNet-50 training 90 epochs\" should be around 76.5 (in stead of the reported 69.2) and the \"DeiT-style training\" should be much higher than the reported number 76.6. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "No. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "FgqLRNzkLT2H", "reviewer": "Reviewer_x4AE", "summary": "The paper proposes an MLP-based architecture for sequence-based DL tasks (vision, NLP). Basically, the idea is to remove the self-attention primitive from Transformers. Instead, it transposes the seq and channel dimension and applies linear projections. This is the same idea as the concurrent work MLP-Mixer from Google. The spatial mixing is now basically a DWise Conv with full receptive field shared across channels. Paper shows reasonable results on ImageNet benchmark (needs a lot more params and worse on throughput for the same top-1 accuracy wrt ViT, and also worse than pure ConvNet baselines). Paper has plenty of ablations on the training recipe and architecture. NLP tasks only include short sequence benchmarks like Translation, with promising signs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Timely paper - along with several concurrent works - on exploring MLPs for sequence models\n2. Removing LayerNorm and CLS tokens\n3. Benchmarking on multiple tasks and modalities\n4. Good ablation studies\n\nCons:\n1. Lukewarm results (not the fault of authors, but it is super clear self-attention is absolutely necessary to get better numbers)\n2. Paper sells the idea of FCNs a lot - but it's pretty clear the architecture is a reparametrized ConvNet in disguise - locality and sparsity are what make it work well and scale well\n3. Should show more results on a different vision task - ex detection/segmentation where pairwise interactions are known to help/matter more\n4. No results on BERT benchmarks would be nice to have.\n5. Studies on pre-training with more data.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "B2OXmLEGYuG", "reviewer": "Reviewer_Q86e", "summary": "This paper introduce the ResMLP architecture. An alternative to self-attention layer in vision transformer using simply a $T \\times T$ linear layer to allow long-range communication between tokens. They show that the performance of this model is comparable to the original vision transformer in supervised, self-supervised and transfer learning as well as knowledge distillation. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI would like to thank the authors for this interesting piece of work.\nThe proposed model is simple and well motivated. The extended ablation studies give a good insight of what makes the success of different token mixing methods.\nThe empirical performance is competitive with Vision Transformers.\nThe main limitation of the proposed architecture with linear token communication is that it operates on fixed size input images, contrary to convolutional or attention networks combined with a global pooling strategy.\nAlso it cannot learn data-dependant long range dependancies, which is verified by the similar performance of conv 3x3.\nI would expect ResMLP to fail on the Long Range Arena [1] but I am curious of the results, maybe you could mention it in future work.\n\n**Concerning communication accross patches.**\nThe extensive ablation studies are very insightful. The comparison with concurent work using MLP mixing is interesting.\nMLP communication improves performance compared to Linear layer with only a minor increase in number of parameter, but with a significant increase in FLOPS.\nThe most insightful discussion is on comparing with conv 3x3: in most aspects convolution is better than linear mixing.\n\n**Concerning self-supervised learning with DINO.**\nOne of the limitation of the linear communication of ResMLP compared to multi-head self-attention in Vision Transformers is that there is no attention maps per head to extract segmentation maps. The data independent ````''single head'' linear mixing would not exhibit object-centric behavior similar to Figure 3 in DINO.\n\n**Application to Neural Machine Translation.**\nIt seems that you did not apply ResMLP to cross-attention, is there a fundamental limitation that could not allow it?\nHow do you handle masking in the decoder? Are you simply masking the weights with a triangular matrix?\n\n\n\nHave you tried to process upscaled images at resolution 384x384 as it is commonly done with vision transformer?\nThe proposed method of only having a linear mixing layer gives a principle way to interpolate the weights of the matrix compared to concurrent patch communication approaches such as MLP-mixer. It might not even require finetuning.\n\nRemarks:\n- Figure 3: can you show the sparsity metric at initialization to have an horizontal bar as a reference?\n- Missing related work [2] on early application of vision transformer. They showed that the learned attention maps were sparse and localised similarly to convolution which complement your finding on ResMLP.\n\n[1] Long Range Arena: A Benchmark for Efficient Transformers\nYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler\nabs/2011.04006\n\n[2] On the Relationship between Self-Attention and Convolutional Layers\nJean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi\nICLR 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In summary, the reviewers see the paper as very close to the acceptance threshold with no reviewer opposing acceptance, but no reviewer strongly arguing for acceptance, either. Half of the reviewers are leaning towards accepting, half are leaning towards rejecting.

The paper discusses an MLP-based architecture for DL tasks, which results in surprisingly good performance on vision and NLP tasks.

Highlights of the paper mentioned by the reviewers include:
* This paper is concurrent (and partially complimentary) work with several other papers exploring a similar direction (as noted in footnote 1 and pointed out in reviews). This may indicate an interest in this line of work in the community.
* The paper is well-written and the model is well-motivated.

The main concerns expressed by the reviewers include the following:
* While the performance that is reached may be interesting, it is not clear whether the architecture can be relevant in practice when compared to ViTs or ResNets.
* Some related work could be discussed more appropriately.

Overall, the paper is very close to the acceptance threshold, but sufficient support for acceptance has not been reached.
Decision: Reject