ResMLP: Feedforward networks for image classification with data-efficient training
Year: 2021

Abstract:
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for 1 image classification. It is a simple residual network that alternates (i) a linear layer 2 in which image patches interact, independently and identically across channels, and 3 (ii) a two-layer feed-forward network in which channels interact independently per 4 patch. When trained with a modern training strategy using heavy data-augmentation 5 and optionally distillation, it attains surprisingly good accuracy/complexity trade6 offs on ImageNet. We also train ResMLP models in a self-supervised setup, to 7 further remove priors from employing a labelled dataset. Finally, by adapting our 8 model to machine translation we achieve surprisingly good results. 9 We will share our code based on the Timm library and pre-trained models. 10

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for1 image classification. It is a simple residual network that alternates (i) a linear layer2 in which image patches interact, independently and identically across channels, and3 (ii) a two-layer feed-forward network in which channels interact independently per4 patch. When trained with a modern training strategy using heavy data-augmentation5 and optionally distillation, it attains surprisingly good accuracy/complexity trade-6 offs on ImageNet. We also train ResMLP models in a self-supervised setup, to7 further remove priors from employing a labelled dataset. Finally, by adapting our8 model to machine translation we achieve surprisingly good results.9
We will share our code based on the Timm library and pre-trained models.10
1 Introduction11
Recently, the transformer architecture [60], adapted from its original use in natural language pro-12 cessing with only minor changes, has achieved performance competitive with the state of the art on13 ImageNet-1k [50] when pre-trained with a sufficiently large amount of data [16]. Retrospectively,14 this achievement is yet another step towards less priors: convolutional neural networks had removed15 a lot of hand-made choices compared to hand-designed pre-CNN approaches, moving the paradigm16 of hard-wired features to hand-designed architectural choices. Vision transformers avoid making17 assumptions inherent to convolutional architectures and noticeably the translation invariance.18
What these recent transformer-based works suggest is that longer training schedules, more parameters,19 more data [16] and/or more regularization [56], are sufficient to recover the important priors for tasks20 as complex as ImageNet classification. See also our discussion of related work in Section 4. This21 concurs with recent studies [2, 15] that better disentangle the benefits from the architectures from22 those of the training scheme.23
In this paper, we push this trend further, and propose Residual Multi-Layer Perceptrons (ResMLP):24 a purely multi-layer perceptron (MLP) based architecture for image classification. We outline our25 architecture in Figure 1 and detail it further in Section 2. It is intended to be simple: it takes image26 patches as input, projects them with a linear layer, and sequentially updates them in turn with two27 residual operations: (i) a simple linear layer that provides interaction between the patches, which28 is applied to all channels independently; and (ii) an MLP with a single hidden layer, which is29 independently applied to all patches. At the end of the network, the patches are average pooled, and30 fed to a linear classifier.31
This architecture is strongly inspired by the vision transformers (ViT) [16], yet it is much simpler32 in several ways: we do not use any form of attention, only linear layers along with the GELU33 non-linearity [25]. Since our architecture is much more stable to train than transformers, we do not34
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
need batch-specific or cross-channel normalizations such as BatchNorm, GroupNorm or LayerNorm.35 Our training procedure mostly follows the one initially introduced for DeiT [56] and CaiT [57].36
Due to its linear nature, the patch interactions in our model can be easily visualised and interpreted.37 While the interaction pattern learned in the first layer is very similar to a small convolutional filter,38 we observe more subtle interactions across patches in deeper layers. These include some form of39 axial filters, and long-range interactions early in the network.40
In summary, in this paper, we show that41
• despite their simplicity, Residual Multi-Layer Perceptrons reach surprisingly good accuracy/com-42 plexity trade-offs with ImageNet-1k training only1, without requiring normalization based on batch43 or channel statistics;44
• these models benefit significantly from distillation methods [56]; they are also compatible with45 modern self-supervised learning methods based on data augmentation, such as DINO [6];46
• thank to its design where patch embeddings simply “communicate” through a linear layer, we can47 make observations on the spatial interaction that the network learns across layers;48
• we adapt ResMLP to machine translation, and again obtain surprisingly good results.49
2 Method50
Our model, depicted in Figure 1, is inspired by the ViT model, from which we adopt the columnar51 structure with fixed-resolution feature maps. We proceed two drastic simplifications. We refer the52 reader to Dosovitskiy et al. [16] for more details about the ViT architecture.53
The overall ResMLP architecture. Our model, denoted by ResMLP, takes a grid of N×N non-54 overlapping patches as input, where the patch size is typically equal to 16×16. The patches are then55 independently passed through a linear layer to form a set of N2 d-dimensional embeddings.56
The resulting set of N2 embeddings are fed to a sequence of Residual Multi-Layer Perceptron layers57 to produce a set of N2 d-dimensional output embeddings. These output embeddings are then averaged58 as a d-dimension vector to represent the image, which is fed to a linear classifier to predict the label59 associated with the image. Training uses the cross-entropy loss.60
The Residual Multi-Perceptron Layer. Our network is a sequence of layers that all have the same61 structure: a linear sublayer followed by a feedforward sublayer. Similar to the Transformer layer,62 each sublayer is paralleled with a skip-connection [23]. We do not apply Layer Normalization [1]63 because training is stable without it when using the following Affine transformation:64
Affα,β(x) = Diag(α)x+ β, (1)
where α and β are learnable weight vectors. This operation simply rescales and shifts the input65 element-wise. Moreover, it has no cost at inference time, as it can absorbed in the adjacent linear66 layer. Note, when writing Aff(X) the operation is applied independently to each column of X.67 While similar to BatchNorm [30] and Layer Normalization [1], the Aff operator does not depend on68 any batch statistics. Therefore, it is closer to the recent LayerScale method [57], which improves the69
1Concurrent work by Tolstikhin et al. [55] brings complementary insights to ours: they achieve interesting performance with larger MLP models pre-trained on the larger public ImageNet-22k and even more data with the proprietary JFT-300M. In contrast, we focus on faster models trained on ImageNet-1k. Other concurrent related work includes that of Melas-Kyriazi [39] and the RepMLP [14] and gMLP [38] models.
optimization of deep transformers when initializing α to a small value. Note, LayerScale does not70 have a bias term.71
We apply this transformation twice for each residual block. As as a pre-normalization Aff replaces72 the LayerNormalization, and avoids using channel-wise statistics. Here, we initialize α = 1, and73 β = 0. As a post-processing of the residual block, Aff implements LayerScale and therefore we74 follow the same small value initialization for α as in [57] for the post-normalization.75
Finally, we follow the same structure for the feedforward sublayer as in the Transformer; we only76 replace the ReLU non-linearity by a GELU function [25].77
Overall, our Multi-layer perceptron takes a set of N2 d-dimensional input features stacked in a78 d×N2 matrix X, and outputs a set of N2 d-dimension output features, stacked in a matrix Y with79 the following set of transformations:80
Z = X+ Aff ( (A Aff ( X)⊤ )⊤ ) , (2)
Y = Z+ Aff (C GELU(B Aff(Z))) , (3)
where A, B and C are the main learnable weight matrices of the layer. The dimensions of the81 parameter matrix A are N2×N2, i.e., this sublayer exchanges information across all the locations,82 while the feedforward sublayer works per location. As a consequence, the intermediate activation83 matrix Z has the same dimensions as the matrices X and Y. Finally, the weight matrices B and C84 have the same dimensions as in a Transformer layer, which are 4d×d and d×4d, respectively.85
The main difference compared to a Transformer layer is that we replace the self-attention by the86 linear interaction defined in Eq. (2). While self-attention computes a convex combination of other87 features with coefficients that are data dependent, the linear interaction layer in Eq. (2) computes88 a general linear combination using learned coefficients that are not data dependent. As compared89 to a convolutional layers which have local support and share weights across space, our linear90 patch interaction layer offers a global support and does not share weights, moreover it is applied91 independently across channels.92
Relationship to the Vision Transformer. Our model can be regarded as a drastic simplification of93 the ViT model by Dosovitskiy et al. [16]. We depart from this model as follows:94
• We do not include any self-attention block. Instead we have a linear patch interaction layer without95 any non-linearity.96
• We do not have the extra “class” token that is typically used in these models to aggregate information97 via attention. Instead, we simply use average pooling. We do, however, also consider a specific98 aggregation layer as a variant, which we describe in the next paragraph.99
• We do not include any form of positional embedding: the linear communication module between100 patches implicitly takes into account the patch position.101
• Instead of pre-LayerNormalization, we use a simple learnable affine transform, thus avoiding any102 form of batch and channel-wise statistics.103
Class-MLP. As an alternative to average pooling, we also experimented with an adaptation of the104 class-attention introduced in CaiT [57]. In CaiT, this consists of two layers that have the same105 structure as the transformer, but in which only the class token is updated based on the frozen patch106 embeddings. We translate this method to our architecture, except that, after aggregating the patches107 with a linear layer, we replace the attention-based interaction between the class and patch embeddings108 by simple linear layers, still keeping the patch embeddings frozen. This increases the performance, at109 the expense of adding some parameters and computational cost. We refer to this pooling variant as110 “class-MLP”, since the purpose of these few layers is to replace average pooling.111
3 Experiments112
In this section, we present experimental results for our ResMLP architecture for image classification.113 We also study the impact of the different components in a series of ablations. We consider three114 training paradigms in our experiments:115
• Supervised learning: We train ResMLP from labeled images with a softmax classifier and cross-116 entropy loss. This paradigm is the main focus of our work.117
• Self-supervised learning: We train the ResMLP architecture without labels. We consider the DINO118 method of Caron et al. [6] that trains a network by distilling knowledge from previous instances of119 the same network, leading to a form of self-distillation without labels.120
• Knowledge distillation: We employ the knowledge distillation procedure proposed by Touvron et121 al. [56] to guide the supervised training of ResMLP with a convnet.122
3.1 Experimental setting123
Datasets. We train our models on the ImageNet-1k dataset [50], that contains 1.2M images evenly124 spread over 1,000 object categories. In the absence of an available test set for this benchmark, we125 follow the standard practice in the community by reporting performance on the validation set. This126 is not ideal since the validation set was originally designed to select hyper-parameters. Comparing127 methods on this set may not be conclusive enough because an improvement in performance may not128 be caused by better modeling, but by a better selection of hyper-parameters. To mitigate this risk, we129 report additional results in transfer learning and on two alternative versions of ImageNet that have130 been built to have distinct validation and test sets, namely the ImageNet-real [3] and ImageNet-v2 [49]131 datasets. We also report a few data-points when training on ImageNet-21k. Our hyper-parameters are132 mostly adopted from Touvron et al. [56, 57].133
Hyper-parameter settings. In the case of supervised learning, we train our network with the Lamb134 optimizer [63] with a learning rate of 5× 10−3 and weight decay 0.2. We initialize the LayerScale135 parameters as a function of the depth by following CaiT [57]. The rest of the hyper-parameters follow136 the default setting used in DeiT [56]. For the knowledge distillation paradigm, we use the same137 RegNety-16GF [48] as in DeiT with the same training schedule. The majority of our models take two138 days to train on eight V100-32GB GPUs.139
3.2 Main Results140
In this section, we compare our architecture to models with more conventional network architectures141 of comparable size and throughput on ImageNet.142
Comparison with Transformers and convnets in a supervised setting. In Table 1, we compare143 ResMLP with different convolutional and Transformer architectures. For completeness, we report144
the best-published numbers obtained with a model trained on ImageNet alone. As expected, in145 terms of the trade-off between accuracy, FLOPs, and throughput, ResMLP is not as efficient as146 convolutional networks or Transformers. However, their accuracy is encouraging: we compare them147 with architectures that have benefited from years of research and careful optimization towards these148 trade-offs. Overall, our results suggest that the structural constraints imposed by the layer design do149 not have a drastic influence on performance, especially when training models with enough data and150 recent advances in training and regularization.151
Self-supervised pre-training of ResMLP. We explore the possibility of training ResMLP using152 DINO, a recent self-supervised learning approach [6]. We pre-train ResMLP-S12 models with this153 approach during 300 epochs. We report our results in Table 2. As expected given the supervised154 classification results, the accuracies obtained with ResMLP are less good than with ViT. Nevertheless,155 the performance is surprisingly high for a pure MLP architecture and competitive with Convnet in156 knn evaluation. We hope that these result will serve as a baseline for future work.157
After self-supervised pre-training, we also fine-tune the network on ImageNet using ground truth158 labels. This pre-training substantially improves the accuracy, when comparing with the same model159 ResMLP-S24 solely trained with labels (top-1 acc. of 79.9% on ImageNet-val instead of 79.4% for160 ResMLP-S24, with the same total number of epochs). Results on ImageNet-v2 suggest that it reduces161 overfitting (68.6% on ImageNet-v2, vs 67.9% with supervised training only).162
Improving models with knowledge distillation. We study our model when training following the163 knowledge distillation approach of Touvron et al. [56]. In their work, the authors show the impact of164 training a ViT model by distilling it from a RegNet. In this experiment, we explore if ResMLP also165 benefits from this procedure and summarize our results in Table 3 (Blocks “Baseline models” and166 “Training”). We observe that similar to DeiT models, ResMLP greatly benefits from distilling from a167 convnet. This result concurs with the observations made by d’Ascoli et al. [13], who used convnets168 to initialize feedforward networks. Even though our setting differs from theirs in scale, the problem169 of overfitting for feedforward networks is still present on ImageNet. The additional regularization170 obtained from the distillation is a possible explanation for this improvement.171
Visualisation. Because they are linear, our patch interaction layers from Eq. (2) are easily inter-172 pretable. In Figure 2 we visualise the rows of the interaction matrices A as N×N images, for our173 ResMLP-S24 model. The early layers show convolution-like patterns: the learned weights resemble174 shifted versions of each other and have local support. Interestingly, in many layers, the support also175 extends along both axes, most prominently seen in layer seven. The last seven layers of the network176 are different: they consist of a spike for the patch itself and a diffuse response across other patches177 with larger or smaller magnitude; see layer 20.178
3.3 Visualization & analysis of the linear interaction between patches179
Measuring sparsity of the weights. The visualizations described above suggest that the linear180 communication layers are sparse. We analyze this quantitatively in more detail in Figure 3. We181 measure the sparsity of the matrix A, and compare it to the sparsity of B and C from the per-patch182 MLP. Since there are no exact zeros, we measure the rate of components whose absolute value is183 lower than 5% of the maximum value. Note, discarding the small values is analogous to the case184 where we normalize the matrix by its maximum and use a finite-precision representation of weights.185 For instance, with a 4-bits representation of weight, one would typically round to zero all weights186 whose absolute value is below 6.25% of the maximum value.187
The measurements in Figure 3 show that all three matrices are sparse, with the layers implementing188 the patch communication being significantly more so. This suggests that they may be compatible with189 parameter pruning, or better, with modern quantization techniques that induce sparsity at training190 time, such as Quant-Noise [20] and DiffQ [19]. The sparsity structure, in particular in earlier layers,191 see Figure. 2, hints that we could implement the patch interaction linear layer with a convolution. We192 provide some results for convolutional variants in our ablation study. Further research on network193 compression is beyond the scope of this paper, yet we believe it worth investigating in the future.194
Communication across patches if we remove the linear interaction layer (linear → none), we195 obtain substantially lower accuracy (-20% top-1 acc.) for a “bag-of-patches” approach. We have tried196 several alternatives for the linear patch interaction layer, which are presented in Table 3 (block “patch197 communication”). Amongst them, using the same MLP structure as for patch processing (linear →198 MLP), which we analyze in more details in the supplementary material. The simpler choice of a199 single linear square layer led to a better accuracy/performance trade-off – considering that the MLP200 variant requires compute halfway between ResMLP-S12 and ResMLP-S24 – and requires fewer201 parameters than a residual MLP block.202
The visualization in Figure 2 indicates that many linear interaction layers look like convolutions. In203 our ablation, we replaced the linear layer with different types of 3×3 convolutions. The depth-wise204 convolution does not implement interaction across channels – as our linear patch communication205 layer – and yields similar performance at a comparable number of parameters and FLOPs. While206 full 3×3 convolutions yield best results, they come with roughly double the number of parameters207
and FLOPs. Interestingly, the depth-separable convolutions combine accuracy close to that of full208 3×3 convolutions with a number of parameters and FLOPs comparable to our linear layer. This209 suggests that convolutions on low-resolution feature maps at all layers is an interesting alternative210 to the common pyramidal design of convnets, where early layers operate at higher resolution and211 smaller feature dimension.212
3.4 Ablation studies213
Table 3 reports the ablation study of our base network and a summary of our preliminary exploratory214 studies. We discuss the ablation below and give more detail about early experiments in Appendix A.215
Control of overfitting. Since MLPs are subject to overfitting, we show in Fig. 4 a control experiment216 to probe for problems with generalization. We explicitly analyze the differential of performance217 between the ImageNet-val and the distinct ImageNet-V2 test set. The relative offsets between curves218 reflect to which extent models are overfitted to ImageNet-val w.r.t. hyper-parameter selection. The219 degree of overfitting of our MLP-based model is overall neutral or slightly higher to that of other220 transformer-based architectures or convnets with same training procedure.221
Normalization & activation. Our network configuration does not contain any batch normalizations.222 Instead, we use the affine per-channel transform Aff. This is akin to Layer Normalization [1],223 typically used in transformers, except that we avoid to collect any sort of statistics, since we do224 no need it it for convergence. In preliminary experiments with pre-norm and post-norm [24], we225 observed that both choices converged. Pre-normalization in conjunction with Batch Normalization226 could provide an accuracy gain in some cases, see Appendix A.227
We choose to use a GELU [25] function. In Appendix A we also analyze the activation function:228 ReLU [22] also gives a good performance, but it was a bit more unstable in some settings. We did229 not manage to get good results with SiLU [25] and HardSwish [28].230
Pooling. Replacing average pooling with Class-MLP, see Section 2, brings a significant gain for a231 negligible computational cost. We do not include it by default to keep our models more simple.232
Patch size. Smaller patches significantly increase the performance, but also increase the number of233 flops (see Block "Patch size" in Table 3). Smaller patches benefit more to larger models, but only234 with an improved optimization scheme involving more regularization (distillation) or more data.235
Training. Consider the Block “Training’ in Table 3. ResMLP significantly benefits from modern236 training procedures such as those used in DeiT. For instance, the DeiT training procedure improves237
the performance of ResMLP-S12 by 7.4% compared to the training employed for ResNet [23]2.238 This is in line with recent work pointing out the importance of the training strategy over the model239 choice [2, 48]. Pre-training on more data and distillation also improve the performance of ResMLP,240 especially for the bigger models, e.g., distillation improves the accuracy of ResMLP-B24/8 by 2.6%.241
Other analysis. In our early exploration, we evaluated several alternative design choices. As in242 transformers, we could use positional embeddings mixed with the input patches. In our experiments243 we did not see any benefit from using these features, see Appendix A. This observation suggests that244 our linear patch interaction layer provides sufficient spatial communication, and referencing absolute245 positions obviates the need for any form of positional encoding.246
3.5 Transfer learning247
We evaluate the quality of features obtained from a ResMLP architecture when transferring them to248 other domains. The goal is to assess if the features generated from a feedforward network are more249 prone to overfitting on the training data distribution. We adopt the typical setting where we pre-train250 a model on ImageNet-1k and fine-tune it on the training set associated with a specific domain. We251 report the performance with different architectures on various image benchmarks in Table 4, namely252 CIFAR-10 and CIFAR-100 [34], Flowers-102 [42], Stanford Cars [33] and iNaturalist [27]. We refer253 the reader to the corresponding references for a more detailed description of the datasets. We observe254 that the performance of our ResMLP is competitive with the existing architectures, showing that255 pretraining feedforward models with enough data and regularization via data augmentation greatly256 reduces their tendency to overfit on the original distribution. Interestingly, this regularization also257 prevents them from overfitting on the training set of smaller dataset during the fine-tuning stage.258
3.6 Machine translation259
We also evaluate the ResMLP transpose-mechanism to replace the self-attention in the encoder and260 decoder of a neural machine translation system. We train models on the WMT 2014 English-German261 and English-French tasks, following the setup from Ott et al. [45]. We consider models of dimension262 512, with a hidden MLP size of 2048, and with 6 or 12 layers. Note that the current state of the art263 employs much larger models: our 6 layers model is more comparable to the base transformer model264 from Vaswani et al. [60], which serves as a baseline, along with pre-transformer architectures such as265 Recurrent and convolutional neural networks. We use Adagrad with learning rate 0.2, 32k steps of266 linear warmup, label smoothing 0.1, dropout rate 0.15 for en-de and 0.1 for en-fr. We initialize the267 LayerScale parameter to 0.2. We generate translations with the beam search algorithm, with a beam268 of size 4. As shown in Table 5, the results are at least on par with other architectures:269
4 Related work270
We review the research on applying Fully Connected Network (FCN) for computer vision problems271 as well as other architectures that shares common modules with our model.272
Fully-connected network for images. Many studies have shown that FCNs are competitive with273 convnets for the tasks of digit recognition [11, 51], keyword spotting [7] and handwritting recogni-274 tion [4]. Several works [37, 40, 59] have questioned if FCNs are also competitive on natural image275 datasets, such as CIFAR-10 [34]. More recently, d’Ascoli et al. [13] have shown that a FCN initialized276 with the weights of a pretrained convnet achieves performance that are superior than the original277 convnet. Neyshabur [41] further extend this line of work by achieving competitive performance by278 training an FCN from scratch but with a regularizer that constrains the models to be close to a convnet.279 These studies have been conducted on small scale datasets with the purpose of studying the impact of280 architectures on generalization in terms of sample complexity [18] and energy landscape [31]. In281 our work, we show that, in the larger scale setting of ImageNet, FCNs can attain surprising accuracy282 without any constraint or initialization inspired by convnets.283
Finally, the application of FCN networks in computer vision have also emerged in the study of284 the properties of networks with infinite width [43], or for inverse scattering problems [32]. More285 interestingly, the Tensorizing Network [44] is an approximation of very large FCN that shares286 similarity with our model, in that they intend to remove prior by approximating even more general287 tensor operations, i.e., not arbitrarily marginalized along some pre-defined sharing dimensions.288 However, their method is designed to compress the MLP layers of a standard convnets.289
Other architectures with similar components. Our FCN architecture shares several components290 with other architectures, such as convnets [35, 36] or transformers [60]. A fully connected layer is291 equivalent to a convolution layer with a 1× 1 receptive field, and several work have explored convnet292 architectures with small receptive fields. For instance, the VGG model [52] uses 3×3 convolutions,293 and later, other architectures such as the ResNext [62] or the Xception [10] mix 1×1 and 3×3294 convolutions. In contrast to convnets, in our model interaction between patches is obtained via a295 linear layer that is shared across channels, and that relies on absolute rather than relative positions.296
More recently, transformers have emerged as a promising architecture for computer vision [9, 17,297 46, 56, 66]. In particular, our architecture takes inspiration from the structure used in the Vision298 Transformer (ViT) [17], and as consequence, shares many components. Our model takes a set299 of non-overlapping patches as input and passes them through a series of MLP layers that share300 the same structure as ViT, replacing the self-attention layer with a linear patch interaction layer.301 Both layers have a global field-of-view, unlike convolutional layers. Whereas in self-attention the302 weights to aggregate information from other patches are data dependent through queries and keys,303 in ResMLP the weights are not data dependent and only based on absolute positions of patches. In304 our implementation we follow the improvements of DeiT [56] to train vision transformers, use the305 skip-connections from ResNets [23] with pre-normalization of the layers [8, 24].306
Finally, our work questions the importance of self-attention in existing architectures. Similar ob-307 servations have been made in natural language processing. Notably, Synthesizer [54] shows that308 dot-product self-attention can be replaced by a feedforward network, with competitive performance309 on sentence representation benchmarks. As opposed to our work, Synthesizer does use data dependent310 weights, but in contrast to transformers the weights are determined from the queries only.311
5 Conclusion312
In this paper we have shown that a simple residual architecture, whose residual blocks consist of a313 one-hidden layer feed-forward network and a linear patch interaction layer, achieves an unexpectedly314 high performance on ImageNet classification benchmarks, provided that we adopt a modern training315 strategy such as those recently introduced for transformer-based architectures. Thanks to their simple316 structure, with linear layers as the main mean of communication between patches, we can vizualize317 the filters learned by this simple MLP. While some of the layers are similar to convolutional filters,318 we also observe sparse long-range interactions as early as the second layer of the network. We hope319 that our model free of spatial priors will contribute to further understanding of what networks with320 less priors learn, and potentially guide the design choices of future networks without the pyramidal321 design prior adopted by most convolutional neural networks.322
References323
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint324 arXiv:1607.06450, 2016. 2, 7325
[2] Irwan Bello, W. Fedus, Xianzhi Du, E. D. Cubuk, A. Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret326 Zoph. Revisiting ResNets: Improved training and scaling strategies. arXiv preprint arXiv:2103.07579,327 2021. 1, 8328
[3] Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aaron van den Oord. Are we329 done with ImageNet? arXiv preprint arXiv:2006.07159, 2020. 4, 7330
[4] Théodore Bluche. Deep neural networks for large vocabulary handwritten text recognition. PhD thesis,331 Université Paris-Sud, 2015. 9332
[5] A. Brock, Soham De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition333 without normalization. arXiv preprint arXiv:2102.06171, 2021. 4334
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand335 Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294,336 2021. 2, 4, 5337
[7] Clément Chatelain. Extraction de séquences numériques dans des documents manuscrits quelconques.338 PhD thesis, Université de Rouen, 2006. 9339
[8] Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones,340 Niki Parmar, Mike Schuster, Zhifeng Chen, et al. The best of both worlds: Combining recent advances in341 neural machine translation. In Annual Meeting of the Association for Computational Linguistics, 2018. 9342
[9] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse343 transformers. arXiv preprint arXiv:1904.10509, 2019. 9344
[10] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Conference on345 Computer Vision and Pattern Recognition, 2017. 9346
[11] Dan Claudiu Cireşan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuber. Deep big multilayer347 perceptrons for digit recognition. In Neural networks: tricks of the trade, pages 581–598. Springer, 2012. 9348
[12] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data349 augmentation with a reduced search space. arXiv preprint arXiv:1909.13719, 2019. VI350
[13] Stéphane d’Ascoli, Levent Sagun, Joan Bruna, and Giulio Biroli. Finding the needle in the haystack with351 convolutions: on the benefits of architectural bias. In NeurIPS, 2019. 5, 9352
[14] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. RepMLP: Re-parameterizing con-353 volutions into fully-connected layers for image recognition. arXiv preprint arXiv:2105.01883, 2021.354 2355
[15] Piotr Dollár, Mannat Singh, and Ross Girshick. Fast and accurate model scaling. arXiv preprint356 arXiv:2103.06877, 2021. 1357
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas358 Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is359 worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning360 Representations, 2021. 1, 2, 3, 8361
[17] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative362 unsupervised feature learning with convolutional neural networks. In NeurIPS, 2014. 9363
[18] Simon S Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Aarti Singh.364 How many samples are needed to estimate a convolutional neural network? In NeurIPS, 2018. 9365
[19] Alexandre Défossez, Yossi Adi, and Gabriel Synnaeve. Differentiable model compression via pseudo366 quantization noise. arXiv preprint arXiv:2104.09987, 2021. 6367
[20] Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand368 Joulin. Training with quantization noise for extreme model compression. In International Conference on369 Learning Representations, 2021. 6370
[21] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence371 to sequence learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International372 Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages373 1243–1252. PMLR, 06–11 Aug 2017. 8374
[22] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International375 Conference on Machine Learning, 2011. 7376
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.377 In Conference on Computer Vision and Pattern Recognition, 2016. 2, 7, 8, 9, V378
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks.379 arXiv preprint arXiv:1603.05027, 2016. 7, 9380
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415,381 2016. 1, 3, 7382
[26] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your383 batch: Improving generalization through instance repetition. In Conference on Computer Vision and384 Pattern Recognition, 2020. VI385 [27] Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona,386 and Serge J. Belongie. The iNaturalist species classification and detection dataset. arXiv preprint387 arXiv:1707.06642, 2017. 8388 [28] A. Howard, Mark Sandler, G. Chu, Liang-Chieh Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V.389 Vasudevan, Quoc V. Le, and H. Adam. Searching for MobileNetV3. International Conference on Computer390 Vision, 2019. 7391 [29] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic392 depth. In European Conference on Computer Vision, 2016. VI393 [30] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal394 covariate shift. In International Conference on Machine Learning, 2015. 2395 [31] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter396 Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International397 Conference on Learning Representations, 2017. 9398 [32] Yuehaw Khoo and Lexing Ying. Switchnet: a neural network model for forward and inverse scattering399 problems. SIAM Journal on Scientific Computing, 41(5):A3182–A3201, 2019. 9400 [33] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained401 categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13),402 2013. 8403 [34] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, University of404 Toronto, 2009. 8, 9405 [35] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural406 networks. In NeurIPS, 2012. 8, 9407 [36] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to408 document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 9409 [37] Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolution: Improving410 fully-connected networks. arXiv preprint arXiv:1511.02580, 2015. 9411 [38] Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention to MLPs. arXiv preprint412 arXiv:2105.08050, 2021. 2413 [39] Luke Melas-Kyriazi. Do you even need attention? A stack of feed-forward layers does surprisingly well414 on ImageNet. arXiv preprint arXiv:2105.02723, 2021. 2415 [40] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and416 Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by417 network science. Nature communications, 9(1):1–12, 2018. 9418 [41] Behnam Neyshabur. Towards learning convolutions from scratch. In NeurIPS, 2020. 9419 [42] M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In420 Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 8421 [43] Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia,422 Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels423 are Gaussian processes. In International Conference on Learning Representations, 2019. 9424 [44] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks.425 In Neurips, 2015. 9426 [45] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In ACL,427 pages 1–9. Association for Computational Linguistics, 2018. 8428 [46] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin429 Tran. Image transformer. In International Conference on Machine Learning, 2018. 9430 [47] Filip Radenović, Giorgos Tolias, and Ondrej Chum. Fine-tuning CNN image retrieval with no human431 annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7):1655 – 1668, 2018. 4432 [48] Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing433 network design spaces. Conference on Computer Vision and Pattern Recognition, 2020. 4, 8434 [49] B. Recht, Rebecca Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet?435 In International Conference on Machine Learning, 2019. 4, 7436 [50] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,437 Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large438 scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015. 1, 4439 [51] Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks440 applied to visual document analysis. In ICDAR, 2003. 9441 [52] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In442 International Conference on Learning Representations, 2015. 9443 [53] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks.444 arXiv preprint arXiv:1905.11946, 2019. 4, 8445
[54] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking446 self-attention in transformer models. arXiv preprint arXiv:2005.00743, 2020. 9447
[55] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner,448 Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An449 all-MLP architecture for vision. arXiv preprint arXiv:2105.01601, 2021. 2450
[56] Hugo Touvron, M. Cord, M. Douze, F. Massa, Alexandre Sablayrolles, and H. Jégou. Training data-efficient451 image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020. 1, 2, 4, 5, 8,452 9, I, V453
[57] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper454 with image transformers. arXiv preprint arXiv:2103.17239, 2021. 2, 3, 4, I455
[58] Hugo Touvron, Alexandre Sablayrolles, M. Douze, M. Cord, and H. Jégou. Grafit: Learning fine-grained456 image representations with coarse labels. arXiv preprint arXiv:2011.12982, 2020. 8457
[59] Gregor Urban, Krzysztof J Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana,458 Abdelrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convolutional nets really need459 to be deep and convolutional? In International Conference on Learning Representations, 2017. 9460
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz461 Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1, 8, 9462
[61] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim463 Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging464 the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. 8465
[62] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor-466 mations for deep neural networks. In Conference on Computer Vision and Pattern Recognition, 2017.467 9468
[63] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song,469 James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training470 BERT in 76 minutes. In International Conference on Learning Representations, 2020. 4471
[64] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.472 CutMix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint473 arXiv:1905.04899, 2019. VI474
[65] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk475 minimization. arXiv preprint arXiv:1710.09412, 2017. VI476
[66] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In477 Conference on Computer Vision and Pattern Recognition, 2020. 9478
[67] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation.479 In AAAI, 2020. VI480
Checklist481
1. For all authors...482
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s483 contributions and scope? [Yes]484
(b) Did you describe the limitations of your work? [Yes]485
(c) Did you discuss any potential negative societal impacts of your work? [No] We only486 use MLPs for image classification. There is already other system much more powerful487 therefore the impact is negligible488
(d) Have you read the ethics review guidelines and ensured that your paper conforms to489 them? [Yes]490
2. If you are including theoretical results...491
(a) Did you state the full set of assumptions of all theoretical results? [N/A]492
(b) Did you include complete proofs of all theoretical results? [N/A]493
3. If you ran experiments...494
(a) Did you include the code, data, and instructions needed to reproduce the main exper-495 imental results (either in the supplemental material or as a URL)? [Yes] We provide496 code in appendix497
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they498 were chosen)? [Yes]499
(c) Did you report error bars (e.g., with respect to the random seed after running experi-500 ments multiple times)? [No] Because it’s too expensive in computational resources. But501 for some experiments we run multiple times and we have ∼ 0.1 as standard deviation502
(d) Did you include the total amount of compute and the type of resources used (e.g., type503 of GPUs, internal cluster, or cloud provider)? [Yes] The majority of our experiments504 take 2 days on 8 GPUs V100-32GB505
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...506
(a) If your work uses existing assets, did you cite the creators? [Yes]507
(b) Did you mention the license of the assets? [No] But license can be found with the508 references we mention509
(c) Did you include any new assets either in the supplemental material or as a URL? [No]510
(d) Did you discuss whether and how consent was obtained from people whose data you’re511 using/curating? [No] We only use datasets that are widely used in computer vision512 such as ImageNet or CIFAR513
(e) Did you discuss whether the data you are using/curating contains personally identifiable514 information or offensive content? [No] We only use datasets that are widely used in515 computer vision such as ImageNet or CIFAR516
5. If you used crowdsourcing or conducted research with human subjects...517
(a) Did you include the full text of instructions given to participants and screenshots, if518 applicable? [N/A]519
(b) Did you describe any potential participant risks, with links to Institutional Review520 Board (IRB) approvals, if applicable? [N/A]521
(c) Did you include the estimated hourly wage paid to participants and the total amount522 spent on participant compensation? [N/A]523

Review summaries:
The paper proposes an MLP-based architecture for sequence-based DL tasks (vision, NLP). Basically, the idea is to remove the self-attention primitive from Transformers. Instead, it transposes the seq and channel dimension and applies linear projections. This is the same idea as the concurrent work MLP-Mixer from Google. The spatial mixing is now basically a DWise Conv with full receptive field shared across channels. Paper shows reasonable results on ImageNet benchmark (needs a lot more params and worse on throughput for the same top-1 accuracy wrt ViT, and also worse than pure ConvNet baselines). Paper has plenty of ablations on the training recipe and architecture. NLP tasks only include short sequence benchmarks like Translation, with promising signs.

This paper introduces Residual Multi-Layer Perceptrons (ResMLP): a purely multi-layer perceptron (MLP) based architecture for image classification. ResMLP takes image patches as input, projects them with a linear layer, and sequentially updates them in turn with two residual operations: (i) a simple linear layer that provides interaction between the patches, which is applied to all channels independently; and (ii) an MLP with a single hidden layer, which is independently applied to all patches. At the end of the network, the patches are average pooled, and fed to a linear classifier. The main change from ViT to ResMLP is replacing the self-attention layer with a simple linear layer cross patches. There are some other small changes, like pooling layers and normalization layers.  The claimed contributions are: 1. despite their simplicity, Residual Multi-Layer Perceptrons reach surprisingly good accuracy/complexity trade-offs with ImageNet-1k training only. From this point of view, this "surprisingly good" is very subjective. At least, from the results, the accuracy/complexity trade-offs is not as good as Vision Transformers (DeiT models). 2. The ResMLP benefits from distillation methods and also works well with self-supervised method DINO. 3. The simple linear layer enables observations on the spatial interaction that the network learns across layers. 4. The authors also try ResMLP on machine translation.   

Recently, vision Transformers have been popular and achieve SOTA performance on various tasks. This paper proposes a more simple variant of the Transformer architecture, *i.e.*, connect image patches via simple linear/MLP layers. The proposed ResMLP shows promising performance under various setups, *e.g.*, supervised and self-supervised training, and on different datasets. It is a new exploration of the methodology of neural architectures.

This paper introduce the ResMLP architecture. An alternative to self-attention layer in vision transformer using simply a $T \times T$ linear layer to allow long-range communication between tokens. They show that the performance of this model is comparable to the original vision transformer in supervised, self-supervised and transfer learning as well as knowledge distillation.