MLP-Mixer: An all-MLP Architecture for Vision
Authors: Ilya Tolstikhin
Neil Houlsby
Alexander Kolesnikov
Lucas Beyer
Xiaohua Zhai
Thomas Unterthiner
Jessica Yung
Andreas Peter Steiner
Daniel Keysers
Jakob Uszkoreit
Mario Lucic
Alexey Dosovitskiy
Venue: NeurIPS 2021 Poster (2021)
Keywords: computer vision
image recognition
large-scale training
multi-layer perceptrons
transfer learning

Abstract: Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

MLP-Mixer: An alI-MLP Architecture for Vision
Ily Tolstikhin', Neil Houlsby", Alexander Kolesnikov", Lucas Beyer",
Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, Alexey Dosovitskiy
"equal contribution
Google Research, Brain Team
(tolstikhin, neilhoulsby, akolesnikov, lbeyer, xzhai, unterthiner, jessicayung', andstein, keysers, usz, lucic, adosovitskkiy)ggoogeee com
'work done during Google Al Residency
Abstract
Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention- baasdd networks, such as the Vision Transformer, have also become popular. In this paper show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, with modern regularization schemes, MLP-Mixer attains competitive scores image classification benchmarks, with pre-training and inference cost comparable to state-of-the-aa models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.'
Introduction
As the history of computer vision demonstrates, the availability of larger datasets coupled with in- creased computational capacity often leads to a paradigm shift. While Convolutional Neural Networks (CNNs) have been the de-facto standard for computer vision, recently Vision Transformers [14] (ViT), an alternative based on self-attention layers, attained state-of-the-an performance. VIT continues the long-lasting trend of removing hand-crafted visual features and inductive biases from models and relies further on learning from raw data.
We propose the MLP-Mixer architecture (or "Mixer" for short), a competitive but conceptually and technically simple alternative, that does not use convolutions or self-attention. Instead, Mixer's architecture is based entirely multi-layer perceptrons (MLPs) that repeatedly applied across either spatial locations or feature channels. Mixer relies only on basic matrix multipicctioon routines, changes to data layout (reshapes and transpositions), and scalar nonlincarities.
Figure depicts the macro-structure of Mixer. accepts a sequence of linearly projected image patches (also referred to as tokens) shaped as a "patches x channels" table an input, and maintains this dimensionality. Mixer makes use of two types of MLP layers: channel-mixing MLPs and token-mixing MLPs. The channel-mixing MLPs allow communication between different channels;
MLX-Mixer code available ttttp://ggiiin wtwm///oggl--esserre//ttteeeeeee
http::///gtttuooooo/oooooreeeeee http:///vvivooottsrrrrereee transformer
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Mixer Layet
Chonnels
Obonnels
MEPT MIPT ML21
Class
Pully cossected
MI.P
Globil
Felly-commecedd
N'x (Mixer Layer
GELU
parch
Polly-commocedd
Figure 1: MLP-Mixer consists of per-patch linear embeddings. Mixer layers, and a classifier head. Mixer layers contain one token-mixing MLP and one channel-mixing MLP, each consisting two fully-connected layers and GELU nonlinearity. Other components include: skip-connections, dropout, and layer norm on the channels.
they operate each token independently and take individual of the table as inputs. The token-mixing MLPs allow communication between different spatial locations (tokens); they operate on each channel independently and take individual columns of the table as inputs. These two types of layers are interleaved to enable interaction of both input dimensions.
In the extreme case, architecture be for channel mixing, and single-channel depth-wise convolutions of full receptive field and parameter sharing for token mixing. However, the converse is not true typical CNNs Mixer. Furthermore, a convolution is more complex than the plain matrix multiplication in MLPs as it requires additional costly reduction matrix multiplication and/or specialized implementation.
very special CNN, which uses 1 x convolutions
special cases of
Despite its simplicity, Mixer attains competitive results. When pre-trained on large datasets (i.e., ~00M images), it reaches near satte-of-the-an performance, previously claimed by CNNs and Transformers, in terms of the accuracy/cost trade-off. This includes 87. 77.4%% top-1 validation accuracy on ILSVRC2012 "ImageNet" [13]. When pre-trained data of more modest scale (i.e., ~1 10M images), coupled with modern regularization techniques [49, 54], Mixer also achieves strong performance. However, similar ViT, it falls slightly short of specialized CNN architectures.
2 Mixer Architecture
Modern deep vision architectures consist of layers that mix features (i) at given spatial location, (ii) between different spatial locations, or both at once. In CNNs, (ii) is implemented with N x N convolutions (for N > 1) and pooling. Neurons in deeper layers have a larger receptive field [1, 29]. At the same time, x convolutions also perform (i), and larger kernels perform both (i) and (ii). In Vision Transformers and other attention-bat -baee architectures, self-attention layers allow both (i) and (ii) and the MI.P-blocks perform (i). The idea behind the Mixer architecture is to clearly separate the per-location (channel-mixingg) operations (i) and cross-location (token- mixing) operations (ii).
Both operations implemented with MLPs. Figure summarizes the architecture.
Mixer takes input sequence of S non-overlapping image patches, each one projected to desired hidden dimension C.. This results in a two-diment ovo-dimensional real-valued input table, X E SxC If the original input image has resolution (H, W), and each patch has resolution (P. P), then the number of patches is S = HW p2. All patches are linearly projected with the same projection matrix. Mixer consists of multiple layers of identical size, and each layer consists two MLP blocks. The first one is the token-mixing MI.: it acts on columns of X (i.e. it is applied to transposed input table X ) maps R >> S and is shared across all columns. The second one is the channel-mixing MIP: acts on rows of X, maps Ro C and is shared across all rows. Each MLP block contains two

---

fully-commen celly-coneettd layers and a nonlinearity applied independently to each row of its input data tensor. Mixer layers can be written as follows (omitting layer indices):
U., = X,, + W2 (W, LayerNorm(X (X)..): for = I...C. C, YJ, U,, Wa a Wa LayerNormi (U)))oommmU)))))) U) for = 1
(I)
Here a is an element-wise nonlinearity (GELU [16]). Ds and De are tunable hidden widths in the token-mixing and channel-mixing MLPss respectively. Note that Ds is selected independently of the number of input patches. Therefore, the computational complexity of the network is linear in the number of input patches, unlike VIT whose complexity is quadratic. Since De is independent of the patch size, the overall complexity is linear in the number of pixels in the image, as for typical CNN.
As mentioned above, the same channel-mixing MLP (token-mixing MLP) is applied to every row (columm) of X. Tying the parameters of the channel-mixing MLPs (within each layer) is a natural choice-it it provides positional invariance, a prominent feature of convolutions. However, tying parameters across channels is much less common. For example, separable convolutions [9, 40], used in some CNNs, apply convolutions to each channel independently of the other channels. However, in separable convolutions, a different convolutional kernel is applied to each channel unlike the token-mixing MLPs in Mixer that share the same kernel (of full receptive field) for all of the channels. The parameter tying prevents the architecture from growing too fast when increasing the hidden dimension C' or the sequence length S and leads to significant memory savings. Surprisingly, this choice does not affect the empirical performance, see Supplementary A.I.
Each layer in Mixer (except for the initial patch projection layer) takes imput of the same size. This "isotropic" design is most similar to Transformers, or deep RNNs in other domains, that also use a fixed width. This is unlike most CNNs, which have a pyramidal structure: deeper layers have lower resolution input, but more channels. Note that while these are the typical designs, other combinations exist, such as isotropic ResNets [38] and pyramidal ViTs [52].
Aside from the MLP layers, Mixer uses other standard architectural components: skip-connec- tions [15] and layer normalization [2]. Unlike ViTs, Mixer does not use position embeddings because the mixing MLPs are sensitive to the order the input tokens. Finally, Mixer uses a standard classification head with the global average pooling layer followed by linear classifier. Overall, the architecture can be written compactly in JAX/Flax, the code is given in Supplementary F.
Experiments
We evaluate the performance of MLP-Mixer models, pre-trained with medium- to large-scale datasets, on a range of small and mid-sized downstream classification tasks. We are interested three primary quantities: (1) Accuracy on the downstream task; (2) Total computational cost of pre-training, which important when training the model from scratch on the upstream dataset; (3) Test-time throughput, which is important to the practitioner. Our goal is not to demonstrate state-of-thee thettrr results, but to show that, remarkably, a simple MLP-based model is competitive with today's best convolutional and attention- based models.
Downstream tasks
We use popular downstream tasks such as ILSVRC2012 "ImageNet" (1.3M
training examples, 1k classes) with the original validation labels [13] and cleaned-up Real labels [5]. CIFAR-10/100 (50k examples, 10/100 cl/sse)l [23], Oxford-IIIT Pets (3.7k examples, 36 classes) [33]. and Oxford Flowers- 102 (2k examples, 102 classes) [32]. We also use the Visual Task Adaptation
Benchmark (-T,B-1k), which consists of 19 diverse datasets, each with 1K training examples [58].
Pre-training We follow the standard transfer learning setup: pre-training followed by fine-tuning on the downstream tasks. We pre-train our models on two public datasets: ILSVRC2021 ImageNet, and ImageNet-21k, a superset of ILSVRC2012 that contains 21k classes and 14M images [13]. To assess performance at larger scale, we also train on JFT-300M, a proprictary dataset with 300M examples and 18k classes [44]. We de-duplicate all pre-training datasets with respect to the test sets of the downstream tasks as done in Dosovitskiy et al. [14], Kolesnikov et al. [22]. We pre-train all models at resolution 224 using Adam with 81 = 0.9, B2 0.999, linear learning rate warmup of 10k steps and linear decay, batch size 096, weight decay, and gradient clipping at global norm 1. For JFT-300M, we pre-process images by applying the cropping technique from Szegedy et al. [45] in addition to random horizontal flipping. For ImageNet and ImageNet-21k,, we employ additional data augmentation and regularization techniques. In particular, we use RandAugment [12], mixup [60].
3

---

Table 1: Specifications of the Mixer architectures. The "B", "L", and "H" (base, large, and huge) model scales follow Dosovitskiy et al. [14]. A brief notation "B/16" means the model of base scale with patches of resolution 16x 16. The number of parameters is reported for an input resolution of 224 and does not include the weights of the classifier head.
Specification
S/32
S/16
B/32
B/16
L/32
L/16
H//4
Number of layers
8
8
12
12
24
24
32
Patch resolution PxP
32x32
16x16
32x32
16x16
32x32
16x16
14x 14
Hidden size C
512
512
768
768
1024
1024
1280
Sequence length $
49
196
49
196
49
196
256
MLP dimension De
2048
2048
3072
3072
4096
4096
5120
MLP dimension Ds
256
256
384
384
512
512
640
Parameters (M)
19
18
60
59
206
207
431
dropout [43], and stochastic depth [19]. This set of techniques was inspired by the timm library [54] and Touvron et al. [48]. More details on these hyperparameters are provided in Supplementary
Fine-tuning We fine-tune using momentum SGD, batch size 512, gradient clipping at global norm 1, and a cosine learning rate schedule with a linear warmup. We do not use weight decay when fine- tuning. Following common practice [22, 48], we also fine-tune at higher resolutions with respect to those used during pre-training. Since we keep the patch resolution fixed, this increases the number of input patches (say from S to S') and thus requires modifying the shape of Mixer's 's moxeng MLP blocks. Formally, the input in Eq. (1) is left-multiplied by weight matrix W1 E I P x and this operation has to be adjusted when changing the input dimension S. For this, we increase the hidden layer width from D3 to Ds in proportion to the number of patches and initialize the (now larger) weight matrix W E TR Dgr X with block-diagonal matrix containing copies of W2 2 on its diagonal. This particular scheme only allows for S' = K2S S with K E N. See Supplementary C for further details. On the VTAB-IK Ik benchmark we follow the BiT-HyperRule [22] and fine-tune Mixer models at resolution 224 and 448 on the datasets with small and large input images respectively.
Metries We evaluate the trade-off between the model's computational cost and quality. For the former we compute two metrics: (1) Total pre-training time on TPU-v3 accelerators, which combines three relevant factors: the theoretical FLOPs for each training setup, the computational efficiency on the relevant training hardware, and the data efficiency. (2) Throughput in images/sec/core on TPU-v3. Since models of different sizes may benefit from different batch sizes, we sweep the batch sizes and report the highest throughput for each model. For model quality, we focus on top-1 downstream accuracy after fine-tuning. On two occasions (Figure 3, right and Figure 4), where fine-tuning all of the models is too costly, we report the few-shot accuracies obtained by solving the fn-regularized linear regression problem between the frozen learned representations of images and the labels.
Models We compare various configurations of Mixer, summarized in Table 1, to the most recent, state-of-the-art CNNs and attention- based models. In all the figures and tables, the MLP-based Mixer models are marked with pink (o), convolution based models with yellow (s). and uttention-based models with blue (o). The Vision Transformers (ViTs) have model scales and patch resolutions similar to Mixer. HaloNets are attention based models that use a ResNet- structure with local self- attention layers instead of 3 33 convolutions [51]. We focus on the particularly efficient "HaloNet-HH H4 (base 128, Conv- 12)" model, which is a hybrid variant of the wider HaloNet-H4 architecture with some of the self-attention layers replaced by convolutions. Note, we mark HaloNets with both attention and convolutions with blue (o). Big Transfer (BiT) [22] models are ResNets optimized for transfer learning. NFNets [7] are normalizer-free ResNets with several optimizations for ImageNet classification. We consider the NFNet-F4+ model variant. We consider MPL [35] and ALIGN [21] for EfficientNet architectures. MPL is pre-trained at very large-scale on JFT-300M images, using meta-pseudo labelling from ImageNet instead of the original labels. We compare to the EfficientNet- B6-Wide model variant. ALIGN pre-train image encoder and language encoder on noisy web image text pairs in a contrastive way. We compare their best EfficientNet-L.2 image encoder.
3.1 Main results
Table 2 presents comparison of the largest Mixer models to state-of-the-arr models from the literature. "ImNet" and "ReaL" columns refer to the original ImageNet validation [13] and cleaned-up ReaL [5]

---

Table 2: Transfer performance, inference throughput, and training cost. The rows are sorted by inference throughput (fifth column). Mixer has comparable transfer accuracy to state-of ttae-off-the-ar models with similar cost. The Mixer models are fine-tuned at resolution 448. Mixer performance numbers are averaged over three fine-tuning runs and standard deviations are smaller than 0.1.
ImNet
ReaL.
Avg lop-1
VTAB-1k
Throughput img/sec/core
TPUv3
top-1
top-1
19 tasks
core-days
Pre Preitrained
ImageNet-211 (public)
HaloNet [51]
85,8
120
0.10k
Mixer-I/16
84.15
87.86
93.91
74.95
105
0.41k
VIT-L/16 [14]
85.30
88.62
94.39
72.72
32
0.18k
BiT-R152x4 [22]
85.39
94.04
70.64
26
0.94k
Pre-trained on JFT-300M (proprietary)
NENet-FF4+ [7]
89.2
46
1.86k
Mixer-H/le
87.94
90.18
95.71
75.33
40
1.01k
BiT-R152x4 [22]
87.54
90.54
95.33
76.29
26
9.90k
ViT-H/14 [14]
88.55
90.72
95.97
77.63
15
2.30k
Pre-trained on unlabelled weakly labelled data (proprietary)
MPL [35]
90.0
91.12
20.48K
ALIGN [21]
88.64
79.99
15
14.82k
labels. "Avg.5" stands for the average performance across all five downstream tasks (ImageNet, CIFAR-10, CIFAR-IOG 100, Pets, Flowers). Figure 2 (left) visualizes the accuracy-computc frontier. When pre-trained on ImageNet-21k with additional regularization, Mixer achieves overall strong performance (84.15% 15% top-1 on ImageNet), although slightly inferior to other models². Regularization in this scenario is necessary and Mixer overfits without which is consistent with similar observations for VIT [14]. The same conclusion holds when training Mixer from random initialization on ImageNet (see Section 3.2): Mixer-B/16 attains reasonable overfit. This score is similar to a vanilla ResNet50, but behind state-of-the-an CNNs/hybrids for the ImageNet "from scratch" setting, e.g. 84.7% BotNet [42] and 86.5% NFNet [7].
of 76.4% at resolution 224, but tends to
When the size of the upstream dataset increases, Mixer's performance improves significantly. par- ticular, Mixer-H/14 achieves 87.94% top- accuracy ImageNet, which is 0.5% better than BiT- ResNet152x4 and only 0.5% lower than ViT-H/14. Remarkably, Mixer-H/14 runs 2.5 times faster than ViT-H/14 and almost twice as fast BIT. Overall, Figure (left) supports our main claim that in terms the sacuracy-compuss trade-off off Mixer competitive with more conventional neural network architectures. The figure also demonstrates clear correlation between the total pre pre-training cost and the downstream accuracy, even across architecture classes.
BiT-ResNet 5225 in the table are pre-trained using SGD with momentum and long schedule. Since Adam tends to converge faster, we complete the picture in Figure (left) with the BiT-R200x3 model from Dosovitskiy et al. [14] pre-trained on JFT-300M using Adam. This ResNet has slightly lower accuracy, but considerably lower pre-training compute. Finally, the results smaller ViT-L/16 and Mixer-L/16 models are also reported in this figure.
3.2
The role of the model scale
The results outlined in the previous section focus on (large) models at the upper end of the compute spectrum. We now turn our attention to smaller Mixer models.
We may scale the model in two independent ways: (1) Increasing the model size (number of layers, hidden dimension, MLP widths) when pre-training; (2) Increasing the input image resolution when
II Table 2 we consider the highest accuracy models in each class for each pre-training dataset. These all the large resolutions (448 and above). However, fine-tuning at smaller resolution can lead to substantial improvements in the test-time throughput, with often only a small accuracy penalty. For instance, when pre- training on ImageNet-211 1l,, the Mixer-L/16 model fine-tuned at 224 resolution achieves 82.84% ImageNet top-1 accuracy throughput 420 img/sec/core; the VIT-L/16 model fine-tuned at 384 resolution achieves 85.15% at 80 img/sec/core [14]: and HaloNet fine-tuned at 384 resolution achieves 85.5% at 258 img/sec/core [51].
5

---

Moarr 1214 JPT)
D- GFT1
Imagekes Imapehestranerr transrer accuracy (Nu
HaleMeet 1214
ALIGIN (web)
Unoa dros imsgete [8]
0236 (FT)
B/32
V/T-8/32
Mixer
VIT-L/32
Mixer-L/16
VIT-L/16
BIT-R152x2
Total pre-treining S0--TPUv3-oree-day
Training Size
150
SEE
Figure 2: Left: ImageNet accuracy/training cost Pareto frontier (dashed line) for the SOTA models in Table 2. Models are pre-trained on ImageNet-21k, or JFT (labelled, or pseudo-lahelled for M. or web image text pairs. Mixer as good as these extremely performant ResNets, ViTs, and hybrid models, and sits on frontier with HaloNet, VIT, NFNet, and MPL. Right: Mixer (solid) catches or exceeds BiT (dotted) and ViT (dashed) the data size grows. Every point on curve uses the same pre-training compute; they correspond to pre-training on 3%, 10%, 30%, and 100% of JFT-300M for 233, 70. 23, and 7 epochs, respectively. Additional points at ~3B correspond to pre-training on an even larger JFT-3B dataset for the same number of total steps. Mixer improves more rapidly with data than ResNets, or even VIT. The gap between large Mixer and ViT models shrinks.
InugeNNe transler accurccy
80
Mixer
15-
Transformer (VIT)
ReaNet (Adam, BiT)
65
TPUv3-core-day
Throughpor (ing/seo/oors)
Figure 3: The role of the model scale. ImageNet validation top-1 accuracy total pre-training compute (left) and throughput (right) of VIT, BiT, and Mixer models at various scales. All models are pre-trained JFT-300M and fine-tuned resolution 224, which is lower than Figure (left).
fine-tuning. While the former affects both pre-training compute and test-time throughput, the latter only affects the throughput. Unless stated otherwise, fine-tune at resolution 224.
We compare various configurations Mixer (see Table 1) to ViT models similar scales and BiT models pre-trained with Adam. The results are summarized in Table 3 and Figure 3. When trained from scratch on ImageNet, Mixer-B/11 achieves a reasonable top-1 accuracy of 76.44%. This 3% behind the VIT BTT6 model. The training curves (not reported) reveal that both models achieve very similar values the training loss. In other words, Mixer-Bi Bi1e overfits more than ViT-B/16. For the Mixer-L/16 and ViT-L/16 models this difference is
pronounced.
As the pre-training dataset grows, Mixer's performance steadily improves. Remarkably, Mixer-H/14 pre-trained on JFT-300M and fine-tuned at 224 resolution is only 0.3% behind VIT-H/14 on ImageNet whilst running 2.2 times faster. Figure clearly demonstrates that although Mixer is slightly below the frontier on the lower end of model scales, sits confidently on the frontier at the high end.
3.3 The role the pre-training dataset size
The results presented thus far demonstrate that pre-training larger datasets significantly improves Mixer's performance. Here, we study this effect in more detail.
To study Mixer's ability to make use of the growing number of training examples we pre-train Mixer-BAk B/32, Mixer-L/32, and Mixer-L/16 models on random subsets JFT-300M containing 3%, 10%, 30% and 100% of all the training examples for 233, 70, 23, and 7 epochs. Thus, every model is pre-trained for the same number total steps. We also pre-train Mixer-L/16 model on an even larger JFT-3B dataset [59] containing roughly 3B images with 30k classes for the same number of total steps.

---

Table 3: Performance of Mixer and other models from the literature across various model and pre- pre-training dataset scales. "Avg. 5" denotes the average performance across five downstream tasks. Mixer and ViT models are averaged over three fine-tuning runs, standard deviations are smaller than 0.15. ( Extrapolated from the numbers reported for the same models pre-trained on JFT-300M without extra regularization.
(a) Numbers provided by authors of Dosovitskiy et [14] through personal communication. Rows are sorted by throughput.
Image size
Pre-Train
ImNet
Real.
Avg.5 Throughput
TPUv3
Epochs
top-1
top-1
top-1 (img/sec/core) core-days
Pre-trained on ImageNet (with extra regularization)
Mixer-B/16
224
300
76.44
82.36
88.33
1384
0.01k
VIT-B/16 (x)
224
300
79.67
84.97
90.79
861
0.02k
Mixer L/16
224
300
71.76
77.08
87.25
419
0.04k (9)
VIT-L/16 (x)
224
300
76.11
80.93
89.66
280
0.05k
Pre-trained on ImageNet-21K (with extra regularization)
Mixer-B/16
224
300
80.64
85.80
0.15k
VIT-B/16 (z)
224
300
84.59
88,93
94.16
861
0.18k
Mixer-L/16
224
300
82.89
87.54
93.63
419
0.41k
ViT-L/16 (x)
224
300
84.46
88.35
94.49
280
0.55k
Mixer-L/K
448
300
83.91
87.75
93.86
105
0.41k
Pre-trained on JFT-300M
Mixer-S/33
224
68.70
75.83
87.13
1 11489
0.01k
Mixer-B/33
224
75.53
81.94
90.99
4208
0.05k
Mixer-S/16
224
73.83
80.60
89.50
3994
0.03k
BiT-R50x1
224
73.69
81.92
2159
0.08k
Tixer-B/16
224
80.00
85.56
92.60
1384
0.08k
Mixer-L/32
224
80.67
85.62
93.24
1314
0.12k
BiT-RI52x1
224
79.12
86.12
932
0.14k
BiT-R50x2
78.92
890
0.14k
BiT-R152x2
14
83.34
88.90
356
0.58k
Mixer-L/16
224
84.05
88.14
94.51
419
0.23k
Mixer-L/16
224
14
84.82
88.48
94.77
419
0.45k
ViT-L/16
224
14
85.63
89.16
95.21
280
0.65k
Mixer-H/14
224
14
86.32
89.14
95.49
194
1.01k
BiT-R200x3
224
14
84.73
89.58
141
1.78k
Mixer-I/16
448
14
86.78
89.72
95.13
105
0.45k
VIT-H/14
224
14
86.65
89.56
95.57
87
2.30k
VIT-L/16 [14]
512
14
87.76
90.54
95.63
32
0.65k
While not strictly comparable, this allows linear 5-shot top-1 accuracy on ImageNet as a proxy for transfer quality. For every pre-training run perform early stopping based on the best upstream validation performance. Results reported in Figure (right), where we also include VIT-B/32, ViT-L/32, ViT-L/16, and BiT-R152x2 models.
further extrapolate the effect of scale. We the
When pre-trained on the smallest subset of JFT-300M, all Mixer models strongly overfit. BiT models also overfit, but to a lesser extent, possibly due to the strong inductive biases associated with the convolutions. As the dataset increases, the performance of both Mixer-L/32 and Mixer-L/16 grows faster than BiT; Mixer-L/16 keeps improving, while the BiT model plateaus.
The same conclusions hold for ViT, consistent with Dosovitskiy et al. [14]. However, the relative improvement of larger Mixer models are even more pronounced. The performance gap between Mixer-I and ViT-L/16 shrinks with data scale. It appears that Mixer benefits from the growing dataset size even more than ViT. One could speculate and explain it again with the difference in inductive biases: self-attention layers in ViT lead to certain properties of the learned functions that less compatible with the true underlying distribution than those discovered with Mixer architecture.
3.4 Invariance to input permutations
In this section, we study the difference between inductive biases of Mixer and CNN architectures. Specifically, we train Mixer-B// and ResNet50x1 models JFT-300M following the pre-training

---

Original
Patch pixel shuffling
Global shuffling
original
global shuffling
patch pixel shurfling
Linear ImageNet
Mixer-Bll B/16
ResNet50xl
0.-
500
100k
500%
100.
Training steps
Training steps
Figure 4: Top: Input examples from ImageNet before permuting the contents (left); after shuffling the 16 16 patches and pixels within the patches (center); after shuffling pixels globally (right).
Bottom: Mixer-B/16 (left) and ResNet50x1 (right) trained with three corresponding input pipelines.
Figure 5: Hidden units in the first (left), second (center), and third (right) voken-mixing MLPs of Mixer-B/16 model trained on JFT-300M. Each unit has 196 weights, one for each of the 14 14 incoming patches. We pair the units to highlight the emergence of kernels of opposing phase. Pairs are sorted by filter frequency. In contrast the kernels of convolutional filters, where each weight corresponds to one pixel in the input image, one weight in any plot from the left column corresponds to particular 16 16 patch
input image. Complete plots in Supplementary D.
setup described in Section 3 and using one of two different input transformations: (1) Shuffle the order 16x 16 patches and permute pixels within each patch with a shared permutation; (2) Permute the pixels globally in the entire image. Same permutation is used across all images. We report the linear 5-shot top-1 accuracy the trained models on ImageNet Figure (bottom). Some original images along with their two transformed versions appear in Figure 4 (top). As could be expected, Mixer is invariant to the order of patches and pixels within the patches (the blue and green curves match perfectly). On the other hand, ResNet's strong inductive bias relies of pixels within an image and its performance drops significantly when the patches are permuted. Remarkably, when globally permuting the pixels, Mixer's performance drops much less (~445% drop) compared to the ResNet (~75% drop).
particular order
3.5 Visualization
It is commonly observed that the first layers of CNNs tend to learn Gabor-like detectors that act on pixels in local regions the image. In contrast, Mixer allows for global information exchange the token-mixing MLPs, which begs the question whether it processes information in similar fashion. Figure shows hidden units of the first three loken-mixing MLPs of Mixer trained on JFT-300M. Recall that the makinn MLPs allow global communication between different spatial locations. Some of learned features operate on entire image, while others operate on smaller regions. Deeper layers appear to have no clearly identifiable structure. Similar to CNNs, we observe many pairs of feature detectors with opposite phases [39]. The structure of learned units depends on the hyprrparameters. Plots for the first embedding layer appear in Figure 2 of Supplementary D.

---

4 Related work
MLP-Mixer is new architecture for computer vision that differs from previous successful architec- tures because it uses neither convolutional nor self-attention layers. Nevertheless, the design choices can be traced back to ideas from the literature on CNNs [24, 25] and Transformers [50].
CNNs have been the de-facto standard in computer vision since the AlexNet model [24] surpassed prevailing approaches based on sand-crafted image features [36]. Many works focused on improving the design of CNNs. Simonyan and Zisserman [41] demonstrated that one can train state-of offtt--f--hh-arr models using only convolutions with small 3x3 kernels. He et al. [15] introduced skip-connections together with the batch normalization [20], which enabled training of very deep neural networks and further improved performance. prominent line of research has investigated the benefits of using sparse convolutions, such as grouped [57] or depth-wise winte [9, 17] variants. In a similar spirit to our token-mixing MLPs, Wu et al. [55] share parameters in the depth-wise convolutions for natural language processing. Hu et al. [18] and Wang et al. [53] propose to augment convolutional networks with non-local operations to partially alleviate the constraint of local processing from CNNs. Mixer takes the idea of using convolutions with small kernels to the extreme: by reducing the kernel size to 1x1 1 it turns convolutions into standard dense matrix multiplications applied independently to each spatial location Ohhannel-mixing MLPs). This alone does not allow aggregation of spatial information and to compensate we apply dense matrix multiplications that are applied to every feature across all spatial locations (token-mixing MLPs). In Mixer, matrix multiplications are applied row-wise or column- on the "patches features" input table, which is also closely related to the work on sparse convolutions. Mixer uses skip-connections [15] and normalization layers [2, 20].
In computer vision, self-attention based Transformer architectures were initially applied for generative modeling [8, 34j. Their value for image recognition was demonstrated later, albeit in combination with a convolution- [14] introduced ViT, pure transformer model that has fewer locality biases, but scales well to large data. ViT achieves state-of-the-an performance on popular vision benchmarks while retaining the robustness of CNNs [6]. Touvron et al. [49] trained ViT effectively on smaller datasets using extensive regularization. Mixer borrows design choices from recent transformer- based architectures. The design of Mixer's MLP-blocks originates in [27, 50]. Converting images to sequence of patches and directly processing embeddings of these patches originates in Dosovitskiy et al. [14].
locality bias [37], or on low-resolution images [10]. Dosovitskiy et al.
Many recent works strive to design more effective architectures for vision. Srinivas et al. [42] replace 3 3x3 convolutions in ResNets by self-attention layers. Ramachandran et al. [37], Tay et al. [47], Li et al. [26], and Bello [3] design networks with new attention-lik mechanisms. Mixer can be seen as a step in an orthogonal direction, without reliance on locality bias and attention mechanisms.
The work of Lin et al. [28] is closely related. It attains reasonable performance on CIFAR- 10 using fully connected networks, heavy data augmentation, and pre-training with an auto-encoder. Neyshabur [31] devises custom regularization and optimization algorithms and trains fully-connected network, attaining impressive performance on small-scale tasks. Instead we rely on token and channel-mixing MLPs, use standard regularization and optimization techniques, and scale to large data effectively.
Traditionally, networks evaluated on ImageNet [13] are trained from random initialization using Inception-style pre-processing [46]. For smaller datasets, transfer of ImageNet models is popular. However, modern state-of-the the-art models typically use either weights pre-trained on larger datasets, or more recent data-augmentatioon and training strategies. For example, Dosovitskiy et al. [14], Kolesnikov et al. [22], Mahajan et al. [30], Pham et al. [35], Xie et al. [56] all advance ofate-of-the-art in image classification using large-scale pre-training. Examples of improvements due to augmentation or regularization changes include Cubuk et al. [11], who attain excellent classification performance with learned data augmentation, and Bello et al. [4], who show that canonical ResNets are still near Kate-of-theeard if one uses recent training and augmentation strategies.
5 Conclusions
We describe a very simple architecture for vision. Our experiments demonstrate that is as good as existing sfate-of-the-art methods in terms of the trade-off between accuracy and computational resources required for training and inference. We believe these results open many questions. On the practical side, it may be useful to study the features learned by the model and identify the main

---

differences (if any) from those learned by CNNs and Transformers. On the theoretical side, we would like understand the inductive biases hidden in these various features and eventually their role in generalization. Most of all, we hope that our results spark further research, beyond the realms of established models based on convolutions and self-attention. It would be particularly interesting to see whether such a design works in NLP or other domains.
Acknowledgments and Disclosure of Funding
The work was performed in the Brain teams in Berlin and Zürich. We thank Josip Djolonga for feedback on the initial version of the paper; Preetum Nakkiran for proposing train MLP-Mixer on input images with shuffled pixels; Olivier Bousquet, Yann Dauphin, and Dirk Weissenborn for useful discussions.
References
[1] A. Araujo, W. Norris, and Sim. Computing receptive fields of convolutional neural net- works. Distill, 2019. doi: 00,23915/distill.00022 URL
rumutting-reeppiive- fields..
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[3] I. Bello. Lewrdss Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602. 2021.
[4] I. Bello, W. Fedus, X. Du, E. D. Cubuk, A. Srinivas, T.-Y. Lin, J. Shlens, and B. Zoph. Revisiting ResNets: Improved training and scaling strategies. arXiv preprint arXiv:2 07579, 2021.
[5] L. Beyer, J. Hénaff, A. Kolesnikov, X. Zhai, and A. van den Oord. Are we done with ImageNet? arXiv preprint arXiv:2006.0713 071590 2020.
[6] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterthiner, and A. Veit. Understanding robustness of transformers for image classification. arXiv preprini arXiv:2103. 14586, 2021.
[7] A. Brock, S. De. S. Smith, and K. Simonyan. performareem large-scale image recognition without normalization. arXiv preprint arXiv:2102.0617 2021.
[8] R. Child, S. Gray, A. Radford, and Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1944.
2019.
[9] F. Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.
[10] J.-B. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self-attention and convolutional layers. In ICLR, 2020.
[11] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning augmentation policies from data. In CVPR, 2019.
[12] E. D. Cubuk, B. Zoph. J. Shlens, and Q. V. Le. RandAugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[13] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
[14] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[15] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR. 2016.
[16] D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs). arXiv preprint 1606.0606.08411 2016.
[17] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and
H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.
[18] J. Hu, L. Shen, and G. Sun. maneezzeandd-xxctation networks. In CVPR, 2018.
10

---

[19] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
[20] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[21] C. Jia, Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and aisionllanuuage representation learning with noisy text supervision. arXiv preprint arXiv: 102.0590 2021.
[22] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver. J. Yung. S. Gelly, and N. Houlsby. Big transfer
(BiT): General visual representation learning. In ECCV, 2020.
[23] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report. University of Toronto, 2009.
[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NeurlPS, 2012.
[25] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Back- propagation applied to handwritten zip code recognition. Newral Computation, 1:541-551, 1989.
[26] D. J. J. Hu, C. Wang, X. Li, Q. She, L. Zhu, T. Zhang, and Q. Chen. Involution: Inverting the inherence of convolution for visual recognition. CVPR, 2021.
[27] M. Lin, Q. Chen, and S. Yan. Network in network. In ICLR, 2014.
[28] Z. Lin, R. Memisevic, and K. Konda. How far can we go without convolution: Improving fullyconnected networks. In ICLR, Workshop Track, 2016.
[29] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS, 2016.
[30] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
[31] B. Neyshabur. Towards learning convolutions from scratch. NeurlPS, 2020.
[32] M. Nilsback and A. Zisserman. Automated flower classification large number of classes.
In ICVGIP, 2008
[33] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. CVPR, 2012.
[34] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image
transformer. In JCML, 2018.
[35] H. Pham, Z. Dai, Q. Xie, M.-T. Luong, and Q. V. Le. Meta pseudo labels. In CVPR, 2021.
[36] A. Pinz. Object categorization. Foundations and Trends in Computer Graphics and Vision, 1(4), 2006.
[37] Ramachandran, N. Parmar, A. Vaswani, Bello, A. Levskaya, and J. Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019.
[38] M. Sandler, J. Baccash, A. Zhmoginov, and Howard. Non-discriminative data or weak model? On the relative importance of data and model resolution. In ICCV Workshop on Real- World Recognition from Low-Quality Images and Videos, 2019.
[39] W. Shang, K. Sohn, D. Almeida, and H. Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In ICML, 2016.
[40] L. Sifre. Rigid-Motion Scattering For Image Classification. PhD thesis, Ecole Polytechnique, 2014.
[41] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[42] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605, 2021.
[43] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, JMLR, 15(56), 2014.
[44] C. Sun, A. Shrivastava, Singh, and Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.

---

[45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed. D. Anguelov, D. Erhan, V. Vanhoucke, and
A. Rabinovich. Going deeper with convolutions. CVPR, 2015.
[46] C. Szegedy, V. Vanhoucke, S. lofte, Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In CVPR, 2016.
[47] Y. Tay, D. Bahri, D. Metzler, D.-C. Juan, Zhao, and C. Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv, 2020.
[48] H. Touvron, A. Vedaldi, M. Douze, and H. Jegou. Fixing the train-test resolution discrepancy. NeurIPS, 2019.
[49] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. In NeurlPS, 2017.
[51] A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman, and J. Shlens. Scaling local self-attention for parameter efficient visual backbones. arXiv preprint 1xXix:2003.12311, 2021.
[52] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 2021.
[53] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In CVPR, 2018.
[54] R. Wightman.
Pytorch image models.
httts:///gtthubb.cco///wgggggi
pytorch- image-models, 2019.
[55] F. Wu, A. Fan, A. Baevski, Y. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
[56] Q. Xie, M.-T. Luong, E. Hovy, and Q. Le. Self-training with noisy student improves imagenet classification. CVPR, 2020.
[57] Xie, R. Girshick, Dollár, Tu, and K. He. Aggregated residual transformations for deep 1.05431, 2016.
neural networks. arXiv preprint
[58] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.0448 2019.
[59] X. Zhai, A. Kolesnikov. N. Houlsby, and L. Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.044
2021.
[60] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2018
12


Reviews:
[{"review_id": "dZRE9sOOTSm", "reviewer": "Reviewer_5HxS", "summary": "The paper proposes to a new architecture for image classification, based on recently proposed Vision Transformers. Instead of employing self-attention as a global operation on patch embeddings, the authors use fully connected layers, followed by channel mixing done with fully connected layers too. The changes are motivated by improving performance on proprietary hardware (TPU), designed for matrix multiplication. Unlike ViT which has quadratic cost on the number of input patches, the proposed Mixer model has linear cost. However, it loses the ability to handle arbitrary image resolutions. The experimental evaluation shows that the proposed model is comparable performance to state-of-the-art convolutional and Transformer-based models on proprietary and public data.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the idea of splitting image classification networks into mixing spatially and channel-wise is not novel and was explored in Network-in-Network [Lin et al, 2014], in which convolutions were interleaved with MLPs too. This important citation and comparison are missing in the paper. The authors motivate Mixer with \"a convolution is more complex than plain matrix multiplication in MLPs as it requires an additional costly reduction to matrix multiplication  and/or specialized implementation\" (lines 36-37). I disagree with this statement. The concepts of matrix multiplication and convolution are both equally simple, and matrix multiplication requires a specialized implementation as well. The motivation to use matrix multiplication comes from the usage of proprietary hardware (TPU), which was originally designed for matrix multiplication, and is not well suited to convolution. The paper thus addresses an issue that only exists within one company which has access to the particular proprietary hardware and large proprietary datasets. This needs to be clearly stated in motivation. Compared to Network-In-Network, the authors show that spatial mixing can be done with MLP instead of convolution, or self-attention as in ViT, without the need of positional encoding, which is an interesting and novel finding. However, such reduction of inductive bias in architecture comes at a price:\n- the model cannot be applied to images of different resolutions easily (as the authors state in section 3), as unlike CNN or ViT it cannot be applied in \"fully-convolutional\" manner. There are at least two very important cases for this, finetuning and object detection;\n- the model becomes more data-hungry;\n- the number of model parameters should increase: that would be expected, and important for deployment of such networks, yet never discussed in the paper.\nThe paper lacks related work on extremely simple and high performant CNN architectures, e.g. RepVGG [Ding et al. 2021], consisting only of convolutions and nonlinearities.\n\nQuality: The authors provide an extensive evaluation and test the model for invariance to input permutations. Tables miss the indication of numbers of parameters for different models. I would be valuable to include throughput on other devices than TPUs, e.g. GPUs.\n\nClarity: the paper is well written and is easy to follow. \n\nSignificance: the finding that image classification can be done with 1x1 convolutions interleaved with MLPs for spatial mixing is signficant, but will mostly be interesting for the users of the proprietary devices for which the model was designed for, as it also loses some desirable features of CNN and ViT.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations and potential negative societal impact of their work adequately.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QxrH9hiFr5K", "reviewer": "Reviewer_MVp5", "summary": "This paper presents an all-MLP based architecture for visual recognition tasks. The presented model is to replace the multi-head self-attention block with a spatial MLP in the original Transformer model. With a large-scale pre-trained dataset, this model with few inductive biases could achieve relatively good performance on image classification tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For originality, the architecture proposed in this paper is simple and somewhat novel. A spatial MLP is an effective approach to perform spatial connections when a large-scale dataset is available.\n\nThe paper is well written and well organized.\n\nWith a large-scale pre-trained dataset, this model with few inductive biases could achieve relatively good performance on image classification tasks. This observation is interesting and valuable to the community.\n\nMy major concern is about the spatial MLP (MLP1). This is a simple design, but the application scenario is also limited due to the fixed size of the input spatial resolution. For example, for some downstream tasks of object detection and semantic segmentation, the input sizes are different to that of the image classification. This architecture could not be directly applied to these tasks.\n\nAlso, it is interesting to see the performance of the proposed approach on natural language processing tasks, such as in comparison to BERT on mask language modeling.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BvKhcVyVBex", "reviewer": "Reviewer_6miq", "summary": "This paper proposes a neural network architecture of all MLP layers without any convolution or attention layers. Although it only has MLP layers, it still needs to exchange the information across spacial locations by token mixing strategy. By pretraining on large-scale private datasets, the proposed method achieve comparable results as the standard CNN and transformer networks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-This paper is the first work show strong results by only using MLP layers. But this idea is not too difficult to think, given the recent great successes of transformer based vision (ViT) models. Most of techniques and experiences are from ViT, and the only changes is to replace the attention layers by the MLP layers. \n\n-The quality of the papers is good. Experiments are sufficient and most of the claims are supported. But this paper is a little bit obsessed with classification task, especially on ImageNet. I don't quite understand the point to push forward the ImageNet top-1 accuracy, e.g. over 90%, by pretraining a super large model on a super large private dataset, especially when ImageNet itself has some annotation errors. And Table 3 reports \"Avg.5\"? VTAB is quite common for this kind of Google papers. \n\n-The paper is quite clear and easy to read.\n\n-I am not confident about the significance of this paper. We can say it is a reduced alternative to ViT, by replacing attention layer with token-mixing MLP layer. But we already have CNN and ViT, why do we still need MLP-Mixer? As claimed in the paper, it is an alternative to CNN and ViT, then what are the advantages and disadvantages of MLP-Mixer, for classification and beyond classification? I don't think MLP-Mixer can replace CNN or transformer and can do some tasks that are unavailable for CNN or transformer.\n\n-I think this paper overclaim the significance of MLP-Mixer as alternative to CNN or transformer. As mentioned in the paper, it is a special case of CNN, with 1x1 and global depth-wise convolution. How could a special case be an alternative for a general case?\n\n================after rebuttal================\n\nI am disappointed by the rebuttal. The rebuttal doesn't have satisfactory responses to the two major concerns: 1) the application limit of this paper and 2) the overclaimed significance. What's worse is that the authors seem not to be willing to improve them. Given these, I will downgrade my rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No much about limitations. See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4n97LFPOonR", "reviewer": "Reviewer_tsff", "summary": "The paper proposes to use MLPs in Image Classification and obtains correct performances on ImageNet and VTAB .\nTheir architecture is similar to vision transformers except that it replaces the self attention with an MLP (similar to the FFN of transformers) applied on the tokens.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- The results obtained are good for an MLP architecture that is simple. This results are very interesting and very aligned with the recent work on vision transformers in computer vision.\n- Results are reported on ImageNet, ImageNet-real and VTAB  with pre-training with different amount of date which is a good practice and very useful. Nevertheless it would be interesting to have also the results on ImageNet-v2 [1].\n- The paper is well written and easy to follow.\n\n\nWeakness:\n- The comparison on ImageNet without extra-training data with other types of architecture is quite incomplete. There are only comparisons with BiT and ViT which are not state of the art in Image classification for convnets and transformers architecture. \nIt would be interesting to have comparisons with EfficientNet-v2 [2], NFNet [3], CaiT [4], Swin [5] or CvT [6]. \nFor the comparison on ImageNet with extra-data, MLP-mixer is compared with more architecture but it lacks some state-of-the-art approaches like EfficientNet-v2 [2], Swin [5] or CvT[6]. \nThis may bias the conclusion: \"MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models\" L9\nNevertheless, the paper indicates that the MLP mixer is still below ViT: \"When trained from scratch on ImageNet, Mixer-B/16 achieves a reasonable top-1 accuracy of 76.44%. This is 3% behind the ViT-B/16 model\" L182\n\nMinor comments:\n\n- The ViT results pre-trained on ImageNet with extra regularization are quite low in comparison to the DeiT paper. One may wonder why the paper does not use the same training procedure.\n\n- FLOPs of the models are not reported. It would be very useful for the comparison with other architecture of the literature since it is a fairly commonly used metric. This metric is not perfect but the throughput also has some flaws as it depends a lot on the implementation, the hardware, ....\n\n[1] Recht et al.,  Do ImageNet Classifiers Generalize to ImageNet? \n[2] Tan et al., EfficientNetV2: Smaller Models and Faster Training\n[3] Brock et al. High-Performance Large-Scale Image Recognition Without Normalization\n[4] Touvron et al. Going deeper with Image Transformers\n[5] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows\n[6] Wu et al. CvT: Introducing Convolutions to Vision Transformers\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, the paper received scores 7, 7, 6, and 5. All reviewers consider the idea is very interesting, experiments are comprehensive, the paper is well-written, and results are strong given the simplicity of the model. At the same time, multiple concerns were raised, specifically pointing out limitations of the model ( e.g., inability to handle multiple resolutions and tasks such as detection/segmentation; the model is data-hungry, parameter-inefficient, and tailored to specialized hardware/TPUs). During the discussion phase, two reviewers were disappointed by the author response, which did not address well questions such as reporting ImageNet results from other architectures, adopting the same training procedure as in DeiT, and over claimed significance of results. They downgraded their rating from 7 to 6, still recommending borderline acceptance. While the concerns raised by the reviewers are all legitimate, the AC considers that the strengths of the paper outweighs its weaknesses, and agrees with the majority that it passes the acceptance bar of NeurIPS. In particular, the proposed architecture has the potential to make an impact and inspire further research in the field, as it is conceptually simple yet achieves strong results. The authors should include in the camera-ready the discussion based on reviewer’s comments, properly convey the limitations of the architecture, and also address the two ethical concerns pointed out in the ethics review.
Decision: Accept (Poster)