Pay Attention to MLPs
Authors: Hanxiao Liu
Zihang Dai
David So
Quoc V Le
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
Attention
MLP

Abstract: Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based solely on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

Pay Attention to MLPs
Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le
Google Research, Brain Team
{hanxiaol, zihangd, favidso,qvl.@@ooo. gy1looooogee.co
Abstract
Transformers [1] have become one the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform well Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.
1 Introduction
Transformers [1] have enabled many breakthroughs in natural language processing (e.g.. (2.3,4, 6]) and have been shown work well for computer vision (e.g., [7, 8, 9, 10]). Thanks this success, Transformers have largely replaced LSTM-RNN [11] as the default architecture in NLP, and have become appealing alternative to ConvNets [12, 13, 14, 15, 16, 17) in computer vision.
The Transformer architecture combines two important concepts: (1) a recurrent-free architecture which computes the representations for each individual token in parallel, and (2) multi-head self- attention blocks which aggregate spatial information across tokens. On one hand, the attention mechanism [18] introduces the inductive bias that the spatial interactions should be dynamically parameterized based on the input representations. On the other hand, it is known that MLPs with static parameterization represent arbitrary functions [19]. It therefore remains open question whether the inductive bias in self- self-attention is essential to the remarkable effectiveness of Transformers.
Here we study the necessity of self-attention modules in key language and vision applications of Trans- formers. Specifically, we propose an ML.P-based alternative to Transformers without self-attention, which simply consists of channel projections and spatial projections with static parameterization. We experiment with several design choices for this architecture and find spatial projections work well when they are linear and paired with multiplicative gating (Figure 1). We name the model gMLP because is built out of basic MLP layers with gating.
We apply gMLP to image classification and obtain strong results on ImageNet. gMLP achieves comparable performance with DeiT [8], namely Vision Transformer (ViT) [7] with improved regular- ization, in a similar training setup. With 66% less parameters, gMLP model is 3% more accurate than MLP-Mixer [20]. Together with Tolstikhin al. [20], Melas-Kyriaz [21], Touvron et [22] and Ding et. al. [23], our results question the necessity of self-attention layers in Vision Transformers.
We apply gMLP masked language modeling (MLM) in the BERT [2] setup, one of the most well- established applications of Transformers, and find that it is as good as Transformers at minimizing perplexity during pretraining. Our experiments indicate that perplexity is only correlated with model capacity and is insensitive to the presence of self-attention As capacity increases, we observe that
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pseudo-code for the gMLP block
def gnlp_blocker d_model, d_ffa):
shortcut
Channel Proj
x norn norm axia="chamnel")
X proj (x, d_ffn, axie="channel")
Spatial Gating Unlt
galu(x)
Spatial Proj
apatial_gatinggunitte
x - proj (x, d_madel, axis="chanmel")
Activation
Norm
return shortcut
split
Channel Proj
def spatial_gating
unit(x): satingg_mit(())
u, split (x, axis=" chanmel"))
Narm
norn(v, axie="chammel")
get_din(v, exie="spatial")
proj(v, axis="spatial", init_ return x
Input Embeddings
Figure 1: Overview of the gMLP architecture with Spatial Gating Unit (SGU). The model consists of stack of blocks with identical structure and size. All projection operations are linear and "O" refers to element-wiss multiplication (linear gating). The input and output protocols follow BERT for NLP and VIT vision. Unlike Transformers, gMLPs do not require positional encodings, nor necessary to mask out the paddings during NLP finetuning.
both pretraining and finetuning metrics for gMLPs improve as quickly as for Transformers. This is remarkable because it indicates gMLPs scale just as well as Transformers despite the absence of self-attention, and any performance gap can always be offset by training a larger model with increased data and compute. With a standard 256-batch size X IM-step training setup as in original BERT, large gMLP model achieves 87.7% accuracy on MNLI and 82. Fl on SQuAD v2.0. Note, these are better than the BERT large results reported in Devlin et al. [2] obtained using Transformers.
lage
For BERT's finetuning. Transformers be more practically advantageous over gMLPs tasks that require cross-sentence alignment (e.g.. by 0.8% on MNLI-m in the 300M-param regime), even with similar pretraining perplexity. This problem larger-3x as large as Transformers. more practical solution is to blend in only a tiny bit of self- attention-a single-head self-attention with size up to 128 sufficient make gMLPs outperform Transformers on all NLP tasks we evaluated with even better parameter efficiency. The improvement sometimes very significant (e.g., +4.4% on SQuAD v2.0 over BERT tane).
be addressed by making MMLPs substantially
Overall, the surprising effectiveness of gMLPs in both vision and NLP domains suggests that self- attention is not necessary ingredient for scaling up machine learning models, although it can be useful addition depending on the task. With increased data and compute, models with simpler spatial interaction mechanisms such as gMIP can be powerful as Transformers and the capacity allocated to self-attention can be either removed or substantially reduced.
2 Model
Our model, gMLP, consists of stack of L blocks with identical size and structure. Let e gaxd be the token representations with sequence length n and dimension d. Each block is defined as:
Z = oXU),
= sZ),
(1)
where a is an activation function such as GeLU [24]. and define linear projections along the channel dimension- the same those in the FFNs of Transformers (c.g., their shapes 768x 3072 and 3072x 768 for BERT Shortcuts, normalizations and biases are omitted for brevity.
A key ingredient in the aforementioned formulation is s(-), layer which captures spatial interactions (see below). When is an identity mapping, the above transformation degenerates to a regular FFN, where individual tokens are processed independently without any cross-token communication. One of our major focuses therefore to design good s capable of capturing complex spatial interactions across tokens. The overall block layout is inspired by inverted bottlenecks [25] which define s(-) as a spatial depthwise convolution. Note, unlike Transformers, our model does not require position embeddings because such information will be captured in s(-).

---

Our model uses exactly the same input and output protocols as BERT (for NLP) and VIT (for vision). For example, when finetuning on language tasks, we concatenate together multiple text segments followed by paddings, and the predictions are deduced from the last-layer representation of reserved <cls> symbol. Although many of these protocols were introduced for Transformers and hence can be suboptimal for gMLPs, strictly following them helps avoid confounding factors in our experiments and makes our layers more compatible with existing Transformer implementations.
2.1 Spatial Gating Unit
To enable cross-token interactions, it is necessary for the layer s(-) to contain a contraction operation over the spatial dimension. The simplistic option would be linear projection:
fw,o(Z) WZ +
where W E R= is a matrix for which the size is the same as the sequence length, n, and b refers token-specific biases. For example, if the padded input sequence has 128 tokens, the shape for W will be x28x 128. Unlike self-attention where (Z) is dynamically generated from Z, the spatial projection matrix W here in Equation (2) independent from the input representations.
In this work, we formulate layer s()) as the output of linear gating:
(2) = 2 Swo(Z)
where denotes element-wise multiplication. For training stability, we find it critical to initialize W as near-zero values and b as ones, meaning that fw,(2) xe 1 and therefore s s(Z) & Z at the beginning of training. This initialization ensures each gMLP block behaves like regular FFN at the early stage of training, where each token is processed independently, and only gradually injects spatial information across tokens during the course of learning.
We further find it effective to split Z into two independent parts (Z,, Z2) along the channel dimension for the gating function and for the multiplicative bypass:
s 22) = Zi O fwo(Zz)
(4)
We also normalize the input to SW. which empirically improves stability of large NLP models. This gives the unit illustrated in Figure which refer to the Spatial Gating Unit (SGU) in the rest of the paper. In Table 3. we provide ablation studies to compare SGU with several other variants of s(.), showing that it works better and narrows the performance gap with self-attention.
Connections to Existing Layers. The overall formulation of SGU resembles Gated Linear Units (GLUs) [26. 27, 28] as well as earlier works including Highway Networks [29] and LSTM-RNNS [11]. A key distinction is that our gating is computed based on projection over the spatial (cross-token) dimension rather than the channel (hidden) dimension. SGU is also related to Squeeze-ang and-Excite (SE) blocks [30] in terms of element-wise multiplication. However, different from SE blocks, SGU does not contain cross-channel projections at all, nor does it enforce permutation invariance (a key feature for content- content-based attentive modules) due to its static parameterization for the spatial transformation. The spatial projection in SGU could theory learn to express superficial depthwise convolutions- unniiko typical depthwise convolutions with channel speecffic filters, SGU learns only single transformation shared across channels. Finally, we note SGUs offer an alternative mechanism to capture high-order relationships other than self-attention. Specifically, the output for Equation (3) contains up to 2nd-order interactions (e.g., 2ı2,) whereas output for self-attention (assuming no nonlinearity) contains up 3rd-order interactions (c.g., vuk) In terms of computation cost, SGU has n e multiply-adds which is comparable to the 2n² dd of dot-product self-attention. Both are linear over the input channel size and quadratic over the sequence length n.
3 Image Classification
Here we examine gMLP in the vision domain by applying it to the image classification task on ImageNet [31] without using extra data. We compare our MLP-like models with recent attentive
'The input channel size e for SGU is typically larger than the input channel size d for self attenattnnion, because the former is applied in the middle of the block after channel expansion.
3

---

models based on vanilla Transformers, including Vision Transformer (ViT) [7], DeiT [8] (ViT with improved regularization), and several other representative convolutional networks.
Table summarizes the configurations of our gMLP image classification models. The input and output protocols follow ViT/B16 where the image is converted into 16x 16 patches the stem. The depth and width are chosen so that the models are comparable with ViT/DeiT in capacity. Like Transformers, we find gMLPs tend drastically overfit the training data. We therefore apply similar regularization recipe as the one used in DeiT.² To avoid extensive tuning, we adjust only the strengths of stochastic depth [32] as we move from smaller larger models in Table 1. All the other hyperparameters remain shared across our three models. See Appendix A.1 for details.
Table Architecture specifications of gMLP models for vision.
#L
dmodel
det
Params (M)
FLOPs (B)
Survival Prob
gMLP-Ti
30
128
768
5.9
2.7
1.00
gMLP
30
256
1536
19.5
8.9
0.95
gMLP-B
30
512
3072
73.4
31.6
0.80
Our ImageNet results are summarized in Table and Figure 2. It is interesting to see that gMMPss are comparable with DeiT [8], namely ViT [7] trained using improved regularization. The results suggest that models without self-attention can be as data-efficient as Transformers for image classification. In fact, when the models are properly regularized, their accuracies seem better correlated with capacity instead the presence of self-attention. Moreover, the accuracy-parameter/Fi surpasses all concurrently proposed MI.P-like architectures [20, 21, 22], which we attribute to the effectiveness of our Spatial Gating Unit (see Table the next section for an ablation). We also note while gMLPs are competitive with vanilla Transformers, their performance is behind the best existing ConvNet models (c.g., [33, 34J) or hybrid models (e.g., [35, 36, 37, 38, 10]).
tradeoff of gMLPs
Table 2: ImageNet- 1K results without extra data.
ImageNet Top-1 (%)
Input Resolution
Params (M)
MAdds (B)
Model
ConvNets
ResNer-152 [16]
78.3
11.3
RegNecY-8GF [39]
81.7
224
8.0
EfficientNet-BO
77.1
224
0.39
LtticiietNet-B33
81.6
300
1.8
EfticientNet-B7 [17]
84.3
600
66
37.0
NFNet-FO [33]
83.6
192
72
12.4
Transformers
VIT-B/16 [7]
77.9
384
55.4
VIT-1/16 [7]
76.5
384
307
190.7
Deif-Ti [8] (ViT+reg)
72.2
224
1.3
DeiT-S [8] (ViTtreg)
79.8
224
22
4.6
DeiT-B (8) (ViT+reg)
81.8
224
86
17.5
MI.P-likef
Mixcr-B166 [20]
76.4
224
12.7
Mixer-B/16 (our setup)
77.3
224
12.7
Mixer-L/16 [20]
71.8
224
207
44.8
ResMLP-12 12 [22]
76.6
224
15
3.0
ResMLP-24 [22]
79.4
224
30
6.0
ResMI. P-36 [22]
79.7
8.9
gMLP-Ti (ours)
72.3
1.4
gMLP-S (ours)
79.6
224
4.5
gMLP-B (ours)
81.6
224
15.8
Standard deviation across multiple independent runs around 0.1.
Tokenization & embedding process at the stem be viewed convolution.
Figure visualizes the spatial projection matrices in gMLP-B. Remarkably, the spatial weights after learning exhibit both locality and spatial invariance. In other words, each spatial projection matrix effectively learns to perform convolution with a data-driven, irregular (non-square) kernel shape.
Unlike DeiT, we do not use repeated augmentation or random erasing. AAditiooal results ImageNet-21KK and COCO available in Appendix E.

---

DeiT-B (ViT+reg))
Layer
gMLP-B
Layer
ImageNet Top-3 Acc (%k
gMLP-5
Teg)
Res Bes MLP36
Layer 11
Mixer-B/16 (our setup)
Mixer-B/16
ResMLP:12
Layer
DelT +reg)
Layer 23
Mixer
DeiT-Ti gMLP-TI TMLP-T/VVT++re)) viT+
ResMLP
Layer
gMLP (ours)
20
60
100
Figure 3: Spatial projection weights in gMLP- B. Each row shows the filters (reshaped into 2D) for a selected of tokens in the same layer.
Params (M)
Figure ImageNet accuracy vs model capacity.
4 Masked Language Modeling with BERT
Here we conduct empirical studies over the masked language modeling (MLM) task. The input/output protocol for both pretraining and finetuning follows BERT [2]. Different from Transformer-based models, we do not use positional encodings. We also find unnecessary to mask out <pad> tokens in gMLP blocks during finetuning the model quickly learn ignore them. For ablations and case studies, all models are trained with batch size 2048, max length 128 for 125K steps over the RealNews-like subset of C4 [5]. For main results, models are trained with batch size 256, max length 512 for IM steps over the full English C4 dataset. See Appendix A.2 for details.
Our preliminary MLM experiments show that gMLPs always learn Toeplitz-like matrices the spatial weights (Appendix C). This means gMLPss are able to learn the notion of shift invariance from data, property naturally implied by the MLM task where any offset of the input sequence does not affect the slot filling outcome. In this case, the learned (w:a(-) acts like a 1-d convolution whose kernel size equals the entire sequence length (unlike depthwise convolution with channel-specific filters, here the same W is shared across channels). In the following MLM experiments, we restrict W to be Toeplitz matrix avoid redundant model parameterization (since W will be Toeplitz-like regardless after learming). Note this constraint is empirically quality-neutral.
4.1 Ablation: The Importance of Gating in gMLP for BERT's Pretraining
In Table below, we establish baselines for our ablation studies. These include:
1. BERT with a Transformer architecture and learnable absolute position embeddings.
2. BERT with Transformer architecture and T5-style learnable relative position biases [5]. The biases are both layer- and head- -rra-sccccficc we find this yields the best results.
3. Same as above, but remove all content-depender terms inside the softmax and only retain the relative positional biases. This baseline a straightforward variant Transformers without self-attention, which can also be viewed as a Random Synthesizer [40].
4. MLP-Mixer [20] which replaces the multi-head self-attention module in Transformers with a two-layer spatial MLP. This model was developed for image classification and here we investigate MLM tasks using the same training setup with BERT and gMLP.
We compare these baselines against several versions of gMLPs with similar sizes in Table 3. Note that Multiplicative, Split (last row) the Spatial Gating Unit we describe in the method section and use in the rest of the paper. First, SGU outperforms other variants in perplexity. Secondly and remarkably, gMIP with SGU also achieves perplexity comparable to Transformer. Note the difference between the strongest baseline (2e.plexit=4.266) and ours (perplexity=4335) is insignificant relative to the
5

---

Table 3: MLM validation perplexities of Transformer baselines and four versions of gMLPs. refers the spatial linear projection Equation (2) with input normalization. The MLP-Mixer baseline model has L=24 layers with mo6de==768 dspatia:=384 and din =3072. Each gMLP model has L=36 layers with m51e and dmn 3072. No positional encodings are used for Mixer or gMLPs
Model
Perplexity"
Params (M)
BERTba buse
4.37
110
BERT buse rel pos
4.26
110
BERT buse pos attn
5.64
96
MLP-Mixer
5.34
112
Linear gMLP, s(2)= f(Z)
5.14
92
Additive gMLP. s(2)
f(2)
4.97
92
Multiplicative gMLP, a() = fZ)
4.53
92
Multiplicatiiv, Split gMLP, s(2) ( f(Zz), 2 = 2
4.35
102
Standard deviation across multiple independent runs is around 0.01.
perplexity change when the models are scaled (see Table the next section). Spatial projection weights learned by gMLPs visualized in Figure 4.
Figure 4: Visualization of the spatial filters in gMLP learned on the MLM task. For each layer in the model we plot the row W associated with the token the middle of the sequence. The x-axis of each subplot has length of 128 which equal the number of tokens in the sequence. The learned filters appear to be smooth and have several types: forward-looking (e.g., 1st 2nd row), backward-looking (e.g., 5th in 2nd row) and bi-directional (e.g., 2nd last in the last row).
4.2 Case Study: The Behavior of gMLP as Model Size Increases
In Table 4, we investigate the scaling properties of Transformers and gMLPs in BERT as their model capacity grows. Specifically, we scale the depth these models by a factor of {0.5, 1, 4) x and report the their pretraining MLM perplexities dev sets of two tasks in GLUE [41]. Note each individual Transformer layer is effectively two consecutive blocks:
the validation set well finetuning results
for self-atted self-attention and one for FFN. In the table below use the notation of 12 + 12 to refer to 12 of self-attention blocks plus 12 FFN blocks in the Transformer baselines.
Table 4: Pretraining and dev-set finetuning results over increased model capacity. We use the relative positional encoding scheme for Transformers which performs the best in Table 3.
Model
#L
Params (M)
Perplexity
SST-2
MNLI-m
Transformer
6+6
67
4.91
90.4
81.5
gMLP
18
59
5.25
91.2
77.7
Transformer
12+12
110
4.26
91.3
83.3
gMLP
36
102
4.35
92.3
80.9
Transformer
24+24
195
3.83
92.1
85.2
gMLP
72
187
3.79
93.5
82.8
Transformer
48+48
365
3.47
92.8
86.3
gMLP
144
357
3.43
95.1
84.6
The results above show that a deep enough gMLP is able to match and even outperform the perplexity of Transformers with comparable capacity." In addition, the perplexity-parameter parameter relationships for
WW also experimented with deeper-and beeper-and-thinerr Transformers (with capacity fixed) but found increasing depth further does not improve perplexity. See Appendix B for more details.

---

both architecture families approximately follow power law (left of Figure 5). This implies the empirical scaling laws originally observed for Transformer-based language models [42] might be broadly applicable across different model families.
Transformer
95
86
Log Perplexity tlog scale)
1.6
gMLP
94
84-
1.5-
MNL --
144
92
91
80
1.3
Transformer
Transformer
90
gMLP
78
gMLP
10a
10a
10a
10a
10a
10a
Params (M) (log scale)
Params (M) (log scale)
Params (M) (log scale)
Figure 5: Scaling properties with respect to perplexity and finetuning accuracies. The figures show that for pretraining, gMLPs are equally good optimizing perplexity Transformers. For finetuning, the two model families exhibit comparable scalability despite task-specific offsets.
Table 4 also leads to an interesting observation that the pretraining perplexities across different model families not equal in terms of finetuning. While gMLPs outperform Transformers SST-2, they are worse on MNLI. The results imply that the finetuning performance for NLP tasks is a function of not only the perplexity but also the inductive bias in the architecture. Figure shows that despite the architecture-specinn discrepancies between pretraining and finetuning, gMLPs and Transformers exhibit comparable scalability (slope) both finetuning tasks. This means one can always offset the gap by enlarging the model capacity. In other words, the results indicate that model scalability with respect downstream metrics can be independent from the presence of self-attention.
4.3 Ablation: The Usefulness of Tiny Attention in BERT's Finetuning
So far we have found that self-attention not required component to achieve strong MLM perplexity or scalability. At the meantime, we also identified NLP finetuning tasks where gMLPs transfer less well than Transformers (Table 4). The fact that our MLP-like model is advantageous on SST-2 but worse MNLI is particularly informative-the former is single-sentence task whereas the latter involves sentence pairs (premise and hypothesis) [43]. We suspect the role of self-attention during finetuning is related to cross senseeeentence alignment.
To isolate the effect of self-attention, we experiment with a hybrid model where a tiny self-attention block is attached to the gating function of gMLP (Figure 6). Since gMLP itself is already capable in capturing spatial relationships, we hypothesize that this extra self-attention module does not have to be heavy, and that its presence is more relevant than its capacity. A typical tiny attention module in our experiments has only single head with size 64, significantly smaller than a typical multi-head self-attention in Transformers with 12 heads and a total size of 768. In the following, refer to the hybrid model, namely gMLP with a tiny self-attention, as aMLP ("a" for attention).
Pseudo-code for the tiny attention module
tiny_attnie d_out, d_atto=64);
qkv (x, d_attn, axis="channel")
Spatial Proj A Norm
Tiny Alin
9. k, split(qkr, 3, axias="hannel""
einsum cinsum("bnd, bmd->ban' "'un q, k)
softmax(u
(sqrtt(d_att))
split
einsum ("ban, bad->bud",
retura proj d_out, axis=* channel")
Figure 6: Hybrid spatial gating unit with a tiny self-attention module. We use the normalized input of the gMLP block (endpoint after the input normalization and right before the channel expansion) as the input to the tiny self-attention. For SGU have dout d2t. 2 due the channel split.
In Figure 7, we investigate the transferability of ML. models via the calibration plots between their pretraining perplexities and finetuning metrics. Models evaluated include BERT base. gMLP
7

---

and its hybrid version aMLP with 64-d single-head self-attention (Figure 6). The data points were collected by varying the model depth by (0.5, 1, 2] or data by [1, 4. 8/x. . can seen that gMLPPs transfer better to SST-2 than Transformers regardless of the presence of self-attention, While gMLP performs worse on MNI., attaching tiny bit of self-attention is sufficient close the gap. In Appendix D we visualize the tiny self-attention modules in aMLP over MNLI examples, showing that they are primarily responsible for the alignment between sentence pairs.
Transformer
gMLP
aMLP
Transformer
gMLP
aMLP
96
88
86
SST-2 Acp (%%
94
MNLI-m Acc (%)
92
-5.0
.4.5
-4.0
-3.5
-5.0
-4.5
-4.0
-3.5
Negative Perplexity
Negative Perplexity
Figure 7: Transferability from MLM pretraining perpexity to finetuning accuracies on GLUE. aMLP refers gMLP enhanced with 64-d single-head self-attention, illustrated in Figure 6. In contrast, each self-attention module in the BERT baseline contains 12 heads with a total size of 768.
In Figure 8 put together the scaling properties of the three models, showing that aMLP (gMLP+ tiny attention) consistently outperforms Transformer on both finetuning tasks.
Log Perplexity dog scale
1.7
Transformer
95
gMLP
1.6
94
1.5-
MNLL-m
82
1.4
92
1.3
Transformer
80
Transformer
gMLP
gMLP
90
1.2
102
10a
10a
10a
10a
10a
Params (log scale)
Params (M) (log scale)
Params (M) (log scale)
Figure 8: Comparing the scaling properties of Transformers, gMLPs and aMLPs (with 64-d, single- head attention). Results were obtained using the same setup in Section 4.2.
4.4 Main Results for MLM in the BERT Setup
Below present pretraining and finetuning results in the full BERT setup. Different from ablation and case studies, here we use the full English C4 dataset and adopt a common MLM setup with batch size 256, max length 512 and IM training steps. For fair comparison, we adjust the depth and width of gMLPs to ensure comparable model capacity with the Transformer baselines. The model specifications
given in Table and hyperparameters we report the dev-set performance for SST-2 and MNLI in GLUE [41] and each result entry was obtained by taking the median of five independent runs. addition, we report finetuning results on SQuAD [44, 45] to test the models' ability in reasoning over a longer context.
detailed in Appendix A.2. For finetuning,
Results presented in Table 6. Consistent with findings earlier in Section 4.1 and Section 4.2, gMLPs are competitive with Transformers in terms of perplexity, especially in the larger scale setup. There several observations related to the finetuning results:
First, on finetuning tasks where gMLPs underperform Transformers, performance gap tends to narrow as the model capacity increases. For example, while gMLP performs worse by 8.5% on SQuAD-v2.0 the base scale, the performance gap relative to the baseline decreases to 2.7% the larger scale. Notably, our MaLP achieves 89.5% Fl on SQuAD-v1.1 without any self-attention or

---

Table 5: Model specifications in the full BERT setup.
Params (M)
FLOPs (B)
#L
dmodel
ditn
BERT hase
110
100.8
12+12
768
3072
eMLP hose
130
158.0
48
512
3072
aMLP Passe
109
128.9
36
512
3072
BERT Turge
336
341.2
24+24
1024
4096
gMLP Pargg
365
430.1
96
768
3072
aMLP Jurge
316
370.3
72
768
3072
gMLP Atrrge
941
1091.3
144
1024
4096
Table 6: Pretraining perplexities and dev-set results for finetuning. "ours" indicates models trained using our setup. We report accuracies for SST-2 and MNLI, and Fl scores for SQuAD v1.1/2.0.
SQuAD
Params
Perplexity
SST-2
MNLI (m/mm)
Attn Size
v1.1
v2.0
(M)
BERT base [2]
92.7
84.4/-
88.5
76.3
768 (64 x 12)
110
BERT hase (ours)
4.17
93.8
85.6/85.7
90.2
78.6
768 (64 X 12)
110
gMLP base
4.28
94.2
83.7/84.1
86.7
70.1
130
aMLP base
3.95
93.4
85.9/85.8
90.7
80.9
64
109
BERT large [2]
93.7
86.6/-
90.9
81.8
1024 (64 x 16)
336
BERT lurge (ours)
3.35
94.3
87.0/87.4 .4
92.0
81.0
1024 (64
16)
336
gMLP large
3.32
94.8
86.2/86.5
89.5
78.3
365
aMLP Plarge
3.19
94.8
88.4/88.4
92.2
85.4
128
316
gML. slarge
2.89
95.6
87.7/87.7
90.9
82.1
941
dynamic spatial parameterization [28), which is well above the 88.5% reported for BERT base in Devlin al. [2] and is only 1.4% away from the original result for BERT iarge- We also include one additional data point by scaling up gMLP even further. The resulting model, gML. starge outperforms BERT large on SQuAD-v2.0-a difficult task involving question-answer pairs- mairs-without any self-attention. While this is not fair comparison due to different model sizes, it is an existence proof that MLP-like models can be competitive with Transformers on challenging NLP tasks.
Furthermore, we show that blending in tiny single-head self-attention of size either 64 or 128 is sufficient make gMLPs outperform Transformers of similar capacity, sometimes by significant margin. For example, our hybrid model aMLP lunge achieves 4.4% higher F1 than Transformers on SQuAD-v2.0. The results suggest that the capacity in the multi-head self-attention of Transformers can be largely redundant, and that the majority of its functionalities can be captured by the spatial gating unit gMLPs. The results also imply that the inductive biases in the spatial gating unit of gMLPs and the tiny attention are complementary each other. While the benefits of architectural inductive bias may vanish over increased compute, tiny attention does improve the practical value of gMLPss in the regime that we investigate in this work.
Conclusion
Since the seminal work of Vaswani et al. [1], Transformers have been widely adopted across NLP and computer vision. This adoption has enabled many impressive results especially in NLP. To date, is still unclear what empowers such success: is the feedforward nature of Transformers or is it the multi-head self-attention layers in Transformers?
Our work suggests a simpler alternative to the multi-head self-attention layers in Transformers. We show that gMLPs, simple variant of MLPs with gating, can be competitive with Transformers in terms of BERT's pretraining perplexity and VIT's accuracy. gMLPs are also comparable with Transformers in terms of the scalability over increased data and compute. As for BERT finetuning,

---

find gMLPs can achieve appealing results on challenging tasks such as SQuAD without self-attention, and can significantly outperform Transformers in certain cases. We also find the inductive bias in Transformer's multi-head self-attention useful on downstream tasks that require cross-sentence align- ment. However those cases, making gMLP substantially larger closes the gap with Transformers. More practically, blending a small single-heus head self-attention into gMLP allows for an even better architecture without the need for increasing model size.
Acknowledgements
We thank Gabriel Bender, Neil Houlsby, Thang Luong, Niki Parmar, Hieu Pham, Jascha Sohl- Dickstein, Noam Shazeer, Ilya Sutskever, Jakob Uszkoreit and Ashish Vaswani for their feedback.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, 2017.
[2] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
[3] Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe
Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019.
[4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.1169, 2019.
[5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
[8] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 2021.
[11] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[12] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012.
[14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[15] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
10

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[18] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICIR, 2015.
[19] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5): -66,,, 1989.
[20] Ily Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.0166 2021.
[21] Luke Melas-Kyriazi. Do you even need attention? a stack of feed-forward layers does surpris- ingly well on imagenet. arXiv preprint arXiv: /05.020005 2021.
[22] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthicu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2/05.03404, 2021.
[23] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Repmlp: Re-
parameterizing convolutions into fully-connected layers for image recognition. arXiv preprint arXiv:2105.0/883, 2021.
[24] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1rXiv:1606.0841. 2016.
[25] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov. and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[26] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, 2017.
[27] Noam Shazeer. Glu variants improve transformer. arXiv preprint urXiv:2002.05202, 2020.
[28] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019.
[29] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv: 1505.00380 2015.
[30] Jie Hu, Li Shen, and Gang Sun. Squeeze-ande and-excitation networks. In CVPR, 2018
[31] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[32] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
[33] Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. figh-pperformnce large- scale image recognition without normalization. arXiv preprint arXiv:2 2021.
[34] Mingxing Tan and Quoc V Le. Efficientnetv2: Smaller models and faster training. In ICML, 2021.
[35] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. In ICLR, 2021.
[36] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In CVPR, 2021.
[37] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:1101.11605. 2021.
[38] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15800 2021.
11

---

[39] Iija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. CVPR, 2020.
[40] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.001 2020.
[41] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-tass tallti-tskkaa benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[42] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.088 000361 2020.
[43] Adina Williams, Nikita Nangia, and Samuel R Bowman. broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2017.
[44] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
[45] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In ACI., 2018.
[46] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[47] Mingxing Tan, Ruoming Pang. and Quoe V Le. Efficientdet: Scalable and efficient object detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781-10790, 2020.
[48] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 12122, 2021.
[49] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-bast based object detection. arXiv preprint arXiv:2012.099 09958, 2020.
12


Reviews:
[{"review_id": "w9jB8orHGqr", "reviewer": "Reviewer_Upxb", "summary": "This paper proposes and studies an interesting research problem whether we can remove the self-attention in Transformer based models. The authors devise a novel architecture called gMLPs by removing the self-attention layer and introducing a new Spatial Gating Unit into the FFN module to well model the interactions between independent tokens. Comprehensive experiments on CV or NLP tasks confirmed that the proposed gMLPs can achieve comparable performances with the standard ViT or BERT models.    ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper, the whole paper is very clear and well-written, and I believe this paper is going to be useful for the community and inspire other researchers’ work on this topic. \n\nBelow are some questions or confusions I had: \n\n1. The proposed architecture are evaluated on encoder-style pre-trained models, e.g. ViT or BERT, will it be possible to extend the method to encoder-decoder or decoder-style models, like T5 or GPT?\n\n2. Transformer-based models tend to have better OOD generalization ability, the authors are suggested to add some OOD evaluation experiments as conducted in the paper “Pretrained Transformers Improve Out-of-Distribution Robustness”. \n\n3. In Tables 4 and 5, it is better to add inference time and flops as extra evaluation metrics. \n\n4. In Figure 4, it seems that the learned filters tend to have local correlations, will this limit the ability of gMLPs in modeling long range dependencies.  \n\n5. In appendix D the learned spatial projection matrices seems to be Toeplitz-like, can you quantitatively evaluate how these matrices be Toeplitz-like?\n\n6. Will the proposed gMLPs model be sensitive to the length of input sentence? The authors are suggested to add some analysis with considering the sequence length factor, e.g. accuracy vs length on downstream tasks.\n\n7. In practice, before deploying to products, pre-trained models often need to be compressed, I wonder whether the gMLPs can be compressed into some tiny models for IoT devices.      \n\n8. It is better to discuss some potential limitations of the proposed gMLPs. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I see no potential negative societal impact of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "three hours "}, "extra_text": {}}, {"review_id": "lyctd9-7YWq", "reviewer": "Reviewer_s4GZ", "summary": "This paper proposes a simple mechanism: gated MLP and questions the necessity of self attention.\ngMLP achieves comparable performance as DeiT on the classification task. Experiments on NLP talk also support the validness of this method. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. Experiments are designed across both vision classification and NLP tasks.\n2. Overall, the experiment results are good.  It achieves similar performance as DeiT on ImageNet 1k classificaition and NLP baseline. gMLP achieves better performance than the MLP baseline MLPmixer.\n3. The motivation is clear and good. However, the claim is too strong because of the weakness. \n\nWeakness:\n1. One big concern  is that the gMLP cannot scale to down stream tasks such as object detection, where different resolution is required during the training.  The spatial_gating_unit function in the pseudo code can only handle fixed length.\n I believe this is one fatal and fundamental drawback and would greatly limit the application of this paper. Note that self attention can handle it easily, and there are a lot of new work [1,2,3] before NeurIPS, which can easily handle this case and achieve impressive results on segmentation and object detection.   In contrast, the gMLP framework cannot deal with it directly.\nPlease correct me if I'm wrong. \n\n2. Some important factors are neglected and maybe leads to unfair comparisons. One important baseline is DeiT, however, many improvements [1,2,3,4,5,6,7] are proposed based on DeiT and further boost the performance. The performance gap between transformer and gMLP are enlarged if they are included. For example,  gMLP utilizes global average pooling GAP in classification, while DeiT doesn't. This will boost transformer about at least 1-1.5% accuracy on ImageNet validation dataset [4].  And fine-tuning the drop path rate (gMLP also do this) [2, 3, 7] can marginally improve the performance of transformer.  I believe proper discussions and comparisons with these work would benefit this paper and community.\n\nMinors:\n1. The gated title is a bit confusing since it doesn't behave like  commonly used meaning. \n2. Inference speed (imgs/s) is not reported (Table 2)\n\n\n\nReference\n\n[1] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions,\n https://arxiv.org/abs/2102.12122\n\n[2]Twins: Revisiting the Design of Spatial Attention in Vision Transformers, https://arxiv.org/abs/2104.13840\n\n[3] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows https://arxiv.org/abs/2103.14030\n\n[4] Conditional Positional Encodings for Vision Transformers, https://arxiv.org/abs/2102.10882\n\n[5]Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet https://arxiv.org/abs/2101.11986\n\n[6] Transformer in Transformer, https://arxiv.org/abs/2103.00112.\n\n[7] Going deeper with Image Transformers https://arxiv.org/abs/2103.17239\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See weakness 1 of the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "GIcnaXpw1_i", "reviewer": "Reviewer_GiR7", "summary": "This paper proposes a MLP-based model that achieves comparable performance with transformers for vision tasks and language modeling. While this MLP-based model performs slighly worse on natural language understanding tasks (e.g. MNLI), introducing a small single-head self-attention module can compenate for most of the performance loss. Compared to other MLP-based models introduced in concurrent works, the proposed model uses a gating module before the spatial projection. Performance wise, the proposed method is stronger than the concurrent MLP variant MLP-mixer on vision and language modeling tasks, showing the advantage of the proposed gating methods\n\nOverall I find the proposed method to be interesting and empirically strong. I am leaning toward accept but also wish the authors had done more work in understanding the proposed models. I offer some suggestions and have some questions below.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n\n- The empirical performance is strong\n- The proposed model is novel in the sense that it combines a gating mechanism with MLP-only model.\n- The submission is well-written and clear.\n\nCons/Questions\n\n- The submission very much focuses on the empirical performance of the proposed method. This is understandable but I also hope the authors had done more analysis work to understand the proposed methods. Here are some suggestions:\n    - Learning dynamics: Are the proposed method noticeably different from transformers in terms of convergence behavior?\n    - Data efficiency: Is there trade-off between transformers and gMLP in terms of data efficiency? For example, does gMLP perform better or worse relative to transformer as we add in more training data?\n- Can the authors document the computational advantage of gMLP? Is the proposed method faster to inference or does gMLP save GPU memory?\n- The fact that attention mechanism is especially important for cross-sentence understanding tasks is very interesting. I wonder if the authors can identify any patterns in the learned single head attention. Or, can the authors offer more discussion on how the attention mechanism is helping?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative societal impact from this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any ethical concern with this work.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DTJzFxlIUS", "reviewer": "Reviewer_5t8k", "summary": "This paper presents a simple architecture based on MLPs, which only adopts channel projections and spatial projections with the spatial gating units. On image classification, the presented gMLP could achieve on par performance with DeiT and ViT, and better than previous MLP-based approaches, such as MLP Mixer. On masked language modeling tasks, gMLP could also achieve on par performance with Transformer on Perplexity, and gMLP could perform slightly worse than Transformer on MNLI.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For originality, this paper presents a new MLP-based approach, e.g. gMLP. But compared to MLP-Mixer, the contribution of this gMLP seems marginal. Only the spatial gating unit is newly designed and verified to be effective. But the experimental study of this paper is sufficient and valuable to the community.\n\nThe quality of this paper is good. From an experimental report perspective, this paper contains sufficient experimental results, verifies important claims, and achieves relatively good performance on both ImageNet classification and masked language modeling in NLP, comparing to the previous approaches. The diversity of the experiments is sufficient.\n\nThe clarity of this paper is good. This paper is well-organized and easy to follow.\n\nMy major concern is about spatial projection. This is a simple design, but the application scenario is also limited due to the fixed size of the input spatial resolution. For example, for some vision downstream tasks of object detection and semantic segmentation, the input sizes are different from that of the image classification. This architecture could not be directly applied to these tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an MLP-based model with gating (gMLP) that achieves comparable performance to Transformers, showing that self-attention is not critical for the success of these models. While multiple concerns were raised regarding the limitations of the proposed method, all four reviewers appreciate the strong empirical results reported in the paper and recommend acceptance. The AC agrees with this decision, and requests the authors to add to the final version the discussion and additional information provided in the rebuttal, as well as more clearly describe the limitations of the proposed approach.
Decision: Accept (Poster)