Scaling MLPs: A Tale of Inductive Bias
Authors: Gregor Bachmann
Sotiris Anagnostidis
Thomas Hofmann
Venue: NeurIPS 2023 poster (2023)
Keywords: MLP
scaling-laws
inductive bias
DL theory

Abstract: In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical insights into MLPs are important for multiple reasons. (1) Given the recent narrative "less inductive bias is better", popularized due to transformers eclipsing convolutional models, it is natural to explore the limits of this hypothesis. To that end, MLPs offer an ideal test bed, as they lack any vision-specific inductive bias. (2) MLPs have almost exclusively been the main protagonist in the deep learning theory literature due to their mathematical simplicity, serving as a proxy to explain empirical phenomena observed for more complex architectures. Surprisingly, experimental datapoints for MLPs are very difficult to find in the literature, especially when coupled with large pre-training protocols. This discrepancy between practice and theory is worrying: \textit{Do MLPs reflect the empirical advances exhibited by practical models?} Or do theorists need to rethink the role of MLPs as a proxy? We provide insights into both these aspects.
We show that the performance of MLPs drastically improves with scale (95% on CIFAR10, 82% on CIFAR100, 58% on ImageNet ReaL), highlighting that lack of inductive bias can indeed be compensated. We observe that MLPs mimic the behaviour of their modern counterparts faithfully, with some components in the learning setting however exhibiting stronger or unexpected behaviours. Due to their inherent computational efficiency, large pre-training experiments become more accessible for academic researchers. All of our experiments were run on a single GPU.

Scaling MLPs: A Tale of Inductive Bias
Gregor Bachmann", Sotiris Anagnostidis', ETH Zürich, Switzerland
Thomas Hofmann
Abstract
In this work we revisit the most fundamental building block deep learning, the multi-layer perceptron (MLP), and study the limits of its performance vision tasks. Empirical insights into MLPs are important for multiple reasons. (1) Given the recent narrative "less inductive bias is better", popularized due transformers eclipsing convolutional models, it is natural to explore the limits of this hypoth- esis. To that end. MLPs offer ideal test bed, they lack any vision-specifie inductive bias. (2) MLPs have almost exclusively been the main protagonist in the deep learning theory literature due their mathematical simplicity, serving proxy to explain empirical phenomena observed for more complex architectures. Surprisingly, experimental datapoints for MLPs are very difficult find in the liter- ature, especially when coupled with large pre-training protocols. This discrepancy between practice and theory is worrying: Do MLPs reflect the empirical advances exhibited by practical models? Or do theorists need to rethink the role of MLPs as a proxy? We provide insights into both these aspects. We show that the performance of MLPs drastically improves with scale (95% on CIFARIO, 82% on CIFARION 58% on ImageNet ReaL), highlighting that lack inductive bias can indeed be compensated. We observe that MLPs mimic the behaviour of their modern counter- parts faithfully, with some components the learning setting however exhibiting stronger unexpected behaviours. Due to their inherent computational efficiency, large pre-training experiments become more accessible for academic researchers. All of our experiments were run single GPU,
262k
524k
1m
2. 1m
4.2m
12m
B. G-WI_256
B. 12-Wi 256
B. 6-WL_112
B. 10-12225500 512
B, 6_6-WL1100 1024
Linear Evalluation Error (%)
B_ 12-12111000 1024
50
35
30
25
0.0.16+0.5
100
Compute (PFLOPS)
Figure 1: Test error on CIFAR100 100 as function of PFLOPS.
"Equal contribution. Correspondence to (gregorb, sanagnos) @ ethz.ch. Code and checkpoints available
https:://ghht http///gtttttttt/// hwtp.///grggggrchcaa//eeeaaaaaae
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original
Convolutiona
Patchified
Flattened
Figure 2: Different architectures process images differently. Convolutions directly operate the image, ViTs and MLP-Mixes work with patches while the MLP takes the flattened image as input.
Introduction
Deep learning has undergone tremendous empirical progress in the last decades. The dominant approaches in practice these days rely on very large, pre-trained models which are then fine-tuned to the specific task at hand. For natural language processing. these models usually are some variant of the Transformer architecture (Vaswani et al., 2017), while in computer vision, both convolutional and transformer- based models are very popular (He et al., 2015; Tan and Le, 2020; Dosovitskiy et al., 2021). The theoretical understanding of these advances on the other hand remains very poor and the gap between the world of theory and practice is growing at an alarming rate. One aspect of this gap is the family of models investigated; due to their mathematical simplicity, theoretical works largely focus on simple multi-layer perceptrons (MLPs). Consisting of a series of unstructured matrix multiplications, interleaved with element- wise non mon-lnnariiies, the MLP serves an ideal test bed to analyze empirical phenomena exhibited by more complicated models employed in practice. Due to their inferior performance, MLPs are rarely used and very little is known regarding their behaviour in more modern settings. For instance, to the best of our knowledge, there is not a single published result showcasing an MLP trained on ImageNetlk, the de-facto standard benchmark in vision, let alone any pre- pee-training/transtte aims to understand the characteristics of modern architectures through the lens of MLPs, yet only little assessments are made regarding how well such a proxy works. This raises the question,
learning studies. This lack of empirical data is concerning as theory
Do MLPs reflect the empirical advances exhibited by practical models?
(1)
Investigating MLPs is not only interesting for theory but also for practice. With the Vision Transformer (ViT) outperforming its convolutional competitors in very large-scale settings, the role of inductive bias has recently been brought into question. Since a ViT is equipped with significantly less inductive bias for vision compared to convolutional models (e.g. it lacks cunslation-equivaiiann narrative has recently emerged:
novel
At large scales of compute, having less inductive bias is beneficial for performance.
(2)
More evidence for this hypothesis has been collected in the form of the MLP-Mixe (Tolstikhin
et al., 2021), architecture with arguably even less inductive bias, solely relying multi-layer perceptrons as patch processors and mixers. The MLP architecture is the ideal candidate to test the limits of such hypothesis, as it exhibits the least inductive bias for vision due to its invariance to permutations of pixels. Unfortunately. the scale where Transformers and MLP-Mixers start to outperform convolutional models is of reach for most researchers, requiring billions of annotated images and thousands of TPUs. We thus expect similar required scales for MLPs and hence instead investigate the following, weaker hypothesis:
Lack of inductive bias can be compensated by scaling compute.
(3)
iec we aim measure to what degree a lack of inductive bias hinders performance even if a model is subjected to a large parameter count and trained on datasets with many examples (albeit smaller than what is employed in Dosovitskiy al. (2021)).
In this work, we provide answers to question and provide further evidence for hypothesis 2 and by investigating how far we can push the empirical performance of models solely built from composing several MI.P blocks. We give largely positive answers to question 1, observing that ML.s behave very similarly to their modern counterparts when subjected to scale, i.c. their performance increases

---

MLP-Mixer
MLP
Patch Mixing
Patch Processing
MLP1
MLP
Patch 2
Pation 3
Patch 4
Figure 3: A simplified depiction of the differences between an MLP-Mixer and an MLP.
predictably as power law in parameter count and sample size, akin to Hestness et al. (2017, 2019); Kaplan et al. (2020); Zhai et al. (2022) (see e.g. Fig. 1). In contrast to previous work, however, we find that compute-optimal MLPs allocate their budget more strongly into sample size, highlighting again their small inductive bias. While regularization the form of data augmentation is also helpful for CNNs, its role is significantly amplified for MLPs degradation turned off. We further investigate how the implicit bias of SGD affects performance, and we make very counter-intuitive discovery: contrary to CNNs, larger batch sizes generalize significantly better for MLPs. This result questions the validity of the proxy role that the MLP plays in theoretical works investigating the implicit bias of SGD. While, expected, the scale employed in this work does not suffice for hypothesis 2, we provide strong evidence for 3, which we view as an important first step. We observe that scale indeed suffices overcome the bad inductive bias present in MLPs, leading to surprisingly strong downstream performance, e.g. R 95% on CIFARIO, z 82% on CIFAR 100 and x 58% ImageNet Real.. In summary, we make the following contributions:
at large sample sizes, leading to fatal
We fill the gap between theory and practice, providing the first results for MLPs trained in modern settings.
We show that MLPs mostly behave comparably to their modern counterparts, making them good proxy for theory. We observe however that the roles of regularization and implicit bias of SGD significantly differ and theory hence needs adapt.
We provide further evidence that inductive bias is not crucial large scales, showing that even "bad" architectures like MLPs can achieve strong downstream performance. We however identify shift in compute-optimaliin compute significantly more into dataset size compared to model size.
showing that optimal MLPs invest their
Background
Theoretical Works. The MLP has served as the main object of study for theoretical works in deep learning across different domains. The cornerstone results for areas such as convergence of SGD-trained neural networks (Mei et al., 2018; Du et al., 2019; Zou et al., 2020; Li and Yuan, 2017; Saxe al., 2014), most generalization bounds (Arora et al., 2019b: Mei and Montanari, 2021; Jacot et al., 2018; Allen-Zhu et al., 2019a), the benefits of overparametrization (Neyshabur et al., 2019; Allen-Zhu et al., 2019b; Arora al., 2018), the implicit bias of SGD towards favourable solutions (Soudry et al., 2018; 8; Neyshabur et al., 2014; Chizat and Bach, 2020), signal propagation properties (Poole et al., 2016; Schoenholz et al., 2017) and scaling laws (Bahri al., 2021; Maloney al., 2022) are all largely obtained for MLPs. To quote the very influential Principles of Deep Learning Theory book (Roberts al., 2022):
"MLPs are the simplest of these neural network architectures that hinge on this stacking idea, and thus provide minimal model for an effective theory of deep learning."
There are also several theoretical works studying more modern setups such convolutional or transformer- based networks including Arora et al. (2019a); Gunasekar et al. (2018); Brutzkus and Globerson (2017); Hron al. (2020) to our knowledge still remains on the MI. architecture. We thus believe it important to explore the limits of such theoretical proxy in realistic settings.
but few, but the main theoretical focus to the best of

---

MLPs. The multi-layer perceptron has its origins in Rosenblatt (1958), serving as an extension to the classic Perceptron with its hidden layers however fixed to random initialization. Ivakhnenko et al. (1965) devised the first method to update the hidden layers through relf-organiztion. Amari (1967) then introduced the idea to train the parameters with stochastic gradient descent. Mathematically, an MLP of depth I. E N can be described very efficiently; given an input æ E IR. it applies a series of linear transformations, interleaved with an element-wis wise non non-linearity a : RR + R:
2(1) = (1
(1) = a (2(1)
where we define x: (0) == x. and W E Rdxxd-1 for I,...., are the learnable weight matrices. For the sake of readability, attractive model to study theoretically (albeit still very far from trivial) and indeed many works frame their results around this general model class. When used for vision, the input tensor E hxx is flattened into a vector vec(z) E Rh Notice how such an architecture completely lacks locality and weight sharing, every unit simply processes the entire image at once. More worryingly, the vectorization vec could be applied in any way, i.e any permutation of x looks identical to an MLP.
omit the biases. This mathematical simplicity makes the MLP very
We want to highlight that MIPs of course are not completely free of inductive bias, in the sense that they encourage learning a hierarchical feature structure. On the other hand, there is no vision-specific inductive bias present in MLPs, which is the main setting we investigate here. We refer to Battaglia et al. (2018) for a more in-depth treatment of inductive bias.
Convolutions. The MLP is very general model and has no structure built into it to make it more suitable for vision tasks. A convolution on the other hand was designed specifically for vision with desirable characteristics incorporated into the model. A convolution can be viewed as special case of an MLP, where the weight matrix W is very structured by being sparse and having shared entries, leading to spatially localized learning. This can be most easily illustrated in the case of convolving 2 x 3 x 1 image & with a 2 x 2 filter f as the following matrix multiplication:
f2
0
f3
fA
0 fa vec(z))
f x Wf voc(z)
f1
f2
O
fs
Here vec denotes the standard, row-wise vectorization-s schhmme to flatten the image. Instead of operating with a dense matrix as the MLP, the convolution uses a structured matrix W, tailored to the task of vision, leading to better inductive bias. Moreover, a convolution exhibits translation- equivariance, i.e. shifts of images are processed equivalently to the original. Crucially, in contrast to the MLP, convolution severely suffers if permutation is applied to the image.
Vision Transformer. Inspired by successes in NLP, the Transformer architecture has been adapted to vision (Dosovitskiy et al., 2021). An image E Rxxxxx3 is broken up into patches (also called tokens) and linearly embedded (see Fig. 2), augmented with a positional embedding, marking its spatial location in the image. The obtained embeddings are processed by self-attention layers where patches interact, and MLP layers, which are shared among patches and transform them individually. While the inductive bias of a ViT is certainly weaker compared to CNN (it lacks translation- equivariance), the patching and parameter sharing still make the architecture suitable for vision.
MLP-Mixer. Similar to the ViT, the MLP-Mixer also works with a patchified image (Tolstikhin et al., 2021). Unlike the ViT, token-mixing is not implemented using self-attention but rather another MLP block is used to exchange information between patches. We want to clearly highlight the difference between an MLP-Mixer and MLP: An MLP-Mixer operates on patches, where in each block it applies a shared MLP to each patch for processing, and another MLP for mixing the patches along the channels. We visualize the differences in Fig. 3 for clarity. We again want to stress that breaking the image into patches and sharing parameters among them significantly enhances the amount of inductive bias, compared
standard MLP.
Patchifiying. As highlighted above, VITs and Mixers largely obtain their inductive biases through breaking the images into patches. This choice seems already possess a strong inductive bias, such as the ConvMixer (Trockman and Kolter, 2022), where convolutions are performed on individual patches. The very recent Metaformer (Yu et al., 2022) further shows that even simple spatial pooling instead of attention can lead to strong performance if the image is patchified. While the success of this mechanism certainly warrants further investigation, in this work we decided to deliberately focus on MLPs as they specifically lack this type of bias.
beneficial even for architectures that

---

CIFARIO
CIFARIO0
TINYIMAGENET
IMAGENET
S-MLP (@100 E)
54.2
28.8
8.5
9.2
S-MLP- + DA 1000 E)
68.9
43.3
25.2
24.3
S-MLP DA 5000 E)
72.3
44.5
27.3
26.8
B-MLP @ 100 E)
58.1
30.5
8.9
8.7
B-MLP + DA (@ 1000 E)
70.1
48.3
27.2
28.7
B-MLP + DA (@5000 E)
75.4
50.4
31.2
31.7
RESNETI8 DA
93.2
75.6
68.9
69.7
Table 1: Test accuracies (in %) without any pre-training. The S-MLP has depth and width 1024 while the B-MLP M. has depth 6, width 1024 and expansion factor of 4.
3 Architecture
We study different variants of the MLP architecture, starting from the standard vanilla setup and then adding more components such as residual connections and bottleneck layers.
Standard MLP. As first starting point, investigate simple MLPs with ReLU activations and isotropic design, i.e. except for the first, every layer has the same width m E N. In order to avoid training instabilities further enhance the standard MI.P with layer normalizations (Ba al., 2016) placed after the activations. We thus compose several blocks of the form
Block(z) (W WWN)))
with W Rmxm To embed the image Rx we a linear layer emb(x: Wemb vec(æ) with Wemb E mm Such an embedding layer is crucial since for high resolution images, 3d² can be quite large and thus m needs to be chosen smaller. We empirically find that such network design the minimal choice order to guarantee successful training across all scales of parameter count and sample size. We will use the short cut S-MLP to denote such an architecture.
Inverted Bottleneck MLP. Inspired by Lin al. (2015); Tolstikhin et al. (2021) we add a bottleneck structure to an MLP block as well as skip connections as follows:
Block(z) z + 8 (W LN
where W E I km expands the dimension to km for back to width m. For most experiments we set k = 4. While the additions of skip connections and bottleneck layers to the architecture arguably add some amount of inductive bias, believe that in comparison to modern architectures such enhancements remain negligible. We will denote this variant by B-MLP.
and (c E gm collapses
Experiments
Setup this work, we solely focus on vision tasks as inductive bias is more readily understood in this setting. Moreover, most theoretical works focus image classification tasks, making it thus natural test bed to assess the performance of MIPs. We study the popular tasks CIFARIO, CIFAR 100 (Krizhevsky, 2009), STL10 (Coates al., 2011). TinylmageNet (Le and Yang. 2015), ImageNetlk for evaluation, as well as ImageNet21k (Deng et al., 2009) for pre-training. To limit the size of the embedding layer and the computational needs, we downscale images resolution 64 64 x (if needed) as done in Chrabaszez et al. (2017). We center and normalize all the images and use random flips and crops well as MixUp (Zhang al., 2018) as data augmentations.
In contrast the MLPs, the ResNet18 was trained at the original image resolutions.

---

CIFARIO
CIFARI00
STL
TINY-IN
IN
REAL
B-6/Wi-1024
69.9+0.1
43.0+0.4
51.540.1
47.140.1
15.2+0.2
20.3++.2
B-6/Wi-1024 + DA
91.5+0.02
76.4+0.2
85.0+0.2
02.7
38.7
47 17.0=2.15
B-12/Wi-1024 DA
94.2+0.05
80.0=0.05
89.9
69.910.4
43.340.06
48.6#0.2
B-12/Wi-1024 DA + TTA
95.510.05
82. 0...
92.2+0.05
73.1
51.5t0.1
57.940.1
Table 2: Fine-tuning Top-1 accuracies (in %) when pretrained on ImageNet21k. Accuracies are averaged over 3 runs. For readability, we abbreviate ImageNet as IN.
4.1 Training from Scratch
We start the empirical exploration of MLPs by training them from scratch (i.e. without any extra data) on popular vision benchmarks. All models were trained with the LION optimizer (Chen et al., 2023) with learning rate 1) = 5e-5. In order to combat overfitting we use strong label smoothing We display the resulting test accuracies in Table 1. We observe that both the standard architecture and the bottleneck without any data augmentation suffer from overfitting, leading suboptimal performance. When turning it on, data augmentation as regularizer however really unfolds its full power, significantly pushing the performance by roughly 20% across all tasks. As observed in Lin et al. (2015), the inverted bottleneck architecture leads to an improvement in performance across all datasets. Learning on the other hand significantly slows down with strong augmentations such as MixUp, enabling training for up to 5000 epochs without suffering from overfitting. However, compared simple modern baselines such as a 8 (He et al., 2015), large discrepancy in performance remains, highlighting the importance of inductive bias in the small sample regime. We remark that ViTs and MI.P-Mixee as well exhibit more learning difficulties the dataset size is small (Dosovitskiy al., 2021; Tolstikhin et al., 2021). We provide more ablation studies in Appendix A.2.
0.3.
4.2 Transfer Learning
In this section, we aim to analyze how transferable features learnt by MLPs are across different vision tasks. Transferability is one of the hallmark characteristics of modern deep learning, enabling practitioners to fine-tune large models on their specific dataset, leading to superior performance. We are, to the best of our knowledge, the first to measure transferability of MI.Ps, which crucial to assess in order to build theoretical understanding of the process. In this section, we focus on the inverted bottleneck MIP as it generalizes better and is easier to optimize. We provide the dual results for the standard MLP in Appendix B. We restrict to k for the expansion factor and denote by B-L/Wi-m& a network with L blocks and width m. For pre-training we use ImageNet2 Ik, the largest publicly available image dataset with annotated classes. After preprocessing the dataset following Ridnik et al. (2021), it consists of roughly 12 million images and 11 thousand classes. We then pre-train the MLP with the cross-entropy loss for 800 epochs, employing label smoothing and the LION optimizer. To guarantee fast data loading we rely on the FFCV framework (Leclere et al., 2023) for all experiments.
In order to measure transferability of the learnt features we fine-tune the network on the new task. We also study training linear layer on top of the embeddings but defer those results to Appendix A.3. We again explore the effects of data augmentation during the pre-training stage. For fine-tuning we use SGD with momentum with a learning rate of Thead = 0.01 for the head and Toody = 0.001 for the encoder for 50 epochs. We upscale CIFAR images to resolution 64 x 64 x 3 at fine-tuning time to guarantee compatibility. We display the fine-tuning results in Table 2. For visualizations of the learnt features, we refer the interested reader to Appendix C. We again observe that using data augmentation during the pre-training phase is essential to successful training, boosting performance up to 30% in case CIFAR 100. Surprisingly, the learnt features are highly transferable, improving the performances reported previously in Table 1 dramatically. While of course pre- trained large quantity of data, we nevertheless want to highlight that such an MLP becomes competitive with R8 trained from scratch for all the datasets, except for ImageNet1k where performance falls surprisingly short. We hypothesize that MLPs struggle with the more fine-grained distinctions between classes, in combination with the reduced resolution of the images.

---

Epoch 25
Epoch
Model
# parameters
100
200
B B-6/Wi-256
9M
B- B-12/Wi-256
12M
B-6/Wi-512
24M
B-12/Wi-512
37M
B-6/Wi-
74M
05
B- B-112Wi-1024
124M
204n
R1R7
16314
Batch Stos
Figure 4: Linear downstream error on CIFARI00 (in %) when pretrained for varying batch-sizes on ImageNet2 Ik, on log-log scale.
Table 3: The different models and the respective parameter counts in millions.
Test- Time-Timm-a Augmentations. For ImageNetlk we further notice that objects tend to not be centered, in contrast to datasets like CIFARIO. We suspect that this might lead to the comparatively weaker performance. To test this, we leverage test-time augmentations (TTA). As introduced by Krizhevsky et al. (2012), for each test image, produce fixed number of 100 random crops and use the averaged logits for prediction. We observe significant improvements across all datasets, especially for ImageNet obtain increase of roughly 8%. This indeed indicates that ML.s struggle localize the object of interest, especially for the more complicated ImageNet task. Using a large number of crops alleviates this problem some degree. This also explains why the gains tasks like CIFAR10 are smaller as the objects there usually are perfectly centered.
ReaL accuary. As observed in (Beyer et al., 2020), the ImageNet labels do not capture that a single image might contain multiple objects of distinct classes. ImageNet accuracy thus be misleading in the sense that model classes such as convolutional networks might have implicitly adapted to the particular labeling strategy due to the repeated benchmarking likely lack such an implicit adaptation as this work is to our knowledge the first to evaluate them on ImageNet k. To address this, Beyer et al. (2020) introduced a novel set of validation labels that better capture the multi-label nature, where prediction is deemed correct if it matches one of the categories present in the image. We observe further very significant improvements of z 7% when employing ImageNet ReaL.
the same validation
MLPs most
Overall, these results underline that bad inductive bias as exhibited by MLP can indeed be overcome if subjected to enough scale. For theory, the results are double-edged; while MLPs prove to be a good proxy to understand transfer learning, data augmentation proves be crucial component. Also test-time augmentations significantly boost performance. Both these components on the other hand remain rather understudied in theoretical works.
Large batch-sizes. We further make the counter-intuitive observation that training with larger batch sizes significantly boosts performance both up- and downstream. In Fig. 4 we plot pre-training batch size against resulting linear downstream accuracy on CIFAR100 for different number of pre- pre-training epochs. We observe that across all training times, using a larger batch size leads to significantly better performance. Moreover, we want to highlight that such a plot is even favoring small batch-sizes since those models perform more gradient updates for fixed number of epochs. This effect is in stark contrast to convolutional architectures where entire lines of works have focused on preserving the performance of the small batch-size regime for larger ones (Goyal et al., 2017; You et al., 2017: Hoffer et al., 2017; Keskar et al., 2017). Training with large batch-sizes without degradation of high interest as it can lead to potentially more efficient training pipelines since computation can be sharded among more devices. This observation about optimal batch-sizes is in-line with similar recent conclusions in Transformers (Kaplan et al., 2020; Touvron et al., 2023).
Role of augmentations. Data augmentation is very pronounced for MLPs, providing indirect inductive bias to the model. Remarkably, model pre-trained on 12 million examples without data augmentation shows inferior performance CIFARI0 compared to network trained from scratch
7

---

with augmentations turned on. This emphasizes that augmentations go beyond merely leading to bigger dataset but provide the model with useful invariances. We investigate the learnt weights in Appendix C, showing very evidently, that more localized features are learnt if data augmentation employed. The power of augmentations has already been demonstrated through the advent of self-super learning (Grill et al., 2020; Caron et al., 2021; Chen et al., 2020). Even when training on purely random labels, it still provides powerful learning signals (Anagnostidis et al., 2023).
4.3 Scaling Laws
One of the key mysteries in deep learning is that networks tend improve in terms of generalization when compute, in the form of parameter count and dataset size, is scaled up. Recently it has been observed in several works that the benefits of scale are highly predictable, i.e. generalization performance cxhibits power-law structure when plotted against compute measured in FLOPS (Rosenfeld et al., 2020; Hestness al., 2017, 2019; Kaplan et al., 2020; Zhai et al., 2022). The functional form has recently been further refined (Caballero et al., 2023). The predictable nature of test performance has even been leveraged to estimate the optimal model before training (Hoffmann et al., 2022; OpenAl, 2023). In order to understand this important characteristic of deep learning theoretically, it important to analyze whether MLPs exhibit similar properties.
260k
524K
12n
B_0-M12000
8 M1 250
B. 12-M1 298
B_ 1.-224M, 208
U,6-W1 312
D. 9.6-M 512
12-41 55112
R. 1.12244, 512
BL5 1024
6,6 M1 1024
B, 12-M1 1024
12-12-W 1024
Lnbar Swtluder Ere
Urwer Evalaaten
20
70
E..0.0 + 0.41C-0.29
E-0.29 56IC 1.3)
100
500
Compute (PFLOPS)
Figure 5:
Test error (in %) on CIFARIO (left) and ImageNetlk (right) when linearly transferred as a function of PFLOPS, measured according to (4), on a log-log scale.
Compute. Following OpenAl (2018) we define the computational cost C' incurred from training a model f on N examples for T epochs
C FLOP(F) x 3 XT,
(4)
where FLOP(F) denotes the number of FLOPs needed to complete the forward pass of f for a single example. We note that the number of parameters P present in f enters this equation implicitly in the form of FLOP & P. Observe that a given level of compute can be achieved in different ways, i.e. using more parameters P. training on more examples N, or training for longer time T. When allocating given level of compute optimally, it is observed that for convolutional and transformer-hase architectures, the test error FC(C) as a function of compute behaves as a power-law
E(C) = a(b + C) + Eso:
(5)
where a, b, Eoo E R4, and the irreducible error, i.c. even if infinite compute were employed, the performance remains imperfect. The test error can be measured upstream (i.e. on the pre-training task) or downstream when fine-tuning on different task. We investigate various pre-training schemes with different number of examples, parameter counts and training times. We subsample ImageNet21k proportionally across classes and pre-train variously sized inverted bottleneck MLPs. We summarize the configurations in Table 3. We then measure test error on the downstream task of CIFAR 100 Fig. as well as CIFARI and Imag in Fig. 5 by linearly transferring the learnt features (without test-time augmentations). The plotting style inspired by Zhai et al. (2022). Each point in the curve is the downstream performance of MLP, where the color of the point indicates the model type (blue denotes smaller
is the scaling coefficient determining the rate of decay. Ex denotes
8

---

12 2008
9_5-H,, 512
9, 12-1, 5112
5-40 1004
B_12-W, 1024
Unar Enakutattion Entw
Eussseto Froor
05
202144
E-023 2.00P+
E-000 127)
824250
1045570
30
2007162
4194506
11901660
25M
# Panmeter (P)
Sarrppee
Figure 6: Power law in linear evaluation error CIFAR100 (in %) when either bottlenecked by the number of parameters (left) or the number of examples (right), on log-log scale. The dotted line visualizes the fitted functional form.
and red larger models) and the size of the point indicates the number of pre-training examples. Points connected by a line indicates longer training times where T E {50, 100, 200, 400, 800} is measured employ data augmentation for pre-training. We observe that
in epochs. In all experiments, the compute-optimal performance of MLPs strongly exhibits characteristics of a power-law with coefficients {0.12, 0.25, 0.35). This is very encouraging for future theoretical work, showing that MIPs indeed mirror the scaling behaviour of modern models. We provide the dual results for the standard MLPs in Appendix B.2, noting that they exhibit essentially the same scaling behaviour, albeit with a slightly weaker slope and intercept.
We further study how performance E' evolves when compute is either bottlenecked by the number of parameters or the dataset size N. We visualize the resulting scaling laws in Fig. 6. We find a very steep decay rate in terms of parameters P where roughly ap whereas for dataset size N we identify a significantly slower rate of aN ze 0.35. This shows that the performance of MLPs is significantly
limited by the dataset size, which in-line with the fact that MIPs exhibit bad inductive bias. We investigate the role of dataset size and parameters more in the next paragraph.
Parameters or examples. Given a fixed level of compute C, what is the optimal way to allocate to parameter count P and number of examples N? To be more comparable to previous work, assume a fixed training time = 50. To answer this question, we follow the approach outlined in Hoffmann al. (2022) and plot the optimal compute models identified Fig. both against model size P and number of examples N and visualize the results in Fig. 7. We empirically observe that the optimal parameter count p* (C) and dataset size (C) as a function of compute C' exhibit lawer-law behaviour of the approximate form
(C(C) CO.35 (C x C00.6
While for transformers, the number of examples (or tokens) N and parameters P are scaled equally (Hoffmann et al., 2022) (i.e. op s o Rs 0.5), in contrast we observe that the optimal strategy for MLPs invests significantly more compute into dataset size N. This is further evidence for the weaker inductive bias present MLPs, which needs more examples in order to be compensated for.
4.4
Computational Feasibility
We believe that a further exciting feature of our study is its computational feasibility, while at the same time preserving the main characteristics of large-scale pre-training. All of our experiments were conducted on single NVIDIA RTX A5000 GPU with 24GB of memory. In conjunction with the strongly optimized FFCV dataloading framework (Leclere et al., 2023) and the inherent efficiency of MLPs, we are able to perform very rapid training. For instance we complete single epoch on ImageNet21k with the B-/2/Wi- 1024 architecture, equipped with 124 million parameters, only roughly 450 seconds, while the smaller variant B-6/Wi-1024 at parameter count of 74 million requires roughly 250 seconds on the specified hardware. Low memory requirements allow us to train with batch-size of 16384 without having to shard computation among multiple GPUs. We

---

Exampies (Milions))
D 74
10 Compute (PFI OPs)
Compune (PFLOPe)
Figure 7: Optimal model size (left) and number of examples (right) for a given level of compute for linear evaluation on CIFAR100, on a log-log scale.
compare the computational efficiency of MLPs with contemporary networks of similar size such as ResNet-152, ViT-B/4 and VIT-B/8 in Appendix A.5.
Related Works
There are some prior works that investigate MLPs on vision tasks. Lin et al. (2015) study the performance of MLPs on small scale datasets such as CIFARIO. They observe similar improvements when using inverted bottleneck layers but do not study surger-scale setups, transfer-leaming nor do they discuss the implications for theoretical works. The bottleneck structure used in this work also been investigated theoretically (Parhi and Nowak, 2021; Shenouda et al., 2023; Parkinson et al., 2023), further highlighting that such an architecture exhibits desirable properties. Urban et al. (2017) study to what degree convolutions are necessary for good performance and conclude that even with distillation techniques it remains very difficult train performant MI.s on CIFAR10. Other approaches have focused on sparsifying fully-connected layers through evolutionary training (Mocanu et al., 2018; Fernando et al., 2016), aiming to learn a good inductive bias from scratch. Similarly, Neyshabur (2020) study how the inductive bias of MLPs can be improved by systematically sparsifying them with a LASSO-type algorithm, making them more convolution-like d' d'Ascoli et al. (2019) on the other hand first train convolutional network for certain duration and then subsequently continue training the network as an MLP (by using the correspondence between CNNs and MLPs highlighted in Sec. 2). They show that good performance can be reached if the network was trained long enough as a CNN. In contrast to these works, our goal is not to enhance the inherent inductive bias ML.s but study whether it can be overcome with enough scale.
The advent the MLP-Mixer (Tolstikhin et al., 2021) has led to a series of follow-up work, similarly using MLPs as a patch processor and token mixer (Touvron et al., 2021; Chen et al., 2022; Lian et al., 2022; Guo et al., 2021; Liu et al., 2021). Again, we remark that these architectures all possess significantly more inductive bias.
Finally, we would like to remark that MLPs are successfully used in other areas such as novel view synthesis (e.g. NeRF (Mildenhall et al., 2021)).
Discussion
In this work, we have explored the limits of the multi-layer perceptron as an architecture for vision tasks. Our study reveals that (1) lack of inductive bias can be compensated by scale and (2) MLPs constitute a (largely) accurate proxy for modern architectures, further cementing their role as the main theoretical object study. The role of data augmentation and the implicit bias of SGD however strongly differ for MLPs in the setting considered in this work and theoretical works should take this into account. Large-scall scale pre-training of MLPs proves be very efficient, enabling researchers with less access to computational resources to study this very exciting line of work. While lack of inductive bias does not prevent MLPs from reaching impressive performance, it leads to an interesting shift in compute-optimality towards more training examples. Subjecting MLPs to even larger amounts of compute similar Zhai al. (2022), especially in the form of more training examples, remains as very interesting future work.
10

---

References
Allen-Zhu, Z., Li, Y., and Liang, Y. (2019a). Learning and generalization in overparameterized neural networks, going beyond two layers. Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F,, Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Allen-Zhu, Z., Li, Y., and Song. Z. (2019b). A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pages 242-252 252. PMLR.
Amari, S. (1967). A theory of adaptive pattern classifiers. IEEE Transactions Electronic Computers, EC-1663):9993307.
Anagnostidis, S., Bachmann, G., Noci, L., and Hofmann, T. (2023). The curious case of benign memorization. In The Eleventh International Conference on Learning Representations.
Arora, S., Cohen, N., and Hazan, E. (2018). On the optimization of deep networks: Implicit acceleration by overparameterization.. In International Conference on Machine Learning, pages 244-253. PMLR.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang. R. (2019a). On exact computation with infinitely wide neural net. Neural Information Processing Systems.
Arora, S., Du, S. S., Hu, W., Li, Z., and Wang, R. (2019b). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Learning.
Ba, I.,, Kiros, R., and Hinton, (2016). Layer normalization.
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. (2021). Explaining neural scaling laws.
Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer, J., Dahl, G., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D., Kohli, P., Botvinick, M., Vinyals, O., Li, Y., and Pascanu, R. (2018). Relational inductive biases, deep learning, and graph networks.
Beyer, L.. Hénaff, O. J., Kolesnikov, A., Zhai, X., and van den Oord, A. (2020), Are we done with imagenet?
Brutzkus, A. and Globerson, (2017). Globally optimal gradient descent for a ConvNet with Gaussian inputs. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 605-614. PMLR.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. (2023). Broken neural scaling laws. In The Eleventh International Conference Learning Representations.
Caron, M., Touvron, H., Misra, I., J'gg, H., Mairal, J., Bojanowski. P., and Joulin, A. (2021). Emerg- ing properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640.
Chen, S., Xie, E., GE, C., Chen, R., Liang, D., and Luo, P. (2022). CycleMLP: A MLP-like architecture for dense prediction. In International Conference on Learning Representutions.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020), simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. (2023). Symbolic discovery of optimization algorithms.
11

---

Chizat, L. and Bach, F. (2020). Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Abernethy, and Agarwal, S., editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 1305-1338. PMIR.
Chrabaszez, P., Loshchilov, I., and Hutter, F. (2017). A downsampled variant of imagenet as an alternative to the cifar datasets.
Coates, A., Ng. A., and Lee, H. (2011). An analysis of single-layer networks in unsupervised feature learning. In Gordon, G., Dunson, D., and Dudík, M., editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statisties, volume 15 Proceedings of Machine Learning Research, pages 1--22,, Fort Lauderdale, FL, USA. PMLR.
Deng, J., Dong, W. Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
A'soolli S., Sagun, I.,, Biroli, G., and Bruna, J. (2019). Finding the needle in the haystack with convolutions: on the benefits of architectural bias. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Du, S. S., Zhai, X., Poczos, B., and Singh, A. (2019). Gradient descent provably optimizes over- parameterized neural networks. International Conference on Learning Representations.
Fernando, C., Banarse, D., Reynolds, M., Besse, F., Pfau, D., Jaderberg, M., Lanctot, M., and Wierstra, D. (2016). Convolution by evolution: Differentiable pattern producing networks. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO '16 page 109-116, New York, NY, USA. Association for Computing Machinery.
Goyal, P., Dollár, Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). Accurate, large minibatch sgd: Training imagenet hour.
Grill, J.-B.. Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires,
B., Guo, Z.., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. (2020). Bootstrap your own latent new approach self-supervised learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages
Curran Associates, Inc.
Gunasekar, S., Lee, D., Soudry, D., and Srebro, N. (2018). Implicit bias of gradient descent on linear convolutional networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., cisac,,, N., and Gamett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Guo, J., Tang, Y., Han, K., Chen, X., Wu, H., Xu, C., Xu, C., and Wang. Y. (2021). Hire-mlp: Vision mlp via hierarchical rearrangement. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 816-826.
He, K., Zhang, X., Ren. S., and Sun, J. (2015). Deep residual learning for image recognition. 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Hestness, J., Ardalani, N., and Diamos, G. (2019). Beyond human-level accuracy: Computational challenges in deep learning.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017), Deep learning scaling is predictable, empirically.
12

---

Hoffer, E.. Hubara, I., and Soudry, D. (2017). Train longer, generalize better: closing the general- ization gap in large batch training of neural networks. Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Hoffmann, J., Borgeaud, S., MeMscM, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbi, J., Clark. A., Henmigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.. (2022). Training compute-optimal large language models.
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, (2020). Infinite attention: NNGP and NTK for deep attention networks. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 19 of Proceedings of Machine Learning Research, pages 4376-4386. PMLR.
Ivakhnenko, A., Lapa, V., and ENGINEERING. P. U. L. S. Devices. JPRS 37, 803. Joint Publications Research Service [available from the Clearinghouse for Federal Scientific and Technical Information].
S. O. (1965). Cybernetic Predicting
Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Dessa-Binchhi N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2017). On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations.
Krizhevsky, A. (2009). Learning multiple layers of features from tiny images.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolu- tional neural networks. In Pereira, F., Burges, C., Bottou, I.,, and Weinberger, K., editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc.
Le, Y. and Yang, X. S. (2015). Tiny imagenet visual recognition challenge.
Leclere, G., Ilyas, A., Engstrom, L., Park, S. M., Salman, H., and Madry, A. (2023). FFCV: Accelerating training by removing data bottlenecks.
Li, Y. and Yuan, Y. (2017). Convergence analysis two-layer neural networks with relu activation. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Lian, D., Yu, Z.. Sun, X., and Gao, (2022). AS-MLP: An axial shifted MLP architecture for vision. In International Conference on Learning Representations.
Lin, Z.,, Memisevic, R., and Konda, K. R. (2015). How far can go without convolution: Improving fully-connected networks. ArXiv, 0ax//551100255
Liu, H., Dai, Z., So, D., and Le, Q. V. (2021). Pay attention to MLPs. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems.
Liu, Z., Mao, H., Wu, C.-Y,, Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986.
Maloney, A., Roberts, D. A., and Sully, J. (2022). solvable model of neural scaling laws.
Mei, and Montanari, A. (2021). The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75.
13

---

Mei, S., Montanari, A., and Nguyen, P.-M. (2018), A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 11563))
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng. R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99-106.
Mocanu, D., Mocanu, E., Stone, P., Nguyen, P., Gibescu, M., and Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9.
Neyshabur, B. (2020). Towards learning convolutions from scratch. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 8078-808 8088. Curran Associates, Inc.
Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., and Srebro, N. (2019). The role of over- parametrization in generalization of neural networks. In International Conference on Learning Representations.
Neyshabur, B., Tomioka, R., and Srebro, N. (2014). In search of the real inductive bias: On the role of implicit regularization in deep learning.
OpenAl (2018). Ai and compute.
OpenAl (2023). Gpt-4 technical report.
Parhi, R. and Nowak, R. D. (2021). What kinds of functions deep neural networks learn? insights from variational spline theory. S.AM Math. Data Sci., 4:464-489.
Parkinson, S.. Ongie, G., and Willett, R. (2023). Linear neural network layers promote learning single- and multiple-index models.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., Vito. Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32,
pages 8024 8035. Curran Associates, Inc.
Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2016). Exponential expressivity in deep neural networks through transient chaos. In Lee, D., Sugiyama. M., Luxburg, U., Guyon, and Garnett, R., editors, Advances in Neural Information P Processing Systems, volume 29. Curran Associates, Inc.
Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik, L. (2021). Imagenet-21k pretraining for the masses. In Vanschoren, J. and Yeung, S., editors, Proceedings the Neural Information Processing Systems Truck on Datasets and Benchmarks, volume 1. Curran.
Roberts, D. A., Yaida, S., and Hanin, B. (2022). The Principles of Deep Learning Theory: An Effectivve Theory Approach to Understanding Neural Networks. Cambridge University Press.
Rosenblatt, E. (1958). The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2020). A constructive prediction of the generalization error across scales. International Conference on Learning Representations.
Saxe, A. M., McClelland, J. L.,, and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.
Schoenholz, S. S., Gilmer, J., Ganguli. S., and Sohl-Dickstein, J. (2017). Deep information propaga- tion. In International Conference on Learning Representations.
Shenouda, J., Parhi, R., Lee, K., and Nowak, (2023). Vector-valued variation spaces and width bounds for dnns: Insights on weight decay regularization. ArXiv, abs/2305.16534.
14

---

Soudry, D., Hoffer, E., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. In International Conference on Learning Representations.
Tan, M. and Le, Q. V. (2020). Efficientnet: Rethinking model scaling for convolutional neural
networks.
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L.,, Zhai, X., Unterthiner, T., Yung, J., Steiner, A. P., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. (2021). MLP-mixer: An all-MLP architecture for vision. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems.
Touvron, H., Bojanowski, P., Caron, M., Cord, M.. El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., and Jegou, H. (2021). ResMLP: Feedforward networks for image classification with data- eata-effficien training.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
Trockman, A. and Kolter, J. Z. (2022). Patches are all you need?
Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Mohamed, A., Philipose, M., Richardson, M., and Caruana, R. (2017). Do deep convolutional nets really need to be deep and convolutional? In International Conference on Learning Representations.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio. S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, I., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M.. Ribeiro, A. H.. Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272.
You, Y., Gitman, I., and Ginsburg. B. (2017). Large batch training of convolutional networks.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. (2022). Metaformer is actually what you need for vision.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022). Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12104-12113.
Zhang, H., Cisse, M., Dauphin, Y. and Lopez-Paz. D. (2018). mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Zou, D., Cao, Y., Zhou, D., and Gu, Q. (2020). Gradient descent optimizes over-parameterized deep relu networks. Machine Learning, 109:1-26.
15

---

Appendix
A Experimental Details
A.1 Resources
For all experiments we rely on NVIDIA RTX A5000 GPU with 24GB of memory. Every experiment be performed single GPU. We leverage the FFCV dataloader framework since the transfer time of the data to the GPÚ becomes the bottleneck in terms of training time case of MLPs. All of experiments were performed in PyTorch (Paszke et al., 2019).
A.2 Additional Ablations
Ablations. We provide some more ablations Fig. More specifically, for (approximate) fixed budget of compute, we investigate different architecture and optimization choices, when pretraining on ImageNetlk and performing linear probing on CIFARI00.
Impal image
Flatlen
Logits projection
018 CZU
8248 6BU
$3.22
23.05
Ardiieetio
Wy ge a MTT
W, ge
W. Reti AT
82x8,9, CEU
82xi JR, AN, ARR
57.1%
X lnssar ligers
JAOT
No AUG
34:13
X lnnsst tejert
M0
50,50
RobeYY: (AoT
Wesee We gpirt ACT
pat's splco
tnvenedd Bollawek
Adam/
DIART0 Downstraar Acouracy
WD No weight D'CCPOUT Adaetii Lien with Acteri
Figure 8: Ablations on different architectures and optimizations choices when training on ImageNet. Numbers indicate linear probing Top- accuracies on CIFAR100.
Normalization. We investigate the importance of the normalization method (LayerNorm Batch- Norm) in more detail in Table 4. We pre-train two B-MLPs on ImageNet21k with layer normalization and batch normalization and compare the fine-tuning performance on various tasks. We find that the techniques perform similarly, which layer normalisation having slight edge.
CIFAR-10
CIFAR-100
TINYIMAGENET
IMAGENET
LAYERNORM
90.0
74.6
39.6
36.2
BATCHNORM
89.4
73.8
57.7
35.9
Table 4: Pretraining a B-6/Wi-1024 B-MLP with BatchNorm and LayerNorm on ImageNer21k and subsequently fine-tuning.
Label smoothing. We further ablate the influence of label smoothing on the downstream perfor- mance. We pre-train B-MLPs with varying amounts of label smoothing (a E (0.0, 0. 0.3)) and evaluate the resulting down-stream fine-tuning performance. We report the results in Table 5. While label smoothing does provide some boost in performance, gains are very modest. Label smoothing is thus helpful but not essential for training MLPs.
16

---

CIFARIO
CIFARI00
TINYIMAGENET
IMAGENET
0.3
90.0
74.6
59.6
36.2
a 0.1
89.5
73.7
58.2
36.0
a = 0.0
89.2
72.2
57.1
35.7
Table 5: Pretraining a B-6/Wii-122 B-MLP with different amounts of label smoothing on Ima- geNet21k and subsequently fine-tuning.
Architecture. We make the following observatoons/reomneeddiiioo to boost the model's per- formance, in line with results reported in the literature (Liu et al., 2022); (1) replacing ReLUs and GELUs boosts results significantly, (2) adding skip connections every layers helps with optimization, especially for deeper networks. (3) Using an inverted bottleneck increases performance even more. (4) Using normalization layer in the PRE-LN configuration helps with optimization and (4) layer normalization leads to significantly better results compared to batch normalization, while also being more stable during training.
Optimization. As discussed in the main text, augmentations are crucial, and disabling them can have a detrimental effect. We also found that clipping gradients, using weight decay and dropout have a small positive effect on downstream performance. Finally, replacing LION (Chen al., 2023) with Adam(W), leads to a decrease in performance.
A.3 Linear Probing
We showcase the transferability of our MIPs by training linear classifier top of the frozen features. For training the linear layer, we use the LION optimizer with learning rate of r = 0.00001 for 50 epochs. We display the results in Table 6. We observe very strong down-stream performance
CIFARIO
CIFARI00
STLI0
TINYIMAGENET
IMAGENET
B-6/Wi-1024
65.1
41.3
53.4
45.6
13.0
1-6/Wi-1024 DA
87.8
73.2
85.2
61.3
39.2
B- 1-122Wi-104 DA
90.6
74.5
88.3
68.5
40.7
Table 6: Linear probing Top-1 accuracies when pretraining on ImageNet2
even this more limited setting, highlighting how transferable the features learnt by MI.Ps are.
A.4 Scaling Laws
Implementation Details. For the scaling law plots, we trained all the models with batch-size 16384 and the LION optimizer with learning rate r 0.00001 and weight decay of strength 0.001. We further use label smoothing of strength 0.3. We again use augmentations in the form of random flips and crops well MixUp with strength 0.8. We rely the curvefit function from the SciPy library (Virtanen et al., 2020) to fit powerlaws of the form E(C') a(b + +C)- Ecc.
A.5
Computational Efficiency
We highlight the fact that although MLPs require lot of training data, inference is extremely efficient from computational perspective. To illustrate this, we embark on the following comparison; we study inference on 64 x 64 resolution images in MI. vs other popular vision architectures of similar size and complexity in Table 7. More specifically, we compare against ResNet- /52, where replace the stride in the first convolutional layer and remove the first max-pooling operation to
17

---

compensate for the smaller image size. We also compare against a base VTT and Mixer model, where we extract patches from regions in the original image.
As it quickly becomes eminent, MLPs require significantly less FLOPs to make predictions on individual images, in essence utilizing their parameters lot methodically. As result, lateney and throughput are significantly better compared to other candidate architectures. We measure throughput using the optimal batch size on NVIDIA RTX A5000. We highlight, that our MLPs, in contrast to the other architectures are memory bound, meaning that their throughput is determined by the prefetching bandwidth of our GPU. Hardware advancement and specialized architectures could significantly mitigate this effect. Neglecting memory transfer time by propagating the same input through our network gives further 6-fold increase in the potential throughput.
LATENCY (MSEC)
THROUGHPUT (IMAGES/SEC)
PARAMETERS
FLOPS PER FORWARD PASS
B-12/Wi-768
66.89 M
21.2
16063
66.8 M
ResNer-152
60.19 M
423
506
13.07 G
VIT-B/4
86.06 M
424
222
23.08
Mixer-B/4
63.82 M
400
319
19.36
Table 7: Various measures assessing the computational efficiency of different architectures.
B Results for Standard MLPs
B.1 Transfer Learning
For completeness we also analyze the transfer performance of standard MLPs when pre-trained ImageNet21k. We compare a S-MLP of depth 6 and width 2048 against a B-MLP of depth 6 and width 1024, ensuring that both models roughly have a parameter count of around 70 million. We display the results in Table 8. We observe that even the features of standard MLP (i.c. without residual connections and bottleneck structure) transfer very well on different downstream task. Tthe ewerted-bottleneee MLP however still remains superior.
CIFARIO
CIFAR100
TINYIMAGENET
IMAGENET
S-MLP
87.1
68.3
52.1
30.2
B-MLP
90.0
74.6
59.6
36.2
Table 8: Comparing a S-MLP of width 2048 and depth 6 pre-trained on ImageNet21k, with a B-6/Wi- 1024 B-MLP (both models around 70M params) in terms of fine-tuning performance
B.2 Scaling Laws
We also evaluate the scaling law of standard MLPs by training variously sized models on different subsets of ImageNet21k and subsequently linearly probing the features on CIFARION The setting is identical the one described in 4.3. We observe that also standard MLPs exhibit power-law behaviour. The slope (0.22 vs 0.25) and the intercept (0.18 vs 0.16) are however worse when compared against wverted-bottlenecl MLP.
C Weight visualizations
We visualize the first layer weights (1) e 3w by reshaping them back Rexhx3xm We then produce a I xxxx representation by taking the maximal value along the channel dimension.
18

---

b24k
2.1m
4.2m
12m
S-6. 6-WA-512
S-B. 6-Wi-1024
S-B 12-W-1024
6-N6-2048
S-B. 12-W)-048
S-8. --8_6-W1-1096
S-B 12-2-9999
50
40
35
E= 0.18 0.58CC 0.4)-222
100
500
2000
Compute (PFLOPS)
Figure Test error standard MLPs on CIFARI00 when linearly transferred as a function of PFL.OS, measured according to (4,, on log-log scale.
Random Initialization
Pre-trained on 250k samples
Pre-trained 500k samples
Pre-trained on 1M samples
Pre-trained on 4M samples
Pre-trained on 12M samples
Figure 10: Visualization of the first layer weights for different pre-training dataset sizes.
We display such visualizations of the first 5 5 25 "filters" for different pre-training sizes, also including the weights at random initialization in Fig. 10. All models were trained with data augmentation. We observe that filters increasingly develop structure as we increase the dataset size and become more and more localized. We further compare against models that were pre-trained on the full ImageNer211 k, with and without data augmentation in Fig. 11. We observe that even though provide the model with abundance of samples, the weights still remain largely structure-less
19

---

Random Initialization
Without Augmentation
with Augmentation
Figure 11: Visualization of the first layer weights for models trained with and without data augmenta- tion.
and have developped any locality properties. On the other hand, using data augmentation leads to more adapted filters.
D Inverted Bottleneck MLP Code
We provide PyTorch-style pseudo-code for the inverted bottleneck MLP highlight its simplicity.
from torch import nn
class Block (nn. Module)
def __init__ (self, dim, expansion_factor-4, dropout=0.).
super -....-innt_d self
fn in
mn. Sequential(
mn Linear (dim, int (expansion_facttrr * dim)), GELU
un Dropout (dropout), nn nn. Linear (int eexpansion_factor dim), dim)
10
mn Dropott (dropout)
self In mn. LayerNorm (dim)
14
15
def forward (self, x):
return self fn(self (n(x))
19
def MLP (image_size channels, dim, depth, num_classes.
expansion_ Aaaasion_factor=4, factor-4, dropout-0.): return mn. Sequential
Flatten (start_dim=;, dim=-11=
22
nn Linear (image_ size mize channels, dim),
23
[Block (dim, expansion_factor dropout) for in range (depth)
},
nn. Linear (dim, num_classes)
20


Reviews:
[{"review_id": "rpelRLOLtD", "reviewer": "Reviewer_xaDV", "summary": "This paper studies the abilities of Vanilla MLPs and a variant of the MLP with residual connections, LayerNorm, and inverted bottlenecks, performs when scaling in size, compute, batch size, and dataset size. Many learning theory papers study MLP-based settings and claim to motivate their results by deep learning results that are usually run on CNNs or transformers. Somewhat counterintuitively, there are not many clear results on MLPs that showcase all of the relevant behaviors of interest that are studied on more complicated architectures. To justify the theoretical avenues of research further, we would hope that scaling MLPs behaves similarly to scaling CNNs and transformers. Furthermore, one goal of this paper is to understand the claim that the transformer is better due to lack of inductive bias, being a move from a strong inductive bias of CNNs to a weaker inductive bias. Due to the lack of inductive bias, it is often claimed that transformers require further pretraining, but in this regime end up performing better.\nThe authors study a variety of these claims as they pertain to MLPs and show that their “modern” version of the MLP can, indeed, showcase powerlaw generalization scaling with total compute, as well as improved performance with size of the model and dataset size, as well as the ability to be pretrained and effectively transferred to downstream tasks (all on visual recognition tasks).\n\nEDIT: Thank you for your rebuttal, I think with these changes and clarifications it is a good contribution to NeurIPS and I intend to keep my score and continue to lean towards accept rather than reject.", "questions": "- Nitpicky: I am not sure I agree that MLP-Mixer has “arguable even less inductive bias” than a Vision Transformer, but this is a very subjective opinion and I could see an argument for it, but given that data is pre-processed into patches and there is mixing between patches and features, it feels like there is quite a similar level of “inductive bias” between the two, and so I am hesitant to accept this claim at face value, but I don’t mind if it is left in the paper either\n- The authors make the statement at least twice early on in the paper (e.g. top of page 3) that “MLPs invest their compute significantly more into dataset size compared to model size” and I don’t think I understand what this means, nor did I feel I had much clarity on this statement after reading the experiments. Is it saying that scaling dataset size is more important than scaling model size? If that is the case, I don’t think any of the current plots very clearly justify this point, perhaps you can find a way to make it more clearly\n- Why do you use LayerNorm and not BatchNorm? Do you compare to BatchNorm? Does it matter, and if it does how much? How does this relate to normalizations and choices in other models? E.g. does CNN or ViT have less susceptibility to be affected (in how it scales) by normalization layer choices? \n- Do you need to use a bottleneck embedding layer? This seems like a pretty significant requirement. Do these scalings not work without a bottleneck embedding? If that is the case I think it is important to be very clear that a specific MLP gives this scaling, but then if you were to take an MLP with wider layers than input dimension (no bottleneck embedding) these results don’t hold. This seems very important to emphasize in your study, as many learning theory papers (that I’ve come across) don’t necessarily distinguish that MLPs need to have a bottleneck embedding layer\n- Strong label smoothing also seems like a strong constraint, does this work without it? I think generally label smoothing I can believe is practical but not so aggressive as to use alpha=0.3. If these behaviors are only specific to bottleneck embeddings with strong label smoothing I think that should really change the tenor of the main claims, because you are studying a highly specific setting in which MLPs showcase such scalings, but that an average MLP trained on one-hot encoded labels with wide layers, for instance, does not do this (this seems very important to make clear)\n- Perhaps there is a citation or another line of work that gives precedent for aggressive label smoothing like this? You don’t provide any citation or justification of this choice\n- A bit related to the last point, you mention combatting overfitting and severe overfitting, but you haven’t reported any training accuracies nor generalization gaps. Throughout Table 1 it is hard for me to totally understand what is overfit vs. underfit when you only report test accuracies. \n- Furthermore, you don’t mention whether you use a validation set or any cross-validation when experimenting with different choices, and then report the final error on a held-out test set. Can you clarify your experimental methodology?\n- In Table 1 why do you train Vanilla at 100 and then inverted + DA at 1000 and 5000, what happened to Vanilla at 1000 and 5000? Plus Vanilla + DA, Inverted, and ResNet18 + DA don’t report how many epochs it is trained for. Maybe reporting train accuracies and/or generalization gap will help.\n- It seems like it would be useful, when comparing different settings, to stop model training at the same train accuracy to control for other factors, is this something you do? Or how do you justify the comparisons you are making? It is hard for me to get a total sense of all of the dynamics at play here purely from test accuracies.\n- One of your claims is that larger batches significantly boosts both up and downstream performance, and Figure 3 is plotting Error vs. batch size showing this but the line in page 6 says “we plot pre-training batch size against resulting linear downstream accuracy…” which I just found a bit confusing because when I read that and looked at the downward trend in Figure 3 my immediate thought was that accuracy was going down, but then I saw you had error on the y-axis and not accuracy, so maybe you can reword this line\n- In the “Role of augmentations” section you mention that data augmentation gives indirect inductive bias, but you don’t really qualify what this means nor give any citation or justification of this. I guess it reads just like a statement you can make because it’s vague, but I don’t know what exactly you are trying to say to me and why it is correct. Perhaps you can give some intuition on what is left implicit here", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a question I myself was excited to see studied. I bid on this paper as I am highly curious to what extent MLPs share the same behaviors as their more modern counterparts. Having seen many learning theory papers that study the MLP, it is a very natural question to know whether they share some of the interesting dynamics that CNNs and transformers show. I think the experiments are largely beginning to answer this question, though I do have many questions and some potential concerns about experiments and the presentation of results in the paper (which I have written in the questions section) and would appreciate more clarity in order to feel very confident in accepting this paper.\n\nI am generally impressed by the results that are shown, especially the transfer learning results, in that an MLP can, in some settings, achieve such accuracies on tasks like CIFAR-10 and CIFAR-100. In some of these settings I am not yet sure the authors have gone deep enough (discussed more below), but I am overall excited about this type of work and want to see more of it. It seems natural to have tried many of these experiments, and yet I can’t recall any paper that has done so. \n\nClearly, the authors have identified a few key examples of settings in which model scaling is commonly studied, and they have given compelling evidence that some MLPs do indeed follow our existing intuitions. I would love to see a lot more evaluations and discussions, and I think the current results could be a launching pad for more thorough work. As such, I feel that this paper leaves a lot of questions to be answered, but in that sense I think it might still be a worthwhile contribution to the community in its current form, and will help motivate a much-needed bridge between theory and practice, and further inspire more detailed empirical research. However, a revision that carefully addresses the questions I’ve left below would go a long way to convince me that it is a very strong piece of work.", "weaknesses": "The authors are motivated by the question of whether MLPs reflect the empirical observations of practical models, but I have some concern of how they qualify practical. These results show some settings of interest that to me seem fairly specific, and try to discuss what is practical or not. I think it is overall a little bit shallow in total experimentation, given it is a purely empirical paper and experiments are all using one GPU I think more can be done. Generally, I am averse to criticizing a paper for not enough experiments and this paper, I feel, has shown sufficient evidence to convince me there is something interesting. But I think that a lot more work has to be done to really show many of the dimensions by which MLPs scale, and how it relates and compares to modern architectures. \n\nI know the comment above may be a little vague, so I list below the various specific questions (in the questions section) that I have upon reading this paper, which I hope the authors will find helpful. I think the questions below subsume anything I could write on weaknesses, and I think the paper would be very strong if it has answers to these questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very extensive feedback, and we are very happy to hear that the reviewer is excited about this research direction. We address the questions below.\n\n**Mixer:** We agree, it is unclear how the token mixing (attention vs MLP) influences the inductive bias. While [1] does argue that Mixers have less inductive bias, we will highlight that this is debatable. \n\n**Compute vs samples:** We were referring to lines 257-267, where we more closely dissect how compute-optimal MLPs trade-off parameter count and number of samples. We find that for a given level of compute $C$, an optimal MLP invests $\\propto C^{0.35}$ into parameter count, and $\\propto C^{0.65}$ into sample size. This is in contrast to other architectures like transformers, where both quantities scale roughly the same as $\\propto C^{0.5}$  (see [7]). \n\n**Normalization:** We have experimented with both Layer and BatchNorm and found that LayerNorm performs slightly better when pre-training on IN1k (see Fig. 7 in Appendix). We have extended this result now to IN21k, see Table 4 in pdf. Concretely, we pre-train a B-6/Wi-1024 B-MLP with BatchNorm on IN21k and fine-tune on all datasets. We again find that LayerNorm slightly outperforms BatchNorm across all the tasks.  \nWe have also decided to focus on LayerNorm due to its simplicity (no discrepancy for train and inference), as well as to be compatible with more modern architectures like ViT. To our knowledge, ViTs rely on LayerNorm largely due to the fact that there is no batch dependence, allowing more efficient training on multiple GPUs.\n\n**Bottleneck:** This is a great question. We managed to significantly improve performance of V-MLPs (the training protocol was sub-optimal before). We observe that they become more competitive with B-MLPs when training from scratch, thus downplaying the importance of the bottleneck layer (see Table 1). We also pre-trained a V-MLP of width 2048 and depth 6 on IN21k and compared it with a B-6/Wi-1024 B-MLP (both models around 70M params) in terms of fine-tuning (see Table 3). We observe that B-MLPs still generalize better but V-MLPs also reach strong performance. We further find that V-MLPs also exhibit very similar power-laws (see Fig. 1, left) with a slightly worse slope than B-MLPs (0.22 vs 0.25), as expected. We think this further strengthens our claims regarding inductive bias and also makes our findings more directly compatible with theory settings.\n\n**Label Smoothing:** Upstream label smoothing does help performance but is not crucial. We have added results for more smoothing amounts in Table 2. We see that across all tasks, performance improves with stronger smoothing upstream, but only moderately. Thus, we are very confident that the scaling laws still apply even for no smoothing, albeit with a slightly weaker slope. \nStrong amounts of smoothing have been used previously, for instance [2] use smoothing of 0.2, notably also for IN21k. We believe this is effective as samples from IN21k often contain multiple objects, whereas only one ground truth label is provided, as also argued in [2]. Label smoothing in general has seen a resurgence, see e.g. [3, 4, 5].\n\n**Train accuracies:** In Table 5 (pdf) we show training accuracies both on the augmented and un-augmented set as well as test accuracies on IN21k. A moderately large B-MLP trained without augmentations easily overfits IN21k, leading to a large generalization gap. The training accuracies on the augmented set are small largely due to MixUp. We have also added a ViT B/8 baseline to highlight that “underfitting” the original training data when training with augmentations is typical. \n\n**Validation:** For downstream tasks we do not use a validation set but rather fix the number of epochs such that training accuracy saturates. We do not tune hyperparameters but choose a configuration that allows training across all tasks.  For upstream we use a validation set to track progress. We experiment with different hyper-parameters upstream and have stopped runs early in training if upstream validation accuracy does not improve. Tuning hyper-parameters upstream with this strategy (if at all) is common in previous papers investigating scaling laws see e.g. [6, 7, 8].\n\n**V-MLP:**  We did not report later epochs for the V-MLP since it already fits the training data perfectly (also applies for the improved version), and later epochs only deteriorate the performance. We have added more epochs in Table 1. The ResNet was trained for 100 epochs.\n\n**Stop wrt train accuracy:** For downstream training as well as training from scratch for small datasets, we indeed rely on training accuracy/loss (and their saturation) to determine the length of training. When employing augmentations, the models improve more slowly, hence we choose to also evaluate at later epochs. \n\n**Batch size:** We apologize, the plot depicts downstream test error as a function of pre-training batch size, not accuracy. \n\n**Augmentation:** We agree that we were not precise. Augmentations encode invariances in the data (e.g. rotating an image does not affect its content). By training on such augmentations, we try to instill symmetries into the network, and thus arguably add some inductive bias implicitly. We believe that also the noisy nature of augmentations helps with overfitting. \n\\\n\\\nWe will include these results in the text and thank the reviewer again for all the questions. We truly believe that addressing them helped improve our work significantly.\n\n\\\n[1] MLP-Mixer: An all-MLP Architecture for Vision, Tolstikhin et al.\n\n[2] ImageNet-21K Pretraining for the Masses, Ridnik et al.\n\n[3] Training data-efficient image transformers, Touvron et al.\n\n[4] ResNet strikes back, Wightman et al.\n\n[5] How to train your ViT? Steiner et al.\n\n[6] Scaling laws for large language models, Kaplan et al.\n\n[7] Training Compute-Optimal Large Language Models, Hoffmann et al\n\n[8] Scaling Laws for Autoregressive Generative Modeling, Henighan et al.\n"}, {"review_id": "j7UgnI93pc", "reviewer": "Reviewer_bQKr", "summary": "This paper studies scaling of MLPs on various image classification datasets with and without pretraining and data augmentation.", "questions": "The abstract states \"Given the recent narrative less inductive bias is better ... it is natural to explore the limits of this hypothesis\" in L4 but this claim is not supported. Rather, a weaker version stated in L52 is investigated in this work, which could be clarified.\n\nThe paper states in L7-12 that while deep learning theory literature almost always studies MLPs, there are no empirical datapoints connecting to theory. This sounds like the paper would cite relevant theory works and try to confirm/refute them, but this is not the case. The paper explores scaling of MLPs. At least, the thought in L7-12 does not connect smoothly to the question raised in L12-13. Perhaps, this statement could be softened a bit.\n\nL15 Perhaps, it should be clarified that pretraining on ImageNet21k is used: \"... drastically improves with scale [and pretraining ...]\"\n\nIn Table 3, the parameter count (in the first layer) is resolution dependent, but the resolution is not given.\n\nL194. MLP does indeed become better, but not enough to state that it's competitive with a ResNet.\n\nL231-232. It is common knowledge that FLOP = 2P for MLP\n\nL236. (5) doesn't have the coefficient c. Also, power-law is defined as `a * x ^ power + b` without the affine offset for `x`.\n\nL296. Discussion should be Conclusion.\n\nI don't fully understand why Block(z) = z + ... is called an inverted bottleneck MLP. Residual MLP seems more natural\n", "limitations": "No, the authors didn't write potential societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "MLP scaling trends are a welcome data point for practitioners and theoreticians alike.", "weaknesses": "The paper does not carefully tune the learning rate, weight decay, and other hyperparameters, so the trends shown in this work may not be a Pareto frontier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback on our work. We will address the concerns in the following.\n\\\n\\\n**Weaknesses:**\n\n\n**Not tuning hyper-parameters:** We apologize for not clearly stating this in the paper but we did tune hyper-parameters employed upstream based on the upstream validation accuracy. We only fully trained models that improve over the current optimal model and otherwise ended runs early on in training. We agree that solely relying on such upstream-tuning might not always be optimal for downstream performance but given our limited computing budget, fully training every hyper-parameter configuration upstream is impossible and we believe that our approach is a reasonable approximation. [1] explicitly explore this question for Vision Transformers on ImageNet21k and find that \n\n*“The results are mixed but generally reflect that the cheaper strategy [upstream validation] works equally well as the more expensive strategy [downstream validation].”* \n\nWe also want to highlight that this is a common strategy in many previous works on scaling laws where performing multiple runs for huge models is also impossible [2, 3, 4]. \n\\\n\\\n\\\nGiven that this was the only concern the reviewer raised, we would be very grateful if the reviewer could further voice avenues for potential improvement of our work or consider raising the score.\n\\\n\\\n[1] How to train your ViT? Steiner et al., 2021\n\n[2] Scaling laws for large language models, Kaplan et al., 2020\n\n[3] Training Compute-Optimal Large Language Models, Hoffmann et al., 2022\n\n[4] Scaling Laws for Autoregressive Generative Modeling, Henighan et al., 2020\n\\\n\\\n**Questions:**\n\n\n**Weaker Hypothesis:** Thank you for pointing this out, we will clarify our focus in this work in the abstract.\n\n**DL Theory:** We apologize that we didn’t write this clearly and will improve the text. We do not aim to question the validity of the theoretical works themselves directly. The cited theoretical works focus on strongly simplified settings with the MLP serving as the model of choice (e.g. studying generalization in a synthetic toy task instead of realistic image classification). These simplified settings are still far from real-world settings (e.g. classification on ImageNet) but with time, progress towards that goal can hopefully be made. Our work serves more as a check to ensure that the MLP actually exhibits the interesting characteristics of modern models in the relevant realistic settings. This is the end goal that theoretical works (slowly) try to build up towards. We will make this clearer in the text.\n\n**L15:** Thank you for highlighting this, we will clarify that pre-training on ImageNet21k is used.\n\n**L194:** We will weaken the phrasing in order to avoid confusion. \n\n**Resolution-Dependence:** Thank you for pointing this out, we will clarify the dependence on the resolution as well as the value of the resolution, which is 64 x 64.\n\n**Minor Changes:** Thank you for the improvements on the manuscript, we will incorporate them."}, {"review_id": "R9udVlMOhy", "reviewer": "Reviewer_RAGY", "summary": "This paper presents a comprehensive study on the scaling capabilities of MLPs, challenging the conventional belief that this architecture is limited in its performance on vision tasks. The authors employ recent advancements in deep learning architectures to modernize MLPs while preserving their core characteristics. The experimental results obtained in this study are remarkable, indicating that tasks previously deemed impossible for MLPs can now be accomplished.\n\nTo achieve these results, the authors incorporate several crucial components of modern deep learning architectures into MLPs. They introduce normalization layers, skip connections, bottlenecks, and a large batch size, along with a modern optimizer. These additions ensure stable training, thus allowing to train on large datasets. Additionally, the authors demonstrate that despite the absence of inherent inductive bias in MLPs, data augmentation and sufficient data volume effectively compensate for this limitation. Consequently, scaled MLPs perform competitively with architectures possessing stronger inductive biases, such as CNNs or ViTs, on several datasets, while being much more computationally efficient at a fixed resolution.\n\nFinally, the authors assess the scaling properties of MLPs providing accurate estimates of their scaling laws in terms of data budget and compute budgets. Unlike other architectures, in scaling MLPs larger datasets are more important than compute budget. The findings open up new possibilities for utilizing MLPs in vision tasks and highlight the importance of reconsidering their scaling capabilities in the broader context of deep learning research.", "questions": "I would encourage the authors to address the stated weaknesses in the rebuttal and in any future revision of the manuscript. I honestly believe doing so will make the paper better.", "limitations": "Overall, I believe the paper has addressed well most of its limitations and that the authors have been open and honest in describing the shortcomings of their results. However, as mentioned in Weakness 2, I believe it is important that they also discuss how resolution and different image sizes matter in their results.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "#### 1. **Addressing a clear research gap in the literature**:\nThis paper successfully completes the research program started with the ViT paper and continued with the MLP-Mixer and ConvMixer that challenges common assumptions regarding the benefits of embedding inductive biases in architectures. By answering the question of whether inductive bias can be fully circumvented with more data affirmatively, this paper fills a significant void in the literature. Although the results might have been expected by many, the clear empirical validation of this hypothesis makes this paper an important contribution.\n\n#### 2. **Thorough experiments with solid results**:\nThe execution of the experiments and the manner in which they are conducted are exemplary. The paper provides a comprehensive ablation analysis across various settings, thoroughly examining and validating each claim. The results obtained are both convincing and transparent. Notably, the authors openly discuss the relatively poorer performance on ImageNet and offer insightful explanations for this observation, enhancing the credibility of their findings.\n\n#### 3. **Outstanding presentation**:\nThe paper exhibits outstanding writing quality, making it a pleasure to read. The clarity of the language and the structure of the paper contribute to its overall excellence. The authors effectively communicate their ideas, methods, and results, ensuring that readers can easily comprehend and appreciate the significance of their work.", "weaknesses": "#### 1. **Missing discussion regarding the importance of processing images in patches**\nIn my opinion, the paper is missing an important discussion regarding the critical factor that makes MLP-mixers superior to the presented full MLP architecture, i.e., the processing of images in patches. Although I understand a very in-depth empirical analysis of these differences might be outside of the scope of the paper, it is important to fully acknowledge the importance of this key design choice. In particular, I believe it is important to also reference the [\"Patches are all you need?\"](https://openreview.net/forum?id=rAnB7JSMXL) paper which also tried to fill a research gap in this sense prior to this work.\n\n#### 2. **Missing discussion on the effect of image resolution on the results**\nIt seems that an important design decision in this paper has been the downscaling of all datasets to a 64x64 resolution. I believe this decision deserves further motivation in the text as well as some reflection on the fact that MLPs cannot be applied to inputs of different sizes. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough assessment of our work and the very positive feedback. We will address the questions in the following.\n\n**Role of patch size:** We completely agree with the reviewer and also believe that breaking the images into patches plays a fundamental role for all these architectures, as it adds a strong inductive bias. Unfortunately, investigating the role of patchifying is out of the scope of our current work but makes for very exciting future work. Especially investigating the role of the patch-size in ViTs and MLPMixers is very interesting as it modulates the degree of inductive bias in the architecture. We will definitely expand the corresponding discussion and add the very relevant proposed related work. Another interesting work in that regard is [1], which shows that even simple spatial pooling for token-mixing suffices for good performance. The spatial pooling of course adds back a lot of inductive bias, but again relying on patches seems crucial for good performance. \n\n**Resolution:** We completely agree with the reviewer and we will add a paragraph to discuss the role of the resolution. In our work, this decision was largely made due to computational constraints, as a high resolution leads to a very large first layer. We will highlight that MLPs indeed cannot be used directly for different resolutions, thank you for pointing this out! Inspired by this comment, we adapted the resizing technique from FlexiViT [2] in order to enable the MLP to handle different resolutions. We take a B-12/Wi-1024 B-MLP which is pre-trained on ImageNet21k at a resolution of $64 \\times 64$ and fine-tune it on ImageNet1k at a resolution of $64 \\times 64$. We then evaluate test performance on ImageNet1k for resolutions $16,32,48,64,96,128$ while resizing the embedding layer as outlined in [2] in Section 3.4 (without any further training). We show the results in Figure 1, on the right. We observe that test performance remains stable for all resolutions and slowly starts deteriorating only for the smallest ones. We will include these results in the discussion.\n\n\n[1] MetaFormer is Actually All You Need for Vision, Yu et al. \n\n[2] FlexiViT: One Model for All Patch Sizes, Beyer et al."}, {"review_id": "lR9CmYB1Ev", "reviewer": "Reviewer_EAzm", "summary": "This paper argues that MLP networks can perform well on challenging tasks when they are scaled up. Thus, they are able to compensate for their lack of inherent inductive bias when trained on a large amount of data with many model parameters. The authors show through experiments on popular image classification datasets that indeed, MLPs can perform surprisingly well when scaled up. Along the way, the authors make a number of other observations regarding the role of MLP architecture, data augmentations and scaling laws.", "questions": "Can performance on ImageNet be improved to near state-of-the-art for MLPs?\n\nWhat is the relative significance of scale vs. data augmentation/model enhancements for MLPs?", "limitations": "It would be good if the authors could mention some of the limitations listed above in the discussion section.\n\nNo potential negative societal impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "**Originality**\n\nThe question the authors aim to answer is quite interesting. As far as I can tell, few empirical studies have approached the question of whether MLPs can be made to perform competitively on standard benchmarks under large-scale data and model size (although studies have trained MLPs on standard benchmarks for other reasons).\n\n**Quality**\n\nThe experiments are extensive and thorough. The authors train on ImageNet21k and evaluate on standard image classification benchmarks; in my view, these are exactly the most valuable experiments to run to investigate the question the authors aim to answer.\n\n**Clarity**\n\nThe paper is quite well-written. The key motivation of the paper and background are made very clear, and figures are well-illustrated. As a minor suggestion, moving Table 1 and 2 closer to the experiment section would be helpful.\n\n**Significance**\n\nOverall, if the paper can demonstrate its claims properly, the paper would be quite significant to the field. Showing that the inductive bias of CNNs or Transformers can be compensated with scale will perhaps focus the attention of the field toward building inductive biases that can't as easily be compensated by scale, as well as validating (or invalidating) the idea of applying theory done on MLPs to other models.\n", "weaknesses": "My main concern with the paper is with the interpretation of the experimental results; I'm concerned that the results are not enough to support the authors' hypothesis 3.\n\nFirst, the authors only conduct experiments on image classification tasks. If the authors wished to make a more general claim about inductive bias in MLPs vs other models, it would be best to conduct experiments on other domains. Alternatively, the authors may wish to restrict their claims to the image domain.\n\nSecond, the empirical results on ImageNet do not appear quite strong. In Table 4, the authors find a maximum ImageNet top-1 accuracy of 41%; this does not seem very competitive. If the authors wanted to show that MLPs can indeed circumvent their lack of inductive bias, I would expect to see a result showing near state-of-the-art (or at least competitive results) on ImageNet. I may be missing some key empirical results though.\n\nThird, the authors find that data augmentation is critical to the performance of MLPs. Data augmentation, as the authors note, allows the model to learn invariances that would otherwise be built into a model like a CNN. Thus, data augmentation is a powerful inductive bias. However, this fact seems to undermine the authors' claim that scale alone is sufficient to compensate for lack of inductive bias. The authors may wish to reformulate their claim to suggest that architectural inductive biases can be compensated by a combination of scale and other inductive biases.\n\nSimilarly, the authors find that using a specific inverted bottleneck MLP architecture is important for performance. The authors seem to view this as a minor addition to the MLP, but given the large performance improvement, this addition is actually quite significant. Indeed, in my view, architectural changes that are simple but lead to large performance gains are exactly those that are most valuable to study!\n\nAlso, as a more minor point, the authors claim that MLPs are \"completely free of any inductive bias.\" This seems to be too strong of a claim: they are often one of the simplest models to implement, but this doesn't mean they are free of any inductive bias. The claim quoted by the authors from the *Principles of Deep Learning Theory* book seems to be a more fair statement to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and the interesting points raised. We will address them in the following.\n\n**Image classification:** We agree with the reviewer and we will highlight that we focus on image classification. We largely do this since inductive bias is clearer and more intuitively defined in the image domain, compared to other domains as for instance natural language processing.\n\n**ImageNet performance:** We agree with the reviewer that ImageNet performance is not competitive with standard baselines. We suspect several reasons for this:\n\n1. Due to computational constraints, we down-sized images to 64 x 64, making the fine-grained classification required for ImageNet more difficult.\n2. We also want to highlight that our work is the first with the aim of making MLPs competitive. Other architectures like convolutions have seen a decade of development from the entire ML community, leading to very specialized recipes (optimizers, learning rate schedules, normalization methods etc.). While AlexNet still outperforms our MLPs (achieving around 55% https://pytorch.org/hub/pytorch_vision_alexnet/ ), this gap is significantly smaller and we believe that similar progress could be made for MLPs with time.\n\nFinally, we also want to stress that MLPs achieve very strong downstream performance on every other task, thus indeed highlighting that lack of inductive bias can be overcome. We hypothesize that training on a larger dataset like (the private) JFT-3B with 3 billion images would significantly boost the performance on ImageNet. \n\n**Data augmentation:** This is a good question! We agree with the reviewer that data augmentation allows us to implicitly encourage networks to learn invariances relevant to the task, thus adding some level of inductive bias. We believe however that the regularizing nature of augmentations is more crucial for the case of MLPs, since training accuracy remains far from perfect even after long periods of training (see Table 5 in the rebuttal pdf). The MLPs hence never really learn the invariance in the data but at the same time also don’t overfit, which helps performance. We do however acknowledge that training with data augmentation does add inductive bias implicitly. In this work we largely focus on the inductive bias of the model architecture, as the reviewer correctly highlights. Moreover, even if data augmentation is employed, scale is still needed to reach decent performance and overcome the lack of inductive bias in the model. We will rephrase the text to make this clearer.\n\n**Bottleneck MLP:** This is a great question. We managed to significantly improve performance of V-MLPs (the training protocol was sub-optimal before). We observe that they become more competitive with B-MLPs when training from scratch, thus downplaying the importance of the bottleneck layer (see Table 1 in attached pdf). We also pre-trained a vanilla MLP of width 2048 and depth 6 on ImageNet21k and compare it with a B-6/Wi-1024 BottleneckMLP in terms of fine-tuning performance (see Table 3 in attached pdf). We chose different widths to make the parameter count comparable (both around 70M). We observe that BottleneckMLPs generalize better in accordance with previous literature but vanilla MLPs also achieve strong performance, further showing that this design choice helps but is not crucial. We have also evaluated the scaling laws for vanilla MLPs on CIFAR100 (see Figure 1 on the left) and find that it is very similar with a slightly worse slope (0.22 vs 0.25), as expected. We think this further strengthens our claims regarding inductive bias and also makes our findings more directly compatible with theory settings.\n\n**Inductive bias of MLP:** We agree with the reviewer and we regret our overly strong wording regarding the inductive bias of MLPs. Comparatively to CNNs and ViT, MLPs have a very weak inductive bias, they are for instance not vulnerable to permutations of pixels. However, they still encourage a hierarchical order of features, which does serve as an inductive bias (albeit not specialized towards vision in particular). We will rephrase the text accordingly.\n\n\nWe again thank the reviewer for the interesting points raised, leading to a refinement of our claims and statements, as well as disentangling the roles of different components more carefully. \n"}]

Author rebuttal:
We thank all the reviewers for their valuable feedback. We reply to each reviewer’s questions in the individual responses and have added tables and figures in the attached rebuttal pdf which we reference and explain in the responses. For convenience we also provide a detailed summary below.
\
\
Due to space constraints, we had to compress notations. For datasets we denote CIFAR10 → C-10, CIFAR100 → C-100, TinyImageNet → T-IN and ImageNet → IN. We further use V-MLP for vanilla MLP and B-MLP for Bottleneck MLP.  We also denote upstream by $\uparrow$ as well as downstream by $\downarrow$. \
To denote the architecture of V-MLPs, we slightly abuse notation and also denote by B-L/Wi-M a vanilla MLP of depth $L$ and width $M$. Notice that we used the same notation for Bottleneck MLPs but a block there is slightly different (i.e. first maps to $4 \times M$ and then to $M$). We refer to the main text for the more precise description.
\
\
To summarize the results in the pdf, we have found the following.

1. We managed to improve performance of the **Vanilla MLPs** on small datasets (the training protocol was sub-optimal before) making them more competitive with Bottleneck-MLPs when training from scratch (see Table 1 in rebuttal pdf). To further investigate the scaling behaviour of Vanilla MLPs, we pre-trained them on ImageNet21k and fine-tuned them on several tasks (see Table 3 in rebuttal pdf). Comparing against a similar sized Bottleneck-MLP, we see that the B-MLP still performs better, but the V-MLP also achieves very strong performance. To complete the picture, we produce the same scaling plot as done for B-MLPs (Figure 4 in paper) in Figure 1 (left) of the rebuttal pdf, i.e. we pre-train variously-sized V-MLPs on several subsets of ImageNet21k and plot the amount of compute (in FLOPs) against the test error of linear evaluations on CIFAR-100. We observe that V-MLPs exhibit the same power-law behaviour as B-MLPs with a slightly weaker slope (0.22 vs 0.25). While the bottleneck layer thus helps in terms of absolute performance, it is not essential to the scaling behaviour. We think this further strengthens our claims regarding inductive bias and also makes our findings more directly compatible with theory settings.

2. We ablated the effect of **label smoothing** upstream by pre-training B-MLPs on ImageNet21k with various smoothing amounts and subsequently fine-tuning them on several datasets. We found that smoothing does help performance but only very marginally so (see Table 2 in rebuttal pdf), e.g. a B-MLP B-6/Wi-1024 without any smoothing performs almost on-par compared to smoothing with a value of 0.3 (90% vs 89.2% on CIFAR-10). Smoothing is thus not an essential component to our findings.


3. We extended the results of Figure 7 in the Appendix and **compared LayerNorm and BatchNorm** by pre-training B-MLPs on ImageNet21k and subsequently fine-tuning them on several tasks (see Table 4 in rebuttal pdf). We confirm that LayerNorm has a slight edge over BatchNorm in terms of downstream performance, but only marginally so (e.g. 90.0% vs 89.4% on CIFAR-10).


4. We have added **upstream training and test accuracies** on ImageNet21k in Table 5 of the rebuttal pdf. We evaluate training accuracy both for augmented and un-augmented versions. We find that a moderately large B-MLP B/6-Wi-1024 easily overfits the training set if trained without augmentations, leading to strong overfitting behaviour. Overfitting is reduced by employing data augmentation, which serves as a regularizer in this setting.

5. In Figure 1 (right) we show how MLPs are robust to **different resolutions** when the embedding layer is adequately resized, adapting the approach of FlexiViT [1]. We take a B-12/Wi-1024 B-MLP which is pre-trained on ImageNet21k at a resolution of $64 \times 64$ and fine-tune it on ImageNet1k at a resolution of $64 \times 64$. We then evaluate test performance on ImageNet1k for resolutions $\{16, 32, 48, 64, 96, 128\}$ while resizing the embedding layer as outlined in [1] in Section 3.4 (without any further training). We observe that test performance remains stable for all resolutions and slowly starts deteriorating only for the smallest ones.

We again thank the reviewers for the very helpful feedback and we truly believe that addressing the questions significantly improved our work.

Finally, we agree with the reviewers and we will add a separate **Limitations section**.
\
\
[1] FlexiViT: One Model for All Patch Sizes, Beyer et al.


Meta-review: Reviewers find this paper studying interesting and under-explored questions about scaling behavior of simple MLP models and its similarity to popular architectures (e.g. Transformers) behavior with scale. They find the paper results, though preliminary, insightful and encouraging warranting further studies.

Reviewers raised some valid criticisms of the paper. 1) Some claims around MLPs having less inductive bias are questionable as Transformers are  also universal approximator function class. 2) Evaluations are limited to image classification and small scale datasets/models. 3) Low accuracies on ImageNet.

While I agree with limitations of this paper, I think the direction is interesting and the paper takes a good enough step towards it. Hence I suggest acceptance. It is important that authors update the paper addressing the limitations as suggested by the reviewers and toning down some of the claims.
Decision: Accept (poster)