SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gregor Bachmann*, Sotiris Anagnostidis*, Thomas Hofmann
In this work we revisit the most fundamental building block in deep learning, the multi-layer perceptron (MLP), and study the limits of its performance on vision tasks. Empirical insights into MLPs are important for multiple reasons. (1) Given the recent narrative "less inductive bias is better", popularized due to transformers eclipsing convolutional models, it is natural to explore the limits of this hypothesis. To that end, MLPs offer an ideal test bed, as they lack any vision-specific inductive bias. (2) MLPs have almost exclusively been the main protagonist in the deep learning theory literature due to their mathematical simplicity, serving as a proxy to explain empirical phenomena observed for more complex architectures. Surprisingly, experimental datapoints for MLPs are very difficult to find in the literature, especially when coupled with large pre-training protocols. This discrepancy between practice and theory is worrying: Do MLPs reflect the empirical advances exhibited by practical models? Or do theorists need to rethink the role of MLPs as a proxy? We provide insights into both these aspects. We show that the performance of MLPs drastically improves with scale (95% on CIFAR10, 82% on CIFAR100, 58% on ImageNet ReaL), highlighting that lack of inductive bias can indeed be compensated. We observe that MLPs mimic the behaviour of their modern counterparts faithfully, with some components in the learning setting however exhibiting stronger or unexpected behaviours. Due to their inherent computational efficiency, large pre-training experiments become more accessible for academic researchers. All of our experiments were run on a single GPU.
Figure 1: Test error on CIFAR100 as a function of PFLOPS.
*Equal contribution. Correspondence to {gregorb, sanagnos}@ethz.ch. Code and checkpoints available at https://github.com/gregorbachmann/scaling_mlps
Figure 2: Different architectures process images differently. Convolutions directly operate on the image, ViTs and MLP-Mixers work with patches while the MLP takes the flattened image as input.
Deep learning has undergone tremendous empirical progress in the last decades. The dominant approaches in practice these days rely on very large, pre-trained models which are then fine-tuned to the specific task at hand. For natural language processing, these models usually are some variant of the Transformer architecture (Vaswani et al., 2017), while in computer vision, both convolutional and transformer-based models are very popular (He et al., 2015; Tan and Le, 2020; Dosovitskiy et al., 2021). The theoretical understanding of these advances on the other hand remains very poor and the gap between the world of theory and practice is growing at an alarming rate. One aspect of this gap is the family of models investigated; due to their mathematical simplicity, theoretical works largely focus on simple multi-layer perceptrons (MLPs). Consisting of a series of unstructured matrix multiplications, interleaved with element-wise non-linearities, the MLP serves as an ideal test bed to analyze empirical phenomena exhibited by more complicated models employed in practice. Due to their inferior performance, MLPs are rarely used and very little is known regarding their behaviour in more modern settings. For instance, to the best of our knowledge, there is not a single published result showcasing an MLP trained on ImageNet1k, the de-facto standard benchmark in vision, let alone any pre-training/transfer learning studies. This lack of empirical data is concerning as theory aims to understand the characteristics of modern architectures through the lens of MLPs, yet only little assessments are made regarding how well such a proxy works. This raises the question,
$$
D o M L P s \text {r e f l e c t}
$$
Investigating MLPs is not only interesting for theory but also for practice. With the Vision Transformer (ViT) outperforming its convolutional competitors in very large-scale settings, the role of inductive bias has recently been brought into question. Since a ViT is equipped with significantly less inductive bias for vision compared to convolutional models (e.g. it lacks translation-equivariance) a novel narrative has recently emerged:
$$
\text {A t l a r g e s c a l e s o f c o m p u t e , h a v i n g l e s s i n d u c t i v e b i a s i s b e n e c i f i c a l f o r p e r f o r m a n c e .} \tag {2}
$$
More evidence for this hypothesis has been collected in the form of the MLP-Mixer (Tolstikhin et al., 2021), an architecture with arguably even less inductive bias, solely relying on multi-layer perceptrons as patch processors and mixers. The MLP architecture is the ideal candidate to test the limits of such a hypothesis, as it exhibits the least inductive bias for vision due to its invariance to permutations of pixels. Unfortunately, the scale where Transformers and MLP-Mixers start to outperform convolutional models is out of reach for most researchers, requiring billions of annotated images and thousands of TPUs. We thus expect similar required scales for MLPs and hence instead investigate the following, weaker hypothesis:
$$
L a c k \quad \text {i f} \quad \text {i n d u c t i v e} \quad \text {b i a s} \quad \text {c a n} \quad \text {b e} \quad \text {c o m p e n s a t e d} \quad \text {b y} \quad \text {s c a l i n g} \quad \text {c o m p u t e}. \tag {3}
$$
i.e. we aim to measure to what degree a lack of inductive bias hinders performance even if a model is subjected to a large parameter count and trained on datasets with many examples (albeit smaller than what is employed in Dosovitskiy et al. (2021)).
In this work, we provide answers to question 1 and provide further evidence for hypothesis 2 and 3 by investigating how far we can push the empirical performance of models solely built from composing several MLP blocks. We give largely positive answers to question 1, observing that MLPs behave very similarly to their modern counterparts when subjected to scale, i.e. their performance increases
Figure 3: A simplified depiction of the differences between an MLP-Mixer and an MLP.
predictably as a power law in parameter count and sample size, akin to Hestness et al. (2017, 2019); Kaplan et al. (2020); Zhai et al. (2022) (see e.g. Fig. 1). In contrast to previous work, however, we find that compute-optimal MLPs allocate their budget more strongly into sample size, highlighting again their small inductive bias. While regularization in the form of data augmentation is also helpful for CNNs, its role is significantly amplified for MLPs even at large sample sizes, leading to fatal degradation if turned off. We further investigate how the implicit bias of SGD affects performance, and we make a very counter-intuitive discovery: contrary to CNNs, larger batch sizes generalize significantly better for MLPs. This result questions the validity of the proxy role that the MLP plays in theoretical works investigating the implicit bias of SGD. While, as expected, the scale employed in this work does not suffice for hypothesis 2, we provide strong evidence for 3, which we view as an important first step. We observe that scale indeed suffices to overcome the bad inductive bias present in MLPs, leading to surprisingly strong downstream performance, e.g.  $\approx 95\%$  on CIFAR10,  $\approx 82\%$  on CIFAR100 and  $\approx 58\%$  on ImageNet ReaL. In summary, we make the following contributions:
- We fill the gap between theory and practice, providing the first results for MLPs trained in modern settings.
- We show that MLPs mostly behave comparably to their modern counterparts, making them a good proxy for theory. We observe however that the roles of regularization and implicit bias of SGD significantly differ and theory hence needs to adapt.
- We provide further evidence that inductive bias is not crucial at large scales, showing that even "bad" architectures like MLPs can achieve strong downstream performance. We however identify a shift in compute-optimality, showing that optimal MLPs invest their compute significantly more into dataset size compared to model size.
Theoretical Works. The MLP has served as the main object of study for theoretical works in deep learning across different domains. The cornerstone results for areas such as convergence of SGD-trained neural networks (Mei et al., 2018; Du et al., 2019; Zou et al., 2020; Li and Yuan, 2017; Saxe et al., 2014), most generalization bounds (Arora et al., 2019b; Mei and Montanari, 2021; Jacot et al., 2018; Allen-Zhu et al., 2019a), the benefits of overparametrization (Neyshabur et al., 2019; Allen-Zhu et al., 2019b; Arora et al., 2018), the implicit bias of SGD towards favourable solutions (Soudry et al., 2018; Neyshabur et al., 2014; Chizat and Bach, 2020), signal propagation properties (Poole et al., 2016; Schoenholz et al., 2017) and scaling laws (Bahri et al., 2021; Maloney et al., 2022) are all largely obtained for MLPs. To quote the very influential Principles of Deep Learning Theory book (Roberts et al., 2022):
"MLPs are the simplest of these neural network architectures that hinge on this stacking idea, and thus provide a minimal model for an effective theory of deep learning."
There are also several theoretical works studying more modern setups such as convolutional or transformer-based networks including Arora et al. (2019a); Gunasekar et al. (2018); Brutzkus and Globerson (2017); Hron et al. (2020) to name but a few, but the main theoretical focus to the best of our knowledge still remains on the MLP architecture. We thus believe it is important to explore the limits of such a theoretical proxy in realistic settings.
MLPs. The multi-layer perceptron has its origins in Rosenblatt (1958), serving as an extension to the classic Perceptron with its hidden layers however fixed to random initialization. Ivakhnenko et al. (1965) devised the first method to update the hidden layers through self-organization. Amari (1967) then introduced the idea to train the parameters with stochastic gradient descent. Mathematically, an MLP of depth  $L \in \mathbb{N}$  can be described very efficiently; given an input  $x \in \mathbb{R}^d$ , it applies a series of linear transformations, interleaved with an element-wise non-linearity  $\sigma : \mathbb{R} \to \mathbb{R}$ :
$$
\boldsymbol {z} ^ {(l)} = \boldsymbol {W} ^ {(l)} \boldsymbol {x} ^ {(l - 1)} \longrightarrow \boldsymbol {x} ^ {(l)} = \sigma \left(\boldsymbol {z} ^ {(l)}\right)
$$
where we define  $\pmb{x}^{(0)}\coloneqq \pmb{x}$  and  $\pmb{W}^{(l)}\in \mathbb{R}^{d_l\times d_{l - 1}}$  for  $l = 1,\dots ,L$  are the learnable weight matrices. For the sake of readability, we omit the biases. This mathematical simplicity makes the MLP a very attractive model to study theoretically (albeit still very far from trivial) and indeed many works frame their results around this general model class. When used for vision, the input tensor  $\pmb {x}\in \mathbb{R}^{h\times w\times 3}$  is flattened into a vector  $\operatorname {vec}(\pmb {x})\in \mathbb{R}^{3hw}$ . Notice how such an architecture completely lacks locality and weight sharing, every unit simply processes the entire image at once. More worryingly, the vectorization vec could be applied in any way, i.e. any permutation of  $\pmb{x}$  looks identical to an MLP.
We want to highlight that MLPs of course are not completely free of inductive bias, in the sense that they encourage learning a hierarchical feature structure. On the other hand, there is no vision-specific inductive bias present in MLPs, which is the main setting we investigate here. We refer to Battaglia et al. (2018) for a more in-depth treatment of inductive bias.
Convolutions. The MLP is a very general model and has no structure built into it to make it more suitable for vision tasks. A convolution on the other hand was designed specifically for vision with desirable characteristics incorporated into the model. A convolution can be viewed as a special case of an MLP, where the weight matrix  $\mathbf{W}$  is very structured by being sparse and having shared entries, leading to spatially localized learning. This can be most easily illustrated in the case of convolving a  $2 \times 3 \times 1$  image  $\mathbf{x}$  with a  $2 \times 2$  filter  $\mathbf{f}$  as the following matrix multiplication:
$$
\boldsymbol {f} * \boldsymbol {x} = \boldsymbol {W _ {f}} \operatorname {v e c} (\boldsymbol {x}) = \left( \begin{array}{c c c c c c} f _ {1} & f _ {2} & 0 & f _ {3} & f _ {4} & 0 \\ 0 & f _ {1} & f _ {2} & 0 & f _ {3} & f _ {4} \end{array} \right) \operatorname {v e c} (\boldsymbol {x})
$$
Here vec denotes the standard, row-wise vectorization-scheme to flatten the image. Instead of operating with a dense matrix as the MLP, the convolution uses a structured matrix  $W_{f}$  tailored to the task of vision, leading to a better inductive bias. Moreover, a convolution exhibits translation-equivalence, i.e. shifts of images are processed equivalently to the original. Crucially, in contrast to the MLP, a convolution severely suffers if a permutation is applied to the image.
Vision Transformer. Inspired by successes in NLP, the Transformer architecture has been adapted to vision (Dosovitskiy et al., 2021). An image  $\pmb{x} \in \mathbb{R}^{h \times w \times 3}$  is broken up into patches (also called tokens) and linearly embedded (see Fig. 2), augmented with a positional embedding, marking its spatial location in the image. The obtained embeddings are processed by self-attention layers where patches interact, and MLP layers, which are shared among patches and transform them individually. While the inductive bias of a ViT is certainly weaker compared to a CNN (it lacks translation-equivalence), the patching and parameter sharing still make the architecture suitable for vision.
MLP-Mixer. Similar to the ViT, the MLP-Mixer also works with a patchified image (Tolstikhin et al., 2021). Unlike the ViT, token-mixing is not implemented using self-attention but rather another MLP block is used to exchange information between patches. We want to clearly highlight the difference between an MLP-Mixer and an MLP: An MLP-Mixer operates on patches, where in each block it applies a shared MLP to each patch for processing, and another MLP for mixing the patches along the channels. We visualize the differences in Fig. 3 for clarity. We again want to stress that breaking the image into patches and sharing parameters among them significantly enhances the amount of inductive bias, compared to a standard MLP.
Patchifying. As highlighted above, ViTs and Mixers largely obtain their inductive biases through breaking the images into patches. This choice seems to be beneficial even for architectures that already possess a strong inductive bias, such as the ConvMixer (Trockman and Kolter, 2022), where convolutions are performed on individual patches. The very recent Metaformer (Yu et al., 2022) further shows that even a simple spatial pooling instead of attention can lead to strong performance if the image is patched. While the success of this mechanism certainly warrants further investigation, in this work we decided to deliberately focus on MLPs as they specifically lack this type of bias.
CIFAR10	CIFAR100	TINYIMAGENET	IMAGENET
S-MLP (@100 E)	54.2	28.8	8.5	9.2
S-MLP + DA (@1000 E)	68.9	43.3	25.2	24.3
S-MLP + DA (@5000 E)	72.3	44.5	27.3	26.8
B-MLP (@100 E)	58.1	30.5	8.9	8.7
B-MLP + DA (@1000 E)	70.1	48.3	27.2	28.7
B-MLP + DA (@5000 E)	75.4	50.4	31.2	31.7
RESNET182+DA	93.2	75.6	68.9	69.7
Table 1: Test accuracies (in %) without any pre-training. The S-MLP has depth 6 and width 1024 while the B-MLP has depth 6, width 1024 and an expansion factor of 4.
3 Architecture
We study different variants of the MLP architecture, starting from the standard vanilla setup and then adding more components such as residual connections and bottleneck layers.
Standard MLP. As a first starting point, we investigate simple MLPs with ReLU activations and isotropic design, i.e. except for the first, every layer has the same width  $m \in \mathbb{N}$ . In order to avoid training instabilities we further enhance the standard MLP with layer normalizations (Ba et al., 2016) placed after the activations. We thus compose several blocks of the form
$$
\operatorname {B l o c k} (\boldsymbol {z}) = \sigma \left(\boldsymbol {W} \operatorname {L N} (\boldsymbol {z})\right)
$$
with  $W \in \mathbb{R}^{m \times m}$ . To embed the image  $\pmb{x} \in \mathbb{R}^{d \times d \times 3}$  we use a linear layer  $\operatorname{emb}(\pmb{x}) = W^{emb} \operatorname{vec}(\pmb{x})$  with  $W^{emb} \in \mathbb{R}^{m \times 3d^2}$ . Such an embedding layer is crucial since for high resolution images,  $3d^2$  can be quite large and thus  $m$  needs to be chosen smaller. We empirically find that such a network design is the minimal choice in order to guarantee successful training across all scales of parameter count and sample size. We will use the short cut  $S$ -MLP to denote such an architecture.
Inverted Bottleneck MLP. Inspired by Lin et al. (2015); Tolstikhin et al. (2021) we add a bottleneck structure to an MLP block as well as skip connections as follows:
$$
\operatorname {B l o c k} (z) = z + W ^ {c} \sigma \left(W ^ {e} \ln (z)\right)
$$
where  $W^{e} \in \mathbb{R}^{km \times m}$  expands the dimension to  $km$  for  $k \in \mathbb{N}$  and  $W^{(c)} \in \mathbb{R}^{m \times km}$  collapses it back to width  $m$ . For most experiments we set  $k = 4$ . While the additions of skip connections and bottleneck layers to the architecture arguably add some amount of inductive bias, we believe that in comparison to modern architectures such enhancements remain negligible. We will denote this variant by  $B-MLP$ .
Setup In this work, we solely focus on vision tasks as inductive bias is more readily understood in this setting. Moreover, most theoretical works focus on image classification tasks, making it thus a natural test bed to assess the performance of MLPs. We study the popular tasks CIFAR10, CIFAR100 (Krizhevsky, 2009), STL10 (Coates et al., 2011), TinyImageNet (Le and Yang, 2015), ImageNet1k for evaluation, as well as ImageNet21k (Deng et al., 2009) for pre-training. To limit the size of the embedding layer and the computational needs, we downscale images to resolution  $64 \times 64 \times 3$  (if needed) as done in Chrabaszcz et al. (2017). We center and normalize all the images and use random flips and crops as well as MixUp (Zhang et al., 2018) as data augmentations.
In contrast to the MLPs, the ResNet18 was trained at the original image resolutions.
CIFAR10	CIFAR100	STL10	TINY-IN	IN	REAL
B-6/Wi-1024	69.9±0.1	43.0±0.4	51.5±0.1	47.1±0.1	15.2±0.2	20.3±0.2
B-6/Wi-1024 + DA	91.5±0.02	76.4±0.2	85.0±0.2	62.7±0.1	38.7±0.1	47.0±0.15
B-12/Wi-1024 + DA	94.2±0.05	80.0±0.05	89.9±0.1	69.9±0.4	43.3±0.06	48.6±0.2
B-12/Wi-1024 + DA + TTA	95.5±0.05	82.6±0.2	92.2±0.05	73.1±0.5	51.5±0.1	57.9±0.1
Table 2: Fine-tuning Top-1 accuracies (in %) when pretrained on ImageNet21k. Accuracies are averaged over 3 runs. For readability, we abbreviate ImageNet as IN.
4.1 Training from Scratch
We start the empirical exploration of MLPs by training them from scratch (i.e. without any extra data) on popular vision benchmarks. All models were trained with the LION optimizer (Chen et al., 2023) with a learning rate  $\eta = 5\mathrm{e} - 5$ . In order to combat overfitting we use strong label smoothing  $\alpha = 0.3$ . We display the resulting test accuracies in Table 1. We observe that both the standard architecture and the bottleneck without any data augmentation suffer from overfitting, leading to suboptimal performance. When turning it on, data augmentation as a regularizer however really unfolds its full power, significantly pushing the performance by roughly  $20\%$  across all tasks. As observed in Lin et al. (2015), the inverted bottleneck architecture leads to an improvement in performance across all datasets. Learning on the other hand significantly slows down with strong augmentations such as MixUp, enabling training for up to 5000 epochs without suffering from overfitting. However, compared to simple modern baselines such as a ResNet18 (He et al., 2015), a large discrepancy in performance remains, highlighting the importance of inductive bias in the small sample regime. We remark that ViTs and MLP-Mixers as well exhibit more learning difficulties if the dataset size is small (Dosovitskiy et al., 2021; Tolstikhin et al., 2021). We provide more ablation studies in Appendix A.2.
4.2 Transfer Learning
In this section, we aim to analyze how transferable features learnt by MLPs are across different vision tasks. Transferability is one of the hallmark characteristics of modern deep learning, enabling practitioners to fine-tune large models on their specific dataset, leading to superior performance. We are, to the best of our knowledge, the first to measure transferability of MLPs, which is crucial to assess in order to build a theoretical understanding of the process. In this section, we focus on the inverted bottleneck MLP as it generalizes better and is easier to optimize. We provide the dual results for the standard MLP in Appendix B.1. We restrict to  $k = 4$  for the expansion factor and denote by  $B - L / W_i - m$  a network with  $L$  blocks and width  $m$ . For pre-training we use ImageNet21k, the largest publicly available image dataset with annotated classes. After preprocessing the dataset following Ridnik et al. (2021), it consists of roughly 12 million images and 11 thousand classes. We then pre-train the MLP with the cross-entropy loss for 800 epochs, employing label smoothing and the LION optimizer. To guarantee fast data loading we rely on the FFCV framework (Leclerc et al., 2023) for all experiments.
In order to measure transferability of the learnt features we fine-tune the network on the new task. We also study training a linear layer on top of the embeddings but defer those results to Appendix A.3. We again explore the effects of data augmentation during the pre-training stage. For fine-tuning we use SGD with momentum with a learning rate of  $\eta_{\mathrm{head}} = 0.01$  for the head and  $\eta_{\mathrm{body}} = 0.001$  for the encoder for 50 epochs. We upscale CIFAR images to resolution  $64\times 64\times 3$  at fine-tuning time to guarantee compatibility. We display the fine-tuning results in Table 2. For visualizations of the learnt features, we refer the interested reader to Appendix C. We again observe that using data augmentation during the pre-training phase is essential to successful training, boosting performance up to  $30\%$  in case of CIFAR100. Surprisingly, the learnt features are highly transferable, improving the performances reported previously in Table 1 dramatically. While of course pre-trained on a large quantity of data, we nevertheless want to highlight that such an MLP becomes competitive with a ResNet18 trained from scratch for all the datasets, except for ImageNet1k where performance falls surprisingly short. We hypothesize that MLPs struggle with the more fine-grained distinctions between classes, in combination with the reduced resolution of the images.
Figure 4: Linear downstream error on CIFAR100 (in %) when pretrained for varying batch-sizes on ImageNet21k, on a log-log scale.
Model	#parameters
B-6/Wi-256	9M
B-12/Wi-256	12M
B-6/Wi-512	24M
B-12/Wi-512	37M
B-6/Wi-1024	74M
B-12/Wi-1024	124M
Table 3: The different models and the respective parameter counts in millions.