SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Test-Time Augmentations. For ImageNet1k we further notice that objects tend to not be centered, in contrast to datasets like CIFAR10. We suspect that this might lead to the comparatively weaker performance. To test this, we leverage test-time augmentations (TTA). As introduced by Krizhevsky et al. (2012), for each test image, we produce a fixed number of 100 random crops and use the averaged logits for prediction. We observe significant improvements across all datasets, especially for ImageNet we obtain an increase of roughly  $8\%$ . This indeed indicates that MLPs struggle to localize the object of interest, especially for the more complicated ImageNet1k task. Using a large number of crops alleviates this problem to some degree. This also explains why the gains on tasks like CIFAR10 are smaller as the objects there usually are perfectly centered.
ReaL accuary. As observed in (Beyer et al., 2020), the ImageNet labels do not capture that a single image might contain multiple objects of distinct classes. ImageNet accuracy can thus be misleading in the sense that model classes such as convolutional networks might have implicitly adapted to the particular labeling strategy due to the repeated benchmarking on the same validation set. MLPs most likely lack such an implicit adaptation as this work is to our knowledge the first to evaluate them on ImageNet1k. To address this, Beyer et al. (2020) introduced a novel set of validation labels that better capture the multi-label nature, where a prediction is deemed correct if it matches one of the categories present in the image. We observe further very significant improvements of  $\approx 7\%$  when employing ImageNet ReaL.
Overall, these results underline that a bad inductive bias as exhibited by an MLP can indeed be overcome if subjected to enough scale. For theory, the results are double-edged; while MLPs prove to be a good proxy to understand transfer learning, data augmentation proves to be a crucial component. Also test-time augmentations significantly boost performance. Both these components on the other hand remain rather understudied in theoretical works.
Large batch-sizes. We further make the counter-intuitive observation that training with larger batch sizes significantly boosts performance both up- and downstream. In Fig. 4 we plot pre-training batch size against resulting linear downstream accuracy on CIFAR100 for different number of pre-training epochs. We observe that across all training times, using a larger batch size leads to significantly better performance. Moreover, we want to highlight that such a plot is even favoring small batch-sizes since those models perform more gradient updates for a fixed number of epochs. This effect is in stark contrast to convolutional architectures where entire lines of works have focused on preserving the performance of the small batch-size regime for larger ones (Goyal et al., 2017; You et al., 2017; Hoffer et al., 2017; Keskar et al., 2017). Training with large batch-sizes without degradation is of high interest as it can lead to potentially more efficient training pipelines since computation can be sharded among more devices. This observation about optimal batch-sizes is in-line with similar recent conclusions in Transformers (Kaplan et al., 2020; Touvron et al., 2023).
Role of augmentations. Data augmentation is very pronounced for MLPs, providing indirect inductive bias to the model. Remarkably, a model pre-trained on 12 million examples without data augmentation shows inferior performance on CIFAR10 compared to a network trained from scratch
with augmentations turned on. This emphasizes that augmentations go beyond merely leading to a bigger dataset but provide the model with useful invariances. We investigate the learnt weights in Appendix C, showing very evidently, that more localized features are learnt if data augmentation is employed. The power of augmentations has already been demonstrated through the advent of self-supervised learning (Grill et al., 2020; Caron et al., 2021; Chen et al., 2020). Even when training on purely random labels, it still provides powerful learning signals (Anagnostidis et al., 2023).
4.3 Scaling Laws
One of the key mysteries in deep learning is that networks tend to improve in terms of generalization when compute, in the form of parameter count and dataset size, is scaled up. Recently it has been observed in several works that the benefits of scale are highly predictable, i.e. generalization performance exhibits a power-law structure when plotted against compute measured in FLOPS (Rosenfeld et al., 2020; Hestness et al., 2017, 2019; Kaplan et al., 2020; Zhai et al., 2022). The functional form has recently been further refined (Caballero et al., 2023). The predictable nature of test performance has even been leveraged to estimate the optimal model before training (Hoffmann et al., 2022; OpenAI, 2023). In order to understand this important characteristic of deep learning theoretically, it is important to analyze whether MLPs exhibit similar properties.
Figure 5: Test error (in %) on CIFAR10 (left) and ImageNet1k (right) when linearly transferred as a function of PFLOPS, measured according to Eq.(4), on a log-log scale.
**Compute.** Following OpenAI (2018) we define the computational cost  $C$  incurred from training a model  $f$  on  $N$  examples for  $T$  epochs as
$$
C = \operatorname {F L O P} (f) \times 3 \times N \times T, \tag {4}
$$
where  $\mathrm{FLOP}(f)$  denotes the number of FLOPs needed to complete the forward pass of  $f$  for a single example. We note that the number of parameters  $P$  present in  $f$  enters this equation implicitly in the form of  $\mathrm{FLOP}(f) \propto P$ . Observe that a given level of compute can be achieved in different ways, i.e. using more parameters  $P$ , training on more examples  $N$ , or training for a longer time  $T$ . When allocating a given level of compute optimally, it is observed that for convolutional and transformer-based architectures, the test error  $E(C)$  as a function of compute behaves as a power-law
$$
E (C) = a (b + C) ^ {- \alpha} + E _ {\infty}, \tag {5}
$$
where  $a, b, E_{\infty} \in \mathbb{R}_{+}$  and  $\alpha > 0$  is the scaling coefficient determining the rate of decay.  $E_{\infty}$  denotes the irreducible error, i.e. even if infinite compute were employed, the performance remains imperfect. The test error can be measured upstream (i.e. on the pre-training task) or downstream when fine-tuning on a different task. We investigate various pre-training schemes with different number of examples, parameter counts and training times. We subsample ImageNet21k proportionally across classes and pre-train variously sized inverted bottleneck MLPs. We summarize the configurations in Table 3. We then measure test error on the downstream task of CIFAR100 in Fig. 1 as well as CIFAR10 and ImageNet1k in Fig. 5 by linearly transferring the learnt features (without test-time augmentations). The plotting style is inspired by Zhai et al. (2022). Each point in the curve is the downstream performance of an MLP, where the color of the point indicates the model type (blue denotes smaller
Figure 6: Power law in linear evaluation error on CIFAR100 (in %) when either bottlenecked by the number of parameters (left) or the number of examples (right), on a log-log scale. The dotted line visualizes the fitted functional form.
and red larger models) and the size of the point indicates the number of pre-training examples. Points connected by a line indicates longer training times where  $T \in \{50, 100, 200, 400, 800\}$  is measured in epochs. In all experiments, we employ data augmentation for pre-training. We observe that the compute-optimal performance of MLPs strongly exhibits characteristics of a power-law with coefficients  $\alpha \in \{0.12, 0.25, 0.35\}$ . This is very encouraging for future theoretical work, showing that MLPs indeed mirror the scaling behaviour of modern models. We provide the dual results for the standard MLPs in Appendix B.2, noting that they exhibit essentially the same scaling behaviour, albeit with a slightly weaker slope and intercept.
We further study how performance  $E$  evolves when compute is either bottlenecked by the number of parameters  $P$  or the dataset size  $N$ . We visualize the resulting scaling laws in Fig. 6. We find a very steep decay rate in terms of parameters  $P$  where roughly  $\alpha_{P} \approx 1$ , whereas for dataset size  $N$  we identify a significantly slower rate of  $\alpha_{N} \approx 0.35$ . This shows that the performance of MLPs is significantly more limited by the dataset size, which is in-line with the fact that MLPs exhibit a bad inductive bias. We investigate the role of dataset size and parameters more in the next paragraph.
Parameters or examples. Given a fixed level of compute  $C$ , what is the optimal way to allocate it to parameter count  $P$  and number of examples  $N$ ? To be more comparable to previous work, we assume a fixed training time  $T = 50$ . To answer this question, we follow the approach outlined in Hoffmann et al. (2022) and plot the optimal compute models identified in Fig. 1 both against model size  $P$  and number of examples  $N$  and visualize the results in Fig. 7. We empirically observe that the optimal parameter count  $P^{*}(C)$  and dataset size  $N^{*}(C)$  as a function of compute  $C$  exhibit power-law behaviour of the approximate form
$$
P ^ {*} (C) \propto C ^ {0. 3 5} \quad N ^ {*} (C) \propto C ^ {0. 6 5}
$$
While for transformers, the number of examples (or tokens)  $N$  and parameters  $P$  are scaled equally (Hoffmann et al., 2022) (i.e.  $\alpha_{P} \approx \alpha_{N} \approx 0.5$ ), in contrast we observe that the optimal strategy for MLPs invests significantly more compute into dataset size  $N$ . This is further evidence for the weaker inductive bias present in MLPs, which needs more examples in order to be compensated for.
4.4 Computational Feasibility
We believe that a further exciting feature of our study is its computational feasibility, while at the same time preserving the main characteristics of large-scale pre-training. All of our experiments were conducted on a single NVIDIA RTX A5000 GPU with 24GB of memory. In conjunction with the strongly optimized FFCV dataloading framework (Leclerc et al., 2023) and the inherent efficiency of MLPs, we are able to perform very rapid training. For instance we complete a single epoch on ImageNet21k with the  $B - 12 / \mathrm{Wi} - 1024$  architecture, equipped with 124 million parameters, in only roughly 450 seconds, while the smaller variant  $B - 6 / \mathrm{Wi} - 1024$  at a parameter count of 74 million requires roughly 250 seconds on the specified hardware. Low memory requirements allow us to train with a batch-size of 16384 without having to shard computation among multiple GPUs. We
Figure 7: Optimal model size (left) and number of examples (right) for a given level of compute for linear evaluation on CIFAR100, on a log-log scale.
compare the computational efficiency of MLPs with contemporary networks of similar size such as ResNet-152, ViT-B/4 and ViT-B/8 in Appendix A.5.
There are some prior works that investigate MLPs on vision tasks. Lin et al. (2015) study the performance of MLPs on small scale datasets such as CIFAR10. They observe similar improvements when using inverted bottleneck layers but do not study larger-scale setups, transfer-learning nor do they discuss the implications for theoretical works. The bottleneck structure used in this work has also been investigated theoretically (Parhi and Nowak, 2021; Shenouda et al., 2023; Parkinson et al., 2023), further highlighting that such an architecture exhibits desirable properties. Urban et al. (2017) study to what degree convolutions are necessary for good performance and conclude that even with distillation techniques it remains very difficult to train performant MLPs on CIFAR10. Other approaches have focused on sparsifying fully-connected layers through evolutionary training (Mocanu et al., 2018; Fernando et al., 2016), aiming to learn a good inductive bias from scratch. Similarly, Neyshabur (2020) study how the inductive bias of MLPs can be improved by systematically sparsifying them with a LASSO-type algorithm, making them more convolution-like. d'Ascoli et al. (2019) on the other hand first train a convolutional network for a certain duration and then subsequently continue training the network as an MLP (by using the correspondence between CNNs and MLPs highlighted in Sec. 2). They show that good performance can be reached if the network was trained long enough as a CNN. In contrast to these works, our goal is not to enhance the inherent inductive bias of MLPs but study whether it can be overcome with enough scale.
The advent of the MLP-Mixer (Tolstikhin et al., 2021) has led to a series of follow-up work, similarly using MLPs as a patch processor and token mixer (Touvron et al., 2021; Chen et al., 2022; Lian et al., 2022; Guo et al., 2021; Liu et al., 2021). Again, we remark that these architectures all possess significantly more inductive bias.
Finally, we would like to remark that MLPs are successfully used in other areas such as novel view synthesis (e.g. NeRF (Mildenhall et al., 2021)).
In this work, we have explored the limits of the multi-layer perceptron as an architecture for vision tasks. Our study reveals that (1) lack of inductive bias can be compensated by scale and (2) MLPs constitute a (largely) accurate proxy for modern architectures, further cementing their role as the main theoretical object of study. The role of data augmentation and the implicit bias of SGD however strongly differ for MLPs in the setting considered in this work and theoretical works should take this into account. Large-scale pre-training of MLPs proves to be very efficient, enabling researchers with less access to computational resources to study this very exciting line of work. While lack of inductive bias does not prevent MLPs from reaching impressive performance, it leads to an interesting shift in compute-optimality towards more training examples. Subjecting MLPs to even larger amounts of compute similar to Zhai et al. (2022), especially in the form of more training examples, remains as very interesting future work.
Soudry, D., Hoffer, E., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. In International Conference on Learning Representations.
Tan, M. and Le, Q. V. (2020). Efficientnet: Rethinking model scaling for convolutional neural networks.
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A. P., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. (2021). MLP-mixer: An all-MLP architecture for vision. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems.
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., and Jegou, H. (2021). ResMLP: Feedforward networks for image classification with data-efficient training.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Trockman, A. and Kolter, J. Z. (2022). Patches are all you need?
Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Mohamed, A., Philipose, M., Richardson, M., and Caruana, R. (2017). Do deep convolutional nets really need to be deep and convolutional? In International Conference on Learning Representations.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, I., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cirmrnan, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272.
You, Y., Gitman, I., and Ginsburg, B. (2017). Large batch training of convolutional networks.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. (2022). Metaformer is actually what you need for vision.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2022). Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12104-12113.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Zou, D., Cao, Y., Zhou, D., and Gu, Q. (2020). Gradient descent optimizes over-parameterized deep relu networks. Machine Learning, 109:1-26.
A.1 Resources
For all experiments we rely on NVIDIA RTX A5000 GPU with 24GB of memory. Every experiment can be performed on a single GPU. We leverage the FFCV dataloader framework since the transfer time of the data to the GPU becomes the bottleneck in terms of training time in case of MLPs. All of our experiments were performed in PyTorch (Paszke et al., 2019).
A.2 Additional Ablations
Ablations. We provide some more ablations in Fig. 8. More specifically, for a (approximate) fixed budget of compute, we investigate different architecture and optimization choices, when pretraining on ImageNet1k and performing linear probing on CIFAR100.
Figure 8: Ablations on different architectures and optimizations choices when training on ImageNet. Numbers indicate linear probing Top-1 accuracies on CIFAR100.
Normalization. We investigate the importance of the normalization method (LayerNorm vs BatchNorm) in more detail in Table 4. We pre-train two B-MLPs on ImageNet21k with layer normalization and batch normalization and compare the fine-tuning performance on various tasks. We find that the techniques perform similarly, which layer normalisation having a slight edge.
CIFAR-10	CIFAR-100	TINYIMAGENET	IMAGENET
LAYERNORM	90.0	74.6	59.6	36.2
BATCHNORM	89.4	73.8	57.7	35.9
Table 4: Pretraining a B-6/Wi-1024 B-MLP with BatchNorm and LayerNorm on ImageNet21k and subsequently fine-tuning.
Label smoothing. We further ablate the influence of label smoothing on the downstream performance. We pre-train B-MLPs with varying amounts of label smoothing ( $\alpha \in \{0.0, 0.1, 0.3\}$ ) and evaluate the resulting down-stream fine-tuning performance. We report the results in Table 5. While label smoothing does provide some boost in performance, the gains are very modest. Label smoothing is thus helpful but not essential for training MLPs.
CIFAR10	CIFAR100	TINYIMAGENET	IMAGENET
α = 0.3	90.0	74.6	59.6	36.2
α = 0.1	89.5	73.7	58.2	36.0
α = 0.0	89.2	72.2	57.1	35.7
Architecture. We make the following observations/recommendations to boost the model's performance, in line with results reported in the literature (Liu et al., 2022); (1) replacing ReLUs and GELUs boosts results significantly, (2) adding skip connections every two layers helps with optimization, especially for deeper networks. (3) Using an inverted bottleneck increases performance even more. (4) Using a normalization layer in the PRE-LN configuration helps with optimization and (4) layer normalization leads to significantly better results compared to batch normalization, while also being more stable during training.
**Optimization.** As discussed in the main text, augmentations are crucial, and disabling them can have a detrimental effect. We also found that clipping gradients, using weight decay and dropout have a small positive effect on downstream performance. Finally, replacing LION (Chen et al., 2023) with Adam(W), leads to a decrease in performance.
We showcase the transferability of our MLPs by training a linear classifier on top of the frozen features. For training the linear layer, we use the LION optimizer with a learning rate of  $\eta = 0.00001$  for 50 epochs. We display the results in Table 6. We observe very strong down-stream performance
Table 5: Pretraining a B-6/Wi-1024 B-MLP with different amounts of label smoothing on ImageNet21k and subsequently fine-tuning.
CIFAR10	CIFAR100	STL10	TINYIMAGENET	IMAGENET
B-6/Wi-1024	65.1	41.3	53.4	45.6	13.0
B-6/Wi-1024 + DA	87.8	73.2	85.2	61.3	39.2
B-12/Wi-1024 + DA	90.6	74.5	88.3	68.5	40.7
Table 6: Linear probing Top-1 accuracies when pretraining on ImageNet21k.
even in this more limited setting, highlighting how transferable the features learnt by MLPs are.
A.4 Scaling Laws
Implementation Details. For the scaling law plots, we trained all the models with a batch-size 16384 and the LION optimizer with a learning rate  $\eta = 0.00001$  and weight decay of strength 0.001. We further use label smoothing of strength 0.3. We again use augmentations in the form of random flips and crops as well as MixUp with strength 0.8. We rely on the curvefit function from the SciPy library (Virtanen et al., 2020) to fit powerlaws of the form  $E(C) = a(b + C)^{-\alpha} + E_{\infty}$ .
A.5 Computational Efficiency
We highlight the fact that although MLPs require a lot of training data, inference is extremely efficient from a computational perspective. To illustrate this, we embark on the following comparison; we study inference on  $64 \times 64$  resolution images in an MLP vs other popular vision architectures of similar size and complexity in Table 7. More specifically, we compare against a ResNet-152, where we replace the stride in the first convolutional layer and remove the first max-pooling operation to
compensate for the smaller image size. We also compare against a base ViT and Mixer model, where we extract patches from  $4 \times 4$  regions in the original image.
As it quickly becomes eminent, MLPs require significantly less FLOPs to make predictions on individual images, in essence utilizing their parameters a lot more methodically. As a result, latency and throughput are significantly better compared to other candidate architectures. We measure throughput using the optimal batch size on an NVIDIA RTX A5000. We highlight, that our MLPs, in contrast to the other architectures are memory bound, meaning that their throughput is determined by the prefetching bandwidth of our GPU. Hardware advancement and specialized architectures could significantly mitigate this effect. Neglecting memory transfer time by propagating the same input through our network gives a further 6-fold increase in the potential throughput.
PARAMETERS	LATENCY (MSEC)	THROUGHPUT (IMAGES/SEC)	FLOPS PER FORWARD PASS
B-12/Wi-768	66.89 M	21.2	16063	66.8 M
ResNet-152	60.19 M	423	506	13.07 G
ViT-B/4	86.06 M	424	222	23.08 G
Mixer-B/4	63.82 M	400	319	19.36 G
B Results for Standard MLPs
B.1 Transfer Learning
For completeness we also analyze the transfer performance of standard MLPs when pre-trained on ImageNet21k. We compare a S-MLP of depth 6 and width 2048 against a B-MLP of depth 6 and width 1024, ensuring that both models roughly have a parameter count of around  $\approx 70$  million. We display the results in Table 8. We observe that even the features of a standard MLP (i.e. without residual connections and bottleneck structure) transfer very well on different downstream task. The inverted-bottleneck MLP however still remains superior.
Table 7: Various measures assessing the computational efficiency of different architectures.
CIFAR10	CIFAR100	TINYIMAGENET	IMAGENET
S-MLP	87.1	68.3	52.1	30.2
B-MLP	90.0	74.6	59.6	36.2
Table 8: Comparing a S-MLP of width 2048 and depth 6 pre-trained on ImageNet21k, with a B-6/Wi-1024 B-MLP (both models around 70M params) in terms of fine-tuning performance
B.2 Scaling Laws