SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also evaluate the scaling law of standard MLPs by training variously sized models on different subsets of ImageNet21k and subsequently linearly probing the features on CIFAR100. The setting is identical to the one described in 4.3. We observe that also standard MLPs exhibit power-law behaviour. The slope (0.22 vs 0.25) and the intercept (0.18 vs 0.16) are however worse when compared against the inverted-bottleneck MLP.
C Weight visualizations
We visualize the first layer weights  $\mathbf{W}^{(1)} \in \mathbb{R}^{3wh \times m}$  by reshaping them back to  $\mathbb{R}^{w \times h \times 3 \times m}$ . We then produce a  $\mathbb{R}^{w \times h \times m}$  representation by taking the maximal value along the channel dimension.
Figure 9: Test error of standard MLPs on CIFAR100 when linearly transferred as a function of PFLOPS, measured according to Eq.(4), on a log-log scale.
Figure 10: Visualization of the first layer weights for different pre-training dataset sizes.
We display such visualizations of the first  $5 \times 5 = 25$  "filters" for different pre-training sizes, also including the weights at random initialization in Fig. 10. All models were trained with data augmentation. We observe that filters increasingly develop structure as we increase the dataset size and become more and more localized. We further compare against models that were pre-trained on the full ImageNet21k, with and without data augmentation in Fig. 11. We observe that even though we provide the model with an abundance of samples, the weights still remain largely structure-less
Figure 11: Visualization of the first layer weights for models trained with and without data augmentation.
and have not developed any locality properties. On the other hand, using data augmentation leads to more adapted filters.
D Inverted Bottleneck MLP Code
We provide PyTorch-style pseudo-code for the inverted bottleneck MLP to highlight its simplicity.
from torch import nn   
2   
3 class Block(nnModule):   
4 def __init__(self, dim, expansion_factor  $= 4$  , dropout  $= 0$  ):   
5 super().__init_.()   
6 self(fn = nn Sequential(   
7 nn.Linear(dim, int(expansion_factor \* dim)),   
8 nn.GELU(),   
9 nnDDDropoutdropout),   
10 nn.Linear(int(expansion_factor \* dim), dim),   
11 nnDDDropoutdropout)   
12 ）   
13 self.ln  $=$  nn.layersNorm(dim)   
14   
15 def forward(self,x):   
16 return x + self.fn(self.ln(x))   
17   
18   
19 def MLP(image_size, channels, dim, depth, num_classes,   
expansion_factor  $= 4$  , dropout  $= 0$  ):   
20 return nnSequential(   
21 nn Flatten(start_dim  $= 1$  , end_dim  $= -1$  ),   
22 nn.Linear(image_size \* image_size \* channels, dim),   
23 \*[Block(dim,expansion_factor,dropout)for_in range(depth) ],   
24 nn.Linear(dim,num_classes)   
25 ）