SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can observe that the scale of the embeddings does not significantly impact the MLPs' performance when using monotone activation functions that do not flip signs. This is a non-trivial result, as the nonlinearity of the activation function, the presence of bias terms  $b$ , and the absence of normalization layers make it difficult to predict the effect of an input rescaling on the performance of the network. It is particularly interesting to see that the cosine activation function shows an oscillatory performance, comparable to the original embeddings when rescaled by the mean embeddings scale (vertical red line) or its opposite since it is symmetric.
Our findings indicate that, surprisingly, even the internal layers of large deep learning models exhibit a positive scale invariance, as illustrated in Figure 12. The underlying mechanism for this behavior is not straightforward, but we hypothesize that it may result from the interplay between various factors,
Figure 12: Scale invariance of RoBERTa according to the performance of a downstream classifier trained on the encodings of the last attention layer. At each layer (with 0 being the embedding layer and 12 the output one), one for each run, we rescale the encodings by the specified  $\alpha$  and measure its effect on the final accuracy. The performance without any rescaling is 0.92.
Figure 13: Performance comparison of three Multilayer Perceptrons (MLPs) with different activation functions, namely cosine (blue), ReLU (orange), and tanh (green) at different rescaling factors  $\alpha$ . The ReLU and tanh MLPs exhibit scale invariance, while the cosine activation function is only invariant on the mean data scale and its periodic cycles.
such as the choice of activation function, the use of normalization layers, the optimization objective and regularization techniques employed during the training phase. Further research is needed to understand and explain this phenomenon fully.
All the experiments were conducted using a machine equipped with an Intel Core i7-9700k CPU, 64 GB of RAM, and an NVIDIA 2080TI GPU.
Decoder structure The full implementation details can be found in the attached code. The various experiments can be run by their corresponding notebook, while the source code for the package they are built on can be found under the "src" folder.
- Autoencoding. Since the autoencoders were used only on image data, the architecture was a simple sequence of convolutions (in the encoder part) and deconvolutions (in the decoder part). Each interleaved with nonlinear activations.
- Classification. The main manuscript refers to "SVM" as the standard SVM implementation in scikit-learn [Pedregosa et al., 2011], with default parameters. The experiments with "MLP" as a classifier refer to a simple stack of 3 linear layers, interleaved by nonlinear activations.
Software and Technologies The research of this study was facilitated by the use of various technologies and tools, which include:
- NN-Template [GrokAI, 2021], was used to kick-start the project while also ensuring best practices were adhered to;
- DVC [Kuprieiev et al., 2022], was implemented for data versioning;
- PyTorch Lightning [Falcon and The PyTorch Lightning team, 2019], contributed to maintaining the integrity of the results and promoting clean, modular code;
- Weights and Biases [Biewald, 2020], were employed for logging experiments, running comparisons over extensive sweeps, and sharing models;
- Transformers by HuggingFace [Wolf et al., 2020], provided pre-configured transformers for processing both image and text data;
- Datasets by HuggingFace [Lhoest et al., 2021], facilitated access to a majority of NLP datasets and ImageNet for computer vision purposes;
Pre-trained encoders All the pre-trained encoders used come from HuggingFace and are listed in Table 7. They are various both in terms of architecture and encoding size.
Table 7: HuggingFace models used as encoders (feature extractors) in the various experiments, with their encoding dimensionality.
Modality	HuggingFace model name	Encoding Dim
Language	bert-base-cased	768
bert-base-uncased	768
google/electra-base-discriminator	768
roberta-base	768
albert-base-v2	768
xlm-roberta-base	768
openai/clip-vit-base-patch32	768
Vision	rexnet_100	1280
cspdarknet53	768
vit_small_batch16_224	384
vit_base_batch16_224	768
vit_base_batch16_384	768
vit_base_resnet50_384	768
openai/clip-vit-base-patch32	768
Table 8: Cross-architecture stitching for reconstruction tasks. 5 different seeds, 2 different bottleneck sizes (250, 500) for the same architecture. Average over all combinations. 500 anchors used and standard scaling as normalization. The naive absolute baseline is impossible to compute due to the dimensionality mismatch.
MNIST	Fashion MNIST	CIFAR10	CIFAR100
lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse	lcos	lmse	rmse
affine	0.95	0.09	0.02	0.95	0.09	0.04	0.98	0.06	0.05	0.98	0.07	0.06
linear	0.64	1.00	0.11	0.66	1.10	0.16	0.77	0.60	0.16	0.78	0.52	0.16
l-ortho	0.87	0.16	0.03	0.89	0.14	0.06	0.95	0.12	0.08	0.95	0.13	0.08
ortho	0.91	0.14	0.03	0.92	0.13	0.06	0.96	0.12	0.09	0.96	0.12	0.09