SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Evaluation perplexity
Figure 7: We train GPT-2 Small on the wikitext-103-raw-v1 dataset from scratch.
36 data points. We report the average of these 36 experiments in Table 3. We conduct inference on a single A100 with automatic mixed precision (amp) in PyTorch. We further change the precision (disabling the amp), computation platforms (16 threads in AMD EPYC 7742 CPUs), and machine learning frameworks (JAX).
D.3 Numerical Issue
The theoretical arithmetic equivalence cannot be fully translated into equality in practical numerical computation if we use floating numbers. An intuitive example is that  $\mu (\pmb {x} + \pmb {y}) = \mu (\pmb {x}) + \mu (\pmb {y})$  always holds for any vectors  $\pmb {x},\pmb{y}$ . However, if these two vectors are represented as (low precision) floating numbers, this equality is not guaranteed in real-world numerical computation. It is possible that these small discrepancies may be accumulated and enlarged in the large models, further degrading the numerical stability. In our proposed method, we cannot ensure the exact zero-mean in the main branch numerically.
The numerical issue is a common problem in machine learning. A typical example is operator reordering and layer fusion. PyTorch provides a related API officially, named torch.io. quantization.fuse Modules. We can fuse the convolution layer and its following batch normalization layer to simplify the computation. These two layers are separate in training and can be fused to accelerate the inference. The fusion does not break the arithmetic equivalence but changes the numerical results. In spite of the numerical difference, the fusion usually has a neutral impact on task-related performance, such as classification accuracy, even in large models. Fine-tuning or calibration may be helpful in case there is severe performance degradation.
Our proposed methods encounter a similar issue as layer fusion since we modify partial parameters. In our experiments, we can convert the pre-trained Pre-LN ViT-H/14 into Pre-(C)RMS variants without any accuracy change on the ImageNet validation dataset. We observe that replacing PyTorch's official LayerNorm implementation with the apex may have a larger impact on the model performance.
D.4 Verification of Training Equivalence
For validation and reference, we train ViT-S/16 as Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm models on ImageNet-1k following the DeiT-3 setting and achieve similar accuracy. The test accuracy over 800 epochs is shown in Figure 6. Remarkably, these Transformer variants yield similar training performance. We also train three variants of GPT-2 Small (12 layers, 768 hidden dimensions) on the wikitext-103-raw-v1 dataset [23] from scratch. The training and evaluation losses are demonstrated in Figure 7. These performance curves exhibit remarkable similarity, providing empirical evidence for their training equivalence.