SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 26: The BLEU score plot paints a better picture for pre-training than the cross-entropy plot (Figure 1), since pre-training was able to improve the En-Zh BLEU score to be on par with the score of joint training for 600k steps. Results are with 3-layer models.
Figure 27: Our proposed pre-training scheme improves upon the best BLEU score for  $\mathrm{En} \rightarrow \mathrm{Ro}$  without pre-training for both the 3-layer models (left) and 6-layer models (right).
Figure 28: Our proposed pre-training scheme improves upon the best BLEU score for  $\mathrm{En}\rightarrow \mathrm{Hi}$  without pre-training for both the 3-layer models (left) and 6-layer models (right). The improvements are more substantial than for  $\mathrm{En}\rightarrow \{\mathrm{Ro},\mathrm{Fr}\}$ .
B Additional Training Details in Multilingual Training
We use an additionally processed version of the mC4 [32] dataset as proposed in [6] (documents with language ID confidence below 0.95 were filtered).
The model architectures used are the same as mT5 models [32], except that relative position embeddings are not shared across layers. We also use the number of real target tokens as the effective loss normalization instead of using a loss normalization factor.
We use SentencePiece tokenization [19] to generate a vocabulary of size 64,000. The corpus used to generate the vocabulary is sampled from the training data using temperature sampling with  $\tau = 3.33$ .
We use the T5X library [25] to train the models. For all experiments, we use the Adafactor optimizer [27], where we use momentum, and we do not factorize the second moment of the Adafactor states. The baseline run without fine-tuning, and the pre-training phase of our proposed method, was run with a constant learning rate of 0.01 in the first 10,000 steps and inverse square root decay afterwards. For the fine-tuning phase of our method, we reset the optimizer state, and do a 10,000-step linear warmup with inverse square root decay afterwards.
C Discussion on Sampling Rate Schedules
From our preliminary experiments on using schedules for the sampling rates in the NMT workloads, we find that the learning rate schedule must be tuned accordingly, which affects the overall performance of the run. For example, we find that cosine decay schedule performs better than inverse square root decay for scalarization. However, if we use cosine learning rate decay in conjunction with linear sampling rate decay (used by DDS, and defining sampling rate to be for the high-resource language-pair), by the time the sampling rate for low-resource task is high enough, the learning rate has decayed rapidly (by nature of cosine decay), resulting in little learning for the low-resource task. Using inverse square root learning rate decay solves this issue, but this results in overall worse performance due to the suboptimal learning rate schedule. In contrast, our method is free to use any scheduler that maximizes performance in each leg of training (pre-training and fine-tuning). Lastly, when tuning hyperparameters, using dynamic sampling rates requires executing the full training run many times. On the other hand, for our method, we can focus our resources on tuning the fine-tuning phase, (since the pre-training phase has only one task, and is an easier optimization problem) which is shorter than the total training time.