## 6 Experiment Details
### ByT5 Model Fine-Tuning  
We chose the batch size of 256 and the default ByT5 maximum sequence length of 1024. Such a configuration matches the total maximum number of tokens (\(256\times 1024=2048\times 128\)) with the best system for diacritics restoration [46]. The larger sequence length is essential, as our model works on byte-level fine-grained tokens, compared to coarser subword-level models.  
We used a GeForce RTX 2080 Ti GPU. Due to the modest memory size, we employed the gradient accumulation technique. It accumulates gradients in a continuous, rather than in a parallel, fashion. In addition, feeding only a single sample at a time allowed us to avoid padding.  
We trained each model for 2048 steps, each consisting of 256 sentences/samples, with a total of \(2048\times 256=524\),288 sentences, and this took up to 10 h for a single model. For example, for the Lithuanian language, this corresponds to a 0.86 epoch over the total 612,724 sentences in its dataset (Table 1). In our results, we refer to such basic training as being trained for \(\times 1\) the number of sentences (#samples). We fix this training length, irrespective of the available dataset, for each language (Table 1) to make training comparable among languages. In experiments where we trained our models for longer (e.g., \(\times 8\)), we used the whole dataset and passed through it as many times as needed, e.g., for Lithuanian \(\times 8\) corresponds to 6.8 epochs.  
We used the Adafactor [104] optimizer with a constant learning rate of 0.001. The same setup was employed by the ByT5 [8] authors for fine-tuning experiments. Moreover, the Adafactor optimizer also has very little auxiliary storage compared to the other popular optimizer, Adam [103]. More complex learning rate schedules may give a slightly better  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **Number of Edits** & **Collection Method** \\ \hline GitHub Typo Corpus [113] & 350,000 & Keyboard \\ Twitter Typo Corpus [115] & 39,171 & Keyboard \\ Birkbeck Spelling Corpus [116] & 36,133 & Handwritten \\ Holbrook Misspelling Corpus [117; 118] & 1,791 & Handwritten \\ \hline \hline \end{tabular}
\end{table}
Table 2: Related datasets for English misspelling corrections.  
performance, but it would be more difficult to compare our runs, so we adhered to the constant learning rate approach.  
For the diacritics restoration task with each language, we trained three different models. Each model has a different weight initialization, and data sampling is performed differently, according to a given random seed. The results are reported as a mean and a standard deviation over these three runs. In addition, we trained models for simultaneous diacritics and typographical error corrections for each language.  
We also trained several models for a much longer time. First, we continued our basic fine-tuning setup with a batch size of 258 to 6000 steps (all other basic setups are up to 2048). At this stage, the loss became noisy (although it was low), so we increased our batch size to 8192 and continued training further. Due to the change in batch size, we reported our model training steps by how much training data, compared to our basic setup, it consumed. In our results, we reported models trained for \(\times\)8 and \(\times\)19 the number of samples in the basic setup. We chose those ceiling-rounded numbers as a means of convenience in our setup. As long training is very time-consuming, we performed only a few of them. We think that it still sufficiently indicates the scaling effects.  
For text generation, in all our experiments we used a beam size of two. Later runs revealed that there is hardly a difference in size. As a result, for future work, we recommend adhering to a simpler beam size of 1.  
The training script and the Pytorch model implementation were used from the Hugging Face library [95]. If not stated otherwise, we used all default parameters as they are in this library version 4.12.0.