## 7 Results
### Training Longer  
Training for longer is beneficial. As can be seen in Figure 2, testing the alpha-word accuracy for all our models is only increasing with training. The lack of training hurts the performance of the Vietnamese language the most, which is the language with the most diacritics. Training the corresponding model for eight times longer brings substantial improvements of over 3.28%.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Language**} & \multicolumn{2}{c}{**With Diacritics, \%**} & \multicolumn{2}{c}{**Without Diacritics, \%**} & \multirow{2}{*}{**Total Unseen**} \\ \cline{2-2} \cline{5-6}  & **Failed** & **Restored** & **Failed** & **Left Correct** & \\ \hline Croatian & 6.8 \(\pm\) 0.2 & 15.9 \(\pm\) 0.2 & 4.3 \(\pm\) 0.4 & 73.0 \(\pm\) 0.4 & 12,147 \\ Czech & 16.4 \(\pm\) 0.2 & 37.7 \(\pm\) 0.2 & 5.0 \(\pm\) 0.4 & 40.9 \(\pm\) 0.4 & 9398 \\ French & 10.1 \(\pm\) 0.1 & 9.8 \(\pm\) 0.1 & 4.6 \(\pm\) 0.3 & 75.6 \(\pm\) 0.3 & 3794 \\ \hline Hungarian & 10.1 \(\pm\) 0.2 & 58.2 \(\pm\) 0.2 & 2.3 \(\pm\) 0.1 & 29.5 \(\pm\) 0.1 & 16,350 \\ \(\times\)8 \#samples & 7.1 & 61.2 & 1.2 & 30.5 & 16,350 \\ \hline Irish & 12.1 \(\pm\) 0.5 & 25.6 \(\pm\) 0.5 & 5.3 \(\pm\) 1.0 & 57.0 \(\pm\) 1.0 & 29,470 \\ \hline Latvian & 16.8 \(\pm\) 0.7 & 39.9 \(\pm\) 0.7 & 5.5 \(\pm\) 0.6 & 37.9 \(\pm\) 0.6 & 17,449 \\ \(\times\)8 \#samples & 13.6 & 43.1 & 3.9 & 39.4 & 15,547 \\ \hline Lithuanian & 8.1 \(\pm\) 0.4 & 29.7 \(\pm\) 0.4 & 4.4 \(\pm\) 1.4 & 57.7 \(\pm\) 1.4 & 15,547 \\ \(\times\)8 \#samples & 6.0 & 31.9 & 2.8 & 59.3 & 15,547 \\ \hline Polish & 7.0 \(\pm\) 0.1 & 20.4 \(\pm\) 0.1 & 2.3 \(\pm\) 0.2 & 70.3 \(\pm\) 0.2 & 9461 \\ Romanian & 15.6 \(\pm\) 0.5 & 15.1 \(\pm\) 0.5 & 6.9 \(\pm\) 0.9 & 62.4 \(\pm\) 0.9 & 8493 \\ Slovak & 14.4 \(\pm\) 0.3 & 33.7 \(\pm\) 0.3 & 4.9 \(\pm\) 1.6 & 46.9 \(\pm\) 1.6 & 13,357 \\ Spanish & 8.1 \(\pm\) 0.2 & 11.6 \(\pm\) 0.2 & 5.4 \(\pm\) 1.0 & 75.0 \(\pm\) 1.0 & 5115 \\ Turkish & 7.0 \(\pm\) 0.1 & 25.3 \(\pm\) 0.1 & 3.7 \(\pm\) 0.2 & 63.9 \(\pm\) 0.2 & 9594 \\ \hline Vietnamese & 14.4 \(\pm\) 0.3 & 1.7 \(\pm\) 0.3 & 1.9 \(\pm\) 0.4 & 82.1 \(\pm\) 0.4 & 4260 \\ \(\times\)8 \#samples & 13.1 & 2.9 & 2.7 & 81.2 & 4260 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The confusion matrix of the unseen word diacritics restoration performance by the ByT5 model. Unseen words with and without diacritics are presented separately. The last column depicts the total number of unseen words for each language.  
Figure 2: Alpha-word accuracy improvement during diacritics restoration training. Training steps of \(\times 1\) corresponds to 2048 \(\times\) 256 sentences for a given language. There is a visible outlier for the Turkish language at \(\times\)0.5 training steps, but that model regained the accuracy later in training.  
A similar trend is observed for all the models trained on the two tasks simultaneously in Figure 3. Here, the improvements are much larger. On the other hand, languages with fewer diacritics, such as French and Spanish, have diminishing gains from longer training. Overall, longer training is a must for the more difficult tasks.  
Note that while the training is much longer, we still use the same dataset sizes presented in Table 1, but we just iterate over them more times.