## 5 Dataset
### A Realistic Model of Typos  
We produce our pairs of correct (target) and incorrect (input) texts by taking the dataset as the correct (gold) text and by generating the corresponding incorrect text automatically.  
The diacritic removal is straightforward, and is simply done by replacing all diacritic letters with the non-diacritic equivalents.  
However, for typographical error inductions, a dedicated realistic corruption model is required. The approach, taken by other works [78; 87], is to infer probabilities for each error group from the available smaller dataset and to use them to generate errors on the target one. We took the same approach in this work.  
There are four prevailing categories of typographical errors. The authors of [60; 111] reported that more than 80% of errors can be attributed to substitution, deletion, insertion, or transposition errors. This division allows us to model each category separately.  
The physical keyboard layout plays an important role in influencing typos. A single keypress instruction consists of information of which hand, finger, and key row to select.  
\begin{table}
\begin{tabular}{l r r r r r} \hline \hline  & \multicolumn{1}{c}{**Language**} & \multicolumn{4}{c}{**Dataset**} \\ \hline \multirow{3}{*}{**Name**} & **Diacritic** & **Keyboard** & \multirow{3}{*}{**Sentences**} & **Alpha-** & **Diacritic \%** \\  & **Letters** & **Family** & & **Words** & **Words** & **Letters** \\ \hline Croatian & 5 & QWERTZ & 802,610 & 12,914,186 & 14.55 & 2.78 \\ Czech & 19 & QWERTY & 952,909 & 14,730,260 & 48.69 & 12.90 \\ French & 15 & AZERTY & 1,818,618 & 37,612,736 & 16.49 & 3.72 \\ Hungarian & 9 & QWERTZ & 1,294,605 & 17,587,448 & 50.05 & 11.48 \\ Irish & 5 & QWERTY & 50,825 & 1,005,620 & 29.52 & 7.04 \\ Latvian & 15 & QWERTY & 315,807 & 4,244,914 & 48.57 & 10.27 \\ Lithuanian & 9 & QWERTY & 612,724 & 7,096,677 & 38.75 & 7.00 \\ Polish & 9 & QWERTY & 1,069,841 & 16,178,130 & 32.71 & 6.42 \\ Romanian & 6 & QWERTY & 837,647 & 16,050,136 & 27.04 & 5.87 \\ Slovak & 25 & QWERTZ & 613,727 & 9,180,800 & 42.38 & 9.32 \\ Spanish & 7 & QWERTY & 1,735,516 & 42,863,263 & 11.50 & 2.33 \\ Turkish & 11 & QWERTY & 875,781 & 10,785,516 & 31.35 & 6.30 \\ Vietnamese & 67 & QWERTY & 819,918 & 20,241,117 & 81.18 & 25.94 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Languages and the training dataset statistics. Diacritic percentages are calculated among alphabetical words or letters. Alphabetical words (alpha-words) range from 72% to 86% of all the total words, including numbers.  
The authors of [53] argue that the confusion of these instructions is the main culprit of substitution errors, while mixed instruction timing between the two hands (operating on different parts of a keyboard) is the main culprit of transposition errors. While there may be more causes, such as visual and phonological factors [112], we restrict ourselves to the physical keyboard layout influence. This allows us to model typographical errors for all languages, given the distribution of the keyboard errors for a single language. We also make no distinction between physical and touchscreen keyboards, large or small.  
There are only limited misspelling resources for the data-rich English language, as shown in Table 2. The largest one is the Github Typo corpus [113]. Although it contains edits for multiple languages, only the English language is of a significant size. There is also a multilingual Wikipedia edit history, which could be prepared, similar to the GitHub dataset. However, it must be filtered [114] to not include non-typographical error-related examples. Incorporating the Twitter Typo corpus [115] may also not be worth the effort, as the domains are different, as well as the length of text spans (needed to normalize error frequencies). In the end, we used a single GitHub Typo Corpus to derive the probabilities of errors.  
Further details on generating the typos are provided in Section 6.2.