<figure><img src="image_1.png" /></figure>

HAL
open science

# Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Olli Kuparinen, Aleksandra Miletić, Yves Scherrer

► To cite this version:

Olli Kuparinen, Aleksandra Miletić, Yves Scherrer. Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation. Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023, Singapore, Singapore. pp.13814-13828, 10.18653/v1/2023.findings-emnlp.923. hal-05311040

HAL Id: hal-05311040

https://hal.science/hal-05311040v1

Submitted on 13 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation

Olli Kuparinen<fnref n="1" />,²

first.last@tuni.fi

Aleksandra Miletić<fnref n="1" />

first.last@helsinki.fi

Yves Scherrer<fnref n="1" />,³

first.last@ifi.uio.no

¹Department of Digital Humanities, University of Helsinki

²Faculty of Information Technology and Communication Sciences, Tampere University

³Department of Informatics, University of Oslo

Abstract

Text normalization methods have been commonly applied to historical language or user-generated content, but less often to dialectal transcriptions. In this paper, we introduce dialect-to-standard normalization – i.e., mapping phonetic transcriptions from different dialects to the orthographic norm of the standard variety – as a distinct sentence-level character transduction task and provide a large-scale analysis of dialect-to-standard normalization methods. To this end, we compile a multilingual dataset covering four languages: Finnish, Norwegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the most successful sequence-to-sequence model architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that a character-level Transformer trained on sliding windows of three words works best for Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we perform an error analysis to evaluate the effect of different data splits on model performance.

# 1 Introduction

Text normalization refers to a range of tasks that consist in replacing non-standard spellings with their standard equivalents. This procedure is beneficial for many downstream NLP tasks, since it increases data homogeneity and thus reduces the impact of unknown word forms. Establishing identity between different variants of the same form is also crucial for information retrieval tasks, and in particular for building efficient corpus querying systems. Furthermore, it can facilitate building applications for a wider audience, such as spelling and grammar checkers.

Important progress has been made on normalization of historical texts (e.g., Tang et al., 2018; Boll-

mann, 2019) and user-generated content (UGC) in social media (e.g. van der Goot et al., 2021). However, the existing work on dialect normalization (e.g., Scherrer and Ljubešić, 2016; Abe et al., 2018; Partanen et al., 2019) remains fragmented: it typically focuses on a single language and uses different models, experimental setups and evaluation metrics, making direct comparisons difficult.

In this paper, we aim to establish dialect-to-standard normalization as a distinct task alongside historical text normalization and UGC normalization. We make the following contributions:

• We compile a **multilingual dataset** from existing sources and make it available in a unified format to facilitate cross-lingual comparisons. The dataset covers Finnish, Norwegian, Swiss German and Slovene. These languages come from three different families and have different morphological systems. For the two largest datasets (Finnish and Norwegian), we provide **different data splits** corresponding to different use cases for dialect normalization.

• We test a **wide range of sequence-to-sequence models** that performed well in other normalization tasks: statistical machine translation, RNN-based and Transformer-based neural machine translation, and byT5, a pre-trained byte-based multilingual Transformer. We compare character with subword tokenizations as well as full-sentence contexts with sliding windows of three words. We evaluate the models on word accuracy, but also provide character error rates and word error reduction rates to facilitate comparison with previous work. Finally, we provide an error analysis on two of the Finnish data splits.<fnref n="1" />

<footnote n="1">¹The data, scripts and model configurations are available at https://github.com/Helsinki-NLP/dialect-to-standard.</footnote>
# 2 Related Work

## 2.1 Historical text normalization

Historical text normalization consists in modernizing the spelling of the text such that it conforms to current orthographic conventions. Pettersson et al. (2014) evaluate three different normalization methods in a multilingual setup: a simple filtering model, an approach based on Levenshtein distance, and an approach using character-level statistical machine translation (CSMT). They find that CSMT is the overall most promising approach. Scherrer and Erjavec (2016) use CSMT in supervised and unsupervised settings to normalize historical Slovene data.

Tang et al. (2018) and Bollmann (2019) provide multilingual comparisons of neural and statistical MT approaches, whereas Bawden et al. (2022) evaluate different normalization methods on historical French. In most settings, SMT outperformed neural models. In several settings, BPE-based subword segmentation led to better results than character-level segmentation.

## 2.2 Normalization of user-generated content

UGC, typically found on social media, contains various non-standard elements such as slang, abbreviations, creative spellings and typos. De Clercq et al. (2013) present various experiments on normalizing Dutch tweets and SMS messages and show that a combination of character-level and word-level SMT models yields the optimal results. Matos Veliz et al. (2019) follow up on this work and show that data augmentation techniques are crucial for obtaining competitive results with NMT models. The MoNoise model (van der Goot, 2019) significantly improved the state-of-the-art in UGC normalization. It contains several modules such as a spellchecker, an n-gram language model and domain-specific word embeddings that provide normalization candidates.

The first multilingual, homogeneous dataset for UGC normalization was published in the context of the MultiLexNorm shared task in 2021 (van der Goot et al., 2021). The results of the shared task also supported the usefulness of normalization for downstream tasks such as PoS-tagging and parsing. The best-performing submission (Samuel and Straka, 2021) proposed to fine-tune byT5, a byte-level pre-trained model (Xue et al., 2022), in such a way that normalizations are produced one word at a time.

## 2.3 Dialect-to-standard normalization

There has been comparatively less research in the domain of dialect normalization. Scherrer and Ljubešić (2016) apply CSMT to Swiss German. They create models for normalizing individual words and entire sentences and show that the larger context provided by the latter is beneficial for the normalization of ambiguous word forms. Lusetti et al. (2018) work on a different Swiss German dataset and show that neural encoder-decoder models can outperform CSMT when additional target-side language models are included.

Abe et al. (2018) work on the NINJAL corpus (Kokushokankokai Inc, 1980), and propose to translate Japanese dialects into standard Japanese using a multilingual (or rather, multi-dialectal) LSTM encoder-decoder model. The transduction is done on the level of *bunsetsu*, the base phrase in Japanese, which corresponds to a content word, potentially followed by a string of functional words.

Partanen et al. (2019) compare LSTM-based and Transformer-based character-level NMT models for normalizing Finnish. The authors use contexts of one word, three words, or the entire sentence, the best results being achieved using three words. On the contrary, Hämäläinen et al. (2020) report optimal results while using individual words in normalizing Swedish dialects spoken in Finland. Hämäläinen et al. (2022) use generated dialectal Finnish sentences to normalize Estonian dialects.

Machine translation from Arabic dialects to Modern Standard Arabic (MSA) can also be considered a dialect normalization task, although Arabic dialects differ from the standard variety to a greater extent than the languages used in our work and therefore contain lexical replacements and re-orderings. The 2023 NADI shared task (Abdul-Mageed et al., 2023) provides a subtask on Arabic dialect translation. The MADAR corpus (Bouamor et al., 2018) is a popular resource for Arabic dialect translation and covers 25 dialects. Additionally, Eryani et al. (2020) describe the creation of a normalization corpus for five Arabic dialects, but do not report any experiments on automatic normalization. Zhao and Chodroff (2022) similarly report on corpus compilation of Mandarin dialects, but they focus on acoustic-phonetic analysis.

Each of these works focuses on dialects of a single language and uses different models and experimental setups, as well as different evaluation metrics (including BLEU, word error rate and ac-
<table><caption>Table 1: Key figures of the four datasets constituting the multilingual normalization benchmark. Speak. = number of speakers. Loc. = number of locations. MSL<sub>w</sub> = mean sentence length in words. MSL<sub>ch</sub> = mean sentence length in characters. Ambig. = percentage of types having more than one possible normalization.</caption><thead><tr><th></th><th>Speak.</th><th>Loc.</th><th>Texts</th><th>Sentences</th><th>Words</th><th>Types</th><th>MSL<sub>w</sub></th><th>MSL<sub>ch</sub></th><th>Ambig.</th></tr></thead><tbody><tr><td>SKN</td><td>99</td><td>50</td><td>99</td><td>41,407</td><td>630,665</td><td>106,452</td><td>15.23</td><td>88.59</td><td>5.54%</td></tr><tr><td>NDC</td><td>438</td><td>111</td><td>684</td><td>126,460</td><td>1,684,059</td><td>77,950</td><td>13.32</td><td>57.16</td><td>10.14%</td></tr><tr><td>ArchiMob</td><td>6</td><td>5</td><td>6</td><td>10,183</td><td>82,658</td><td>11,110</td><td>8.12</td><td>42.17</td><td>6.36%</td></tr><tr><td>GOS</td><td>36</td><td>10</td><td>24</td><td>8,621</td><td>84,199</td><td>12,616</td><td>9.77</td><td>41.43</td><td>8.85%</td></tr></tbody></table>

<table><caption>Table 2: Normalization examples of the four languages. The top row presents the original phonetic transcription, the middle row the normalized version, and the bottom row provides an English gloss.</caption><tr><td>SKN<br>(Finnish)</td><td>mä oon syänys seittemän silakkaa aiva niin häntä erellä<br>minä olen syönyt seitsemän silakkaa aivan niin häntä edellä<br>'I have eaten seven herrings, that's right, tail first'</td></tr><tr><td>NDC<br>(Norwegian)</td><td>å får eg sje sjøra vår bil før te påske<br>og får jeg ikke kjøre vår bil før til påske<br>'and I don't get to drive our car until Easter'</td></tr><tr><td>ArchiMob<br>(Swiss German)</td><td>ich ha das ales inere kasette won ich de schlüssel nüme ha dezue<br>ich habe das alles in einer kassette wo ich den schlüssel nicht mehr habe dazu<br>'I have it all in a case for which I don't have the key anymore'</td></tr><tr><td>GOS<br>(Slovene)</td><td>se zjemla je prpravlena pugnujena pa ubdajlana pa puvlajčena<br>saj zemlja je pripravljena pognojena pa obdelana pa povlečena<br>'because the soil is prepared, fertilised and tilled and harrowed'</td></tr></table>

accuracy). All of these factors make meaningful comparisons between different approaches difficult.

## 3 Datasets

We propose a multilingual dataset that covers Finnish, Norwegian, Swiss German and Slovene. The dataset is compiled from existing dialect corpora, which are presented in detail below. The languages originate from two language families (Uralic and Indo-European) and three branches (Finnic, Germanic, Balto-Slavic). All languages adopt the Latin script. This enables the comparison of language structure, rather than differences in script. Some quantitative information about the individual corpora is available in Table 1, and Table 2 provides some example sentences.

### 3.1 Finnish

The Samples of Spoken Finnish corpus (Suomen kielen näytteitä, hereafter SKN) (Institute for the Languages of Finland, 2021) consists of 99 interviews conducted mostly in the 1960s.<fnref n="2" /> It includes data from 50 Finnish-speaking locations, with two speakers per location (with one exception). The

interviews have been transcribed phonetically on two levels of granularity (detailed and simplified) and normalized manually by linguists. We only consider the utterances of the interviewed dialect speakers, not of the interviewers. Although the detailed transcriptions have been used for the normalization experiments in Partanen et al. (2019), we use the simplified transcriptions here to make the annotations more consistent with the other languages. The simplified transcriptions do not make certain phonetic distinctions and share the alphabet with the normalized text.

### 3.2 Norwegian

The Norwegian Dialect Corpus (Johannessen et al., 2009, hereafter NDC) was built as a part of a larger initiative to collect dialect syntax data of the North Germanic languages.<fnref n="3" /> The recordings were made between 2006 and 2010, and typically four speakers per location were recorded. Each speaker appears in an interview with a researcher and in an informal conversation with another speaker. We concatenate all utterances of a speaker regardless of the context in which they appear.

<footnote n="2">²http://urn.fi/urn:nbn:fi:lb-2021112221, Licence: CC-BY.</footnote>

<footnote n="3">³http://www.tekstlab.uio.no/scandiasyn/download.html, Licence: CC BY-NC-SA 4.0.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: The three data splits (SKN1 on the left, SKN2 in the center, SKN3 on the right) visualized on a subset of the Finnish dataset. Each dot represents one speaker. There are generally two speakers per location in SKN.</figcaption></figure>

The recordings were transcribed phonetically and thereafter normalized to Norwegian Bokmål. The normalization was first done with an automatic tool developed specifically for this corpus, and its output was manually corrected afterwards. The publicly available phonetic and orthographic transcriptions are not well aligned; we automatically re-aligned them at utterance and word level.<fnref n="4" />

## 3.3 Swiss German

The ArchiMob corpus of Swiss German (Samardžić et al., 2016; Scherrer et al., 2019) consists of oral history interviews conducted between 1999 and 2001.⁵ The corpus contains 43 phonetically transcribed interviews, but only six of them were normalized manually. We use the interviewee's utterances of these six documents for our experiments. The selected texts originate from five dialect areas, covering approximately one third of the German-speaking part of Switzerland.

## 3.4 Slovene

Our Slovene dataset is based on the GOS corpus of spoken Slovene (Verdonik et al., 2013).⁶ The original corpus contains 115h of recordings. Two transcription layers are included: a manually transcribed phonetic layer and a semi-automatically normalized layer (with manual validation).

Since the degree of non-standardness in the full corpus was relatively low (16%), we select a subset of the data for our experiments. We retain speakers whose productions contain at least 30% non-standard tokens and who have produced at least 1000 words. This results in a set of 36 speakers from 10 dialect regions.

## 3.5 Preprocessing

To ensure the datasets are comparable, we have applied several preprocessing steps: removing punctuation and pause markers, substituting anonymized name tags with X, and excluding utterances consisting only of filler words. The Slovene data includes some utterances in Italian and German which are normalized to the corresponding standard. They have been excluded from the data.

# 4 Experimental Setup

## 4.1 Data splits

The Finnish and Norwegian datasets contain multiple speakers per location, which provides a possibility to test the generalization capabilities of the models in different scenarios. We create three different data splits:

1. **Normalizing unseen sentences of seen speakers.** We divide each speaker's data in such a way that 80% of *sentences* are used for training, 10% for development and 10% for testing. The sentences are selected randomly.

2. **Normalizing unseen speakers of seen dialects.** We pick speakers from selected locations for the development and test sets, while the rest of the speakers are used for training. For each location, at least one speaker is present in the training set. In other words, 80% of *speakers* are used for training, 10% for development and 10% for testing.

3. **Normalizing unseen dialects.** All speakers from a given location are assigned to either the training, development or test set. In other words, 80% of *locations* are used for training, 10% for development and 10% for testing.

The different data splits are visualized in Figure 1. For the smaller and less geographically diverse Swiss German and Slovene datasets, we only

<footnote n="4">⁴The alignment script and re-aligned data is available at https://github.com/Helsinki-NLP/ndc-aligned/.

⁵https://www.spur.uzh.ch/en/departments/research/textgroup/ArchiMob.html, Licence: CC BY-NC-SA 4.0.

⁶http://hdl.handle.net/11356/1438, Licence: CC BY-NC-SA 4.0.</footnote>
use split 1. For each split, we create three folds with random divisions into train, development and test sets.

While previous work (Scherrer and Ljubešić, 2016; Partanen et al., 2019) mostly relies on split 1, this setup potentially overestimates the models' normalization capabilities: in a given conversation, utterances, phrases and words are often repeated, so that similar structures can occur in the training and test sets. We argue that splits 2 and 3 more realistically reflect dialectological fieldwork, where new texts are gradually added to the collection and need to be normalized.

## 4.2 Tokenization and context sizes

Text normalization is generally viewed as a character transduction problem (Wu et al., 2021), and it seems therefore most natural to use single characters as token units. Character tokenization has also been shown to work well for normalization tasks in various recent studies.

Most other character transduction problems, such as transliteration or morphological inflection, are modelled out-of context, i.e., one word at a time. This assumption does not seem accurate for text normalization: as shown in Table 1, between 5 and 10% of word types have more than one possible normalization, and disambiguating these requires access to the sentential context. Moreover, depending on the annotation scheme, there are sandhi phenomena at the word boundaries (cf. the SKN example in Table 2: *syänys* instead of *syänyt* because of assimilation with the following s) that cannot be taken into account by models that operate word by word. Thus, the most obvious strategy for text normalization is to consider **entire sentences** and break them up into **single character tokens**.

This strategy combining long contexts with short tokens leads to rather long token sequences, and NMT approaches especially have been shown to underperform in such scenarios (Partanen et al., 2019). We include two alternative ways of addressing this issue: (1) by shortening the instances from full sentences to sliding windows of three consecutive words, and (2) by lengthening the tokens using subword segmentation.

**Sliding windows.** Partanen et al. (2019) propose to split each sentence into non-overlapping chunks of three consecutive words. We adapt this approach and use **overlapping chunks of three words** to ensure that the model always has access to exactly

one context word on the left and one on the right. At prediction time, only the word in the middle of each chunk is considered.

The preprocessing needed to create input for both entire-sentence and sliding-window models is illustrated in Appendix D.

**Subword segmentation.** Tang et al. (2018) and Bawden et al. (2022) found that subword segmentation could outperform character-level segmentation on the task of historical text normalization. We follow this work and experiment with subword segmentation as well. Several segmentation schemes have been proposed for general machine translation, e.g. byte-pair encoding (Sennrich et al., 2016, BPE) or the unigram model (Kudo, 2018). Kanjirangat et al. (2023) found the unigram model to perform better than BPE on texts with inconsistent writing. This is the case for our speech transcriptions and we thus opt to use the unigram model. We train our models with the SentencePiece library (Kudo and Richardson, 2018), and optimize the vocabulary size separately for each dataset.

## 4.3 Evaluation

We evaluate the models on **word-level accuracy**, i.e., the percentage of correctly normalized words. Since the reference normalizations are tied to the words in the source sentences, and since the models' output can differ in length from the source sentence, we need to re-align the model output with the reference normalization. We apply Levenshtein alignment to the entire sequence pair and split the system output at the characters aligned with word boundaries of the input (see Appendix D for an illustration).<fnref n="7" />

Word-level accuracy lacks granularity and does not distinguish between normalizations that are only one character off and normalizations that are completely wrong. Therefore, we include **character error rate** (CER) as a more fine-grained metric.⁸ CER is defined as the Levenshtein distance between the system output and the reference, normalized by the length of the reference. Another advantage of CER is that it can be computed directly on sentence pairs without re-alignment.⁹

<footnote n="7">⁷The re-alignment code is provided at https://github.com/Helsinki-NLP/dialect-to-standard/blob/main/scripts/align.py.

⁸Following Partanen et al. (2019), we use the implementation available at https://github.com/nsmartinez/WERpp.

⁹Re-alignment is still necessary for the sliding window models in order to extract the center word of each chunk.</footnote>
Following van der Goot et al. (2021), we also report error reduction rates in Appendix C.

We compare the systems to two **baselines**: the **leave-as-is** (LAI) baseline corresponds to the percentage of words that do not need to be modified. The **most frequent replacement** (MFR) baseline translates each word to its most frequent replacement seen in the training data, and falls back to copying the input for unseen words.

# 5 Methods and Tools

We utilize both statistical and neural machine translation tools trained from scratch, as well as a pretrained multilingual model. We train (or fine-tune) the models for each of the four languages separately. Our hyperparameter choices largely follow recent related work on text normalization. The main characteristics of the models are summarized below, and a detailed description of the hyperparameters is given in Appendix A. The methods used in the experiments are:

**SMT.** Our statistical machine translation method corresponds mostly to the one implemented in the CSMTiser tool.<fnref n="10" /> It uses the Moses SMT toolkit (Koehn et al., 2007) with a 10-gram KenLM language model trained on the training sets.¹¹ Scherrer (2023) found eflomal (Östling and Tiedemann, 2016) to produce better character alignment than the more commonly used GIZA++, and we adopt this method. Minimum error rate training (MERT) is used for tuning the model weights, using WER (word error rate, which effectively becomes character error rate in a character-level model) as the objective.

**RNN-based NMT.** This model uses a bidirectional LSTM encoder and a unidirectional LSTM decoder with two hidden layers each. The attention mechanism is reused for copy attention.

**TF-based NMT.** This model has 6 Transformer layers in the encoder and the decoder, with 8 heads each. We found in preliminary experiments that position representation clipping was beneficial to the results. All NMT models are trained with the OpenNMT-py toolkit (Klein et al., 2017).

**ByT5.** ByT5 is a multilingual pre-trained model of the T5 family (Raffel et al., 2020). These models

use a Transformer architecture and are pretrained on a masked language modeling task. ByT5 (Xue et al., 2022) is a variant of T5 that does not use any subword tokenization model, but rather encodes all text as UTF-8 encoded byte sequences.<fnref n="12" /> It is pre-trained on the multilingual m4C corpus (Xue et al., 2021), which includes the four languages of our dataset. We use the byt5-base model and fine-tune it separately on each normalization task for ten epochs.

We run all models on our base setup (entire sentence instances with character or byte tokenization). In addition, the SMT, RNN and TF models are also trained on subword-segmented data. Finally, the character-level RNN and TF models are trained on sliding windows.<fnref n="13" />

# 6 Results and Discussion

We evaluate the models presented in Section 5 with the metrics described in Section 4.3. We run the models on the three folds of each data split, and present the average scores and standard deviations for each metric.

The word-level accuracies are presented in Table 3 and the character error rates in Table 4. We also provide accuracy scores for the development sets in Appendix C. Transformer-based methods appear as the most robust: the Transformer trained on sliding windows is best for SKN, ArchiMob and GOS, while the Transformer-based byT5 produces the best results for Norwegian. For all other corpora, byT5 is a close second in accuracy, and is thus the best alternative for entire sentences.

While CSMT and RNN-based methods do not yield the best score for any dataset, they still perform reasonably well. For the NMT models, using the sliding window instead of sentences enhances the results for all corpora but NDC. Regarding tokenization, the subwords improve the performance on the large corpora (SKN and NDC) but worsen it on the small corpora (Archimob and GOS). For all datasets, the best results are obtained on the character (or byte) level.

We expected rising levels of difficulty between data split 1 and 3, but neither the baselines nor the model outputs confirmed our expectations. The

<footnote n="10">¹⁰https://github.com/clarinsi/csmtiser
¹¹Contrarily to related work, we did not include any additional target-side language models.</footnote>

<footnote n="12">¹²In our setup, we view character-level and byte-level segmentation as largely equivalent. Our datasets have vocabulary sizes ranging between 42 and 90 and thus lie largely below the byte-level upper bound of 256.</footnote>

<footnote n="13">¹³In preliminary experiments, the sliding window setup did not yield any benefits for SMT and byT5 models.</footnote>
<table><caption>Table 3: Word-level accuracy (↑). We report averages and standard deviations over the three folds of each data split.</caption><thead><tr><th>Model</th><th>SKN1</th><th>SKN2</th><th>SKN3</th><th>NDC1</th><th>NDC2</th><th>NDC3</th><th>ArchiMob</th><th>GOS</th></tr></thead><tbody><tr><td>LAI</td><td>44.63±0.37</td><td>46.01±1.10</td><td>47.96±2.53</td><td>32.88±0.08</td><td>33.31±1.35</td><td>33.72±0.48</td><td>21.37±0.33</td><td>58.08±0.71</td></tr><tr><td>MFR</td><td>84.87±0.22</td><td>82.75±0.26</td><td>83.19±0.24</td><td>86.90±0.06</td><td>86.48±1.15</td><td>86.72±0.17</td><td>83.82±0.20</td><td>83.93±0.52</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with character or byte segmentation</td></tr><tr><td>SMT</td><td>91.22±0.17</td><td>89.96±0.39</td><td>90.36±0.21</td><td>91.85±0.08</td><td>91.44±0.47</td><td>91.55±0.08</td><td>88.06±0.31</td><td>85.82±0.96</td></tr><tr><td>RNN</td><td>89.90±1.14</td><td>89.35±0.88</td><td>88.56±1.04</td><td>93.42±0.06</td><td>93.03±0.39</td><td>93.21±0.09</td><td>88.98±0.56</td><td>83.59±0.68</td></tr><tr><td>TF</td><td>92.90±0.35</td><td>91.38±0.86</td><td>89.75±1.73</td><td>93.99±0.09</td><td>93.68±0.47</td><td>93.41±0.67</td><td>89.14±0.38</td><td>85.65±0.61</td></tr><tr><td>ByT5</td><td>92.95±0.11</td><td>91.77±1.11</td><td>91.90±0.77</td><td><strong>94.68</strong>±0.10</td><td><strong>94.51</strong>±0.28</td><td><strong>94.54</strong>±0.10</td><td>90.57±0.54</td><td>87.26±0.55</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with subword segmentation</td></tr><tr><td>SMT</td><td>90.18±0.02</td><td>88.64±0.36</td><td>89.26±0.13</td><td>92.57±0.06</td><td>92.16±0.48</td><td>92.20±0.12</td><td>87.38±0.13</td><td>85.23±0.73</td></tr><tr><td>RNN</td><td>91.29±0.09</td><td>89.51±1.33</td><td>88.90±1.85</td><td>92.60±0.20</td><td>92.13±0.07</td><td>92.41±0.10</td><td>86.81±0.38</td><td>80.64±1.34</td></tr><tr><td>TF</td><td>92.23±1.80</td><td>90.84±2.10</td><td>91.21±1.59</td><td>94.20±0.01</td><td>93.85±0.37</td><td>93.96±0.12</td><td>87.63±0.37</td><td>84.72±0.83</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Sliding window models with character segmentation</td></tr><tr><td>RNN</td><td>92.46±0.07</td><td>90.74±0.24</td><td>91.42±0.24</td><td>92.31±0.15</td><td>92.23±0.64</td><td>92.31±0.20</td><td>90.39±0.33</td><td>86.26±0.58</td></tr><tr><td>TF</td><td><strong>93.44</strong>±0.04</td><td><strong>92.10</strong>±0.52</td><td><strong>92.38</strong>±0.35</td><td>93.24±0.10</td><td>93.02±0.40</td><td>93.19±0.09</td><td><strong>90.95</strong>±0.21</td><td><strong>87.30</strong>±0.56</td></tr></tbody></table>

<table><caption>Table 4: Character error rates (↓).</caption><thead><tr><th>Model</th><th>SKN1</th><th>SKN2</th><th>SKN3</th><th>NDC1</th><th>NDC2</th><th>NDC3</th><th>ArchiMob</th><th>GOS</th></tr></thead><tbody><tr><td>LAI</td><td>14.60±0.11</td><td>14.04±0.42</td><td>13.21±0.95</td><td>24.54±0.05</td><td>24.28±0.72</td><td>23.78±0.20</td><td>32.45±0.08</td><td>14.95±0.27</td></tr><tr><td>MFR</td><td>4.68±0.06</td><td>5.35±0.13</td><td>5.18±0.10</td><td>5.71±0.02</td><td>5.94±0.63</td><td>5.75±0.12</td><td>7.10±0.11</td><td>6.31±0.30</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with character or byte segmentation</td></tr><tr><td>SMT</td><td>2.64±0.07</td><td>3.00±0.18</td><td>2.91±0.11</td><td>3.07±0.03</td><td>3.28±0.21</td><td>3.16±0.04</td><td>3.63±0.19</td><td>5.30±0.39</td></tr><tr><td>RNN</td><td>4.46±1.64</td><td>5.88±1.31</td><td>5.21±1.49</td><td>2.65±0.09</td><td>3.49±0.50</td><td>3.10±0.52</td><td>4.21±1.37</td><td>23.20±10.72</td></tr><tr><td>TF</td><td>2.39±0.20</td><td>3.13±0.67</td><td>4.57±1.38</td><td>2.32±0.03</td><td>2.48±0.22</td><td>2.59±0.33</td><td>3.49±0.27</td><td>5.71±0.21</td></tr><tr><td>ByT5</td><td>2.96±0.13</td><td>3.67±1.12</td><td>3.75±0.99</td><td><strong>2.11</strong>±0.06</td><td><strong>2.19</strong>±0.12</td><td><strong>2.19</strong>±0.08</td><td>3.21±0.36</td><td>5.28±0.19</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with subword segmentation</td></tr><tr><td>SMT</td><td>2.97±0.03</td><td>3.46±0.19</td><td>3.30±0.12</td><td>2.96±0.04</td><td>3.20±0.25</td><td>3.14±0.02</td><td>4.20±0.08</td><td>5.92±0.35</td></tr><tr><td>RNN</td><td>7.38±0.44</td><td>9.57±5.20</td><td>12.38±4.46</td><td>3.34±0.65</td><td>3.20±0.17</td><td>3.14±0.03</td><td>4.84±0.34</td><td>28.66±1.06</td></tr><tr><td>TF</td><td>2.23±0.10</td><td>2.82±0.18</td><td>2.93±0.63</td><td>2.27±0.02</td><td>2.44±0.18</td><td>2.37±0.07</td><td>4.64±0.15</td><td>7.05±0.10</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Sliding window models with character segmentation</td></tr><tr><td>RNN</td><td>2.38±0.01</td><td>2.90±0.11</td><td>2.73±0.10</td><td>3.06±0.07</td><td>3.12±0.34</td><td>3.02±0.06</td><td>3.22±0.09</td><td>5.38±0.23</td></tr><tr><td>TF</td><td><strong>2.06</strong>±0.02</td><td><strong>2.73</strong>±0.64</td><td><strong>2.40</strong>±0.13</td><td>2.68±0.06</td><td>2.79±0.20</td><td>2.68±0.01</td><td><strong>3.02</strong>±0.08</td><td><strong>4.98</strong>±0.18</td></tr></tbody></table>

differences between splits are very small, and most models achieve the lowest results on split 2.

The character error rates presented in Table 4 follow the same pattern as the word accuracies when it comes to the best models. However, for SKN, byT5 drops below sliding window RNN and SMT with this metric. Some of the Finnish byT5 models generate much shorter predictions than the other models, but it remains to be investigated why this occurs and why it only affects some training runs. Table 4 also highlights poor performance and large standard deviation with sentence-level RNN on GOS. This is in line with earlier findings

about neural models' tendency to overmodify the predictions (Bawden et al., 2022).

## 6.1 Comparison with previous work

Partanen et al. (2019) worked on the normalization of the Finnish SKN dataset, reporting word error rate (WER) as their main metric. Although they use the detailed SKN transcriptions instead of the simplified ones, their results are roughly comparable with our SKN1 data split (see Table 5). While they were not able to successfully train sentence-level models, our parameterization closes the gap to the chunk models. Their best reported word error rates
<table><caption>Table 5: Comparison with previous work on Finnish (SKN1 data split, all models with character tokenization). Metric: word error rate (↓).</caption><thead><tr><th></th><th>Partanen et al. (2019)</th><th>This work</th></tr></thead><tbody><tr><td>RNN (full sentences)</td><td>46.52</td><td>11.11</td></tr><tr><td>RNN (3-word chunks)</td><td>5.73</td><td>7.59</td></tr><tr><td>TF (full sentences)</td><td>53.23</td><td>7.07</td></tr><tr><td>TF (3-word chunks)</td><td>6.10</td><td>6.59</td></tr></tbody></table>

are however slightly lower than ours.

Scherrer and Ljubešić (2016) presented normalization experiments on the ArchiMob corpus. Our results are largely comparable to theirs. A detailed comparison is provided in Appendix B.

## 6.2 Error analysis

We examine the effects of different data splits by looking at the output of the sliding window Transformer on splits 1 and 3 of the Finnish SKN corpus. As a reminder, the test set in split 1 contains unseen sentences from seen texts (and therefore seen dialects), whereas in split 3 it comes from unseen locations. It can be expected (1) that the model trained on SKN1 performs better on dialect-specific phenomena, such as normalizations involving diphthongs, consonant grade and inflection marks; and (2) that the two models behave similarly on phenomena that are not dialect-specific, such as capitalization and proper names.

<table><caption>Table 6: Comparison of error types between the models trained on SKN1 and SKN3.</caption><thead><tr><th>Error type</th><th colspan="2">SKN1</th><th colspan="2">SKN3</th></tr></thead><tbody><tr><td>Capitalization</td><td>2</td><td>1.1%</td><td>2</td><td>0.6%</td></tr><tr><td>Character</td><td>13</td><td>6.8%</td><td>31</td><td>9.8%</td></tr><tr><td>Consonant grade</td><td>3</td><td>1.6%</td><td>15</td><td>4.7%</td></tr><tr><td>Diphthong</td><td>9</td><td>2.7%</td><td>4</td><td>1.3%</td></tr><tr><td>Indiscernible</td><td>17</td><td>8.9%</td><td>32</td><td>10.1%</td></tr><tr><td>Inflection</td><td>88</td><td>46.3%</td><td>127</td><td>39.9%</td></tr><tr><td>Proper name</td><td>10</td><td>5.3%</td><td>16</td><td>5.0%</td></tr><tr><td>Wrong target</td><td>48</td><td>25.3%</td><td>91</td><td>28.6%</td></tr><tr><td>Total</td><td>190</td><td>100%</td><td>318</td><td>100%</td></tr></tbody></table>

We analyze model output on the sentences that appear in both test sets and focus on words for which at least one model produced an erroneous normalization. We identify 382 such cases. On this set of words, the SKN3 model produces a much higher number of errors (318) than the SKN1

model (190). The higher number of errors on SKN3 is coherent with the intuition that this split is more difficult, but the nature of the errors produced by the two models does not fully conform to the expectations (see Table 6). In absolute terms, the model trained on SKN3 does produce more inflection and consonant grade errors, but fewer diphthong errors. In relative terms, the SKN3 model produces a lower percentage of inflection errors than its counterpart (40% vs 46%). This seems to indicate that split 3 does not preclude the model from learning dialect-related patterns. We hypothesize that this is due to the fact that the training set contains material from the same dialect area as the test set (although not from the exact same location).

To identify the critical point beyond which the cross-lectal model performance would be clearly affected, it could be useful to introduce a fourth data split which would exclude larger dialect areas from the train set and test on unseen dialects.

## 6.3 Comparison with other tasks

<table><caption>Table 7: Word-level accuracy ranges of the leave-as-is baselines (LAI) and word-level error reduction rate ranges (ERR) for historical normalization (Bollmann, 2019), UGC normalization (van der Goot et al., 2021) and dialect normalization (this work, cf. Appendix C).</caption><thead><tr><th>Task</th><th>LAI</th><th>ERR</th></tr></thead><tbody><tr><td>Historical norm.</td><td>17.5 – 85.4</td><td>72.1 – 89.9</td></tr><tr><td>UGC norm.</td><td>63.0 – 93.1</td><td>47.5 – 80.1</td></tr><tr><td>Dialect norm.</td><td>21.4 – 58.1</td><td>69.7 – 92.1</td></tr></tbody></table>

As mentioned above, dialect-to-standard normalization shares fundamental properties with historical text normalization and UGC normalization. Here, we compare the respective difficulties of these three tasks. Table 7 reports the LAI ranges and ERR ranges of the best systems reported in Bollmann (2019) and van der Goot et al. (2021).

It can be seen that dialect normalization has the lowest LAI rates on average and thus requires the most changes of the three tasks. The models perform roughly equally well on both historical and dialectal normalization, whereas UGC normalization seems to be a more difficult task.

## 7 Conclusions

In this paper, we present the dialect-to-standard normalization task as a distinct task alongside historical text normalization and UGC normalization. We
introduce a dialect normalization dataset contain-
ing four languages from three different language
branches, and use it to evaluate various statistical,
neural and pre-trained neural sequence-to-sequence
models.

In our base setup with models trained on entire
sentences with character (or byte) segmentation,
the pre-trained byT5 model performs best for all
languages and data splits. Moving from character
segmentation to subword segmentation increases
the accuracies for the large datasets (SKN and
NDC), but not enough to surpass byT5. In contrast,
the sliding window approach outperforms byT5
on all languages except Norwegian. The superior
performance of byT5 on Norwegian cannot be di-
rectly explained by the amount of training data,<fnref n="14" />
but it is likely that the closely related languages
Swedish and Danish enhance its performance. A
further analysis on character error rate shows that
the neural models sometimes offer very poor pre-
dictions, which are not visible when using accuracy
as a metric.

In this work, we have evaluated the most com-
mon and most popular model architectures, but it
would be interesting to test specific model architec-
tures for character transduction tasks, e.g. models
that put some monotonicity constraint on the at-
tention mechanism (Wu and Cotterell, 2019; Rios
et al., 2021). We defer this to future work.

Another point to be investigated in future work
is data efficiency. Our training sets are relatively
large in comparison with other character transduc-
tion tasks, and it would be useful to see how much
the data requirements can be reduced without sig-
nificantly affecting the normalization accuracy.

Limitations

We see the following limitations of our work:

* The proposed multilingual dataset is biased towards European languages and European dialectal practices. It may therefore not generalize well to the types of dialectal variation present in other parts of the world and to transcriptions in non-Latin scripts. In particular, there is an extensive amount of research on the normalization of Arabic and Japanese dialects (e.g., Abe et al., 2018; Eryani et al., 2020). We address some of these issues in Scherrer et al. (2023).

* We voluntarily restrict our dataset to “clean” corpora, i.e., interviews transcribed and normalized by trained experts. This contrasts with other data collections specifically aimed at extracting dialectal content from social media (e.g., Ueberwasser and Stark, 2017; Mubarak, 2018; Barnes et al., 2021; Kuparinen, 2023). Such datasets compound the features and challenges of both dialect-to-standard normalization and UGC normalization.

* We did not perform extensive hyperparame-
ter tuning in our experiments, but rather use set-
tings that have performed well in other normal-
ization tasks. It is therefore conceivable that the
performance of NMT models in particular could
be improved. Furthermore, specific model architec-
tures for character transduction tasks have been
proposed, e.g. constraining the attention to be
monotonic (Wu and Cotterell, 2019; Rios et al.,
2021). We did not include such architectures in
our experiments since they generally only showed
marginal improvements.

Ethics statement

All our experiments are based on publicly avail-
able datasets that were costly to produce. It
is important to ensure that these are appropri-
ately acknowledged. Anybody wishing to use our
dataset will also need to cite the publications of
the original datasets. Details are given on the
resource download page: https://github.com/
Helsinki-NLP/dialect-to-standard.

The datasets have been anonymized where nec-
essary. Text normalization is explicitly mentioned
as a possible research task in the literature present-
ing the ArchiMob corpus (Samardžić et al., 2016;
Scherrer and Ljubešić, 2016), and the SKN dataset
has been previously used to evaluate normaliza-
tion models. We are not aware of any malicious or
harmful uses of the proposed dialect-to-standard
normalization models.

Acknowledgements

This work has been supported by the Academy of
Finland through project No. 342859 “CorCoDial –
Corpus-based computational dialectology”.

The authors wish to acknowledge CSC – IT Cen-
ter for Science, Finland, for generous computa-
tional resources.

<footnote n="14">¹⁴German accounts for 3.05% of the byT5 training data, Finnish for 1.35%, Norwegian for 1.33% and Slovene for 0.95%.</footnote>
## References

Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, ElMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task. In *Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023)*.

Kaori Abe, Yuichiroh Matsubayashi, Naoaki Okazaki, and Kentaro Inui. 2018. Multi-dialect neural machine translation and dialectometry. In *Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation*, Hong Kong. Association for Computational Linguistics.

Jeremy Barnes, Petter Mæhlum, and Samia Touileb. 2021. NorDial: A preliminary corpus of written Norwegian dialect use. In *Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)*, pages 445–451, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.

Rachel Bawden, Jonathan Poinhos, Eleni Kogkitsidou, Philippe Gambette, Benoît Sagot, and Simon Gabay. 2022. Automatic normalisation of early Modern French. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 3354–3366, Marseille, France. European Language Resources Association.

Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 3885–3898, Minneapolis, Minnesota. Association for Computational Linguistics.

Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Miyazaki, Japan. European Language Resources Association (ELRA).

Orphée De Clercq, Sarah Schulz, Bart Desmet, Els Lefever, and Véronique Hoste. 2013. Normalization of Dutch user-generated content. In *Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013*, pages 179–188, Hissar, Bulgaria.

Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple Arabic dialects. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4130–4138, Marseille, France. European Language Resources Association.

Mika Hämäläinen, Khalid Alnajjar, and Tuuli Tuisk. 2022. Help from the neighbors: Estonian dialect

normalization using a Finnish dialect generator. In *Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing*, pages 61–66, Hybrid. Association for Computational Linguistics.

Mika Hämäläinen, Niko Partanen, and Khalid Alnajjar. 2020. Normalization of different Swedish dialects spoken in Finland. In *Proceedings of the 4th ACM SIGSPATIAL Workshop on Geospatial Humanities*, GeoHumanities’20, page 24–27, New York, NY, USA. Association for Computing Machinery.

Institute for the Languages of Finland. 2021. Samples of Spoken Finnish, VRT Version.

Janne Bondi Johannessen, Joel James Priestley, Kristin Hagen, Tor Anders Åfarli, and Øystein Alexander Vangsnes. 2009. The Nordic Dialect Corpus – an advanced research tool. In *Proceedings of the 17th Nordic Conference of Computational Linguistics (NODALIDA 2009)*, pages 73–80, Odense, Denmark. Northern European Association for Language Technology (NEALT).

Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, and Fabio Rinaldi. 2023. Optimizing the size of subword vocabularies in dialect classification. In *Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)*, pages 14–30, Dubrovnik, Croatia. Association for Computational Linguistics.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senelart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In *Proceedings of ACL 2017, System Demonstrations*, pages 67–72, Vancouver, Canada. Association for Computational Linguistics.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In *Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions*, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.

Kokushokankokai Inc. 1980. Zenkoku hougen danwa database nihon no furusato kotoba syusei (the national dialect discourse database collection).

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Olli Kuparinen. 2023. Murreviikko - a dialectologically annotated and normalized dataset of Finnish tweets. In *Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023)*, pages 31–39, Dubrovnik, Croatia. Association for Computational Linguistics.

Massimo Lusetti, Tatyana Ruzsics, Anne Göhring, Tanja Samardžić, and Elisabeth Stark. 2018. Encoder-decoder methods for text normalization. In *Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018)*, pages 18–28, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Claudia Matos Veliz, Orphee De Clercq, and Veronique Hoste. 2019. Benefits of data augmentation for NMT-based text normalization of user-generated content. In *Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)*, pages 275–285, Hong Kong, China. Association for Computational Linguistics.

Hamdy Mubarak. 2018. Converting Dialectal Arabic to Modern Standard Arabic. In *Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)*, Paris, France. European Language Resources Association (ELRA).

Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with Markov Chain Monte Carlo. *Prague Bulletin of Mathematical Linguistics*, 106:125–146.

Niko Partanen, Mika Hämäläinen, and Khalid Alnajjar. 2019. Dialect text normalization to normative standard Finnish. In *Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)*, pages 141–146, Hong Kong, China. Association for Computational Linguistics.

Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2014. A multilingual evaluation of three spelling normalization methods for historical text. In *Proceedings of the 8th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH)*, pages 32–41, Gothenburg, Sweden. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *Journal of Machine Learning Research*, 21(140):1–67.

Annette Rios, Chantal Amrhein, Noëmi Aepli, and Rico Sennrich. 2021. On biasing transformer attention towards monotonicity. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4474–4488, Online. Association for Computational Linguistics.

Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob - a corpus of spoken Swiss German. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)*, pages 4061–4066, Portorož, Slovenia. European Language Resources Association (ELRA).

David Samuel and Milan Straka. 2021. ÚFAL at Multi-LexNorm 2021: Improving multilingual lexical normalization by fine-tuning ByT5. In *Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)*, pages 483–492, Online. Association for Computational Linguistics.

Yves Scherrer. 2023. Character alignment methods for dialect-to-standard normalization. In *Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology*, pages 110–116, Toronto, Canada. Association for Computational Linguistics.

Yves Scherrer and Tomaž Erjavec. 2016. Modernising historical Slovene words. *Natural Language Engineering*, 22(6):881–905.

Yves Scherrer and Nikola Ljubešić. 2016. Automatic normalisation of the Swiss German ArchiMob corpus using character-level machine translation. In *Proceedings of the 13th Conference on Natural Language Processing (KONVENS)*.

Yves Scherrer, Aleksandra Miletić, and Olli Kuparinen. 2023. The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline. In *Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023)*. (accepted).

Yves Scherrer, Tanja Samardžić, and Elvira Glaser. 2019. Digitising Swiss German: how to process and study a polycentric spoken language. *Language Resources and Evaluation*, 53(4):735–769.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normalization. In *Proceedings of the 27th International Conference on Computational Linguistics*, pages 1320–1331, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Simone Ueberwasser and Elisabeth Stark. 2017. What's up, Switzerland? A corpus-based research project in a multilingual country. *Linguistik Online*, 5(84).

Rob van der Goot. 2019. *MoNoise: A multi-lingual and easy-to-use lexical normalization tool*. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 201–206, Florence, Italy. Association for Computational Linguistics.

Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko. 2021. *MultiLexNorm: A shared task on multilingual lexical normalization*. In *Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)*, pages 493–509, Online. Association for Computational Linguistics.

Darinka Verdonik, Iztok Kosem, Ana Zwitter Vitez, Simon Krek, and Marko Stabej. 2013. Compilation, transcription and usage of a reference speech corpus: The case of the Slovene corpus GOS. *Language Resources and Evaluation*, 47(4):1031–1048.

Shijie Wu and Ryan Cotterell. 2019. *Exact hard monotonous attention for character-level transduction*. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 1530–1537, Florence, Italy. Association for Computational Linguistics.

Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. *Applying the transformer to character-level transduction*. In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pages 1901–1907, Online. Association for Computational Linguistics.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. *ByT5: Towards a token-free future with pre-trained byte-to-byte models*. *Transactions of the Association for Computational Linguistics*, 10:291–306.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. *mT5: A massively multilingual pre-trained text-to-text transformer*. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 483–498, Online. Association for Computational Linguistics.

Liang Zhao and Eleanor Chodroff. 2022. *The ManDi corpus: A spoken corpus of Mandarin regional dialects*. In *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 1985–1990, Marseille, France. European Language Resources Association.

## A Experimental details

We trained all neural models on a single NVIDIA V100 GPU. The SMT models were trained on a Xeon Gold 6230 CPU. Table 8 presents the average training time and number of parameters for a single fold of the largest of our datasets, the Norwegian NDC.

<table><caption>Table 8: Training runtime (average) and number of parameters for a single character-level NDC model.</caption><thead><tr><th>Model</th><th>Runtime (hh:mm)</th><th>Parameters</th></tr></thead><tbody><tr><td>SMT</td><td>28:16</td><td>—</td></tr><tr><td>RNN (full sentences)</td><td>12:12</td><td>9.6 M</td></tr><tr><td>RNN (sliding window)</td><td>42:28</td><td>9.6 M</td></tr><tr><td>TF (full sentences)</td><td>17:27</td><td>25.4 M</td></tr><tr><td>TF (sliding window)</td><td>30:19</td><td>25.4 M</td></tr><tr><td>ByT5</td><td>27:27</td><td>581 M</td></tr></tbody></table>

Additional details about the model architectures and hyperparameter settings are provided in Table 12.

## B Comparison with previous work on Swiss German

Scherrer and Ljubešić (2016) presented normalization experiments with CSMT models on the Archi-Mob corpus, albeit with a different data split than in our present work. The most comparable model, a CSMT model trained on entire sentences, obtains an almost identical accuracy score compared to our results (see Table 9). Their best model, a CSMT model with an additional language model and constraints, achieves performance on par with our best model, the Transformer trained on sliding windows.

<table><caption>Table 9: Comparison with previous work on Swiss German. Metric: accuracy (↑). 1LM = one language model, 2LM = two language models.</caption><thead><tr><th>Scherrer and Ljubešić (2016)</th><th>This work</th></tr></thead><tbody><tr><td>CSMT 1LM</td><td>87.59</td><td>88.06</td></tr><tr><td>CSMT 2LM+constraints</td><td>90.46</td><td>—</td></tr><tr><td>TF (sliding window)</td><td>—</td><td>90.95</td></tr></tbody></table>

## C Additional results

Table 10 shows the word-level error reduction rates (ERR). ERR was introduced by van der Goot
<table><caption>Table 10: Error reduction rates (↑) relative to the LAI baseline.</caption><thead><tr><th>Model</th><th>SKN1</th><th>SKN2</th><th>SKN3</th><th>NDC1</th><th>NDC2</th><th>NDC3</th><th>ArchiMob</th><th>GOS</th></tr></thead><tbody><tr><td>MFR</td><td>72.67±0.27</td><td>68.04±0.99</td><td>67.63±1.96</td><td>80.48±0.09</td><td>79.74±1.33</td><td>79.97±0.22</td><td>79.40±0.17</td><td>61.67±1.08</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with character or byte segmentation</td></tr><tr><td>SMT</td><td>84.14±0.21</td><td>81.42±0.37</td><td>81.43±1.05</td><td>87.87±0.11</td><td>87.17±0.49</td><td>87.25±0.05</td><td>84.79±0.44</td><td>66.20±1.82</td></tr><tr><td>RNN</td><td>81.75±2.14</td><td>80.29±1.27</td><td>78.01±1.98</td><td>90.20±0.10</td><td>89.56±0.38</td><td>89.76±0.10</td><td>85.97±0.80</td><td>60.84±2.06</td></tr><tr><td>TF</td><td>87.19±0.57</td><td>84.05±1.30</td><td>80.34±2.75</td><td>91.04±0.12</td><td>90.53±0.51</td><td>90.05±0.96</td><td>86.17±0.56</td><td>65.77±1.28</td></tr><tr><td>ByT5</td><td>87.26±0.12</td><td>84.74±2.33</td><td>84.38±2.21</td><td><strong>92.07</strong>±0.14</td><td><strong>91.78</strong>±0.30</td><td><strong>91.76</strong>±0.17</td><td>87.99±0.76</td><td>69.60±1.34</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with subword segmentation</td></tr><tr><td>SMT</td><td>82.27±0.08</td><td>78.96±0.28</td><td>79.33±0.94</td><td>88.93±0.09</td><td>87.97±0.16</td><td>88.29±0.11</td><td>77.53±0.25</td><td>64.77±1.56</td></tr><tr><td>RNN</td><td>84.27±0.18</td><td>80.59±2.11</td><td>78.66±3.58</td><td>88.98±0.31</td><td>88.32±0.06</td><td>88.51±0.22</td><td>83.21±0.47</td><td>53.79±3.78</td></tr><tr><td>TF</td><td>87.79±0.31</td><td>84.94±0.44</td><td>84.14±1.27</td><td>91.35±0.02</td><td>90.57±0.06</td><td>90.89±0.30</td><td>84.25±0.38</td><td>63.54±2.23</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Sliding window models with character segmentation</td></tr><tr><td>RNN</td><td>86.37±0.08</td><td>82.83±0.51</td><td>83.48±0.88</td><td>88.54±0.23</td><td>88.36±0.72</td><td>88.41±0.22</td><td>87.76±0.45</td><td>67.22±1.07</td></tr><tr><td>TF</td><td><strong>88.16</strong>±0.09</td><td><strong>85.37</strong>±0.67</td><td><strong>85.31</strong>±1.31</td><td>89.93±0.15</td><td>89.54±0.40</td><td>89.73±0.13</td><td><strong>88.48</strong>±0.33</td><td><strong>69.72</strong>±0.95</td></tr></tbody></table>

<table><caption>Table 11: Accuracies obtained on the development sets (↑).</caption><thead><tr><th>Model</th><th>SKN1</th><th>SKN2</th><th>SKN3</th><th>NDC1</th><th>NDC2</th><th>NDC3</th><th>ArchiMob</th><th>GOS</th></tr></thead><tbody><tr><td>LAI</td><td>44.74±0.33</td><td>46.90±1.56</td><td>45.13±2.77</td><td>32.61±0.06</td><td>32.79±0.52</td><td>32.70±0.88</td><td>21.38±0.34</td><td>57.29±0.48</td></tr><tr><td>MFR</td><td>84.94±0.20</td><td>83.42±1.14</td><td>81.88±2.53</td><td>86.84±0.09</td><td>86.45±0.17</td><td>86.14±0.17</td><td>83.59±0.15</td><td>83.92±0.38</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with character or byte segmentation</td></tr><tr><td>SMT</td><td>91.83±0.79</td><td>90.55±0.25</td><td>89.87±1.98</td><td>91.75±0.40</td><td>91.64±0.45</td><td>90.92±0.21</td><td>88.43±0.19</td><td>86.03±0.15</td></tr><tr><td>RNN</td><td>90.09±1.58</td><td>90.02±1.06</td><td>89.18±2.15</td><td>93.45±0.15</td><td>93.20±0.38</td><td>92.62±0.14</td><td>89.33±0.27</td><td>82.63±0.86</td></tr><tr><td>TF</td><td>92.89±0.38</td><td>92.07±0.58</td><td>90.12±3.09</td><td>94.04±0.07</td><td>93.80±0.42</td><td>92.76±0.36</td><td>89.46±0.30</td><td>85.58±0.10</td></tr><tr><td>ByT5</td><td>93.31±0.13</td><td>92.34±1.45</td><td>91.72±1.70</td><td><strong>94.64</strong>±0.12</td><td><strong>94.58</strong>±0.33</td><td><strong>93.85</strong>±0.16</td><td>90.55±0.51</td><td>86.95±0.87</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Full sentence models with subword segmentation</td></tr><tr><td>SMT</td><td>90.34±0.16</td><td>89.39±0.40</td><td>88.53±2.17</td><td>92.62±0.13</td><td>92.20±0.47</td><td>91.52±0.19</td><td>87.62±0.31</td><td>85.35±0.40</td></tr><tr><td>RNN</td><td>91.51±0.18</td><td>90.20±0.99</td><td>89.52±2.15</td><td>92.60±0.30</td><td>92.34±0.28</td><td>91.67±0.25</td><td>87.22±0.34</td><td>80.48±1.00</td></tr><tr><td>TF</td><td>93.51±0.08</td><td>92.55±0.41</td><td>91.76±1.89</td><td>94.24±0.08</td><td>93.96±0.36</td><td>93.29±0.14</td><td>88.13±0.65</td><td>84.49±0.44</td></tr><tr><td colspan="9" style="text-align: center; font-weight: bold;">Sliding window models with character segmentation</td></tr><tr><td>RNN</td><td>92.58±0.12</td><td>91.39±0.35</td><td>90.71±2.18</td><td>92.32±0.05</td><td>92.25±0.46</td><td>91.64±0.23</td><td>90.28±0.06</td><td>86.03±0.24</td></tr><tr><td>TF</td><td><strong>93.56</strong>±0.04</td><td><strong>92.79</strong>±0.11</td><td><strong>91.79</strong>±1.93</td><td>93.23±0.17</td><td>93.07±0.38</td><td>92.53±0.23</td><td><strong>90.91</strong>±0.16</td><td><strong>87.33</strong>±0.26</td></tr></tbody></table>

(2019) and it represents, roughly speaking, the improvement of a model relative to the LAI baseline. Thus, it makes it easier to compare models across datasets, which may not be the case with accuracy due to different LAI values. ERR is defined as follows:

$$ERR = \frac{Accuracy_{system} - Accuracy_{baseline}}{1.0 - Accuracy_{baseline}}$$

By and large, these results follow the same pattern as the accuracy and CER scores reported in Section 6.

Table 11 shows the accuracies on the development sets. They are comparable with the test set
<table><caption>Table 12: Hyperparameter settings.</caption><thead><tr><th>Model</th><th>Parameter</th><th>Selected values</th><th>Considered alternatives</th></tr></thead><tbody><tr><td rowspan="3">Subwords</td><td>Methodology</td><td>Unigram (SentencePiece)</td><td>BPE</td></tr><tr><td rowspan="2">Vocabulary size</td><td>200 (SKN)</td><td>200, 500, 1000, 2000,</td></tr><tr><td>500 (Archimob, GOS)</td><td>4000</td></tr><tr><td rowspan="6">SMT</td><td>Alignment tool</td><td>Eflomal</td><td>GIZA++</td></tr><tr><td>Alignment symmetrization</td><td>grow-diag-final-and</td><td>—</td></tr><tr><td>Language model n-gram size</td><td>10</td><td>—</td></tr><tr><td>Maximum phrase length</td><td>10</td><td>—</td></tr><tr><td>Distortion</td><td>disabled</td><td>—</td></tr><tr><td>Tuning method</td><td>MERT</td><td>—</td></tr><tr><td rowspan="12">RNN</td><td>Encoder + decoder layers</td><td>2 + 2</td><td>—</td></tr><tr><td>Encoder + decoder types</td><td>Bi-LSTM + LSTM</td><td>—</td></tr><tr><td>Attention type</td><td>MLP with reused copy attention</td><td>—</td></tr><tr><td>Embedding dimensions</td><td>512</td><td>—</td></tr><tr><td>Hidden layer dimensions</td><td>512</td><td>—</td></tr><tr><td>Dropout</td><td>0.1</td><td>—</td></tr><tr><td>Optimizer</td><td>Adagrad</td><td>—</td></tr><tr><td>Batch size / accumulate gradient</td><td>1 * 10000 tokens</td><td>1 * 5000, 1 * 25000</td></tr><tr><td>Learning rate</td><td>0.5</td><td>0.01, 0.1, 0.2</td></tr><tr><td>Max. training sequence length</td><td>1000</td><td>—</td></tr><tr><td>Max. prediction sequence length</td><td>1000</td><td>—</td></tr><tr><td>Early stopping</td><td>10 * 500 steps</td><td>—</td></tr><tr><td>Early stopping criterion</td><td>validation accuracy</td><td>—</td></tr><tr><td>Maximum training time</td><td>50000 steps</td><td>—</td></tr><tr><td rowspan="15">TF</td><td>Encoder + decoder layers</td><td>6 + 6</td><td>—</td></tr><tr><td>Attention heads</td><td>8</td><td>—</td></tr><tr><td>Embedding dimensions</td><td>512</td><td>—</td></tr><tr><td>Hidden layer dimensions</td><td>512</td><td>—</td></tr><tr><td>Position representation clipping</td><td>4</td><td>no clipping</td></tr><tr><td>Dropout</td><td>0.1</td><td>—</td></tr><tr><td>Label smoothing</td><td>0.1</td><td>—</td></tr><tr><td>Optimizer</td><td>Adam</td><td>—</td></tr><tr><td>Adam β2</td><td>0.998</td><td>—</td></tr><tr><td>Batch size / accumulate gradient</td><td>8 * 5000 tokens</td><td>8 * 1000, 8 * 10000</td></tr><tr><td>Batch normalization</td><td>tokens</td><td>—</td></tr><tr><td>Initial learning rate</td><td>4</td><td>0.1, 1.0, 2.0</td></tr><tr><td>Decay</td><td>Noam, 8000 warmup steps</td><td>—</td></tr><tr><td>Max. training sequence length</td><td>1000</td><td>—</td></tr><tr><td>Max. prediction sequence length</td><td>1000</td><td>—</td></tr><tr><td>Early stopping</td><td>10 * 500 steps</td><td>—</td></tr><tr><td>Early stopping criterion</td><td>validation accuracy</td><td>—</td></tr><tr><td>Maximum training time</td><td>50000 steps</td><td>—</td></tr><tr><td rowspan="7">ByT5</td><td>Foundation model</td><td>google/byt5-base</td><td>google/byt5-small</td></tr><tr><td>Max. training sequence length</td><td>512 (SKN), 256 (others)</td><td>—</td></tr><tr><td>Max. prediction sequence length</td><td>1024</td><td>—</td></tr><tr><td>Batch size</td><td>4 (SKN), 8 (others) sentences</td><td>—</td></tr><tr><td>Early stopping</td><td>disabled</td><td>—</td></tr><tr><td>Training time</td><td>10 epochs</td><td>—</td></tr><tr><td>Model selection criterion</td><td>validation loss</td><td>—</td></tr></tbody></table>
D Data pre- and post-processing

<figure><img src="image_4.png" /></figure>

Resulting training instance for the full sentence models (original sentence in the top row, normalized sentence in the bottom row).

<figure><img src="image_5.png" /></figure>

Original verticalized
data, one word per line.
Gloss: 'and I also delivered
a sewing machine to her.'

Resulting training instances for the sliding window models.
The number of training instances corresponds to
the number of source words in the verticalized data.

<figure><img src="image_6.png" /><figcaption>Figure 2: Data preprocessing for full sentence and sliding window models, illustrated on an example of the Swiss German ArchiMob corpus.</figcaption></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>Figure 3: Realignment of normalization output (full sentence model in top half, sliding window model in bottom half). The SRC row represents the source text, the OUT row represents the system output (with erroneous *deren* instead of *dieser*). The two strings are character-aligned using Levenshtein distance, and the alignment links of the SRC word boundaries are used to separate the output words, leading to the verticalized representations.</figcaption></figure>