Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation
Authors: Olli Kuparinen
Aleksandra Miletić
Yves Scherrer
Venue: EMNLP 2023 Findings (2023)
Keywords: text normalization
multilingual evaluation
multilingual datasets
linguistic variation
dialects and language varieties
Finnish
Norwegian
Slovene
Swiss German

Abstract: Text normalization methods have been commonly applied to historical language or user-generated content, but less often to dialectal transcriptions. In this paper, we introduce dialect-to-standard normalization – i.e., mapping phonetic transcriptions from different dialects to the orthographic norm of the standard variety – as a distinct sentence-level character transduction task and provide a large-scale analysis of dialect-to-standard normalization methods. To this end, we compile a multilingual dataset covering four languages: Finnish, Norwegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the most successful sequence-to-sequence model architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that a character-level Transformer trained on sliding windows of three words works best for Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we perform an error analysis to evaluate the effect of different data splits on model performance.

Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation
Olli Kuparinen lastétun.
Aleksandra Miletić
Yves Scherrer Airst.lastiffi
first. first.last@hee www.@halsssiiiii....
Department Digital Humanities, University of Helsinki
Faculty of Information Technology and Communication Sciences, Tampere University Department of Informatics, University of Oslo
Abstract
mann, 2019) and user-generated content (UGC) in social media (e.g. van der Goot et al., 2021). However, the existing work on dialect normaliza- tion (e.g., Scherrer and Ljubešić, 2016; Abe et al., 2018; Partanen et al., 2019) remains fragmented: it typically focuses on a single language and uses dif- ferent models, experimental setups and evaluation metrics, making direct comparisons difficult.
Text normalization methods have been com- monly applied to historical language user- generated content, but less often dialectal transcriptions. In this paper, introduce dialect-to-stae ttattadd normalization - i.e., map- ping phonetic transcriptions from different di- alects to the orthographic norm of the standard variety- as a distinct sentence- character transduction task and provide large-scale anal- ysis of dialect-to-siie to-anaddrd normalization meth- ods. this end, we compile multilingual dataset covering four languages: Finnish, Nor- wegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the model
In this paper, we aim to establish dialect-to- standard normalization as a distinct task alongside historical text normalization and UGC normaliza- tion. We make the following contributions:
We compile a multilingual dataset from ex- isting sources and make it available a unified format to facilitate cross-lingual comparisons. The dataset covers Finnish, Norwegian, Swiss German and Slovene. These languages come from three different families and have different morphologi- cal systems. For the two largest datasets (Finnish and Norwegian), we provide different data splits corresponding to different use cases for dialect nor- malization.
most successful requence-to-sequees architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that character- level Transformer trained on sliding windows three words works best Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we per- form error analysis to evaluate the effect of different data splits on model performance.
We test wide range of sequence-to- sequence models that performed well in other nor- malization tasks: statistical machine translation. RNN-based and Transformer-be translation, and byT5, a pre-trained byte-based mul- tilingual Transformer. We compare character with subword tokenizations as well as full-sentence con- texts with sliding windows of three words. We evaluate the models word accuracy, but also pro- vide character error rates and word error reduction rates to facilitate comparison with previous work. Finally, we provide an error analysis on two of the Finnish data splits.'
1 Introduction
Text normalization refers range of tasks that consist in replacing non-standard spellings with their standard equivalents. This procedure is ben- eficial for many downstream NLP tasks, since increases data homogeneity and thus reduces the impact of unknown word forms. Establishing iden- tity between different variants of the same form is also crucial for information retrieval tasks, and in particular for building efficient corpus querying systems. Furthermore, plications for a wider audience, such as spelling and grammar checkers.
neural machine
facilitate building ap-
The data, scripts and model configurations are available at https: /ttppt// thub. com/Helsinki-NW dialect-tor tenecttt-sttadde standard.
Important progress has been made on normaliza- tion of historical texts (e.g.. Tang et al., 2018; Boll-

---

2 Related Work
2.3 Dialect-to-standard normalization
2.1 Historical text normalization
There has been comparatively less research in the domain of dialect normalization. Scherrer and LjubeSić (2016) apply CSMT to Swiss German. They create models for normalizing individual words and entire sentences and show that the larger context provided by the latter is beneficial for the normalization of ambiguous word forms. Lusetti et al. (2018) work different Swiss German dataset and show that neural encoder-decoder mod- els can outperform CSMT when additional target- side language models are included.
Historical text normalization consists in modern- izing the spelling of the text such that it conforms current orthographic conventions. Pettersson et al. (2014) evaluate three different normalization methods a multilingual setup: simple filtering model, an approach based on Levenshtein distance, and an approach using character-leved statistical machine translation (CSMT). They find that CSMT is the overall most promising approach. Scher- rer and Erjavec (2016) use CSMT in supervised and unsupervised settings to normalize historical Slovene data.
Abe et al. (2018) work on the NINJAL cor- pus (Kokushokankokai Inc, 1980), and propose translate Japanese dialects into standard Japanese using a multilingual (or rather, multi-dialeche LSTM encoder-decoder model. The transduction is done on the level of bunsetsu, the base phrase in Japanese, which corresponds to a content word, po- tentially followed by string of functional words.
Tang et al. (2018) and Bollmann (2019) provide multilingual comparisons of neural and statistical MT approaches, whereas Bawden et al. (2022) eval- uate different normalization methods on historical French. In most settings, SMT outperformed neu- ral models. In several settings, BPE-based subword segmentation led to better results than character- level segmentation.
Partanen et al. (2019) compare LSTM-based and Transformer-based character-level NMT models for normalizing Finnish. The authors use contexts of one word, three words, or the entire sentence, the best results being achieved using three words. On the contrary, Hämäläänen et al. (2020) report optimal results while using individual words in normalizing Swedish dialects spoken in Finland. Hämäälänen et al. (2022) use generated dialectal Finnish sentences to normalize Estonian dialects.
2.2 Normalization of user-generated content UGC, typically found on social media, contains various non-standard elements such as slang, ab- breviations, creative spellings and typos. De Clereq et al. (2013) present various experiments on nor- malizing Dutch tweets and SMS messages and show that combination of character-level and word-level SMT models yields the optimal results. Matos Veliz et al. (2019) follow up on this work and show that data augmentation techniques are crucial for obtaining competitive results with NMT models. The MoNoise model (van der Goot, 2019) significantly improved the statc-of-the-art in UGC normalization. It contains several modules such spellchecker, an n-gram language model and dopain-specific word embeddings that provide nor- malization candidates.
Machine translation from Arabic dialects to Modern Standard Arabic (MSA) can also be con- sidered dialect normalization task, although Ara- bic dialects differ from the standard variety to a greater extent than the languages used in our work and therefore contain lexical replacements and re- orderings. The 2023 NADI shared task (Abdul- Mageed et al., 2023) provides a subtask on Arabic dialect translation. The MADAR corpus (Bouamor et al., 2018) is popular resource for Arabic dialect translation and covers 25 dialects. Additionally, Eryani al. (2020) describe the creation of nor- malization corpus for five Arabic dialects, but do not report any experiments on automatic normal- ization. Zhao and Chodroff (2022) similarly report on corpus compilation of Mandarin dialects, but they focus on acoustic-phonetic analysis.
The first multilingual, homogeneous dataset for UGC normalization was published in the context of the Leorr shared task in 2021 (van der Goot et al., 2021). The results of the shared task also supported the usefulness of normalization for downstream tasks such as PoS-tagging and pars- ing. The best-performing submission (Samuel and Straka, 2021) proposed to fine-tune byT5, byte- level pre-trai model (Xue al., 2022), in such a way that normalizations are produced one word at a time.
Each of these works focuses on dialects of a single language and uses different models and ex- perimental setups, as well as different evaluation metrics (including BLEU, word error rate and ac-

---

Speak.
Loc.
Texts
Sentences
Words
Types
MSL
MSLch
Ambig.
SKN
99
50
99
41, 41,407
630,665
106,452
15.23
88.59
5.54%
NDC
438
111
684
126,460
1,684,059
77,950
13.32
57.16
10.14%
ArchiMob
5
6
10,183
82,658
11,110
8.12
42.17
6.36%
GOS
36
10
24
8,621
84,199
12,616
9.77
41.43
8.85%
Table 1: Key figures of the four datasets constituting the multilingual normalization benchmark. Speak. number speakers. Loc. = number locations. MSL mean sentence length in words. MSLch mean sentence length in characters. Ambig. = percentage of types having more than one possible normalization.
SKN
oon syänys seittemän silakkaa aiva nii häntä erellä
(Finnish)
mină olen syönyt seitsemän silakkaa aivan niin häntä edellá
"I have caten seven herrings, that's right, tail first'
NDC (Norwegian)
sjøra vår bil for te påske og får jeg ikke kjøre
får eg
vår bil for til påske
aand I don't get drive
until Easter'
ArchiMob ich ha das
ales inere kasette won ich de schlüssel nüme ha
dezue
(Swiss German)
ich habe das alles in ciner kassette wo ich den schliissel nicht mehr
hahe
dazu
II have it all in a case for which don't have the key anymore"
GOS (Slovene)
se zjemla je prpravlena pugnujena pa ubdajlana pa puvlajčena saj zemija je pripravljena pognojena pa obdelana pa povlečena
'because the soil prepared, fertilised tilled harrowed'
Table 2: Normalization examples of the four languages. The top presents the original phonetic transcription, the middle row the normalized version, and the bottom row provides an English gloss.
curacy). All of these factors make meaningful com- parisons between different approaches difficult.
interviews have been transcribed phonetically two levels granularity (detailed and simplified) and normalized manually by linguists. We only consider the utterances of the interviewed dialect speakers, not of the interviewers. Although the detailed transcriptions have been used for the nor- malization experiments in Partanen et al. (2019), the simplified transcriptions here to make the annotations more consistent with the other lan- guages. The simplified transcriptions do not make certain phonetic distinctions and share the alphabet with the normalized text.
3 Datasets
We propose a multilingual dataset that covers Finnish, Norwegian, Swiss German and Slovene. The dataset is compiled from existing dialect cor- pora, which are presented in detail below. The languages originate from two language families (Uralic and Indo-European) and three branches (Finnic, Germanic, Balto-Slavic). All languages adopt the Latin script. This enables the compari- son of language structure, rather than differences in script. Some quantitative information about the in- dividual corpora is available in Table 1, and Table provides some example sentences.
3.2 Norwegian
The Norwegian Dialect Corpus (Johannessen al., 2009, hereafter NDC) was built part of a larger initiative to collect dialect syntax data of the North Germanic languages." The recordings were made between 2006 and 2010, and typically four speak- eTs per location were recorded. Each speaker ap- pears in an interview with researcher and in an informal conversation with another speaker. We concatenate all utterances of speaker regardless of the context in which they appear.
3.1 Finnish
The Samples of Spoken Finnish corpus (Suomen kielen näytteitä, hereafter SKN) (Institute for the Languages of Finland, 2021) consists of 99 inter- views conducted mostly in the 1960s.2 includes data from 50 Finnish- Finnish-speaking speaking locations, with two speakers per location (with one exception). The
'nttp://urn. //urn. fi/urn:nbn: fi fi: cence: CC-BY.
:bl-222111111 20211 12221, Li-
httpp: http:://wwwweeee tekstlab. uio. na/ bo/scandiasyn! download, html, Licence: CC BY-NC-SA 4.0.

---

Suit Just
Figure I: The three data splits (SKNI on the left, SKN2 in the center, SKN3 on the right) visualized on subset of the Finnish dataset. Each dot represents one speaker. There are generally two speakers per location in SKN.
The recordings were transcribed phonetically and thereafter normalized to Norwegian Bokmål. The normalization was first done with automatic tool developed specifically for this corpus, and its output was manually corrected afterwards. The publicly available phonetic and orthographic tran- scriptions are not well aligned; we automatically re-aligned them at utterance and word level.
3.5 Preprocessing
To ensure the datasets are comparable, we have ap- plied several preprocessing steps: removing punctu- ation and pause markers, substituting anonymized name tags with X, and excluding utterances consist- ing only of filler words. The Slovene data includes some utterances in Italian and German which are normalized to the corresponding standard. They have been excluded from the data.
3.3 Swiss German
The ArchiMob corpus of Swiss German (Samardžic et al., 2016; Scherrer et al., 2019) consists of oral history interviews conducted between 1999 and 2001. The corpus contains 43 phonetically transcribed interviews, but only six of them were normalized manually. We use the interviewee's 's utterances of these six documents for our experiments. The selected texts originate from five dialect areas, covering approximately one third of the German-speaking part of Switzerland.
Experimental Setup
4.1 Data splits
The Finnish and Norwegian datasets contain multi- ple speakers per location, which provides possi- bility to test the generalization capabilities of the models in different scenarios. We create three dif- ferent data splits:
1. Normalizing unseen sentences of seen speakers. We divide each speaker's data in such way that 80% of sentences used for training, 10% for development and 10% for testing. The sentences are selected randomly.
3.4 Slovene
Our Slovene dataset is based on the GOS corpus of spoken Slovene (Verdonik et al., 2013). The original corpus contains 115h of recordings. Two transcription layers are included: a manually tran- scribed phonetic layer and semi-automatically normalized layer (with manual validation).
2. Normalizing unseen speakers seen di- alects. We pick speakers from selected locations for the development and test sets, while the rest of the speakers are used for training. For each loca- tion, at least one speaker is present in the training set. In other words, 80% of speakers are used for training. 10% for development and 10% for testing.
Since the degree of non-standardness in the full corpus was relatively low (16%), we select subset of the data for experiments. We retain speak- ers whose productions contain at least 30% non- standard tokens and who have produced at least 1000 words. This results in a set of 36 speakers from 10 dialect regions.
3. Normalizing unseen dialects. All speakers from a given location are assigned to either the training, development or test set. In other words, 80% of locations are used for training, 10% for development and 10% for testing.
*The alignment script and re-aligned data is available at https:/ thwbb..tttttt..../ www//elsinkiiiiieeeeeeoepo...cee MaaiiniiiLPPMMA hhttps: //ww:///wwwww/ww spwr..spu.... wazz.cc//en. departments/ resear htt//ttttrroo//rchiooo html. Licence: BY-NC-SA 4.0.
The different data splits are visualized in Fig- ure 1. For the smaller and less geographically di- verse Swiss German and Slovene datasets, we only
htttp: handle. menndle.ne//1130 net//1336//111s Licence: CC
BY-NC-SA 4.0.

---

use split 1. For each split, we create three folds with random divisions into train, development and test sets,
one context word on the left and one on the right. At prediction time, only the word in the middle of each chunk considered.
While previous work (Scherrer and Ljubešić, 2016; Partanen et al., 2019) mostly relies on split 1, this setup potentially overestimates the models' normalization capabilities: in a given conversation, utterances, phrases and words are often repeated, so that similar structures can occur in the training and test sets. We argue that splits 2 and 3 more re- alistically reflect dialectological fieldwork, where new texts are gradually added to the collection and need to be normalized.
The preprocessing needed to create input for both entire-sentence and sliding-windows models is illustrated in Appendix D.
Subword segmentation. Tang et al. (2018) and Bawden et al. (2022) found that subword segmenta- tion could outperform character-level segmentation on the task of historical text normalization. We follow this work and experiment with subword seg- mentation well. Several segmentation schemes have been proposed for general machine transla- tion, e.g. byte-pair encoding (Sennrich et al., 2016, BРЕ) or the unigram model (Kudo, 2018). Kan- jirangat et al. (2023) found the unigram model to perform better than BPE on texts with inconsistent writing. This is the case for our speech transcrip- tions and we thus opt to use the unigram model. We train our models with the SentencePiece library (Kudo and Richardson, 2018), and optimize the vocabulary size separately for each dataset.
4.2 Tokenization and context sizes
Text normalization is generally viewed as a charac- ter transduction problem (Wu et al., 2021), and it seems therefore most natural to use single charac- ters as token units. Character tokenization has also been shown to work well for normalization tasks in various recent studies.
Most other character transduction problems, such as transliteration or morphological inflection, are modelled out-of context, i.e., one word at a time. This assumption does not seem accurate for text normalization: as shown in Table 1, between 5 and 10% of word types have more than one possible nor- malization, and disambiguating these requires ac- cess the sentential context. Moreover, depending on the annotation scheme, there are sandhi phenom- ena at the word boundaries (ef. the SKN example in Table 2: syänys instcad of syänyt because of assimilation with the following s) that cannot be taken into account by models that operate word by word. Thus, the most obvious strategy for text normalization is to consider entire sentences and break them up into single character tokens.
4.3 Evaluation
We evaluate the models on word-level accuracy. i.e., the percentage of correctly normalized words. Since the reference normalizations are tied the words in the source sentences, and since the mod- els' output can differ in length from the source sen- tence, we need to re-align the model output with the reference normalization. We apply Levenshtein alignment to the entire sequence pair and split the system output at the characters aligned with word boundaries of the input (sce Appendix D for illustration).
Word-level accuracy lacks granularity and does not distinguish between normalizations that are only one character off and normalizations that are completely wrong. Therefore, we include char- acter error rate (CER) as more fine-grained metric. CER is defined as the Levenshtein dis- tance between the system output and the reference, normalized by the length of the reference. Another advantage of CER is that it can be computed di- rectly on sentence pairs without re-alignment."
This strategy combining long contexts with short tokens leads to rather long token sequences, and NMT approaches especially have been shown to underperform in such scenarios (Partanen et al., 2019). We include two alternative ways of address- ing this issue: (1) by shortening the instances from full sentences to sliding windows of three consecu- tive words, and (2) by lengthening the tokens using subword segmentation.
"The re-alignment code is provided at https: bl pee.tubb.om//Helsiiiiiii httpeeesssiiiiiiiiiaaeeeee---e-e sotialeectto-staddrddd standard/
Sliding windows. Partanen al. (2019) propose split each sentence into non-overlapping chunks of three consecutive words. We adapt this approach and use overlapping chunks of three words to ensure that the model always has access to exactly
//gi thub.
&Following Partanen et al. (2019), we use implementa- tion available at ht tps: http://git //gi www.ihh..ccooooeeeeeeeeeeeeeeeee com/nsmart tinez/ Re-alignment is still necessary for the sliding window models in order to extract the center word of each chunk.

---

Following van der Goot al. (2021), we also report error reduction rates in Appendix C.
use Transformer architecture and are pretrained on masked language modeling task. ByT5 (Xue et al., 2022) is variant of T5 that does not use any subword tokenization model, but rather encodes all text as UTF-8 encoded byte sequences." 12 It is pre-trained on the multilingual m4C corpus (Xue et al., 2021), which includes the four languages of our dataset. We use the byt5-base model and fine-tune separately each normalization task for ten epochs.
We compare the systems to two baselines: the leave-as-is (LAT) baseline corresponds to the per- centage of words that do not need to be modified. The most frequent replacement (MFR) baseline translates each word to its most frequent replace- ment seen in the training data, and falls back to copying the input for unseen words.
5 Methods and Tools
We run all models on our base setup (entire sen- tence instances with character or byte tokenization). In addition, the SMT, RNN and TF models are also trained on subword-segmenter data. Finally, the character-level RNN and TF models are trained on sliding windows." 13
We utilize both statistical and neural machine trans- lation tools trained from scratch, as well as a pre- trained multilingual model. We train (or fine-tune) the models for each of the four languages sepa- rately. Our hyperparameter choices largely follow recent related work on text normalization. The main characteristics of the models are summarized below, and a detailed description the hyperpa- rameters is given in Appendix A. The methods used in the experiments are:
6 Results and Discussion
We evaluate the models presented in Section with the metrics described in Section 4.3. We run the models on the three folds of each data split, and present the average scores and standard deviations for each metric.
SMT. Our statistical machine translation method corresponds mostly to the one imple- mented in the CSMTiser tool. 10 uses the Moses SMT toolkit (Koehn et al., 2007) with 10-gram KenLM language model trained on the training sets."! 11 Scherrer (2023) found eflomal (Östling and Tiedemann, 2016) to produce better character alignment than the more commonly used GIZA++, and we adopt this method. Minimum error rate training (MERT) is used for tuning the model weights, using WER (word error rate, which effectively becomes character error rate in character-level model) as the objective.
The word-level accuracies are presented in Ta- ble 3 and the character error rates in Table 4. We also provide accuracy scores for the development sets in Appendix C. Transformer-based methods appear as the most robust: the Transformer trained on sliding windows best for SKN, ArchiMob and GOS, while the Transformer-basee byT5 produces the best results for Norwegian. For all other cor- pora, byT5 is a close second in accuracy, and is thus the best alternative for entire sentences.
While CSMT and RNN-based methods do not yield the best score for any dataset, they still per- form reasonably well. For the NMT models, using the sliding window instead of sentences enhances the results for all corpora but NDC. Regarding tok- enization, the subwords improve the performance on the large corpora (SKN and NDC) but worsen it on the small corpora (Archimob and GOS). For all datasets, the best results are obtained on the character (or byte) level.
RNN-based NMT. This model uses a bidirec- tional LSTM encoder and unidirectional LSTM decoder with two hidden layers each. The attention mechanism is reused for copy attention.
Ta-based NMT. This model has 6 Transformer layers in the encoder and the decoder, with 8 heads cach. We found in preliminary experiments that position representation clipping was beneficial to the results. All NMT models are trained with the OpenNMT-py toolkit (Klein al., 2017).
We expected rising levels of difficulty between data split and 3, but neither the baselines nor the model outputs confirmed our expectations. The
ByT5. ByT5 is multilingual pre-trained model the T5 family (Raffel al., 2020). These models
12 setup, view charucter-level and byte-level seg- mentation as largely equivalent. Our datasets have vocabulary sizes ranging between and 90 and thus largely below the byte-level upper bound 256.
10 tps: / //github. ctclb/.coocccoooeo cow/clarinee ittssniiniiisrrrrr Contrarily to related work, we did not include any addi- tional target-side language models.
in preliminary experiments, the sliding window sctup did not yield any benefits for SMT and byT5 models.

---

Model
SKNI
SKN2
SKN3
NDC1
NDC2
NDC3
ArchiMob
GOS
LAI
44.6340.37
46.01=1.10
47.96+2.53
32.88+0.0
33.31=1.35
33.7240048
21.3740.33
58.08 58.08+0.71
MFR
84.8740.22
82.75 20.26
83.1920.24
86.9020.06
86.48=1.15
86.72
83.8240.20
83,93=0.52
Full sentence models with character or byte segmentation
SMT
91.22+0.17
89.96-0.38 -0.39
90.36+0.21
91.85+0.08
91.44=0.47
91.55+0.08
08.06+0..3
$5.82+0.96
RNN
89.90A1.14
$9.35=0.88
38.5641.04
93.4240.06
93.0320.39
93.21 49.00
28.9840.56
83,5940.68
TF
92.90+0.35
91.38 00.6
89.75 89.75#1.73
93.99 03.99=0.00
93.68-0.47
93.41 40.67
89. 19.14
85, 85,65#0.61
BYT5
92.9540.11
91. 91.77=1.11
91.90#0.77
94.68
94.51 -0
94.54
90.5749.54
87 87.2640.55
Full sentence models with subword segmentation
SMT
90.18+0.02
88.64=0.36
89.20 39.26*0.13
92.57=0.06
92. 16=0.44
92.20*0.12
87.38+0.13
85 85.23+0.73
RNN
91 91.29+0.09
09.51=1.33
18.90=1.85
92.60=0.20
92. 92. 13=0.07
92.41 10.10
80.64=1.34 86.8 +00.3
86.81
92.2341.0
90.84=2.10
01.21
94.20*0.01
93.85=0.37
93.9640.12
87.6340.37
84,7240.83
Sliding window models with character segmentation
RNN
92.46+0.07
90.74=0.24
91.42+0.24
92.31
92.23=0.64
92.31 +0.20
90.39
86.26+0.58 87.30+0.56
93.44
92.10=0.52
92.38+0.35
93.24+0.10
93.02=0.40
93.19
90.95+0.21
Table 3: Word-level accuracy (T). We report averages and standard deviations over the three folds of cach data split.
Model
SKNI
SKN2
SKN3
NDC1
NDC2
NDC3
ArchiMob
GOS
LAI
14.60=0.11
14.04#042
1
24.54 24.54+0.05
24.28+0.72
23. 23.78400.0
32.45=0.08
14.95=0.27 6.31
MFR
4.68=0.06
5.35
5.18
5.71
5.9440.63
5.7540.12
7.10
Full sentence models with character or byte segmentation
SMT
2.64=0.07
3.00=0.18
2.9120.11
3.0720.03
3.28x0.21
3.1620.04
3.63=0.19
5,30=0.39
RNN
4.46*1.64
5.88+1.31
5.21
2.65
3,49
3.10
1.33
23.20 23,20=10.72
2.29-0.20
3.
4.57
+0.2+
2.48+0.22
2.59+0.33
3.49+0.27
5.71-0.21
ByTS
2.96-0.13
3.67+1.12
3,75+0.99
2.114
2.1940.12
2.1940.08
3.21
5.28-0.19
Full sentence models with subword segmentation
SMT
2.97=0.03
3.46+0.19
3.30
2.96+0.04
3.20
3.14+0.02
4,20=0.08
5.92-0.35
RNN
7.38=0.44
9.5745.20
12.38+4.46
3.34
3.20
3.1440.03
4.84+0.34
28.66=1.06
TF
2.23-0.10
2.82
2.9340.63
2.2740.02
2.4440.18
2.3749.07
4.64#0.15
7.05=0.10
Sliding window models with character segmentation
RNN
2.38=0.01
2.90=0.11
2.73
3.0610.07
3.1240.34
3.0220.06
3.22
5.38=0.23
2.06=0.02
2..3
2.13
2.68+0.06
2.79#0.20
2. 2.68#0.01
3.02
4.98-0.18
Table 4: Character error rates (L).
differences between splits are very small, and most models achieve the lowest results on split 2.
about neural models' tendency to overmodify the predictions (Bawden et al., 2022).
The character error rates presented in Table follow the same pattern as the word accuracies when it comes to the best models. However, for SKN, byT5 drops below sliding window RNN and SMT with this metric. Some of the Finnish byT5 T5 models generate much shorter predictions than the other models, but remains be investigated why this occurs and why it only affects some training runs. Table also highlights poor performance and large standard deviation with sentence-leve RNN on GOS. This is line with earlier findings
6.1 Comparison with previous work
Partanen et al. (2019) worked on the normalization of the Finnish SKN dataset, reporting word error rate (WER) their main metric. Although they use the detailed SKN transcriptions instead of the sim- plified ones, their results are roughly comparable with our SKNI data split (see Table 5). While they were not able to successfully train sentence-level models, our parameterization closes the gap to the chunk models. Their best reported word error rates

---

Partanen et al. (2019)
This work
model (190). The higher number of errors SKN3 coherent with the intuition that this split is more difficult, but the nature of the errors produced by the two models does not fully conform to the expec- tations (see Table 6). In absolute terms, the model trained on SKN3 does produce more inflection and consonant grade errors, but fewer diphthong errors. In relative terms, the SKN3 model produces lower percentage of inflection errors than its counterpart (40% vs 46%). This seems to indicate that split 3 does not preclude the model from learning dialect- related patterns. We hypothesize that this is due to the fact that the training set contains material from the same dialect area as the test set (although not from the exact same location).
RNN (full sentences)
46.52
11.11
RNN (3-word chunks)
5.73
7.59
TF (full sentences)
53.23
7.07
TF (3-word chunks)
6.10
6.59
Table 5:
Comparison with previous work on Finnish (SKN1 data split, all models with character tokeniza- tion). Metric: word error rate (L).
are however slightly lower than ours. Scherrer and Ljubešié (2016) presented normal- ization experiments on the ArchiMob corpus. Our results are largely comparable to theirs. A detailed comparison provided in Appendix B.
To identify the critical point beyond which the cross-lectal model performance would be clearly affected, it could be useful to introduce fourth data split which would exclude larger dialect areas from the train set and test on unseen dialects.
6.2 Error analysis
We examine the effects of different data splits by looking at the output of the sliding window Trans- former on splits and 3 of the Finnish SKN corpus. As reminder, the test set in split contains un- seen sentences from seen texts (and therefore seen dialects). whereas in split 3 comes from unseen locations. It can be expected (1) that the model trained on SKN1 performs better on dialect-specific phenomena, such as normalizations involving diph- thongs, consonant grade and inflection marks; and (2) that the two models behave similarly on phe- nomena that are not dialect-specific, such as capi- talization and proper names.
6.3 Comparison with other tasks
Task
LAI
ERR
Historical norm.
17.5 85.4
72.1 89.9
UGC norm.
63.0 93.1
47.5 80.1
Dialect norm.
21.4 58.1
69.7 92.1
Table Word-level accuracy ranges of the leave-as- is baselines (LAT) and word-level error reduction rate ranges (ERR) for historical normalization (Bollmann, 2019), UGC normalization (van der Goot al., 2021) and dialect normalization (this work, cf. Appendix C),
Error type
SKN1
SKN3
Capitalization
2
1.1% %
2
0.6%
As mentioned above, dialect-to-standard normal- ization shares fundamental properties with histor- 4.7%
Character
13
6.8%
31
9.8%
Consonant grade
3
1.6%
15
Diphthong
2.7%
4
1.3%
ical text normalization and UGC normalization. Here, compare the respective difficulties of these three tasks. Table reports the LAI ranges and ERR ranges of the best systems reported in Bollmann (2019) and van der Goot et al. (2021).
Indiscernible
17
8.9%
32
10.1%
Inflection
88
46.3%
127
39.9%
Proper name
10
5.3%
16
5.0%
Wrong target
48
25.3%
91
28.6%
It can be seen that dialect normalization has the lowest LAI rates on average and thus requires the most changes of the three tasks. The models per- form roughly equally well on both historical and dialectal normalization, whereas UGC normaliza- tion seems to be more difficult task.
Total
190
100%
318
100%
Table 6: Comparison of error types between the models trained SKNI and SKN3.
We analyze model output on the sentences that appear in both test sets and focus on words for which least one model produced an erroncous normalization. We identify 382 such cases. On this set of words, the SKN3 model produces a much higher number of errors (318) than the SKNI
7 Conclusions
In this paper, we present the coalect-to-standaae nor- malization task as a distinct task alongside histori- cal text normalization and UGC normalization. We

---

introduce dialect normalization dataset contain- ing four languages from three different language branches, and use to evaluate various statistical, neural and pre-trained neural sequence-to-sern to-squeenc models.
We voluntarily restrict our dataset to "clean" corpora, ie., interviews transcribed and normalized by trained experts. This contrasts with other data collections specifically aimed at extracting dialectal content from social media (e.g., .., Ueberwasser and Stark, 2017; Mubarak, 2018; Barnes et al., 2021; Kuparinen, 2023). Such datasets compound the features and challenges of both dialect-to-standee normalization and UGC normalization.
In our base setup with models trained on entire sentences with character (or byte) segmentation, the pre-trained byT5 model performs best for all languages and data splits. Moving from character segmentation subword segmentation increases the accuracies for the large datasets (SKN and NDC), but not enough to surpass byT5. contrast, the sliding window approach outperforms byT5 on all languages except Norwegian. The superior performance of byT5 on Norwegian cannot be di- rectly explained by the amount of training data," 14 but it is likely that the closely related languages Swedish and Danish enhance its performance. A further analysis on character error rate shows that the neural models sometimes offer very poor pre- dictions, which are not visible when using accuracy as metric.
We did not perform extensive hyperparame- ter tuning in our experiments, but rather use set- tings that have performed well in other normal- ization tasks. It is therefore conceivable that the performance of NMT models in particular could be improved. Furthermore, specific model architec- tures for character transduction tasks have been proposed, e.g. constraining the attention to be monotonic (Wu and Cotterell, 2019; Rios al., 2021). We did not include such architectures our experiments since they generally only showed marginal improvements.
In this work, we have evaluated the most com- mon and most popular model architectures, but it would be interesting test specific model architec- tures for character transduction tasks, e.g. models that put some monotonicity constraint on the at- tention mechanism (Wu and Cotterell, 2019; Rios et al., 2021). We defer this to future work.
Ethics statement
All our experiments are based on publicly avail- able datasets that were costly to produce. It is important to ensure that these are appropri- ately acknowledged. Anybody wishing to our dataset will also need to cite the publications of the original datasets. Details are given on the resource download page: https://gi tugb com/
Another point to be investigated future work is data efficiency. Our training sets are relatively large in comparison with other character transduc- tion tasks, and it would be useful to see how much the data requirements can be reduced without sig- nificantly affecting the normalization accuracy.
The datasets have been anonymized where nec- essary. Text normalization is explicitly mentioned as a possible research task in the literature present- ing the ArchiMob corpus (Samardžić al., 2016; Scherrer and Ljubešić, 2016), and the SKN dataset has been previously used to evaluate normaliza- tion models. We are not aware of any malicious or harmful uses of the proposed sialect-to-standard normalization models.
Limitations
We see the following limitations of our work:
The proposed multilingual dataset is biased towards European languages and European dialec- tal practices. It may therefore not generalize well to the types of dialectal variation present in other parts of the world and to transcriptions in non-Latin scripts. In particular, there is an extensive amount of research on the normalization of Arabic and Japanese dialects (e.g., Abe et al., 2018; Eryani et al., 2020). We address some of these issues in Scherrer et al. (2023).
Acknowledgements
This work has been supported by the Academy Finland through project No. 342859 "CorCoDial- Corpus-based computational dialectology". The authors wish acknowledge CSC-IT IT Cen- ter for Science, Finland, for generous computa- tional resources.
14German accounts for 3.05% of the byT5 training data, Finnish for 1,35%, Norwegian for 1.33% and Slovene for 0.95%.

---

References
normalization using a Finnish dialect generator. Proceedings of the Third Workshop on Deep Learn- ing for Low-Resource Natural Language Processing, pages 61-66, Hybrid, Association for Computational Linguistics.
Muhammad Modul-Mageed. AbdelRahim Elmadany. Chiyu Zhang, BlMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The First Arabic Nut- ural Language Processing Conference (ArabicNLP 2023).
Mika Hämäläinen, Niko Partanen, and Khalid Alnaj- jar. 2020. Normalization of different Swedish di- alects spoken in Finland. In Proceedings of the 4th ACM SIGSPATIAL Workshop on Geospatial Humani- ties, GeoHumanities 20, page 24-27, New York, NY, USA. Association for Computing Machinery.
Kaori Abe, Yuichiroh Matsubayashi, Naoaki Okazaki, and Kentaro Inui. 2018. Multi-dialect neural machine translation and dialectometry. In Proceedings of the 32nd Pacific Asia Conference on Language, Informa- tion and Computation, Hong Kong. Association for Computational Linguistics.
Institute for the Languages of Finland. 2021. Samples of Spoken Finnish, VRT Version.
Jeremy Barnes, Petter Mæhluum, and Samia Touileb. 2021. NorDial: A preliminary corpus of written Nor- wegian dialect use. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaL- iDa), pages 445-451, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Janne Bondi Johannessen, Joel James Priestley, Kristin Hagen, Tor Anders Áfarli, and Øystein Alexander Vangsnes. 2009. The Nordic Dialect Corpus au ad- vanced research tool. In Proceedings of the 17th Nordic Conference of Computational Linguistics (NODALIDA 2009), pages 73-80, Odense, Denmark. Northern European Association for Language Tech- nology (NEALT)..
Rachel Bawden, Jonathan Poinhos, Eleni Kogkitsidou, Philippe Gambette, Benoît Sagot, and Simon Gabay. 2022. Automatic normalisation of early Modern French. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3354- 3366, Marseille, France. European Language Re- sources Association.
Vani Kanjirangat, Tanja Samardžie, Ljiljana Dolamic, and Fabio Rinaldi. 2023. Optimizing the size of subword vocabularies in dialect classification. In Tenth Workshop on NLP for Similar Languages, Va- rieties and Dialects (VarDial 2023). pages 14-30, Dubrovnik, Croatia. Association for Computational Linguistics.
Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In Proceedings of the 2019 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Minneapolis, Min- nesota. Association for Computational Linguistics.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel- lart, and Alexander Rush. 2017. OpenNMT: Open- source toolkit for neural machine translation. In Pro- ceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Compu- tational Linguistics.
Short Papers), pages
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdul- rahim, Ossama Obeid. Salam Khalifa, Fadhl Eryani. Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Pro- ceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Orphée De Clereq, Sarah Schulz, Bart Desmet, Els Lefever, and Véronique Hoste. 2013. Normaliza- tion of Dutch user-generated content. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013, pages 188, Hissar, Bulgaria.
Kokushokankokai Inc. 1980. Zenkoku hougen danwa database nihon no furusato kotoba syusei (the na- tional dialect discourse database collection).
Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple Arabic dialects. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4130-4133 Marseille, France. European Language Resources Association.
Taku Kudo. 2018 Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia, Association for Computational Linguistics.
Mika Hämäläinen, Khalid Alnajjar, and Tuuli Tuisk. 2022. Help from the neighbors: Estonian dialect

---

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Annette Rios, Chantal Amrhein, Noëmi Aepli, and Rico Sennrich. 2021. On biasing transformer attention towards monotonicity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4474-4488 Online. Association for Computational Linguisties.
Olli Kuparinen. 2023. Murreviikko- dialectologicalyy annotated and normalized dataset of Finnish tweets. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 31-39, Dubrovnik, Croatia. Association for Computational Linguistics.
Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob a corpus of spoken Swiss German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 4061-4066 Portorož, Slovenia. European Lan- guage Resources Association (ELRA).
David Samuel and Milan Straka. 2021. ÚFAL at Multi- LexNorm 2021: Improving multilingual lexical nor- malization by fine-tuning ByT5. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 483-492, Online. Association for Computational Linguistics.
Massimo Lusetti, Tatyana Ruzsies, Anne Göhring, Tanja Samardžić, and Eliisabeth Stark. 2018. Encoder- decoder methods for text normalization. In Proceed- ings of the Fifth Workshop on NLP for Similar Lan- guages, Varieties and Dialects (VarDial 2018), pages 18 18-28, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yves Scherrer. 2023, Character alignment methods for dialect-to-standee normalization. In Proceed- ings of the 20th SIGMORPHOO workshop on Com- putational Research in Phonetics, Phonology, and Morphology, pages 1 110-116, Toronto, Canada. Asso- ciation for Computational Linguistics.
Claudia Matos Veliz, Orphee De Clercq, and Veronique Hoste. 2019. Benefits of data augmentation for NMT- based text normalization of nser-generated content. In Proceedings of the 5th Workshop on Noisy User- generated Text (W-NUT 2019). pages 275-285, Hong Kong, China. Association for Computational Linguis- tics.
Yee Scherrer and Tomaž Erjavec. 2016. Modernising historical Slovene words. Natural Language Engi- neering, 22(6):881-905,
Hamdy Mubarak. 2018. Converting Dialectal Arabic Modern Standard Arabic. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA).
Yves Scherrer and Nikola Ljubešić. 2016. Automatic normalisation of the Swiss German ArchiMob cor- pus using character-level machine translation. In Proceedings of the 13th Conference on Natural Lan- guage Processing (KONVENS).
Robert Östling and Jörg Tiedemann. 2016. Effi- cient word alignment with Markov Chain Monte Carlo. Prague Bulletin of Mathematical Linguistics, 106:125-146.
Yves Scherrer, Aleksandra Miletić, and Olli Kuparinen. 2023. The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). (accepted).
Niko Partanen, Mika Hämäläinen, and Khalid Alnaj- jar. 2019. Dialect text normalization to normative standard Finnish. In Proceedings of the Sth Work- shop on Noisy User- sherrggeeeeated Text (W-NUT 2019), pages 141-146, Hong Kong, China. Association for Computational Linguistics.
Yves Scherrer, Tanja Samardžić, and Elvira Glaser. 2019. Digitising Swiss German: how to process and study a polycentric spoken language. Language Resources and Evaluation, 53(4)7335-76.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Eva Pettersson, Beáta Megyesi, and Joakim Nivre. 2014. multilingual evaluation three spelling normalisa- tion methods for historical text. In Proceedings of the 8th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH), pages 32-41, Gothenburg, Sweden. Association for Computational Linguistics.
Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normaliza- tion. In Proceedings of the 27th International Con- ference on Computational Linguistics, pages 1320- 1331, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text uranffrmer. Journal of Machine Learning Research, 21 (140):1-67.

---

Simone Ueberwasser and Elisabeth Stark. 2017. What's up. Switzerland? A corpus- corpus-based research project in a multilingual country. Linguistik Online, 5(84).
A Experimental details
We trained all neural models on single NVIDIA V100 GPU. The SMT models were trained on a Xeon Gold 6230 CPU. Table 8 presents the average training time and number of parameters for single fold of the largest of our datasets, the Norwegian NDC.
Rob van der Goot. 2019. MoNoise: A multi-lingual and casy-to-use lexical normalization tool. Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstra- 206,, Florence, Italy. Association for
tions, pages 201-206, Computational Linguistics.
Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Bar- bara Plank, Benjamin Muller, Iñaki San Vicente Ron- cal, Nikola Ljubešić, Özlem Çetinoglu, Rahmad Ma- hendra, Talha Çolakoğlu, Timothy Baldwin, Tom- maso Caselli, and Wladimir Sidorenko. 2021. Mul- tiLexNorm: A shared task on multilingual lexical normalization. In Proceedings of the Seventh Work- shop Noisy User- generated Text (W-NUT 2021). pages 493-504-000 509, Online. Association for Computa- tional Linguistics.
Model
Runtime (hh:mm)
Parameters
SMT
28:16
RNN (full sentences)
12:12
9.6 M
RNN (sliding window)
42:28
9.6M
TF (full sentences)
17:27
25.4 M
TF (sliding window)
30:19
25.4
ByT5
27:27
581 M
Table 8: Training runtime (average) and number of parameters for single character-level NDC model.
Darinka Verdonik, Iztok Kosem, Ana Zwitter Vitez, Simon Krek, and Marko Stabej. 2013. Compilation, transcription usage of a reference speech corpus: The case of the Slovene corpus GOS. Language Resources and Evaluation, 47(4):1031-100
Additional details about the model architectures and hyperparameter settings are provided in Ta- ble 12.
Shijie Wu and Ryan Cotterell. 2019. Exact hard mono-
tonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 1530- 1537, Florence, Italy. Association for Computational Linguistics.
B Comparison with previous work on Swiss German
Scherrer and Ljubešić (2016) presented normaliza- tion experiments with CSMT models on the Archi- Mob corpus, albeit with a different data split than in our present work. The most comparable model, CSMT model trained on entire sentences, ob- tains almost identical accuracy score compared to our results (see Table 9). Their best model, a CSMT model with an additional language model and constraints, achieves performance on par with our best model, the Transformer trained on sliding windows.
Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Ap- plying the transformer character-leve transduction. Proceedings of the 16th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1901-1907, Online. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByTS: Towards token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306.
Scherrer and Ljubešić (2016)
This work
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m55 A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2027 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line, Association for Computational Linguistics.
CSMT ILM CSMT MLM
87.59
88.06
90.46
TF (sliding window)
90.95
Table 9: Comparison with previous work on Swiss Ger- man. Metrie: accuracy (T). 1LM one language model, 2LM = two language models.
Liang Zhao and Eleanor Chodroff. 2022. The ManDi corpus: A spoken corpus of Mandarin regional di- alects. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1985- 1990, Marseille, France. European Language Re- sources Association.
C Additional results
Table 10 shows the word-level error reduction rates (ERR). ERR was introduced by van der Goot

---

Model
SKNI
SKN2
SKN3
NDC1
NDC2
NDC3
ArchiMob
GOS
MFR
72.67=0.27
68. 98.04
67.6 67.63*1.96
80.48+0.09
79.74*1.38
79.97*0.22
79.40*0.17
61.67=1.08
Full sentence models with character or byte segmentation
SMT
84.14.0.21
81.4240.37
81.43+1.05
87.8740.11
87.1740.49
87.2540.05
84.7940.44
66.2041.82
RNN
81.7542.14
80.29#1.27
78.014199
90.20+0.10
89.5640.38
89.7644.10
85,9740.8l
60.8442.06
TF
87.19=0.57
84.05+1.30
80,34+2.75
91.04+0.12
90.53+0.51
90.05#0.90
86. 86.17=0.56
63.7741.28
ByT5
87.2640.12
84.7442.33
84.3842.21
92.07+0.14
91.78++0.0
91.76*0.17
87.9940.76
69.6041.34
Full sentence models with subword segmentation
SMT
82.27=0.08
78.96#0.28
79.33=0.94
88.93+0.09
87.9740.16
88.29=0.11
77.53+0.25
64.77*1.56
RNN
84.27=0.18
80.59+2.11
78.66*3.58
88.98+0.31
88.32+0.06
88.51
83.21+0.47
53.79+3.78
TF
87.7940.31
84.9440.44
84. 84.1441.27
91.3540.02
90.5740.06
90.89+0.30
84.25
63.5442.23
Sliding window models with character segmentation
RNN
86.37=0.08
82.83+0.51
83.48+0.88
88.54
88.36+0.72
88. 88.41+0.22
87.76+0.45
67 67.22
TF
88.16
85.37 0.67
85.31 +1.31
89.9340.15
89.5440.40
89.73
88.48
69.72
Table 10: Error reduction rates (1) relative to the LAI baseline.
Model
SKNI
SKN2
SKN3
NDC1
NDC2
NDC3
ArchiMob
GOS
LAI
44.74=0.33
1..90
45.13 81.88+2.53
32.61 10.06
32.7940.52
32.70
21.3820,34
$7.2940.48
MER
84.94=0.20
83.42
86,84+0.09
86.45
16.14
83.59*0.15
83.92+0.38
Full sentence models with character or byte segmentation
SMT
91.83+0.79
90.55+0.25
89.87*1.98
91.75+0.40
91.64+0.45
90.92+0.21
88.43+0.19
86.03+0.15
RNN
90.09*1.58
90.02=1.06
89.18+2.15
93.45+0015
93.20+0038
92.62=0.14
89.33+00.7
82.63+0086
TF
92.89-
92.07 +0.58
90.12
94.04+0.07
93.80+0.42
92.76+0.36
89.46+0.30
85.58+0.10
ByT5
93.31 20.13
92.34=1.45
91.7241.70
94.64 +0.12
94.58+0.33
93.85+0.16
90.55+0.51
86.95+0.87
Full sentence models with subword segmentation
SMT
90.34-0.16
89.39+0.40 +0.40
88.53+2.17
92.62
92.20+0.47
91.52+0.19
87.62
85.35+0.40
RNN
91.51 10.18
90.90
2..5
92.60+0.30
92.34+0.28
91.67+0.25
80.22
80.48+1.00
TF
93.51 =0.08
92.55=0.41
91.76+1.89
94.24+0.08
93.96+0.36
93.29+0.14
88. 88.13+0.65
84.49+0.44
Sliding window models with character segmentation
RNN
92.58+0.12
91.39+0.35
90.7142.18
92.3240.05
92.25+0.46
91.6440.23
90.28+0.06
86.0340.24
TF
93.56-0.04
92.79=0.11
91.79#1.93
93.23+0.17
93.07+0.38
92.53+0.23
90.9120.16
87.3310.26
Table 11: Accuracies obtained on the development sets (T).
(2019) and it represents, roughly speaking, the im- provement of model relative to the LAI baseline. Thus, makes easier to compare models across datasets, which may not be the case with accuracy due to different LAI values. ERR is defined as follows:
accuracies.
ERR- Accuracysysten - Accuracybaselive
1.0 Accuracybaseline
By and large, these results follow the same pat- tern the accuracy and CER scores reported in Section 6.
Table shows the accuracies on the develop- ment sets. They are comparable with the test set

---

Model
Parameter
Selected values
Considered alternatives
Subwords
Methodology
Unigram (SentencePiece)
ВPE
Vocabulary size
200 (SKN)
200, 500, 1000, 2000, 4000
500 (Archimob, GOS)
2000 (NDC)
SMT
Alignment tool
Eflomal
GIZA++
Alignment symmetrization
final-and
Language model n-gram size
10
Maximum phrase length
10
Distortion
disabled
Tuning method
MERT
RNN
Encoder + decoder layers
2+2
Encoder + decoder types
Bi-LSTM + LSTM
Attention type
MLP with reused copy attention
Embedding dimensions
512
Hidden layer dimensions
512
Dropout
0.1
Optimizer
Adagrad
Batch size / accumulate gradient
10000 tokens
* 5000, 1 * 25000
Learning rate
0.5
0.01, 0.1, 0.2
Max. training sequence length
1000
Max. prediction sequence length
1000
Early stopping
10 500 steps
Early stopping criterion
validation accuracy
Maximum training time
50000 steps
TF
Encoder + decoder layers
6+6
Attention heads
Embedding dimensions
512
Hidden layer dimensions
512
Position representation clipping
4
no clipping
Dropout
0.1
Label smoothing
0.1
Optimizer
Adam
Adam B2
0.998
Batch size / accumulate gradient
5000 tokens
8 * 1000, 8 10000
Batch normalization
tokens
Initial learning rate
0.1, 1.0, 2.0
Decay
Noam, 8000 warmup steps
Max. training sequence length
1000
Max. prediction sequence length
1000
Early stopping
10 * 500 steps
Early stopping criterion
validation accuracy
Maximum training time
50000 steps
ByT5
Foundation model
google/byt5-ban
google/byt5-small
Max. training sequence length
512 (SKN), 256 (others)
Max. prediction sequence length
1024
Batch size
4 (SKN), 8 (others) sentences disabled
Early stopping
Training time
10 epochs
Model sclection criterion
validation loss
Table 12: Hyperparameter settings.

---

D Data pre- and post-processig
gliferet g refeertt
Resulting training instance the sentence models (original sentence in the top row, normalized sentence in the bottom row).
d dère- dère hani
und dère
und dieser habe ich auch eine nähmaschine geliefert
uund dies eeeer habe i icc. dère hanin fleser-habe ich. auch-
hani
a maani a
a habe ich auch eine- n è a schiine ine
nēémaschine gliferet
u e e n h m a s c iihie
é
h mhine ne & f
Original verticalized data, one word line. Gloss: 'and I also delivered sewing machine to her.'
i n n ä a c h mnine g l lliffen è a n
eerrt_ Deliefert..
ä h
Resulting training instances the sliding window models.
The numher training instances corresponds the number of source words verticalized
Figure 2: Data preprocessing for full sentence and sliding window models, illustrated on example of the Swiss German ArchiMob corpus.
feret_
OUT:
uun
und
und
dêre deren hani habe ich auch
eine néémaschine nähmaschine
gliferet geliefert
SRC:
Gnnd dere
ünd und dère deren hani habe ich
OUT:
uund
doren
SRC:
han
auch
OUT:
e
eine nähmaschine
SRC:
nèèmaschine
gliferet
geliefert
OUT:
Figure 3: Realignment of normalization output (full sentence model in top half, sliding window model in bottom half). The SRC row represents the source text, the OUT row represents the system output (with erroneous deren instead of dieser). The two strings are character-aligned using Levenshtein distance, and the alignment links of the SRC word boundaries used separate the output words, leading the verticalized representations.


Reviews:
[{"review_id": "OTHWlOShbg", "reviewer": "Reviewer_fuGZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your feedback. Since the reason to reject you cite is shared by Reviewer 2, the detailed response to this criticism will be given in our response to Reviewer 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is nicely written and includes a thorough discussion of the results and error analysis. ", "reasons_to_reject": "The paper focuses on 4 European languages, which the authors already acknowledge in their limitations, and as such it is difficult to predict how well the proposed methodology would work in other languages.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses normalizing dialectal text to standard orthography. It presents a method based on sequence-to-sequence transformation using MT models, and it includes a multilingual dataset covering 4 European languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rgFeU2sMH6", "reviewer": "Reviewer_cje3", "summary": "", "questions": "Question A: Do the authors conduct multilingual training/fine-tuning experiments ? Can monolingual performance be improved with resources from closely-related variants ?\n\nQuestion B: The study focuses on clean transcripts (i.e. utterances normalized by trained experts). How would it generalize to informal scenarios ?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__*Lack of linguistic diversity in the sample and unclear generalization capacities of our models*__\n\nYou state that our sample lacks linguistic diversity since it only uses data in Latin script in four languages “from the European family” which you consider “highly related”. We contest this characterization. Finnish is typologically and phylogenetically distinct from the other three languages. It belongs to the Uralic language family, and not the Indo-European family. It is also an agglutinative language, while the other three are fusional. And while Norwegian and Swiss German are both from the Germanic branch of the Indo-European family, Slovene belongs to the Balto-Slavic branch, known to have a richer inflectional morphology. These differences are highly relevant for the normalization task (and NLP in general) because these morphological systems tend to yield different amounts of unique surface forms. We recognize that this aspect of the dataset is not sufficiently described in the current version of the paper. We will amend this in the final version. \n\nWhen it comes to working only on Latin-based scripts, we are well aware of this aspect of our work and address it in the Limitations section (l. 573). We do not make claims about the generalizability of our methods to other writing systems. Indeed, it can be reasonably expected that non-alphabetic writing systems would require different techniques. We therefore preferred to focus on an in-depth evaluation and analysis of the four languages in our dataset. Introducing other scripts and languages is a promising avenue for future work but is out of the scope of this paper. \n\nFinally, note that the data used in Abe et al. (2018) is available under restrictive licensing conditions that would have precluded us from remixing and redistributing the dataset. \n\n\n\n__*Not exploring multilingual learning approaches*__  \n__*Question A: Do the authors conduct multilingual training/fine-tuning experiments ? Can monolingual performance be improved with resources from closely-related variants ?*__\n\nWe respectfully disagree with your statement that we do not explore multilingual settings. In fact, one of the four models we use is byT5, which is a multilingual language model. It is actually our best performing model across the board. It is only surpassed by other techniques when sliding trigram windows are used as context, and even in this setup it scores first on Norwegian. We explicitly discuss the presence of related languages in byT5’s training data as the reason for its performance (l.541-553). \n\nGiven the linguistic differences between these four languages that we cited above, we do not expect significant effects through cross-lingual transfer learning based on them. Furthermore, the languages we work with can be considered as high resource when it comes to dialectal data. This is why Hämäläinen et al. (2018) use Finnish to boost performances on Estonian. We believe it is unlikely that working in the opposite direction would yield significant improvements. \n\n__*Marginal contributions when it comes to the dataset*__\n\nWe explicitly state in the Introduction (l.055) that the dataset is built from existing resources. Our goal was not to produce a novel resource but rather to unify the existing ones and make them available in a format that will facilitate further multilingual research and cross-lingual comparisons for this task. We consider our main contributions to be the definition of the evaluation protocol and the tested models. We will clarify this in the final version of the paper. \n\n__*Question B*__: We have also conducted experiments on dialectal data collected from social networks in one of the languages used in the present study. The best overall results were achieved by byT5 and SMT, with Transformer- and RNN-based models scoring significantly lower. For some dialects, there were also significant differences between the top two methods. We will add the reference to the final version of the paper. \n\n__*Missing References*__: We kindly thank you for pointing out that this work is not referenced in our paper. We will add it to the final version. \n\nTo conclude, we agree that some aspects of our work are suboptimal, but we challenge their status as reasons to reject. We believe that these limitations are not fundamental flaws in our research design that affect the soundness of our work. Since you do not cite any major concerns with our experimental setup or the interpretation of the results, we find it difficult to understand the soundness grade of 2 and the reproducibility grade of 3. We would appreciate any clarifications on these points to help us improve the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper compiles a multilingual dataset, and conducts consistent benchmarking across modeling methods, with detailed error analyses. The obtained results are comparable with previous works, while being more unified and observable across languages.", "reasons_to_reject": "The paper introduces a multilingual dataset, compiled entirely of previous works on dialectal research. Regarding this aspect, the paper makes marginal contributions, as the employed corpora are derived completely from previous works, albeit with a few modifications in splitting and preprocessing.\n\nIn addition, given the availability of dialectal corpora in other languages, especially those with non-latin scripts [2][3], the compiled multilingual dataset is limited in linguistic diversity, as it’s composed from four European languages with latin scripts only, and thus the obtained findings can be biases towards these languages and do not necessarily generalize to others. The study depicts **large-scale evaluation** mostly in terms of **sample sizes**, rather than **broad linguistic coverage**.\n\nAlthough the study denotes multilingual evaluation, and the languages being used are related (most belong to the Indo-European family), the experiments focus on monolingual settings rather than multilingual ones (i.e. the study benchmarks models across languages, but put little effort in cross-language linking). There were no discussions nor experiments discussing the inter-relation between languages and how multilingual training/fine-tuning, or knowledge transfer methods,  can affect monolingual performance on the task, especially between closely-related languages. In contrast, previous research has reported complementary improvement, albeit in a small-scale setting [1]. As the paper centers on multilingual evaluation, these aspects should have received more attention.\n\n\n\n[1] Mika Hämäläinen, Khalid Alnajjar, and Tuuli Tuisk. 2022. Help from the Neighbors: Estonian Dialect Normalization Using a Finnish Dialect Generator. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 61–66, Hybrid. Association for Computational Linguistics.\n\n[2] Kaori Abe, Yuichiroh Matsubayashi, Naoaki Okazaki, and Kentaro Inui. 2018. Multi-dialect Neural Machine Translation and Dialectometry. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, Hong Kong. Association for Computational Linguistics.\n\n[3] Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A Spelling Correction Corpus for Multiple Arabic Dialects. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4130–4138, Marseille, France. European Language Resources Association.", "questions_to_authors": "Question A: Do the authors conduct multilingual training/fine-tuning experiments ? Can monolingual performance be improved with resources from closely-related variants ?\n\nQuestion B: The study focuses on clean transcripts (i.e. utterances normalized by trained experts). How would it generalize to informal scenarios ?", "paper_topic_and_main_contributions": "The paper explores a large-scale multilingual study of dialect-to-standard text normalization, covering four languages. The authors compile a multilingual dataset from previous available corpora and conduct detailed experiments. With consistent settings across languages, the authors benchmark popular baselines in a unified manner and derive meaningful findings.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Mika Hämäläinen, Khalid Alnajjar, and Tuuli Tuisk. 2022. Help from the Neighbors: Estonian Dialect Normalization Using a Finnish Dialect Generator. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 61–66, Hybrid. Association for Computational Linguistics."}}, {"review_id": "jX3OkKThCK", "reviewer": "Reviewer_xDQG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for recognizing positive aspects of our work and for acknowledging the evolution of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written paper and was easy to follow and understand. There are several strengths of this work:\n\n- Providing different splits of the same dataset to test different scenarios. This is usually rare in many NLP papers but is very important and usually highlights any artificial gains/losses in systems that can go unseen. Even though in this specific work, the splits did not affect the performance of the system, it is still helpful for future system evaluations.\n- Appropriateness of the baselines and the different configurations for the systems used for this task.\n- The multilingualism of the benchmark.\n\n", "reasons_to_reject": "I have reviewed an earlier version of this paper and all the concerns that I had were addressed. No major reasons to reject this work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a benchmark for evaluating multilingual (four European languages from different families) dialect-to-standard normalization. The main task is to convert a broad phonetic transcription of dialect(s) into orthography of the standard variety. The authors evaluated several seq-2seq and MT systems and different splits and normalizations on existing datasets.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a benchmark for evaluating multilingual (four European languages from different families) dialect-to-standard normalization. The main task is to convert a broad phonetic transcription of dialect(s) into orthography of the standard variety. The authors evaluated several seq-2seq and MT systems and different splits and normalizations on existing datasets.

Reasons To Accept:
- The paper compiles a multilingual dataset, and conducts consistent benchmarking across modeling methods, with detailed error analyses. The obtained results are comparable with previous works, while being more unified and observable across language
- Providing different splits of the same dataset to test different scenarios. This is usually rare in many NLP papers but is very important and usually highlights any artificial gains/losses in systems that can go unseen. Even though in this specific work, the splits did not affect the performance of the system, it is still helpful for future system evaluations.
- Appropriateness of the baselines and the different configurations for the systems used for this task.
- The multilingualism of the benchmark.


Reasons To Reject:
- The paper focuses on 4 European languages, which the authors already acknowledge in their limitations, and as such it is difficult to predict how well the proposed methodology would work in other languages.
- The paper introduces a multilingual dataset, compiled entirely of previous works on dialectal research. Regarding this aspect, the paper makes marginal contributions, as the employed corpora are derived completely from previous works, albeit with a few modifications in splitting and preprocessing.
Decision: Accept-Findings