# The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline

Yves Scherrer, Aleksandra Miletić, Olli Kuparinen

► To cite this version:

Yves Scherrer, Aleksandra Miletić, Olli Kuparinen. The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline. Proceedings of ArabicNLP 2023, Dec 2023, Singapore (Hybrid), Singapore. pp.670-677, 10.18653/v1/2023.arabicnlp-1.73. hal-05311034

HAL Id: hal-05311034

https://hal.science/hal-05311034v1

Submitted on 13 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The Helsinki-NLP Submissions at NADI 2023 Shared Task:
## Walking the Baseline

Yves Scherrer¹,²

Aleksandra Miletić¹

Olli Kuparinen¹,³

first.last@ifi.uio.no

first.last@helsinki.fi

first.last@tuni.fi

¹Department of Digital Humanities, University of Helsinki

²Department of Informatics, University of Oslo

³Faculty of Information Technology and Communication Sciences, Tampere University

## Abstract

The Helsinki-NLP team participated in the NADI 2023 shared tasks on Arabic dialect translation with seven submissions. We used statistical (SMT) and neural machine translation (NMT) methods and explored character- and subword-based data preprocessing. Our submissions placed second in both tracks. In the open track, our winning submission is a character-level SMT system with additional Modern Standard Arabic language models. In the closed track, our best BLEU scores were obtained with the leave-as-is baseline, a simple copy of the input, and narrowly followed by SMT systems. In both tracks, fine-tuning existing multilingual models such as AraT5 or ByT5 did not yield superior performance compared to SMT.

## 1 Introduction

This paper presents the Helsinki-NLP submissions to the NADI 2023 shared tasks. We participated in Subtasks 2 and 3, which consisted in translating dialectal data into Modern Standard Arabic (MSA) (Abdul-Mageed et al., 2023). This was the first time the NADI shared task involved translation, following past tasks on dialect identification and sentiment analysis (Abdul-Mageed et al., 2020, 2021, 2022).

The Arabic dialectal continuum stretches from Morocco in the west to Oman in the east. Various classifications of the dialects have been proposed, ranging from large regions to country-level or even city-level divisions (Bouamor et al., 2018; Habash, 2022). The Arabic language area is also well known for its diglossic situation. While Modern Standard Arabic is used in education, media and culture across the continuum, it is not native to any of the dialectal regions.

The translation subtasks focused on four Arabic dialects: Egyptian, Emirati, Jordanian, and Palestinian. The shared task organizers provided the

MADAR corpus (Bouamor et al., 2018) as the training material for the closed track (Subtask 2), which did not allow for the use of additional training data. The Subtask 3 was described as open track where any additional training material was allowed.

Since our initial experiments showed that neural models were particularly affected by the small size of the MADAR training data, a large part of our efforts went into creating additional parallel data for the in Subtask 3 models. In particular, we focused on freely available monolingual MSA corpora, which we then back-translated to three target dialects, grouping Jordanian and Palestinian together. Adding the back-translated data to the original training corpus allowed our neural models to perform on par with less data-hungry statistical models.

We participated in Subtask 2 with three submissions and in Subtask 3 with four submissions. Our submissions can be divided into four different approaches:

* LAI – the leave-as-is baseline consisting of a copy of the input text,

* SMT – character-level statistical machine translation models;

* NMT – Transformer-based neural machine translation models;

* ByT5 and AraT5 – pretrained sequence-to-sequence models fine-tuned with task-specific data.

Our best performing translation system was SMT for both subtasks, but it was not able to outperform the LAI baseline in Subtask 2, at least in terms of the BLEU score (Papineni et al. 2002; see Section 5.1 for a critical discussion of evaluation measures). Our submissions placed second on both subtasks.

Section 2 describes the data collection and preparation whereas Section 3 outlines the proposed models in more detail. Our results are presented in Section 4 and further discussed in Section 5.
Section 6 offers conclusions of our work.

# 2 Data Collection and Augmentation

## 2.1 MADAR3

The training resource provided by the organizers was the MADAR corpus (Bouamor et al., 2018). The dataset contains the same sentences in different Arabic dialects from 25 cities, as well as in English, French and MSA. The corpus was created by translating sentences from the Basic Traveling Expression Corpus (BTEC) (Takezawa et al., 2007) into Arabic dialects.

We found in early experiments that our models achieved better results when excluding the data from Maghrebi and Yemeni dialects, since the development and test sets do not cover these dialect groups. Therefore, for all our submissions, we use the subset of MADAR that covers the Nile Basin, Levant and Gulf regions. We refer to this subcorpus as MADAR3 throughout the paper.

## 2.2 MSA data

Considering that parallel resources for the target dialects are hard to come by, we focused our collection efforts on monolingual MSA data, taking inspiration from the AraT5 pretraining setup (Nagoudi et al., 2022). In particular, we used the following resources:

**AraNews** is a collection of Arabic newspaper texts from 15 Arab nations, the United States of America and the United Kingdom (Nagoudi et al., 2020).

**Leipzig News** is a dataset of Arabic news curated by the Leipzig Corpora Collection. The data comes from mostly Saudi Arabian news outlets (Goldhahn et al., 2012).

**OSIAN** is an Arabic news corpus crawled from the web (Zeroual et al., 2019). It contains articles from 31 international Arabic news broadcasting platforms.

**Tatoeba** is a project collecting translations of sentences in the web. The data is available in OPUS (Tiedemann, 2012).

**TED** is a corpus of translated subtitles from over 4000 TED talks (Reimers and Gurevych, 2020). The data is available in OPUS (Tiedemann, 2012).

**Wikipedia** is a Wikipedia-based corpus we extracted from the Arabic Wikipedia using WikiEx-tractor (Attardi, 2015)<fnref n="1" />. The extracted data was subsequently sentence-segmented and deduplicated at sentence level (only the exact matches were removed).

<table><caption>Table 1: Size of additional datasets</caption><thead><tr><td>Corpus</td><td>Sentences</td><td>Words</td></tr></thead><tbody><tr><td>AraNews</td><td>59,270</td><td>2,643,313</td></tr><tr><td>Leipzig News</td><td>1,000,000</td><td>23,972,851</td></tr><tr><td>OSIAN</td><td>1,000,000</td><td>21,532,389</td></tr><tr><td>Tatoeba</td><td>47,471</td><td>231,507</td></tr><tr><td>TED</td><td>403,845</td><td>5,652,867</td></tr><tr><td>Wikipedia</td><td>11,368,818</td><td>193,912,867</td></tr></tbody></table>

An overview of corpus sizes is given in Table 1.<fnref n="2" /> Of these resources, AraNews, OSIAN and Wikipedia were used to pretrain AraT5 (V1).

## 2.3 Backtranslation of MSA data

While monolingual target-side data can easily be included into SMT systems in the form of additional language models, this is more difficult for neural models. The most common approach in this situation is to produce synthetic parallel data using backtranslation (Sennrich et al., 2016).

To this end, we reversed our dialect-specific SMT-mono models from Subtask 2 (see Section 3.2 for details) to produce three dialectal versions of all monolingual MSA data presented in Section 2.2. The backtranslated data was used to train or fine-tune the neural models for Subtask 3 (see Sections 3.3, 3.4 and 3.5).

The quality of the backtranslations is most likely poor, but we nevertheless expect backtranslation to work better than simpler data augmentation methods such as noise injection. Since the authors are not speakers of Arabic, the quality of the backtranslations could not be evaluated.

# 3 Models

## 3.1 LAI

As the shared task organizers did not provide an official baseline, we propose the leave-as-is (LAI)

<footnote n="1">¹The extraction was done from the Wikimedia data dump arwiki-20230801-pages-articles-multistream.xml.bz2</footnote>

<footnote n="2">²Note that we did not perform full tokenization of the corpora: the word counts in the table are based on whitespace-delimited tokens.</footnote>
<table><caption>Table 2: Overview of the tested models and their BLEU scores (↑) on the development and test sets. MSA: monolingual MSA data, BT MSA monolingual MSA data back-translated to three dialects. ✓: used for pre-training, ✓: used for training or fine-tuning. EGY: Egyptian, EMI: Emirati, JOR: Jordanian, PAL: Palestinian. The horizontal line separates closed (Subtask 2) from open (Subtask 3) submissions according to the organizer-defined criteria.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Training data</th><th colspan="6">Development set BLEU</th><th colspan="6">Test set BLEU</th></tr><tr><th>MADAR3</th><th>MSA</th><th>BT MSA</th><th>Overall</th><th>EGY</th><th>EMI</th><th>JOR</th><th>PAL</th><th>Subm.</th><th>Overall</th><th>EGY</th><th>EMI</th><th>JOR</th><th>PAL</th></tr></thead><tbody><tr><td>LAI</td><td>—</td><td>—</td><td>—</td><td><strong>15.78</strong></td><td>14.87</td><td><strong>26.31</strong></td><td>12.75</td><td><strong>12.90</strong></td><td>2.2</td><td>14.28</td><td>12.22</td><td><strong>23.13</strong></td><td>11.15</td><td>13.41</td></tr><tr><td>SMT-multi</td><td>✓</td><td>—</td><td>—</td><td>15.62</td><td>15.01</td><td>25.74</td><td>12.52</td><td>12.64</td><td>2.1</td><td>13.60</td><td>12.02</td><td>21.82</td><td>10.46</td><td>12.66</td></tr><tr><td>SMT-mono</td><td>✓</td><td>—</td><td>—</td><td>15.39</td><td><strong>15.91</strong></td><td>17.94</td><td><strong>14.84</strong></td><td>11.78</td><td>2.3</td><td>12.53</td><td>11.91</td><td>16.50</td><td>9.83</td><td>11.42</td></tr><tr><td>NMT</td><td>✓</td><td>—</td><td>—</td><td>2.61</td><td>3.24</td><td>2.52</td><td>0.00</td><td>2.32</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>ByT5</td><td>✓</td><td>✓</td><td>—</td><td>6.63</td><td>6.89</td><td>4.86</td><td>4.94</td><td>7.60</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>AraT5 V2</td><td>✓</td><td>✓</td><td>—</td><td>7.41</td><td>7.61</td><td>5.55</td><td>5.98</td><td>8.01</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>Best competitor</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td><strong>14.76</strong></td><td><strong>16.04</strong></td><td>14.30</td><td><strong>12.55</strong></td><td><strong>13.55</strong></td></tr><tr><td>SMT-multi</td><td>✓</td><td>✓</td><td>—</td><td><strong>19.19</strong></td><td>18.88</td><td>25.66</td><td>17.24</td><td><strong>17.16</strong></td><td>3.1</td><td>17.69</td><td>16.11</td><td>25.81</td><td>15.60</td><td>15.91</td></tr><tr><td>SMT-mono</td><td>✓</td><td>✓</td><td>—</td><td>18.61</td><td><strong>19.03</strong></td><td>26.89</td><td>13.12</td><td>17.14</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td></tr><tr><td>NMT</td><td>✓</td><td>—</td><td>✓</td><td>18.40</td><td>16.78</td><td>25.34</td><td><strong>19.59</strong></td><td>14.36</td><td>3.2</td><td>16.88</td><td>15.17</td><td>24.77</td><td>15.41</td><td>14.45</td></tr><tr><td>ByT5</td><td>✓</td><td>✓</td><td>✓</td><td>17.69</td><td>16.68</td><td>24.90</td><td>16.01</td><td>14.03</td><td>3.3</td><td>16.10</td><td>15.55</td><td>21.79</td><td>13.73</td><td>13.34</td></tr><tr><td>AraT5 V2</td><td>✓</td><td>✓</td><td>✓</td><td>19.14</td><td>18.49</td><td><strong>28.25</strong></td><td>17.19</td><td>14.80</td><td>3.4</td><td>17.46</td><td>15.50</td><td>25.06</td><td>15.97</td><td>15.06</td></tr><tr><td>Best competitor</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td><strong>21.10</strong></td><td><strong>17.65</strong></td><td><strong>28.46</strong></td><td><strong>22.03</strong></td><td><strong>17.29</strong></td></tr></tbody></table>

baseline: an unchanged copy of the input file. We do not suggest that LAI is a potential solution to the task; rather, we introduce it as a way of estimating the task difficulty.

We were unable to beat this baseline with the systems that only use the MADAR corpus for training or fine-tuning in terms of BLEU score. Therefore, we decided to submit LAI as one of our contributions. We think it is interesting to also compare the other participants' systems with this baseline. For example, even the best submitted subtask 2 system scores behind LAI on the Emirati dialect (see Table 2).

## 3.2 SMT

We use a character-level statistical machine translation model based on the Moses toolkit. We split all sentences into character sequences and treat each character as a separate translation unit.<fnref n="3" />

We provide two variants of the SMT approach. **SMT-multi** is a single model trained on all dialects from MADAR3. **SMT-mono** is a collection of 3 models, each of which is trained on the MADAR texts of one major dialect area (Nile Basin, Levant, Gulf). At prediction time, the relevant model is chosen according to the provided dialect labels.

Furthermore, each of the two models is made available in a closed and an open variant. The closed variant contains a single language model

trained on the MSA side of MADAR. The open variant contains a total of 7 language models, corresponding to the different MSA corpora listed in Section 2.2 in addition to MADAR.

## 3.3 NMT

Our neural machine translation method is based on the Transformer architecture. The model was trained with OpenNMT-py (Klein et al., 2017).<fnref n="4" />

We tokenized the data using the unigram model implemented in the SentencePiece library (Kudo and Richardson, 2018), as it has outperformed BPE-based segmentation when the studied texts include inconsistent writing or non-standard language (Kanjirangat et al., 2023). We experimented with three different vocabulary sizes (300, 500, 1000) and found the smallest (300) to offer the best performance.

Furthermore, we found that the NMT model's performance was enhanced by adding a dialect tag at the beginning of the source sentence. We used the three dialect labels of MADAR3.

The NMT model trained on MADAR3 alone did not produce competitive scores. We only submitted an NMT model trained both on MADAR3 and on the backtranslations.

## 3.4 ByT5

ByT5 (Xue et al., 2022) is a multilingual pre-trained model of the T5 family (Raffel et al., 2020)

<footnote n="3">³Character-level models outperformed SMT models with words and subwords in preliminary experiments. Model parameters are presented in Table 5 in the Appendix.</footnote>

<footnote n="4">⁴Experimental details for each model are provided in Table 5 in Appendix A.</footnote>
that encodes all text as UTF-8 encoded byte sequences. It is pre-trained on the multilingual m4C corpus (Xue et al., 2021), with 1.66% of the data in Arabic. ByT5 was used by the winning team (Samuel and Straka, 2021) in the MultiLexNorm shared task (van der Goot et al., 2021), in which the participants had to normalize social media texts of various languages. We expect that Arabic dialect-to-standard translation consists to a large extent of local changes of individual characters. We therefore find that a byte-based model is a good fit for this task.

We fine-tuned the byt5-base model with MADAR3, but found the performance subpar. For our submission, due to computational limitations, we fine-tuned the byt5-small model with a random sample of 1M sentences from our backtranslated data and MADAR3.

## 3.5 AraT5

AraT5 (Nagoudi et al., 2022) is a pre-trained model of the T5 family specifically focused on Arabic, enabling tasks like machine translation into and out of Arabic, summarization, transliteration and other sequence-to-sequence transformation tasks. During the competition, the second version AraT5-V2 was made available. We use the AraT5v2-base-1024 foundation model for our experiments.

Fine-tuning AraT5-V2 on MADAR3 only did not yield competitive results. Instead, we submitted a model fine-tuned on MADAR3 and a random sample of the backtranslations, with a total of 1.4M sentence pairs (15% of the full dataset).<fnref n="5" />

# 4 Results

## 4.1 Results on the development set

Our results on the development set are shown in the middle panel of Table 2 with the official evaluation metric BLEU. Our best submission in Subtask 2 is the leave-as-is baseline (LAI; Section 3.1). The fact that unmodified input achieves better results than machine translation approaches can be taken as an indicator of the difficulty of the closed track task. Note, however, that our best-performing machine translation approach (SMT-multi) is in general less than one BLEU point below LAI.

The inclusion of additional training material in Subtask 3 led to a significant improvement for neu-

ral methods, as illustrated by the results of NMT, ByT5 and AraT5 in the lower part of Table 2. Nevertheless, our best performing approach remains SMT-multi, which scores first overall, and for all individual dialects except for Jordanian. AraT5 is the second best model overall, but note that SMT-mono outperforms it on Egyptian and Palestinian. The scores across different models are the most stable for Egyptian, and they vary the most on Emirati, where the difference between the best (SMT-multi) and worst model (ByT5) is around 4 BLEU points.

## 4.2 Official results

The right-hand panel of Table 2 shows the official results on the test set. For comparison, we added the results of the top-performing system of each subtask.

For Subtask 2, the LAI baseline outperformed both of our SMT systems, and got close to the best submission. It can be noted that our LAI model outperformed the best competitor on Emirati by a large margin, suggesting that models tend to over-normalize this dialect.

For Subtask 3, SMT and AraT5 were our best submissions, as could be expected from the development set scores. However, there is a significant gap to the best competitor, especially for Emirati and Jordanian. We would like to note however that our Subtask 3 submissions rely on similar training data as was used for AraT5 pretraining, but in a smaller volume. In that sense, it may be more relevant to compare our systems with Subtask 2 submissions that are based on AraT5.

Note that in all our experiments we systematically use sentence-level contexts. However, our previous work has shown that contexts of sliding windows of three words can bring significant improvements, especially for the TF-based systems (Kuparinen et al., 2023). This approach requires word-level data alignments which are not trivial to produce. Therefore we defer this to future work.

# 5 Discussion

## 5.1 Evaluation metrics

The BLEU score (Papineni et al., 2002), which was used as the official metric in this shared task, treats each word as an atomic unit and considers a word as wrong even if only one character is incorrect. However, in dialect-to-standard translation tasks, a large amount of differences is expected to concern changes of individual characters. It

<footnote n="5">⁵The samples used for ByT5 and AraT5 differ due to computational time constraints.</footnote>
<table><caption>Table 3: BLEU and chrF scores on the development set.</caption><thead><tr><th>Model</th><th></th><th>Overall</th><th>EGY</th><th>EMI</th><th>JOR</th><th>PAL</th></tr></thead><tbody><tr><td colspan="7" style="text-align: center;">BLEU</td></tr><tr><td>2.2</td><td>LAI</td><td><strong>15.78</strong></td><td>14.87</td><td><strong>26.31</strong></td><td>12.75</td><td><strong>12.90</strong></td></tr><tr><td>2.3</td><td>SMT-mono</td><td>15.39</td><td><strong>15.91</strong></td><td>17.94</td><td><strong>14.84</strong></td><td>11.78</td></tr><tr><td>2.1</td><td>SMT-multi</td><td>15.62</td><td>15.01</td><td>25.74</td><td>12.52</td><td>12.64</td></tr><tr><td colspan="7" style="text-align: center;">chrF</td></tr><tr><td>2.2</td><td>LAI</td><td>45.02</td><td>46.56</td><td>49.47</td><td>40.85</td><td>44.10</td></tr><tr><td>2.3</td><td>SMT-mono</td><td><strong>46.96</strong></td><td><strong>49.11</strong></td><td><strong>51.11</strong></td><td><strong>43.81</strong></td><td><strong>44.69</strong></td></tr><tr><td>2.1</td><td>SMT-multi</td><td>44.96</td><td>46.60</td><td>49.37</td><td>40.81</td><td>43.94</td></tr></tbody></table>

might therefore be interesting to consider metrics that reflects this better, for example the chrF score (Popović, 2015), which is based on the precision and recall of character n-grams.

Table 3 compares the development set BLEU scores with the chrF scores of our Subtask 2 submissions. According to BLEU, LAI is the best performing system, mostly thanks to its good performance on Emirati. SMT-mono is the worst of the three despite winning on two individual dialects. In contrast, according to chrF, SMT-mono outperforms all other systems on all four dialects. The large variation on Emirati has also disappeared.

This suggests that our SMT-mono system could in fact be perceived as better than the higher-ranked LAI baseline. It would be instructive to see which of the two evaluation metrics correlates better with human assessment on this particular task.

## 5.2 Test data domains

While the MADAR corpus contains relatively short and simple sentences from the travel domain, the development and test data provided for the NADI shared task comes from a different source and text domain. It can be interesting to see how the proposed translation models fare on both domains.

To this end, we extracted the test instances from the MADAR3 corpus (which were held out from model training) and evaluated some of our submissions on them. Table 4 provides a comparison of the results on the NADI test data and the MADAR3 test data.

There is a striking difference in terms of LAI BLEU between the two datasets: NADI seems to be much “easier” than MADAR, in the sense that fewer replacements are required. For both datasets, the closed-track SMT model does not do any better than the baseline. The two selected open-track models have very similar performances on the NADI test set, but differ greatly on their performance on

<table><caption>Table 4: Overall BLEU scores for the NADI and MADAR3 test sets.</caption><thead><tr><th>Model</th><th></th><th>NADI</th><th>MADAR3</th></tr></thead><tbody><tr><td>2.2</td><td>LAI</td><td>14.28</td><td>3.48</td></tr><tr><td>2.1</td><td>SMT-multi</td><td>13.60</td><td>3.53</td></tr><tr><td>3.1</td><td>SMT-multi</td><td>17.69</td><td>10.22</td></tr><tr><td>3.4</td><td>AraT5 V2</td><td>17.46</td><td>17.85</td></tr></tbody></table>

MADAR. AraT5, presumably thanks to the large amount of pretraining data, generalizes much better to the more difficult MADAR test set.

## 6 Conclusions

In this paper, we described our participation in the NADI shared task, where we submitted seven systems to two tracks. Our submissions placed second on both tracks. Our strongest translation method was SMT in both tracks, but given the difficulty of the task, it was outperformed by an LAI baseline in the closed track. Neural models closed the gap to the SMT models only with large amounts of additional parallel data obtained through backtranslation.

We would like to note again that our open track submissions do not use any human-translated parallel training data besides MADAR, and that the total amount of training data is smaller than what was used for AraT5 pre-training. This makes our models, in particular the SMT ones, more data efficient than large pretrained models such as AraT5 or ByT5.

We also showed that the participating systems could have been ranked differently with a character-based evaluation metric, which underlines the importance of the selected metrics.

## Acknowledgements

This work has been supported by the Academy of Finland through project No. 342859 “CorCoDial – Corpus-based computational dialectology”.

The authors wish to acknowledge CSC – IT Center for Science, Finland, for computational resources.

## References

* Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, ElMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The Fourth Nuanced Arabic Dialect Identification
Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).

Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics.

Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.

Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Giuseppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor.

Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In International Conference on Language Resources and Evaluation.

Nizar Y Habash. 2022. Introduction to Arabic natural language processing. Springer Nature.

Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic, and Fabio Rinaldi. 2023. Optimizing the size of subword vocabularies in dialect classification. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 14–30, Dubrovnik, Croatia. Association for Computational Linguistics.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senelart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics.

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Olli Kuparinen, Aleksandra Miletić, and Yves Scherer. 2023. Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation. In Findings of EMNLP2023. (accepted).

El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.

El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. Machine generation and detection of Arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

David Samuel and Milan Straka. 2021. ÚFAL at Multi-LexNorm 2021: Improving multilingual lexical normalization by fine-tuning ByT5. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 483–492, Online. Association for Computational Linguistics.
Yves Scherrer. 2023. Character alignment methods for dialect-to-standard normalization. In Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 110–116, Toronto, Canada. Association for Computational Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.

Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA).

Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Barbara Plank, Benjamin Muller, Iñaki San Vicente Roncal, Nikola Ljubešić, Özlem Çetinoğlu, Rahmad Mahendra, Talha Çolakoğlu, Timothy Baldwin, Tommaso Caselli, and Wladimir Sidorenko. 2021. MultiLexNorm: A shared task on multilingual lexical normalization. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 493–509, Online. Association for Computational Linguistics.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics.
## A Experimental Details

We trained all neural models on a single NVIDIA V100 GPU. The SMT models were trained on a Xeon Gold 6230 CPU. We will make the training scripts and the additional data publicly available for the final submission.

<table><caption>Table 5: Hyperparameter settings.</caption><thead><tr><th>Model</th><th>Parameter</th><th>Selected values</th><th>Considered alternatives</th></tr></thead><tbody><tr><td rowspan="7">SMT</td><td>Subword tokenization</td><td>characters</td><td>words, unigram subwords</td></tr><tr><td>Alignment tool</td><td>eflomal</td><td>— (Scherrer, 2023)</td></tr><tr><td>Alignment symmetrization</td><td>grow-diag-final-and</td><td>—</td></tr><tr><td>Language model n-gram size</td><td>10</td><td>—</td></tr><tr><td>Maximum phrase length</td><td>10</td><td>—</td></tr><tr><td>Distortion</td><td>disabled</td><td>—</td></tr><tr><td>Tuning method</td><td>MERT</td><td>—</td></tr><tr><td rowspan="14">NMT</td><td>Subword tokenization</td><td>unigram subwords</td><td>characters</td></tr><tr><td>Encoder + decoder layers</td><td>6 + 6</td><td>—</td></tr><tr><td>Attention heads</td><td>8</td><td>—</td></tr><tr><td>Embedding dimensions</td><td>512</td><td>—</td></tr><tr><td>Hidden layer dimensions</td><td>512</td><td>—</td></tr><tr><td>Position representation clipping</td><td>4</td><td>—</td></tr><tr><td>Dropout</td><td>0.1</td><td>—</td></tr><tr><td>Label smoothing</td><td>0.1</td><td>—</td></tr><tr><td>Optimizer</td><td>Adam</td><td>—</td></tr><tr><td>Adam β2</td><td>0.98</td><td>0.998</td></tr><tr><td>Batch size / accumulate gradient</td><td>2 * 5000 tokens</td><td>—</td></tr><tr><td>Initial learning rate</td><td>0.1</td><td>0.01, 2.0</td></tr><tr><td>Decay</td><td>Noam, 10000 warmup steps</td><td>—</td></tr><tr><td>Max. training sequence length</td><td>1000</td><td>—</td></tr><tr><td>Max. prediction sequence length</td><td>1000</td><td>—</td></tr><tr><td>Training time</td><td>100000 steps</td><td>—</td></tr><tr><td rowspan="6">ByT5</td><td>Foundation model</td><td>google/byt5-small</td><td>google/byt5-base</td></tr><tr><td>Max. sequence length</td><td>512</td><td>—</td></tr><tr><td>Batch size</td><td>8 sentences</td><td>—</td></tr><tr><td>Early stopping</td><td>disabled</td><td>—</td></tr><tr><td>Training time</td><td>5 epochs</td><td>—</td></tr><tr><td>Model selection criterion</td><td>validation loss</td><td>—</td></tr><tr><td rowspan="6">AraT5</td><td>Foundation model</td><td>UBC-NLP/AraT5v2-base-1024</td><td>UBC-NLP/AraT5-base</td></tr><tr><td>Max. sequence length</td><td>256</td><td>—</td></tr><tr><td>Batch size</td><td>12 sentences</td><td>—</td></tr><tr><td>Early stopping</td><td>5 epochs</td><td>—</td></tr><tr><td>Max. training time</td><td>20 epochs</td><td>—</td></tr><tr><td>Model selection criterion</td><td>validation loss</td><td>—</td></tr></tbody></table>