LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages
Authors: Milind Agarwal
Md Mahfuz Ibn Alam
Antonios Anastasopoulos
Venue: EMNLP 2023 Main (2023)
Keywords: language identification
resource creation
machine translation
low-resource languages
TL;DR: Hierarchical method (LIMIT) based on learned confusion patterns of a root model for language identification of low-resource languages

Abstract: Knowing the language of an input text/audio is a necessary first step for using almost every NLP tool such as taggers, parsers, or translation systems. Language identification is a well-studied problem, sometimes even considered solved; in reality, due to lack of data and computational challenges, current systems cannot accurately identify most of the world's 7000 languages. To tackle this bottleneck, we first compile a corpus, MCS-350, of 50K multilingual and parallel children's stories in 350+ languages. MCS-350 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in low-resource Indian and African languages. Second, we propose a novel misprediction-resolution hierarchical model, LIMIT, for language identification that reduces error by 55% (from 0.71 to 0.32) on our compiled children's stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark. Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch.

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages
Milind Agarwal
Md Mahfuz Ibn Alam Antonios Anastasopoulos
Department of Computer Science, George Mason University
[magarwa, malam21, (aronis)egguu segmuuedd. edu
Abstract
Knowing the language of an input text/audio is necessary first step for using almost every NLP tool such as taggers, parsers, or transla- tion systems. Language identification is well- studied problem, sometimes even considered solved; in reality, due to lack data and com- putational challenges, current systems cannot accurately identify most of the world's 7000 languages. To tackle this bottleneck, we first compile corpus, MCS-350, of 50K multilin- gual and parallel children's stories in 350+ lan- guages. NCS-358 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in low- resource Indian and African languages. Second, propose novel misprediction-resoleee erarchical model, LIMIT, for language identifi- cation that reduces error by 55% (from 0.71 to 0.32) our compiled children's 's stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark. Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch.'
Figure 1: Most languages in our dataset are from the Indian Subcontinent and Sub-Saharan Africa, with sig- nificant minorities from Europe (primarily in the role of the high-resource language parallel translation available for each story). Color broadly indicates continent region (North America, South America, Africa, Europe, Asia, Oceania) and size indicates number of languages per country in our dataset.
hi-
ate amount of computational resources, further dis- incentivizing researchers to work towards including these languages in modern NLP systems.
A popular data collection approach is large- scale web mining (Tiedemann and Nygaard, 2004; Bañón et al., 2020; Schwenk et al., 2021b), where large parts of the internet are scoured to find train- ing data for data-hungry NLP algorithms. When faced with sentence phrase, such algorithms must know how to reliably sort this text into the ap- propriate language bucket. Since the web is replete with content in variety of languages, model needs to recognize text in a sufficiently large num- ber of these languages with high accuracy. Identi- fying parallel bitext is even more demanding as a machine translation system must also be available to correctly identify and align parallel data (Vegi et al., 2022; Kunchukuttan et al., 2018). This data- collection paradigm becomes inaccessible for low- resource languages because high-quality transla- tion models usually require substantial amounts of parallel data for training, which is often unavailable. Without high-quality language identification and
1 Introduction
Building natural language processing (NLP) tools like machine translation, language identification, part of speech (POS) taggers, etc. increasingly requires more and more data and computational resources. To attain good performance on a large number of languages, model complexity and data quantity must be increased. However, for major- ity of the world's 7000 languages, large amounts of data are often unavailable which creates high bar- rier of entry (Blasi et al., 2022; Joshi et al., 2020; Khanuja et al., 2023). Increasing model complexity for large-scale models also requires disproportion-
Data, code, and models are publicly available on GitHub under permissive licenses. Repository: https://gtt thub. com/nagarw/limit

---

translation system, it becomes practically impossi- ble to mine the internet for relevant text during such collection efforts. Additionally, mispredictions by language identification and data collection algo- rithms can increase inter-class noise, reducing the crawled data's 's quality, and harming performance in downstream tasks without strong quality evaluation metrics (Kocyigit et al., 2022).
Family
Languages
Sentences
Niger-Congo
129
142605
Indo-European
84
169823
Nilo-Saharan
22
23204
Sino-Tibetan
21
19264
Austronesian
18
28096
Afro-Asiatic
15
20266
Dravidian
13
35638
How can we address these challenges and build high-quality identification and translation for low-
Austro-Asiatic
10
22989
resource languages?
Table I: Our compiled dataset MCS-350 350 contains stories from diverse set of languages families, mostly coming from Africa and India. Prominent language families with with 20K+ sentences across languages shown.
Resource Creation Highlighting the need for resource creation in low-resource languages, first share new parallel children's stories dataset, MCS-350, created using two resources: African Sto- rybooks Initiative and Indian non-profit publish- ing outfit Pratham Books' digital repository Story- weaver (available under permissive Creative Com- mons licenses). The combined dataset includes original and human-translater parallel stories in over 350 languages (visualized in Figure 1) and we merge, preprocess, and structure it it eas- ily utilizable by NLP researchers for training and benchmarking 2$.).
ing large multilingual models for a new set of languages and still outperforms existing systems. Large multilingual models often require thousands of sentences for training, et al., 2022) collects and trains tences per language. On the other hand, for many low-resource languages in India and Aftica, we may not even be able to collect 1000 sentences at first 2. Also, contrast with other recent work hierarchical language identification (Goutte et al., 2014; Lui et al., 2014; Bestgen, 2017; Jauhiainen et al., 2019), our work stands out because it ac- counts for mispredictions made by existing trained models. Unlike other work, it does not predict group/language family first, but rather directly learns confusion relationships between language pairs (which may not be from the same language family). By leveraging hierarchically organized units on top of a root model, we avoid complete retraining, saving computational resources, while increasing coverage into many new and understud- ied languages and language pairs (especially those between two low-resource languages) ($4).
AfroLID (Adebara 4000 sen-
Machine Translation Armed with parallel sto- ries in many low-resource African and Indian lan- guages, we tackle machine translation in resource- constrained situations next. If we aim to collect parallel data low-resource languages, language identification itself is insufficient and we need high- quality translation models as well. We utilize a pre- trained multilingual translation model (Alam and Anastasopoulos, 2022) and explore training with hierarchical language-leve. and language family- level adapter units to translate children's stories at the page level ($3).
Language Identification Finally, we take on the biggest bottleneck in low-resource language data collection efforts language identification. We pro- pose LIMIT iisiidentification-aasse hierarchi- cal modeling approach for language identification, that utilizes data and computational resources ef- ficiently and shows cross-domain generalization. The proposed approach is exciting because unlike previously published language identification mod- els like AfroLID (A (Adebara et al., 2022), CLD3 (Sal- cianu et al., 2020) and Franc LIMIT avoids train-
To summarize, our main contributions are:
1. We compile MCS-350, dataset of 50K+ parallel children's 's stories from African Storybooks Ini- tiative and Storyweaver in 350+ languages ($2).
2. We share machine translation benchmark en- abling translation evaluation in more than 1400 new translation directions ($3).
3. We propose LIMIT, misidentiifiation-baan hierarchical model. that can use limited data to better identify low-resource languages ($4).
Pht //ww:: afri www.ccnstorybook.... htt //stoyywveer. org. //ttps://github... com/wocorm/franc/

---

Dataset
New languages
New pairs
Script
Languages
Examples
Microsoft
67
2835
Devanagari
38
Hindi, Marathi
FLORES-200
51
1449
Cyrillic
14
Russian, Bulgarian
OPUS
82
2853
Arabic
8
Arabic, Persian
Tibetan
3
Tibetan, Ladakhi
Table 2: MCS-350 enables MT evaluation between 1400+ new pairs compared existing benchmarks.
Telugu
3
Telugu, Konda
Odia
Odia, Ho, Kui
Table Our dataset contains stories in many writing systems other than Latin, especially those from the In- dian Subcontinent. Prominent non-Latin writing sys- tems in MCS-350 shown above.
2 MCS-350 Data Curation
We identify two large-scale parallel repositories African Storybooks Initiative and Pratham Books' Storyweaver, both under permissive Creative Com- mons Licenses, with their storybooks available for non-commercial and research use. African Story- books Initiative hosts parallel translated and human- verified children's stories in over 200 African lan- guages. Pratham Books a non-profit Indian pub- lisher that aims to increase literacy of children and adults alike in Indian languages. Their digital repos- itory, Storyweaver, publishes parallel translated sto- ries in 300+ languages. This includes not only In- dian languages but also African, European, and Indigenous languages from the Americas.
2.2 Multilingual Documents
MCS-350 contains multilingual stories with lan- guage identifiers denoted by Lı for story mul- tilingual in L1 and L2. Such stories include text in multiple languages within the same page. Text may be code-mixed or consecutively presented. To ex- tract as many parallel sentences as possible sup- port vulnerable languages and also create new trans- lation directions, we employ string-similarity based matching to identify the segments corresponding to the high-resource language in the pair, and there- fore automatically generating parallel sentences from 10K pages across 52 languages. E.g. through this process, we extracted 1000+ sentences in Kui (0 sentences pre-extraction), a minority Dravidian language with about 900K native speakers. We manually verified all extracted monolingual text after using string matching on multilingual stories.
2.1 Parallel Dataset
We collect stories through a mix of web scraping and public APIs, preprocess them to remove mis- matched/incorrect text, extract monolingual text for language identification and parallel text for ma- chine translation. We maintain metadata about au- thors, translators, illustrators, reading level, parallel translations, and copyrights for each story. We re- move stories that are either empty or those from non-English languages that have over 50% pages containing majority English text with 90% con- fidence using langdetect (Nakatani, 2010). This leaves us with ~52K stories.
2.3 Language Varieties/Lects
We attempt to separate language varieties/lects into unique prediction classes if there is sufficient train- ing data for them (2 1000 sentences). If an ISO code is unavailable for the lect, we assign class name with the ISO code and the subdivision speci- fied as: ISO ISO_ SUBDIVISION, For instance, we sep- arated Gondi's South Bastar lect (GON BASTAR, 4000+ sentences) from the generic language code for Gondi (GON). For fair evaluation and compar- ison, we provide manual mappings for any non- standard identifiers from the output space of vari- ous language identification tools. Lects with too lit- tee data are merged into their parent language, e.g.. "Bangla (Bangladesh)" merged into "Bengali".
Note that both African Storybooks Initiative and Pratham Storyweaver human verify stories and lan- guage. However, there are several abandoned trans- lation projects and completed but unverified stories that need automated checking. Therefore, our pre- processing is meant for unverified stories, and may introduce noise in the collected data. By improv- ing the preprocessing filters, we can likely further improve the quality of the unverified stories in the corpus. Collected stories in the pre-merge stage are available with their associated metadata in the repository.
2.4 Data Overview
MCS-358 covers over 350 languages from a diverse pool of language families. In Table I, we share the

---

Model
AvgAL.
Avg BAFRI-AFRI
AV8x-RNG
AVg
AVBENG-X
Avgy-PR
AVgERA AYBERA-Y
Baseline
11.87
10.19
18.79
13.20
15.64
12.55
(6.31)
(5.06)
(7.75)
(8.19)
(5.22)
(5.81)
L-Fine
19.52
18.21
30.38
17.46
21.93
17.86
(10.33)
(10.06)
(13.63)
(8.46)
(4.87)
(6.86)
F-Fine
24.93
23.58
35.66
25.26
27.06
21.36
(11.74)
(11.31)
(14.36)
(13.72)
(6.00)
(7.32)
Unique Pairs
88
58
16
16
14
14
Table 4: spBLEU across 176 translation directions involving African languages, that including phylogenctic information helps in translation, with the family-basee F-Fine model showing the best performance, on average. AFrirIFa denotes the overall average spBLEU of translation between two African languages. AVEx/Y-ENGFFRA and Avg BeNNiRRA-X/V denote translating into and out of English/French respectively. Parentheses below the averages represent standard deviations. Basel ine refers to a DeltaLM model finetumed on 26 languages without adapters. We can that is harder translate out of English than into English.
Lang Pair
A
Lang Pair
A
Machine Translation Benchmark
ENG-XHO
20.1
ENG-HAU
18.8
While it is true that resource creation in low- resource languages requires fine-grained and high- quality language identification, collecting parallel data additionally requires high-quality MT ($1). this section, we explore phylogeny-base -bsed hierarchi- cal adapter units to improve translation quality be- tween two African languages, and between African languages and English/French.
FRA-LUG
3.6
NSO-LUG
3.0
LUG-KIN
2.9
KIN-LUG
2.4
NYA-LUG 2.1
ENG-KAM
1.8
IBO-LUG 1.7
ENG-LUG
1.5
ZUL-LUG 1.5
FRA-TSO
1.3
XHO-LUU 1.2
FRA-YOR
1.1
NSO-TSO 1.0
AMH-LUG
1.0
Table 5: Despite MCS-350 and FLORES-200 having widely different domains, several translation directions see cross donss- improvements. A indicates spBLEU improvements in the F-F ine model over the Baseling ine
Data
We exploit the parallel nature of children's stories in MCS-350 and ensure that all training stories are separate from test (1000 pages) stories. This is done to get a more realistic estimate of translation quality on new stories. For languages with 1000 pages across stories, we use 500-page test sets.
number of languages and the number of sentences in each language family in the dataset. The data is roughly evenly split between stories from the large Niger-Cone Congo and Indo-Europcan language families, with sizeable minority in other language families like Nilo-Saharan, Sino-Tibetan, Austrone- sian, Dravidian, Creole, etc. About 70% of the dataset's languages the Latin script or its tended variants with diacritics. However, the data is still quite typographically rich, and stories with non-Latin scripts are abundance, enumerated in Table
3.2 Experimental Settings
As our baseline, we used the model from Alam and Anastasopoulos (2022), which the best- performing publicly available model from the WMT Shared Task on Large Scale Evaluation for African Languages (Adelani et al., 2022). 6 They first fine-tuned the DeltaLM² model (Ma et al., 2021) in 26 languages. After that, they added lightweight anguaag-sspecffi adapter layers (Pfeif- et al., 2022) and fine-tuned only the adapters in those 26 languages. We can either use sin- gle adapter per language (L-Fine) or organize the adapters in phyylogenetically
Compared to highly multilingual translation benchmarks like NTREX (parallel data of 128 lan- guages; Federmann et al., 2022), FLORES-200 (n- way, 200 languages; NLLB Team et al., 2022), or OPUS-100 (parallel data for 99 languages to/from English; Aharoni al., 2019), our benchmark in- troduces up to 82 new languages leading to more than 1400 new language pairs (see Table 2).
hierarchy
Ranked third the Shared Task. Top two systems were industry submissions that are not publicly available.
? Phttps: /akka ms/ ms/deltalm

---

Model
Fi
Supported
Common
Total (with LIMIT)
CLD3 (Salcianu et al., 2020)
0.11
101
81
376
langid.py (Lui and Baldwin, 2012)
0.09
97
73
380
Frane
0.18
369
116
609
fastText (Joulin et al., 2017)
0.10
176
117
415
HeLI-OTS (Jauhiainen et al., 2022a)
0.13
200
81
475
Table 6: This table shows different popular language identification ssytems, their Fi scores MCS-350. supported
languages, common languages, and total coverage with LIMIT. Franc, trained on UDHR data, outperforms other systems both on performance and coverage, and will serve as the root model for our experiments. Macro Fi score is computed across all 355+ languages to identify system with the best overall coverage and accuracy.
(F-Fine) so that similar languages share language- family and genus-level adapters (Faisal and Anas- tasopoulos, 2022). We perform both L-Fine and F-Fine experiments using the publicly available code 8 and also share an additional baseline by finetuning the DeltaLM model without adapters. Details phylogenetic trees and reproducibility are in Appendix $A.3.
tice cross-domain improvement some translation directions involving two African languages such as Franda-Kiyyrrannde (2.9 points) and Northern Sotho-Ganda (3.0 points). Exhaustive results for other language pairs can be found in Appendix $B.
Language (Miisddentificatioo Benchmark
Language identification (LID) affects low-resource language resource creation efforts severely (Jauhi- ainen et al., 2019; Schwenk al., 2021a) because to collect data, we need accurate language iden- tifiers that themselves need high-quality data to train(Burchen et al., 2023). creating a vicious cy- cle. Low-quality systems often make mispredic- tions which increases inter-class noise and reduces the crawled data's quality (Kocyigit et al., 2022; Burchell al., 2023) both for the predicted lan- guage and the true language. To correct mispredic- tions and improve accuracy in supported languages with limited data, we propose hicrarchical model- ing approach.
3.3 Evaluation
Table 4, we show the performance of our L-Fine and F-Fine models compared to the base- line on our test set. We evaluate using three well- known MT metrics: BLEU (Papineni et al., 2002), CHRF++ (Popović, 2017), and spBLEU (NLLB Team al., 2022). For spBLEU, RES200 SPM model to create subwords.
the FLOO-
Based on all three metrics, our L-Fine model outperforms the Baseline model consistently by 4.0-11.5 spBLEU points by just fine-tuning with language-specific adapters. Our F-Fine model out- performs the L-Fine model by 5.0-7.5 spBLEu points by fine-tuning only some shared parameters among languages and language-specifte adapters. We also test our models on public benchmark, FLORES200 (Appendix $B), and observe that due the domain shift, L-Fine and F-Fine models under-perform the Bascline.
Hierarchical modeling is an extremely popular choice for wide variety of algorithmic tasks and it has been explored for language identification well (Goutte et al., 2014; Lui et al., 2014; Best- gen, 2017; Jauhiainen et al., 2019). However, pre- vious work has focused on predicting language group/family first, followed by finer-grained pre- dictions with a smaller set of classes. Our work departs from this paradigm in two ways first, we bring focus onto expanding language identification coverage in pre-trained or off-the-shelf systems without retraining, and second, predict prior and posterior language based confusion and mis- prediction patterns of the model directly (without predicting language family/group first).
Despite this domain shift, several low-resource language pairs benefit from adapter fine-tuning across domains. We report these language pairs and their respective spBLEU gains for the F-F'ine model in Table 5. We get the highest gains for English-Xhose (20.1 points) and English-Hausa (18.8 points) across domains, both of which had poor performance from the Baseline model with spBLEU of 3.5 and 4.5, respectively. We also no-
Under our technique, we first choose a well- performing root model with high-coverage that
"https://gi httmm//aauuiiilaalaaaaaegeg L.Afrrcaa_laagueee

---

gui
Multilingual Root Model
08
lefr
Predicted Labels
amhh Bhilori
0
Gujarati
Amharic
Tigrinya
br
GUJ KFR-BHILORI Unit
AMH-TIR-SLL Unit
02
stv
guj
True Labels
Gujarati
Kutchi
Bhilori
Amharic
Tigrinya
Silt'c
Figure 2: Subset of the multilingual root model's confusion matrix (6 languages). Using the confusion matrix, clusters of highly confused languages identified and confusion-resolltion units trained according to the tree shown on the right. The tree, for demonstration purposes, a subset of the entire tree which has confusion- resolution units
provides us with the base/prior prediction. Such base predictions are obtained for a sample of MCS-350's training set, allowing us to identify systemic confusion patterns embedded within the model using confusion matrix. Based the iden- tified misprediction patterns (which may or may not be between languages in the same family), we train lightweight confusion-resolutinn subunits that can be attached onto the root model to make the posterior prediction. Our results show that, with this architecture, a small sample of data is sufficient to investigate pretrained, off-the-shelf, or blackbox commercial models and identify systemic mispre- diction patterns across domains.
N-Way Multilingual multi Model To contrast our work with typical large multilingual model- ing where there is no architectural class hierarchy, train large fastText multilingual model with all 350+ languages (multi). With large number of classes, we know that low-resource languages suffer due to class imbalance, even with upsam- pling. But, we still include performance results from multi in Table to compare it with the two hierarchical approaches.
LIMIT'S Confusion- Units We fasttext (Joulin et al., 2017) to train small mod- els that will specialize in distinguishing between 2-3 highly-confused languages each. Up to 1000 sentences/language are used for training, and 100 randomly selected sentences across stories are re- served as the final test set. We train our own embed- dings because existing multilingual embeddings (Devlin al., 2019) are not trained sufficiently wide low-resource language data.
4.1 Experimental Settings
Wide-Coveres root Model
To pick an appro- our misidentification-
priate root model based hierarchical approach, we compare sev- eral statc-of-the-arr pre-trained models ($4.2) and choose the system with the highest macro-Fi score, giving equal importance to all languages in MCS-350.
Evaluation Metric To select a root model, per- formance is compared based on aggregated macro- Fi scores across languages (Table 6). To compare the performance of the root model and LIMIT (our proposed approach) on MCS-350, our benchmark dataset, and the existing FLORES-200 benchmark. we report language-level F1 scores (Table 7).
Traditional Hierarchical group-first Model Classical hierarchical models predict language fam- ily/group first, followed by the specific language (Goutte et al., 2014; Lui et al., 2014; Bestgen, 2017; Jauhiainen al., 2019). These groups of- ten have phylogenetic backing and not learned through the output distribution of the root model. For benchmarking, we train this traditional hierar- chical group model as well (Table 7).
"embedding dim= 100, learning rate= 0.5, loss= negative sampling. We explored several starting learning rates and set- tled on Ir=0.5 due to optimal performance on a small sumpled dev set. FastText's IrUpdate parameter, by default. reduces the starting learning rate gradually over epochs. We don't optimize for emhedding dimension and use Tast'ext'ss default, 100, for supervised training.

---

MCS-350 (LIMIT's Domain)
FLORES-200 200 (Out-of-Domain)
Lang
root
multi
group
LIMIT
multi root
group
LIMIT
GUJ
0.49
0.44
0.58
0.63
1.00
0.81
0.99
0.99
KFR
0.78
0.83
0.80
BHIL
0.63
0.03
0.28
AMH
0.20
0.81
0.78
0.83
0.60
0.95
0.93
0.99
TIR
0.56
0.93
0.94
0.85
0.99
0.96
0.95
0.95
LSTV
0
0
0.00
BEN
0.47
0.82
0.05
0.85
1.00
0.94
0.96
0.99
ASM
0.63
0.89
0.89
0.00
0.98
0.96
0.66
CDZ
0.57
0.87
0.93
ZHO
0.61
0
0
0.68
0.99
0
0.01
0.99
L YUL
0
0.02
0.14
0.00
0
0
0.00
TEL.
0.92
0.75
0.80
0.94
1.00
0.83
0.91
1.00
KFC
0.66
0.66
0.66
KAN
0.70
0.77
0.78
0.81
1.00
0.96
0.93
0.99
L KFA
0.66
0.68
0.52
TSO
0.49
0.53
0.34
0.67
0.97
0.79
0.72
0.97
TSC
0.74
0.52
0.77
DAGAARE
0.83
0.48
0.54
0.87
L M7M
0.69
0.88
0.82
KAT 0.66
0.46
0.42
0.80
1.00
0.72
0.70
1.00
L BBL
0.82
0.66
0.66
AVG
0.29
0.56
0.47
0.68
0.77
0.70
0.73
0.86
Table 7: LIMIT improves Fi scores over the root, mult: ti, and group models on both our children's 's stories dataset and out-of-domall FLORES-200 The traditional hierarchical approach group underperforms the multilingual model multi on both MCS-350 and F FLORES-200. Empty entries indicate unsupported languages and bolded entries indicate noteworthy differences in ri scores. Nested languages are misidentified as the parent in root. Note that for FLORES-200, the root model gets score ASM and YUE but both languages are covered by the dataset.
4.2 Pre-trained root Models
written sentences in MCS- 350 (all systems achieve an Fi score < 0.20) are worth noting, and indicate that off-the-shelf systems ultimately tend to per- form really well only some languages, despite officially supporting hundreds of languages.
In Table 6, show macro-F, scores across all 350+ languages for popular pretrained identifi- cation systems like Google's CLD3, Langid.py. py, Franc, fastText (Joulin et al., 2017), and HeLI-OTS (Jauhiainen et al., 2022a). Franc, built using the Universal Declaration of Human Rights (UDHR) data, comes out with the best macro-Fj, covering 30% of our languages (105/356 languages). It is derived from guess-language" which uses mix of writing system detection and character-level tri- grams. Hence, we use Franc as the root system for our misprediction- hierarchical modeling experiments. The overall low scores on human-
4.3 Language (Missidentification
Next, we inspect the best-performing root model's confusion matrix on MCS-350's training set (a representative example is shown in Figure 2) to understand and identify misprediction patterns. For each test language, we divide the root model's predictions by the total number of tested examples giving us a hit ratio for each pair. E.g., (Gujarati, Kutchi) would represent the ratio of Kutchi sentences that were misidentified

---

Gujarati. Upon inspection of the confusion matrix, we identified the following 9 clusters with a high confusion ratio (> 0.7). According to our experimental approach outlined in $4.1, we train a lean fastText classifier for each of these clusters, that will specialize in differentiating between these highly-confused languages:
is worth noting that that is the opposite of our find- ings in the machine translation task ($3.3), where translating shorter texts in MCS-350 proved easier than translating FLORES-200 data. Our mispredic- tion-based hierarchical is not only easier to train with limited data, but also brings valuable cross- domain language identification improvements.
1. Gujarati, Kutchi, Bhilori
Related Work
2. Amharic, Tigrinya, Silt'e
5.1 Parallel Datasets
3. Koda, Bengali, Assamese
4. Mandarin, Yue Chinese
Language identification models tend to use pop- ular training datasets like UDHR (Vatanen et al., 2010), Blodgett et al. (2017) for social media, King and Abney (2013) (web-crawl in 30 languages), FLORES-200, JW-300 (Agié and Vulić, 2019) (multilingual articles from Jehovah's 's Witness' web- site) etc.
5. Konda, Telugu
6. Kodava, Kannada
7. Tsonga, Tswa
8. Dagaare, Mumuye
9. Bats, Georgian
4.4 Expanded Language Coverage
A recently published dataset, BLOOM (Leong et al., 2022), leverages text and audio in children's ss stories from similar sources (African Storybooks, The Asia Foundation, Little Zebra Books etc.) to create benchmarks for image captioning and speech recognition. However, their data is monolingual, unaligned, and cannot be used for machine transla- tion. We leveraged the highly parallel nature of the collected storybooks, five times the number of sto- ries in BLOOM, and created test sets and baselines for understudied translation directions.
We report F1 scores for each of the 9 highly con- fused clusters' languages (Table 7) and observe that languages in each cluster share writing sys- tems and are often phylogenetically related. Our misidentification- based model, LIMIT, is success- ful at improving Fi scores on both our newly collected MCS-350 dataset as well as the public benchmark, FLORES-200 200. On MCS-350, LIMIT improves Fi scores from 0.29 to 0.68, 55% error reduction. Of the multidomain data avail- able in FLORES-200 (11/21 languages), LIMIT improves Fi from 0.77 to 0.86, a 40% error reduc- tion, demonstrating that our method's s utility is not restricted to the training data's domain.
It is also important to us to avoid representation washing (Caswell et al., 2020) and we clearly high- light the sources of noise from unverified stories in our merged dataset. With stricter preprocessing filters applied the pre-merge stage, 'cleaner' dataset could be produced, like in Burchell et al. (2023). We provide access to our data at all such timesteps in the preprocessing pipeline so researchers are not required to use the final dataset, but may use an earlier raw version and preprocess it themselves according to their needs.
Note that hicrarchical modeling could be viewed as further complicating simple root model, but we contend that this is valuable when retraining is not an option due lack of data, closed-source code, etc (Section 5). This simple extension allows us to extend a coverroveeege root model to newer languages or domains that have small amounts of training data, while maintaining high-quality pre- dictions. Furthermore, our hierarchical method LIMIT also outperforms a system multi that is trained on all the languages in the test set.
5.2 Machine Translation
Thousands of languages are spoken worldwide, so representing them with bilingual models would require thousands of models. Neither scalabil- ity nor adaptability makes this an ideal solution. Through various training methods (Aharoni et al., 2019; Wang et al., 2020), model structures (Wang et al., 2018: Zhang al., 2021), and data augmen- tation (Tan et al., 2019; Pan et al., 2021) a variety of research has attempted to improve multilingual translation models. Adapter units were initially pro-
4.5 Sentence Length and Domain
For several languages like Gujarati, Amharic, Ben- gali, and Mandarin, low Fi scores for MCS-350 compared to high F1 scores on FLORES-20 indi- cate that shorter texts in the children's 's stories do- main are much harder to identify. This is expected due to limited feature signals in shorter texts but it

---

posed for light-weight domain adaptation (Vilar, 2018) and then also for exlending large pre-trained models to a downstream tasks and using bilinugal adapters (Houlsby et al., 2019; Bapna and Firat, 2019).
5.4 Hierarchical Modeling
Hierarchical approaches have proved successful in solving myriad of computational problems, and have proved useful in language identification previously. The widely used approach first pre- dicts a preliminary language group/family, and then another fine-tuned prediction from a smaller set of output classes contained within the language group/family (Goutte et al., 2014; Lui et al., 2014; Bestgen, 2017: Jauhiainen et al., 2019). In con- trast, our work extends architecture to account for mispredictions made by existing trained models, and does not predict a group/language family first, but rather directly learns confusion relationships between language pairs. Then, similar to Bestgen (2017); Goutte et al. (2014), we train smaller clas- sifiers for a fine-tuned posterior prediction. How- ever, our approach departs from their paradigm in that our classifiers may also distinguish between highly confuu-co languages which belong to differ- ent language families.
5.3 Language Identification
Text-hased language identification is usually mod- elled as a classification task. By increasing the number of languages a classifier must predict, av- erage accuracy generally tends to decrease (Jauhi- ainen et al., 2017), a problem propose tackle by leveraging misprediction-based hierarchical approach. To distinguish between closely related languages, lot of exciting research has been pub- lished at various editions of VarDial The Work- shop on NLP for Similar Languages, Varieties and Dialects (Aepli et al., 2022; Scherrer et al., 2022; Chakravarthi et al., 2021; Zampieri et al., 2020, 2014).
Over the last 3 iterations of VarDial from 2019- 2022, many new datasets and techniques to identify Romance languages (Jauhiainen et al., 2022b; Za- haria et al., 2021), Nordic languages (Haas and Der- czynski, 2021), Uralic languages (Jauhiainen et al., 2020), German lects (Mihaela al., 2021: Siewert al., 2020), and the Slavic language continuum (Popović et al., 2020; Abdullah et al., 2020) were published. In contrast, we see only a handful papers and tasks on Indian languages at the venue with 2 focusing on Indo-Aryan and focusing on Dra- vidian languages (Nath et al., 2022; Bhatia et al., 2021: Jauhiainen et al., 2021: Chakravarthi et al., 2020), and papers tasks, our knowledge, on African languages. Outside the venue, recently published models like AfroLID (Adebara al., 2022) for language identification and IndicTrans2 (AI4Bharat et al., 2023) for Indic- cadic-language transla- tion are great large-scale efforts in the low-resource language space.
Conclusion
In this work, we tackle the lack of resources for many of the world's languages and release a large, massively parallel children's stories dataset, MCS-350, covering languages from diverse lan- guage families, writing systems, and reading lev- els. Since translation is crucial for parallel re- source creation, explore adapter- based networks fine-tuned a phylogenetic architecture, and uti- lize MCS-350 to create new translation benchmarks for vulnerable and low-resource languages. We demonstrate large improvements the children's 's story domain and cross-domain improvement for several language pairs (on the FLORES bench- mark dataset). On the algorithmic front, we in- troduce LIMIT, a hicrarchical, misprediction bbasedd approach to counter the inaccuracies of pre-trained language identification systems. Our method in- creases language coverage and prediction accuracy bypassing complete retraining, and shows cross- domain generalization despite being trained on our MCS-350 dataset.
Brown (2014), a notable technique, trains richer embeddings with non-linear mappings and achieves substantial improvements in downstream language identification on 1400+ languages. How- ever, we do not benchmark with this technique be- cause the paper does not contain any experiments in low-resource training setups. Training data is about 2.5 million bytes/language, while we are working with <50K bytes/language. Therefore, exploring non non-linear embedding mappings in low- low-resource settings (Brown, 2014) (4 is left for future work.
In the future, we hope to further investigate misprediction--rat
hicrarchical language identifi- cation across more datasets, with more configura- tions, and extensions such probabilistic branch- ing, automated constructions etc. As a natural next step, we will utilize LIMIT in a web-crawl to find and collect more low-resource language data.

---

Limitations
PR-276810-21, by the National Science Founda- tion under award IIS-2125466, and by Sponsored Research Award from Meta. Computational re- sources for experiments were provided by the Of- fice of Research Computing at George Mason Uni- versity (URL: https: / http//rr.gg.o ..g. gmu.edu) and funded in part by grants from the National Science Foun- dation (Awards Number 1625039 and 2018631),
Our dataset covers 350+ text-based languages. However, out of the 7000 languages in the world, many are primarily spoken languages and do not have a presence in the form of articles, textbooks, stories etc. Therefore, language identification for speech is crucial and we plan extending our text-based work to speech in future work.
While our proposed method LIMIT shows cross- domain improvements, we acknowledge that our system, like other language identification systems, is not perfect and may still make classification er- rors on new domains, text lengths, or orthogra- phies. We encourage researchers to keep this in mind when applying our proposed method to their work.
References
Badr M. Abdullah, Jacek Kudera, Tania Avgustinova. Bernd Möbius, and Dietrich Klakow. 2020. Redis- covering the Slavic continuum in representations emerging from neural models of spoken language identification. In Proceedings of the 7th Workshop NLP for Similar Languages, Varieties and Di- alects, pages 128-139, Barcelona, Spain (Online). International Committee on Computational Linguis- tics (ICCL).
Ethics Statement
Data used, compiled, and preprocessed in this project is freely available online under Creative Commons licenses (CC BY 4.0). Stories from the African Storybooks Initiative (ASI) are openly li- censed, can be used without asking for permission, and without paying any fees. We acknowledge the writers, authors, translators, illustrators of each of the books and the ASI team for creating such a valu- able repository of parallel storybooks in African languages. Stories from the Pratham Storybooks' Storyweaver portal are available under open licens- ing as well, and we preserve metadata for the au- thor, illustrator, translator (where applicable), pub- lisher, copyright information, and donor/funder for each book, in accordance with Storywcaver's ' guide- lines. Since stories hosted on African Storybooks Initiative and Pratham Books' Storyweaver are in- tended for children and most of them are vetted human-verified do not explicitly check for offensive content.
Iff Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Inciarte. 2022. AfroLID: A neural language identification tool for African lan- guages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1958-1981, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
David Adelani, Md Mahfuz Tbn Alam, Antonios Anas- tasopoulos, Akshita Bhagia, Marta R. Costa-jussá, Jesse Dodge. Fahim Faisal, Christian Federmann, Na- talia Fedorova, Francisco Guzmán, Sergey Koshelev, Jean Maillard. Vukosi Marivate, Jonathan Mbuya, Alexandre Mourachko, Safiyyah Saleem. Holger Schwenk, and Guillaume Wenzek. 2022. Findings of the wmn 22 shared task on large-scale machine trans- lation evaluation for african languages. In Proceed- ings of the Seventh Conference on Machine Trans- lation, pages 773-800, Abu Dhabi. Association for Computational Linguistics.
Noëmi Acpli, Antonios Anastasopoulos, Adrian-Gabriel Chifu, William Domingues, Fahim Faisal, Mihacla Gaman, Radu Tudor Ionescu, and Yves Scherrer. 2022. Findings of the VarDial evaluation campaign 2022. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1-13, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Our language identification models, by design, are meant to provide an alternative to training resource-hungry large-scale multilingual models that require a lot of training data. Such models are inaccessible to many researchers since they require access to specialized computing hardware. Our models are built with sustainability and equity in mind, and can be trained in matter of minutes on CPU on standard laptops.
Želiko Agié and Ivan Vulić. 2019, JW300: A wide- coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3204- 3210, Florence, Italy. Association for Computational Linguistics.
A Acknowledgments
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2079 Conference of the North
This work was generously supported by the Na- tional Endowment for the Humanities under award

---

American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 3874-3884 Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Su Lin Blodgett, Johnny Wei, and Brendan O' CoCnnor 2017. A dataset and classifter for recognizing social media English. In Proceedings of the 3rd Workshop on Noisy Uxer-generated Text. pages 56-61, Copen- hagen, Denmark. Association for Computational Lin- guistics.
AI4Bharat, Jay Gala, Pranjal A. Chitale, Raghavan AK, Sumanth Doddapaneni, Varun Gumma, Aswanth Ku- mar, Janki Nawale, Anupama Sujatha, Ratish Pudup- pully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible ma- chine translation models for all 22 scheduled indian languages.
Ralf Brown. 2014, Non-linear mapping for improved identification of 1300+ languages. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 627- 632. Doha, Qatar. Association for Computational Linguistics.
Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. of the s Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865-879, Toronto, Canada, Association for Computational Linguistics.
Md Mahfuz Ibn Alam and Antonios Anastasopoulos. 2022. Language adapters for large-scale mt: The gmu system for the wmt 2022 large-scale machine translation evaluation for african languages shared task. In Proceedings of the Seventh Conference on Machine Translation, pages 1015-1033, Abu Dhabi. Association for Computational Linguisties.
Proceedings
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Espla-Gomis, Mikel L.. Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz Rojas, Leopoldo Pla Sempere, Gema Ranirez-Sánchez, Elsa Sarrías, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acqui- sition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4555-4567, Online. Association for Computational Linguistics.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unex- pected challenges on the path to a thousand- thousand-language language web text corpus. Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages
Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Bharathi Raja Chakravarthi, Gaman Mihaela, Radu Tu- dor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Ruba Priyadharshini, Christoph Purschke, Eswari Ra- jagopal. Yves Scherrer, and Marcos Zampieri. 2021. Findings of the VarDial evaluation campaign 2021. In Proceedings of the Eighth Workshop NIPP for Similar Languages, Varieties and Dialects, pages 1- 11, Kiyv, Ukraine. Association for Computational Linguisties.
Ankur Bapna and Orhan Firat. 2019. Simple, scal- able adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- pages 1538-
guage Processing 1548, Hong Kong, China. Association for Computa- tional Linguisties.
Bharathi Raja Chakravarthi, Navancethan Ra- jasekaran, Mihael Arcan, Kevin McGuinness, Noel E. O'Connor, and John P. McCrae. 2020. Bilingual lexicon induction across orthographically-isiinn Dravidian languages. In Pro- ceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 57-69, Barcelona, Spain (Online). International Committee Computational Linguistics (ICCL).
Yves Bestgen. 2017. Improving the character ngram model for the DSL task with BM25 weighting and less frequently used feature sets. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects ( (VarDial), pages 115- Va- Iencia, Spain. Association for Computational Lin- guistics.
Kushagra Bhatia, Divyanshu Aggarwal, and Ashwini Vaidya. 2021. Fine-tuning distributional semantic models for closely-related languages. In Proceed- ings of the Eighth Workshop on NLP for Similar Lan- guages, Varieties and Dialects, pages 60-66, Kiyv, Ukraine. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics..
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig, 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5486-5505, Dublin, Ireland. Association for Computational Linguistics.
Fahim Faisal and Antonios Anastasopoulos. 2022. Phylogeny-inspired adaptation of multilingual mod- els to new languages. In Proceedings of the 2nd

---

Conference of the Asia-Pacifi Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Langwage Processing /Volume I: Long Papers), pages 434-452, Online only. Association for Computational Linguis- tics.
21st Nordic Conference on Computational Linguis- tics, pages 183-191, Gothenburg. Sweden. Associa- tion for Computational Linguistics.
Tommi Jauhiainen, Marco Lui, Marcos Zampieri. Timo- thy Baldwin, and Krister Lindén. 2019. Automatic language identification texts: survey. J. Artif. Int. Res., 65(())665--63 682.
Christian Federmann, Tom Koemi, and Ying Xin. 2022. NTREX- 128 news test references for MT evalua- tion of 128 languages. In Proceedings of the First Workshop on Sealing Up Multilingual Evaluation, pages 21-24, Online. Association for Computational Linguisties.
Tommi Jauhiainen, Tharindu Ranasinghe, and Marcos Zampieri. 2021. Comparing approaches to Dravid- ian language identification. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Va- rieties and Dialects, pages I 120-127, Kiyv, Ukraine, Association for Computational Linguisties.
Cyril Goutte, Serge Léger, and Marine Carpuat. 2014. The NRC system for discriminating similar lan- guages. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 139-145, Dublin, Ireland. As- sociation for Computational Linguistics and Dublin City University.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury, 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the S8th Annual Meeting of the Association for Computational Linguistics, pages 0282-6293, Online. Association for Computational Linguisties.
René Haas and Leon Derczynski. 2021. Discriminating between similar Nordic languages, In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 67-75, Kiyv, Ukraine. Association for Computational Linguistics.
Armand Joulin, Edouard Grave. Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pu- pers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. crammeter-fffcieen transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLRR
Simran Khanuja, Sebastian Ruder, and Partha Talukdar. 2023. Evaluating the diversity, equity, and inclu- sion of NLP technology: A case study for Indian languages, In Findings of the Association for Compu- rational Linguistics: EACL 2023, pages 1763-1777, Dubrovník, Croatia. Association for Computational Linguistics.
Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022a. HcLI-OTS, off ohf-the-shelf language identifier for text. In Proceeuiings of the Thirteenth Language Resources and Evaluation Conference, pages 3912-3922, Marseille, France. European Lan- guage Resources Association.
Ben King and Steven Abney. 2013. Labeling the lan- guages of words in mixed-language documents ing weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1110-1 19, Atlanta, Georgia. Association for Computational Lin- guistics.
Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022b. Italian language and dialect iden- tification and regional French variety detection using adaptive naive Bayes. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 11-112, Gyeongju, Republic of Korea, Association for Computational I. Linguisties.
Muhammed Kocyigit, Jiho Lee, and Derry Wijaya. 2022. Better quality estimation for low resource corpus mining. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 533-543, Dublin, Ireland. Association for Computational Lin- guistics.
Tommi Jauhiainen, Heidi Jauhiainen, Niko Partanen, and Krister Lindén. 2020. Uralic language identifi- cation (ULI) 2020 shared task dataset and the wanca 2017 corpora. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Di- alects, pages 173-185, Barcelona, Spain (Online). International Committee Computational Linguis- tics (ICCL).
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhat- tacharyya. 2018. The IIT Bombay English-Hindi parallel corpus. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki. Japan. European Language Resources Association (ELRA).
Tommi Jauhiainen, Krister Lindén, and Heidi Jauhi- ainen. 2017. Evaluation of language identification methods using 285 languages. In Proceedings of the
Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni,, and Daniel White- nack, 2022, Bloom library: Multimodal datasets in

---

300+ languages for variety of downstream tasks, In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 8608-8621, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume I: Long Papers), pages 244-258, Online. Asso- ciation for Computational Linguistics.
Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceed- ings of the ACI. 2012 System Demonstrations, pages 25-30. Jeju Island, Korea. Association for Computa- tional Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Marco Lui, Ned Letcher. Oliver Adams, Long Duong. Paul Cook, and Timothy Baldwin. 2014. Exploring methods and resources for discriminating similar lan- guages. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 129-138, Dublin, Ireland, As- sociation for Computational Linguistics and Dublin City University.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics.
Shuming Ma. Li Dong, Shaohan Huang, Dong- dong Zhang, Alexandre Muzio, Saksham Sing- hal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Encoder-decoder pre- pre-training for language generation and translation by aug- menting pretrained multilingual encoders. CoRR, abs/2106.137388
Maja Popović. 2017. chrF++: words helping charac- ter n-grams. In Proceedings of the Second Confer- ence on Machine Translation, pages 612-618, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Gaman Mihaela. Sebastian Cojocariu, and Radu Tudor lonescu. 2021. UnibucKernel: Geolocating Swiss German jodels using ensemble learning. In Proceed- ings of the Eighth Workshop on NLP for Similar Lan- guages, Varieties and Dialects, pages 84-95, Kiyv, Ukraine. Association for Computational Linguistics.
Maja Popović, Alberto Poncelas, Marija Brkic, and Andy Way. 2020. Neural machine translation for translating into Croatian and Serbian. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 102-113, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics (ICCL).
Shuyo Nakatani. 2010. Language detection library for java. Gitllub repository. Lost accessed: 2023-06-23.
Abhijnan Nath, Rahul Ghosh, and Nikhil Krishnaswamy. 2022. Phonetic, semantic, and articulatory features in Assamese- Bengali cognate detection. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 41-53, Gyeongju, Re- public of Korea. Association for Computational Lin- guistics.
Alex Salcianu, Andy Golding, Anton Bakalov, Chris AL- berti, Daniel Andor, David Weiss, Emily Pitler, Greg Coppola, Jason Riesa, Kuzman Ganchev, Michael Ringgaard, Nan Hua, Ryan McDonald, Slav Petrov, Stefan Istrate, and Terry Koo. 2020. Compact lan- guage detector v3 (cld3). Last accessed: 2023-06-23.
Yves Scherrer, Tommi Jauhiainen, Nikola Ljubešić, Preslav Nakov, Jörg Tiedemann, and Marcos Zampieri, editors. 2022. Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguis- tics, Gyeongju. Republic of Korea.
NLLB Team, Marta R. Costa-jussá, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel I Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe. Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyaah Saleem, Holger Schwenk, and Jeff Wang- 2022. No language left behind: Scaling human- centered machine translation.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021a. Wiki- Matrix: Mining 135M parallel sentences in 1620 lan- guage pairs from Wikipedia. In Proceedings of the 16th Conference the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 1351-1361, Online, Association for Computa- tional Linguistics.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 202 CCMatrix: Mining billions of high-quality
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual

---

parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Compu- national Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Vol- ume I: Long Pupers), pages 6490-6500, Online. As- sociation for Computational Linguisties.
2960. Brussels, Belgium. Association for Computa- tional Linguistics.
George-Eduard Zaharia, Andrei-Marius Avram, Dumitru-Clementi Cercel, and Traian Rebedea. 2021. Dialect identification through adversarial learning and knowledge distillation on Romanian BERT. In Proceedings of the righth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 113-119, Kiyv, Ukraine. Association for Computational Linguistics.
Janine Siewert, Yves Scherrer, Martijn Wieling, and Jörg Tiedemann. 2020. LSDC a comprehensive dataset for Low Saxon dialect classification. In Proceedings of the 7th Workshop NLP for Sim-
ilar Languages, Varieties and Dialects, pages 25-35, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
Marcos Zampieri, Preslav Nakov, Nikola Ljubešić. Jörg Tiedemann, and Yves Scherrer, editors. 2020. Pro- ceedings of the 7th Workshop on NLP for Similar Lon- guages, Varieties and Dialects. International Commit- tee on Computational Linguisties (ICCL), Barcelona, Spain (Online).
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie- Yan Liu. 2019. Multilingual neural machine transla- tion with knowledge distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Marcos Zampieri. Liling Tan, Nikola Ljubešić, and Jörg Tiedemann, editors. 2014. Proceedings of the First Workshop on Applying NLP Tools to Similar Lan guages, Varieties and Dialects. Association for Com- putational Linguistics and Dublin City University, Dublin, Ireland.
Jörg Tiedemann and Lars Nygaard. 2004. The OPUS uio. no/
corpus- parallel and free: opus. In Proceedings of the Fourth International Conference Language Resources and Evaluation (4)EC'04), Lisbon, Portugal. European Language Resources Association (ELRA).
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specingg capacity for multilingual transla- tion. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Tommi Vatanen, Jaakko J. Väyrynen, and Sami Vir- pioja. 2010. Language identification of short text segments with n-gram models. In Proceedings of the Seventh International Conference on Language Re- sources and Evaluation (LREC'I0) 10), Valletta, Malta. European Language Resources Association (ELRA).
Pavanpankaj Vegi, Sivabhavani J, Biswajit Paul, Abhi- nav Mishra, Prashant Banjare, Prasanna Kumar K R, and Chitra Viswanathan. 2022. Webcrawl african A multilingual parallel corpora for african languages. In Proceedings of the Seventh Conference on Ma- chine Translation, pages 1076-1089, Abu Dhabi. As- sociation for Computational Linguistics.
David Vilar. 2018. Learning hidden unit contribution for adapting neural machine translation models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 500-505, New Or- leans, Louisiana, Association for Computational Lin- guistics.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. 2020. Balancing training for multilingual neural machine translation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 8526 8537, Online. Association for Computa- tional Linguisties.
Yining Wang. Jiajun Zhang, Feifei Zhai, Jingfang Xu. and Chengqing Zong. 2018. Three strategies to im- prove one-to-many many multilingual translation. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 2955-

---

A Reproducibility
this section, we outline how to reproduce the dif- ferent aspects our work. Data collection, data pre- processing, machine translation experiments and evaluation, and language identification experiments have been completed in manner that is fully re- producible.
Family
Genus (Group)
Language
Indo- European
English
Germanic
Afrikaans
Romance
French
A.1 Data Curation
Hausa
Hausa
Afro- Asiatic
All data can be replicated and reproduced through code//ata-coollecton diate preprocessing steps can applied through code/preprocessinee merged through moddmmerging, and summary statistics be pro- duced through code/sudme satee/smmarsssttts.. Data paths are set up that any retrieved, preprocessed, merged data is located in data/.
Amharic
Amharic
Interme-
Cushitic
Oromo
Cushitic
Somali
Nilo-Saharan
Luo
Luo
Wolof
Wolof
Atlantic
Fula
Nigerian
Fulfulde
A.2 Language ID
code/ lode/languge--d// contains the relveant scripts replicate all language identification experi- ments, training, model architecture, and results. Relevant language identification data is decou- pled from the code directory and is located in data/Language-id.
Igboid
Igbo
Volta-Niger
Yoruboid
Yoruba
Bangi
Lingala
Shona
Shona
Nyasa
Chichewa
A.3 Machine Translation
Umbundu
Umbundu
Our machine translation experiments are performed using publicly available code from
Tswana
Sotho- Tswana
Bantu
Northern
To produce results regarding novel translation directions enabled by our data, please refer to code/new_lang_paiire Table A.1 shows the phylogeny configuration we use to fine-tune the MT system.
Sotho
Zulu
Nguni-
Xhosa
Tsonga
Swati
Xitsonga
B Supplementary Machine Translation Benchmarks
Kamba
Northeast-
Swahili
On the following pages, we report the aggre- gate evaluation results of our MT models on the FLORES200 devtest of 176 languages (BLEU, CHRF++, spBLEU). We also report BLEU, CHRF++, and spBLEU for baseline, language-fin fine, and family-fine scores for all language pairs we per- form machine translation experiments for (African focus languages from the WMT tasks' focus lan-
Bantu
Kinyarwanda
Luganda
Table A.I: This table highlights the three-tiered phylogeny-i -infomee tree structure we use for L-Fine and F-Fine models. Other than the high-resource lan- guages English, Afrikaans, and French, all other eval- uated languages are from Africa and cover 5 different language families.
guages)

---

Metrics
Models
AvgALL
Avgx- BX-ENG
BEN---X
AVgAFRICAN AFRAIIA
BY-FRA
AVgERA-
Baseline
9.59
17.84
10.56
7.83
13.42
9.79
BLEU
L-Fine
16.57
28.55
13.68
15.28
19.16
14.24
F-Fine
21.52
33.77
21.79
20.01
23.99
17.28
Bascline 29.59
35.47
30.88
28.05
32.54
31.25
CHRF++
L-Fine 37.04
45.54
35.24
35.89
39.11
36.82
F-Fine
41.33
49.83
41.28
40.18
43.27
39.28
Bascline 11.87
18.79
13.20
10.19
15.64
12.55
spBLEU
L-Fine 19.52
30.38
17.46
18.21
21.93
17.86
F-Fine 24.93
35.66
25.26
23.58
27.06
21.36
Table B.I: Evaluation results on our test set of 176 language directions. Avgx-esG denotes the average score of directions between other languages and English. Avgeso-x denotes the average score directions between English and other languages. AFAAIICAN-AARICC denotes the average score of directions between African languages other African languages. Arg. denotes the average score of directions between other languages and French. A denotes the average score of directions between French and other languages. Avgau denotes the average result of all translation directions.
Metries
Models
AvgaLL
Avgx-ENG
Avg AVBENG-X
Avg AVgY-FRA
Avg AVBRRA+Y
Baseline
14.01
28.21
13.68
23.62
11.66
11.23
BLEU
L-Fine
12.91
25.98
14.19
22.21
10.98
10.05
F-Finc
12.25
25.15
13.99
20.98
10.88
9.34
Baseline 39.16
49.54
37.16
46.24
38.03
37.29
CHRF++
L-Fine 37.89
47.28
39.86
44.80
37.26
35.57
F-Fine 37.03
46.57
39.64
43.68
37.41
34.50
Baseline 18.23
30.77
17.22
28.01
16.69
15.64
spBLEU
L-Fine
17.01
28.23
18.91
26.46
15.75
14.22
F-Fine
16.15
27.41
18.69
25.25
15.69
13.21
Table B.2: Evaluation results on FLORES200 devtest of 176 language directions. Avgx-ENG denotes the average of directions between other languages and English. AvgeNo-x denotes the average of directions between English and other languages. AVBaericAN-APRICAM denotes the average score of directions between African languages to other African languages. Avgv denotes the average score of directions between other languages French. AVEERA-Y denotes the average score of directions between French and other languages. Avg AvgaL denotes the average result of all translation directions.

---

BLEU
CHRE++
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
LUG-ENG
10.1
17.3
23.4
33.8
39.6
10.9
18.2
24.5
YOR-ENG
16.2
22.2
26.3
33.6
40.1
43.8
16.3
22.1
26.5
HAU-ENG
15.9
21.5
24.6
33.6
40.7
17.2
22.7
25.7
AMH-ENG
21.4
27.8
30.1
41.3
45.8
47.6
23.2
29
31.3
SWA-ENG
22.5
26.6
33.3
41.1
44.1
49.4
23.4
27.5
34.5
IBO-ENG
15.1
20.9
22.8
34
40.2
41.8
15.8
22.3
24.1
NYA-ENG
17.9
25.2
33.4
37.3
44.5
51.4
18.7
25.5
35.2
ORM-ENG
7.9
11.3
15
24.1
30
33.3
9.4
12.6
16.5
NSO-ENG
22.5
46.7
53.5
39.5
61.7
66.5
23.4
51.1
57.1
XHO-ENG
21.9
36.5
42.2
39
52.1
56.9
22.4
38.2
44.7
TSO-ENG
20.5
48.3
57.1
37.1
62
68.7
21.1
52.1
60.3
KIN-ENG
13.4
22.5
29.1
31.1
40.3
45.8
13.9
23.1
30
KAM-ENG
5.3
11.7
21.8
24.6
29.9
6.1
8.9
12.9
ZUL--ENG
19.5
34.8
41.3
37.9
51.9
20.7
39.1
45.5
SSW-ENG
17.4
41.9
48.8
33.8
56.6
61.9
17.9
45.9
52.5
AFR-ENG
38
45.3
47.7
57.2
61.9
62.9
40.3
47.7
49.3
ENG-SWA
16.8
18.1
24.5
41.4
43
47
20
22.1
27.8
ENG-IBO
13.6
17.6
20.9
33.4
37.4
39.7
17.1
21
24.3
ENG-NYA
10.4
11.7
15.6
33.5
36.3
39.4
12
14.3
18.4
ENG-ORM
1.1
3.1
2.9
14.1
18.5
17.7
1.7
4.4
4.4
ENG-NSO
19.5
25.1
43.2
37.7
43.6
59.9
19.8
26
46.4
ENG-TSO
16
23.6
44.6
36.8
43.8
61.2
16.9
25.6
47.8
ENG-KIN
5.3
7.8
10.1
26.3
31.1
33.5
7.7
11.2
13.9
ENG-KAM
1.6
2.1
2.9
16.3
18.6
20.2
2.4
3.1
4.3
ENG-ZUL
9
11.2
27.8
36.4
38.9
49.3
16
18.7
32,9
ENG-SSW
5.7
9.9
36.2
29.3
36.8
53.5
10.2
16.7
37.9
ENG-AFR
33.1
35.1
40.3
52.8
54.6
57.7
35.6
37.4
42,4
ENG-XHO
4.5
28.2
26.5
37.7
48.2
8.1
18.1
31.5
ENG-LUG
4.2
5.9
25.7
29.8
31.9
9.8
12.3
ENG-YOR
13.3
15.8
20.4
30.9
32.5
35.7
14.8
17.3
20.8
ENG-HAU
10.4
12.4
15
31.3
34,9
37.8
8.7
14.2
17.8
ENG-AMH
4.4
7.9
21.7
26.3
27.7
13.2
19.4
21.3
FRA-SWA
8.9
12.9
17.6
34
39
42.6
11.1
16
21.3
FRA-KIN
5
7.6
9.9
28.4
32
34.5
7.6
11.6
14.5
FRA-HAU
6.5
8.9
11.8
29.5
33.6
36.1
7.8
10.7
14
FRA-NSO
15.2
23.9
29.2
34.2
45.4
49.9
16
26.3
32.3
FRA-AMH
2.9
5.9
17.6
21.7
21.9
10.2
14.3
15.8
FRA-XHO
9.2
12.2
15.8
35.8
40.1
42.1
14.5
19
22.2
FRA-ZUL
6.9
9.4
12.7
35.3
38.8
41.1
13
16.3
20
FRA-LUG
2.3
5.3
7.3
21.9
29.2
31.3
4.3
8.7
11.3
FRA-IBO
13
18.1
20.5
32
37.1
38.6
16
21.2
23.5
FRA-AFR
27.1
30.1
31.6
47.3
50.3
28.5
31.3
33.3
FRA-NYA
11.8
14.9
33.2
36.3
39.1
10.6
13.8
17.4
5.7
12.4
17.7
29.8
39.8
44.2
9.7
18.2
24.1

---

BLEU
CHRF++
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L--Fine
F-Fine
Baseline
L-Fine
F-Fine
FRA-YOR
9.6
14.1
15.5
23,9
28
29.5
9.9
13.8
16
FRA-TSO
15.8
27.6
31.5
34.6
45.5
48.7
16.5
28.9
33.3
HAU-FRA
9
13.9
17.3
26.2
32.5
35.8
11.4
16.8
20.7
NSO-FRA
15.2
23.3
29.9
35.8
44.5
49.7
18
26.3
33
AMH-FRA
12
16.7
19.1
30.2
35.6
37.4
13.4
18.9
21.2
XHO-FRA
16.7
22.1
26.6
37.1
42.9
46.2
18.2
23.9
28.8
ZUL-FRA
14.4
19.9
24.5
36.2
41.5
45.5
16.9
23.1
28.3
LUG-FRA
6.4
14.1
19.5
21.6
32.6
38.5
8.4
16.5
22.2
IBO-FRA
10.9
15.1
17.6
31.5
36.8
38.9
13.6
18.5
21.5
AFR-FRA
26
28.5
31.5
47.8
50.3
51.8
29.6
32.4
35.2
NYA-FRA
14
20.4
27.2
32.7
39.5
45.6
15.7
22.5
29.6
SSW-FRA
14.7
21.4
28.8
34.2
42
48
16.7
24.4
32.3
YOR-FRA
10.4
16
18.2
30.1
35.6
38
11.9
18.2
21
TSO-FRA
18.3
26
35.5
36.1
43.9
51.3
20.4
29
38.5
SWA-FRA
12
15.3
19.2
30.5
34.5
38.4
14.1
18
22.2
KIN-FRA
7.9
15.5
21
25.5
35.4
40.7
10.6
18.5
24.3
TSO-SWA
13.4
17.7
27.2
37.6
41.8
48.8
17.4
22.8
31.9
SSW-TSO
13.7
35.6
40.9
33.5
54.3
58.3
13.8
38.9
44.1
AMH-KIN
2.6
5.8
7.1
21
26.2
27.3
4.6
8.2
9.7
TSO-NYA
6.4
8.7
17
31.5
36.6
43.7
9.4
12.5
22.7
TSO-NSO
18.6
37
43.6
36.4
55.4
60.2
17.8
40.7
46.8
NSO-KIN
4.8
9.2
14.2
27.9
33.9
37.7
8.2
13.7
19.7
YOR-IBO
13.3
24.9
26.7
28.4
39.3
41.5
15.6
26.2
28.4
SSW-SWA
8.6
13.2
23.2
34.1
39.8
48.4
11.7
16.8
28.6
NYA-SWA
9.8
14
22.9
35.1
38.6
45.6
12.2
17.2
27.4
YOR-SWA
10.4
17.8
21.5
30.3
39.1
42.8
12.2
18.5
23.4
SSW-NSO
13.8
38.2
44.9
31.6
54.9
60.4
13.8
40.6
47.5
SSW-NYA
5.9
9.3
17.7
28.1
35.5
42.1
7.9
13.5
22.5
AFR-SWA
11.8
15.5
23.7
38.7
41.7
46
16.1
19.8
28.6
XHO-TSO
13.1
35.6
42.1
34
53.7
59.7
14.1
37.1
44.4
LUG-NYA
2.3
7.4
12.2
20.2
30
34.2
4
9.7
15.3
AMH-AFR
10.4
16.9
19.5
28.6
34.1
36.8
11.9
18.1
21
LUG-NSO
14.6
18.8
20.7
31.9
36.1
5.5
14.8
19.3
NSO-AFR
20.6
30.9
43.3
40
50.9
58.9
21.9
33.4
45.2
2.6
6.1
21.7
26.8
27.6
5.2
8.6
9.5
IBO-SWA
14.1
19.9
26.7
35
39.8
44.6
15.6
21
27.6
AMH-ZUL
3.1
6.7
7.9
26.2
31.2
32.5
7.5
11.8
13.5
LUG-SWA
6.3
11.4
17.7
26.5
34.8
40
8.2
14.7
21.8
LUG-IBO
4.8
13.7
17.3
18
28.6
33
6.8
15.6
19.7
NSO-ZUL
7.8
23.9
27
34.3
45.6
48.8
13.6
27
30.9
ZUL-SWA
9.5
13.7
20.9
34.9
37.9
43.4
13.3
17.7
25.8
XHO-SWA
13
16.6
23,4
37.1
40.2
45.3
16.2
20.1
27.4
LUG-XHO
3.7
7.7
12.4
22.1
29.7
34.9
5.5
11.2
16.8
XHO-NSO
15.1
32.8
40.5
34.5
51.6
57.9
15.5
35.5
43.5

---

BLEU
CHRF++
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
ZUL-NYA
6.1
7.7
11.2
31.2
33.6
36.8
8.8
10.9
15.5
KAM-SWA
4.1
4.4
7.2
22.3
21.2
25.8
5.8
5.8
9.5
XHO-NYA
6.8
10
18
31.6
36.4
43.2
9.7
13.5
22.6
TSO-KIN
4.4
7.1
13
26.2
29.4
36.1
6.8
10.6
18.7
NSO-NYA
5.6
8.1
15
30.2
35.6
40.5
8.1
12.2
19
LUG-AFR
9.7
14.2
22.3
27.9
32.7
39.9
10.6
15.1
23.4
AMH-ORM
0.8
2
3.1
14.3
19.8
22.9
1.3
3.8
5.1
AMH-SWA
5.7
14.8
18.3
27
38
41.1
7.8
19
22.2
SWA-KIN
4.3
5.3
7.3
23.3
24.2
27.5
6.3
7.5
10.5
LUG-ZUT.
3.3
5.7
8.6
22.9
28.1
32.2
5.7
9.3
13.3
NSO-SWA
13.5
19.5
27.9
36.8
42.9
48.4
16.9
24.3
32.4
XHO-SSW
4.5
36.2
40.9
27.4
51.9
56.9
8,9
37.4
43.2
SSW-KIN
5.3
15.3
26.9
33.5
8.2
13.9
20.7
NYA-KIN
4.9
9.2
13.3
26.7
31.8
36.9
12.4
YOR-LUG
2.1
5.7
7.9
18.7
26.8
27.9
7.5
9.7
XHO-ZUL
9.1
32.4
36.1
33.9
49.7
53.5
14.9
33.5
38.4
XHO-AFR
19.7
24.5
33.9
39.8
44.4
51.8
21.4
26.1
36
ZUL-AFR
18
22.4
30.2
38.4
42.8
49.1
20.1
24.4
32.8
TSO-ZUL
6.3
39.4
41.8
31.4
53.7
56
11.2
38
41.5
AFR-KIN
7.6
11.3
29.7
32.9
37.6
7.9
11.6
16.6
HAU-SWA
8.3
10.5
13.5
29.3
32.6
34.5
10
13.6
16.2
ORM-SWA
5.6
8.4
9.6
23.4
27.7
28.1
6.4
10.9
11.6
TSO-AFR
19.9
25.9
36.3
38.4
44.3
52.8
21.1
27.1
38.1
LUG-KIN
1.1
6.3
7.8
15
27.2
29.5
2
8.7
11
ZUL-KIN
4.8
6.8
10.4
27.4
30.6
33.4
7
10.1
14.6
SSW-ZUL.
8.5
31
33.3
33.8
50.6
52.2
13.9
33.3
36
XHO-KIN
4.8
7.6
11.1
27.9
31.3
35.2
7.6
11.6
16.3
LUG-AMH
2.4
8.5
13.6
3.2
8.2
11.4
SSW-AFR
17.1
23.5
36.3
42.1
51.8
18.2
25.1
38.1
NYA-AFR
15.4
20.8
29.7
38.6
45.9
16.7
21.8
31.3
SWA-TSO
13,8
21.1
29.7
35.6
41.6
49.5
15.3
22.7
32.4
TSO-SSW
6.3
31.1
35
31
50
54.2
10.5
33.1
38.3
KIN-AMH
0.7
1.8
2.9
9.7
14.8
16.9
3.5
8.4
11
NYA-TSO
13.3
24.7
31.5
33.2
45.1
50.6
14.2
27.2
34.3
NSO-TSO
15.9
37.5
42.2
36.2
56.5
60.3
16.3
41.3
46.2
KIN-NSO
6.4
17.8
24.2
22.7
36.1
41.8
6.1
17.9
25
IBO-YOR
12.9
17.1
18.3
26.2
29.9
30.3
13
16.7
17.4
SWA-SSW
3,9
6.5
11.4
23.5
28.7
34.4
5.8
10.2
16.1
SWA-NYA
6.5
7.4
11.4
27.6
29
33.4
8.3
9.8
14.3
SWA-YOR
9.3
13.3
15.7
21.3
24.7
28
9.4
13.8
15.5
NSO-SSW
5.7
36.3
40.2
29.7
52.8
56.6
10
36.9
41.4
NYA-SSW
5.1
10.5
19.2
27.3
36.3
44.6
7.7
16
25.9
SWA-AFR
15.2
18.4
25
36.2
41.9
16.8
19.5
26.8
TSO-XHO
8.8
33.3
37.2
34.9
51.3
54.9
13.8
33.2
38.4

---

BLEU
CHRF+
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
NYA-LUG
2.5
4.7
8.1
22.9
26.4
29.7
4.5
7.5
11.3
AFR-AMH
2.6
3.8
5.3
17.4
21.1
23.8
9.2
12.7
15.6
NSO-LUG
2.6
4
5.1
23.3
27.4
28.1
4.6
7
8.9
AFR-NSO
18.6
28.5
34.7
38.3
48.7
53
18.8
29.5
36
KIN-HAU
6.3
8.3
11.1
26
28.4
32.8
7.6
10.3
13.9
SWA-IBO
18.4
23.5
27.3
32.5
37.4
41
20.1
24.7
28.5
ZUL-AMH
4.2
5.4
15.9
19.4
22.8
7.4
11.8
15.8
SWA-LUG
3.5
5.1
7.2
23.2
25.2
28.3
5.8
7.8
10.4
IBO-LJUG
5.5
7.7
20.3
28.2
29.9
2.9
8.2
10.8
ZUL-NSO
15.7
29.1
34.6
36
47.8
52.8
15.9
30.6
36.6
SWA-ZUL
7.2
8.3
11.6
30.2
31.5
34.9
10.9
12.5
16.6
SWA-XHO
8.8
9.8
15.7
33.9
35.1
40.6
13.4
14.8
21.6
XHO-LUG
3.7
5.4
8.1
26.6
30.4
31.9
5.6
8.6
11.9
NSO-XHO
29.9
33.9
34.1
49.2
53
13.5
32.1
36.7
NYA-ZUL
6.9
8.8
14
31.3
34.5
39.4
11.2
14.2
20.1
SWA-KAM
1.5
1.3
1.8
16.2
16.3
17.8
2.3
2
3
NYA-XHO
8.1
11.3
20.5
33.2
38.9
46.1
12.8
18.2
27.5
KIN-TSO
6.2
15.8
22.2
22.9
34
39.8
6.6
17
24.1
NYA-NSO
14.9
24.2
32.9
34
44.2
50.5
15.1
25.8
34.5
AFR-LUG
5.4
28.3
31.2
32.6
6.9
9.3
11.4
ORM-AMH
1.8
2.8
10.5
15.7
16.3
3.2
9.9
10.8
SWA-AMH
2.1
4.4
5.8
14.8
19.4
22.5
7.4
12.9
16.9
KIN-SWA
6.3
11.2
18.7
28.4
34.2
40.1
14.8
23.2
ZUL-LUG
3.2
5.2
25.2
26.3
8.2
SWA-NSO
15.2
20.1
29.4
33.9
39.6
46.8
15.2
21.3
SSW-XHO
6.1
36.4
40.5
29.1
51.4
55.2
10.6
36.6
41.3
KIN-SSW
1.9
10
16.9
19.9
33.6
40.8
4
15.7
24.2
KIN-NYA
2.4
6.2
11.4
22.1
31.5
36.7
3.4
8.4
15.3
LUG-YOR
3.2
8
13.1
12.6
19.1
25.2
3.3
8.4
13.5
ZUL-XHO
8.9
32.6
35.7
34.4
49.8
52.7
14.5
33.4
37.2
AFR-XHO
10.1
12.3
16.2
37.7
40.3
43.2
16.3
19.2
23.4
AFR-ZUL
9.5
11.2
15.1
36.5
38.8
42
15.3
18
22.3
ZUL-TSO
13
39.6
44
34
56.6
59.9
13.8
42.1
46.6
KIN-AFR
10.9
15.3
23.6
29
33.6
41.3
11.3
16
25.3
SWA-HAU
7.8
10.7
13
30.4
31.1
35.1
9.3
12
14.8
SWA-ORM
2.2
3.3
17.9
20.1
22.2
3.3
3.6
5.2
AFR-TSO
18.2
24.9
29.5
39.6
48.2
52.3
19.6
28
33
KIN-LUG
0.3
4.6
6.2
14.6
26.6
28.4
1.4
8
9.6
KIN-ZUL
3.9
6.6
9.5
24.2
30.3
33.2
6.6
11.2
ZUL-SSW
6.4
30.4
33.8
33
51.2
53.9
12.4
32.8
37.2
KIN-XHO
3.4
8.7
12.1
31.3
34.3
5.8
12.8
16.8
AMH-LUG
1.4
4.2
4.6
18.4
24.7
24.9
3.1
6.2
7.2
AFR-SSW
6.4
11.9
18
28.9
37.8
43.4
10
18
25.1
AFR-NYA
6.4
8.4
13
30.1
33.5
37.7
9.6
12.1
17.5
Table B.3: Results of all language pairs on our test set

---

BLEU
CHRF+
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
LUG-ENG
16.1
15.5
15.5
36.6
36
36.2
18.3
17.5
17.7
YOR-ENG
16.7
15.6
16.3
38.6
37.2
38.2
18.9
17.7
18.4
HAU-ENG
27.8
27.2
25.9
50.2
49.1
48
31.1
29.2
27.9
AMH-ENG
31.4
27.8
27.1
55.5
52.1
51.3
34
30.1
29.3
SWA-ENG
41.6
34.2
36.6
62.5
56.5
58.4
43.5
36.2
38.5
IBO-ENG
25.6
24
23.8
48.3
45.3
45.4
28.6
26.3
26.1
NYA-ENG
25.2
23.3
22.9
47.8
45.5
44.9
28.7
26.3
25.7
ORM-ENG
13.2
11.7
10.3
34.6
32.2
29.6
14.5
12.7
11.1
NSO-ENG
34.6
32.5
29.5
55
53
50.2
36.8
34.5
31.5
XHO-ENG
35.1
33.4
31.7
56.2
55.1
53.3
37.9
36.2
34.3
TSO-ENG
28.1
25.8
24.4
49.6
47.2
46.2
30.8
28.2
26.9
KIN-ENG
28.1
24.2
23.3
50
46.4
45.5
30.2
26.2
25.4
KAM-ENG
9.5
10
9.1
28.3
28.2
28.7
12.2
12.3
12.2
ZUL-ENG
35.8
32.3
31.1
57.5
53.8
52.8
38.7
34.6
33.3
SSW-ENG
26.1
25.6
24.1
47.6
47.1
45.7
28.5
27.9
26.3
AFR-ENG
56.5
52.6
50.8
74.4
71.8
70.7
59.6
55.7
53,9
ENG-SWA
33.8
30.8
29.8
59.4
57.3
56.4
38
35.3
34.4
ENG-IBO
15.8
16.1
16.3
39.5
40.1
40.2
18.6
19
19.2
ENG-NYA
14.2
13.8
13.4
44.5
44.5
43.6
18.1
17.7
16.9
ENG-ORM
1.3
0.7
18.2
17.1
15.4
2.4
1.7
1.2
ENG-NSO
23.1
19.1
19.4
47.9
44.9
45.7
24.4
21
21.5
ENG-TSO
16.4
15.6
16.8
43.7
42.5
43.9
19.6
18.2
19.4
ENG-KIN
12.5
11.3
37.9
38.1
38.2
15.9
14.5
14.7
ENG-KAM
2.8
3,9
4.2
19.3
22.4
22.8
3.8
5.4
5.6
ENG-ZUL
16.1
15.3
14.3
50.2
49.5
48.5
27.2
26.2
24.7
ENG-SSW
7.6
39
38.6
38.8
14.7
14.6
14.3
ENG-AFR
40.4
37.5
35.8
65.7
63.6
62.4
46.1
43.4
41.7
ENG-XHO
1.4
12.8
13.9
15.7
46.6
47.6
3.5
22.5
23.6
ENG-LUG
5.4
5.8
6.1
29.8
30.9
31.2
7
8.5
ENG- YOR
3.3
3.3
3.2
19.5
19.2
19.2
5.1
4.6
4.6
ENG-HAU
13.1
22.3
20.7
27.7
46.9
45.6
4.5
24.2
23.3
ENG-AMH
11.6
11.8
10.9
36.6
35.5
34.7
26.8
26.2
25.4
FRA-SWA
23.6
20.5
20.1
50.9
48.6
47.5
28.1
25.1
24.3
FRA-KIN
9.6
8.6
9.1
36.4
34.6
35.6
13.4
11.8
12.2
FRA-HAU
15.4
15.3
15.1
41
40.5
40.7
18.1
17.8
17.7
FRA-NSO
12.8
12.3
13.1
38.6
38
39.4
14.9
14.4
15.3
FRA-AMH
8.5
6.9
6.5
31.7
28.5
27.8
22.2
19.6
19.1
FRA-XHO
10.3
9.1
43.1
41.1
41.2
19.4
17
17.3
FRA-ZUL
11.1
10
9.6
44.9
43.5
42.9
21.6
19.9
19.1
FRA-LUG
2.2
4.3
4.2
22
28
28.7
3
6.4
6.6
FRA-IBO
13.1
12.1
12.5
36.9
35.6
36.4
16.1
14.9
15.3
FRA-AFR
26.7
24.6
22.6
54.9
52.6
50.8
32.7
30.1
28.2
FRA-NYA
11.6
10.1
10.1
42.2
39.7
39.6
15.6
13.4
13.5
FRA-SSW
4.9
4.8
4.9
33.7
34.2
35.1
11.3
11.2
11.4
Table B.4: Results of all language pairs on our test set

---

BLEU
CHRE+
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L- L-Fine
F-Fine
FRA-YOR
2.4
3.2
18.2
18.5
18.7
3.6
4.8
4.7
FRA-TSO
11
11.9
12.5
37.9
38.2
39.4
13.6
14.1
14.9
HAU-FRA
23.2
21.8
21.2
45.7
44.1
43.6
27.4
25.8
25.2
NSO-FRA
24.4
23.2
20.8
46.7
45.5
43.6
29
27.3
25.1
AMH-FRA
25.2
22.7
22.6
49.3
47.3
46.7
29.9
27.4
27.1
XHO-FRA
26.6
25.9
23.4
49.2
48.4
46
31.3
30.2
27.9
ZUL-FRA
28.1
25.3
22.7
51
48.4
46.1
32.5
29.6
27.1
LUG-FRA
13.1
13.6
12.5
33.8
34.3
33.6
16.2
16.6
15.6
IBO-FRA
20.2
18.8
18.1
43.1
41
40.7
24.2
22.5
22.4
AFR-FRA
37.9
37.1
35.8
60.8
59.9
58.9
43.6
42.7
41.5
NYA-FRA
20.5
19.7
18.5
44
42.5
41.2
25,4
24.3
23
SSW-FRA
19.7
20.4
18.3
41.9
43.1
40.9
24.1
24.5
22,4
YOR-FRA
15
13.5
13.5
37
35.4
35.3
18.9
17.5
17.5
TSO-FRA
22.4
20.9
19
44.8
43.5
41.5
26.7
25.2
22.8
SWA-FRA
31.7
27.3
27.7
54.4
50.4
50.8
36.1
31.9
32.2
KIN-FRA
22.7
20.7
19.6
45.7
43.4
42.6
26.8
24.9
23.7
TSO-SWA
19.3
16.3
13.8
45.7
42.8
39.3
22.9
20
17.1
SSW-TSO
12.1
13
12.4
38.6
39.4
38.2
15.1
15.5
14.6
AMH-KIN
8.2
6.7
6.5
35.1
32
31.4
11.6
9.5
8.9
TSO-NYA
10.3
9.9
8.3
39.2
38.4
35.1
13.8
13.2
11.2
TSO-NSO
17.3
15.5
14.1
42
40.6
39.2
18.9
17.3
15,9
NSO-KIN
9.6
9.4
7.9
35
34.6
31.9
12.9
12.4
10.3
YOR-IBO
8.4
7.9
8.2
29.7
29.2
29.4
11.1
10.6
10,9
SSW-SWA
17.4
16.4
13.1
43.5
42.6
38.6
20.6
19.5
16,2
NYA-SWA
17.8
15.1
13.3
44.9
41.4
39
21.6
18.9
16.7
YOR-SWA
11.9
9.4
9.9
37.6
33.4
34.2
14.6
11.9
12.4
SSW-NSO
16.1
14.9
13.8
40.8
39.9
38.1
17.7
16.7
15.3
SSW-NYA
9.2
10.2
8.3
37.4
38.5
35.2
12.5
13.1
AFR-SWA
27.6
24.2
20.9
54.8
51.5
48.4
32.1
28.7
25.7
XHO-TSO
13.8
13.7
13.7
40.5
40
40.1
16.8
16.4
16.2
LUG-NYA
7.2
7
32.5
32.4
30.4
9.4
9.4
8.1
AMH-AFR
19.4
17.3
16.3
46.5
44.1
43.1
23.3
20.8
19.6
LUG-NSO
9.9
10.7
9.5
32
33.7
32.8
10.9
12.1
NSO-AFR
20.9
18.8
16.8
45.8
43.4
41.1
24.2
21.3
19.2
HAU-KIN
10.1
8.4
7.6
36.5
32.7
31.5
13.7
10.8
9.9
IBO-SWA
16.6
15
14.7
44.4
40.5
40.7
20.8
18.1
17.7
AMH-ZUL
8.3
7.5
42.1
40.5
39.4
18.1
16.7
15.5
LUG-SWA
11.9
9.9
8.7
36.7
33.9
32.3
14.2
12.2
10.9
LUG-IBO
7.5
8
7.1
26.6
28.1
27.4
9.8
10.3
9.8
NSO-ZCL.
12.7
10.8
9.3
44.9
42.5
40.2
22
19.7
17.4
ZUL-SWA
25
20.9
17.7
51.6
47.2
43.1
29
24.6
21
XHO-SWA
22.5
20.6
17.8
49.4
47.3
43.5
26.6
24.7
21.4
LUG-XHO
5.1
4.6
4.5
31.8
31.1
30.7
10.3
9.8
8.9
XHO-NSO
18.2
17.5
16.1
43.2
42.2
40.7
19.7
18.8
17.6

---

BLEU
CHRF++
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
ZUL--NYA
12.8
11.6
9.1
42,9
40.8
36.7
16.6
15
12.2
KAM-SWA
8.4
7.1
5.6
30.3
27.5
25.6
10.5
9.2
7.3
XHO-NYA
12.3
11.7
10.2
42.1
40.8
37.7
15.2
13.1
TSO-KIN
10
6.8
36.3
34.2
30.6
14.1
12.3
9.2
NSO-NYA
11.1
10.6
9.4
39.6
38.9
36.5
14.3
13.5
12
LUG-AFR
11.6
10.6
9.9
34
32.9
31.8
13
11.9
AMH-ORM
1.2
0.8
0.9
19.5
16.7
17.9
2.5
1.5
1.6
AMH-SWA
20.2
17.1
16.4
48.4
44.8
43.3
24.3
20.9
19.7
SWA-KIN
12.2
7.4
7.9
39.9
32.1
32.9
16.1
10.2
10.6
LUG-ZUL
5.4
5.3
4.7
33.1
32.6
31.1
11.9
11.4
10.2
NSO-SWA
21.7
19
15.3
48.2
45.1
40.6
25.2
22.3
18.4
XHO-SSW
7.2
6.3
37.7
37.2
35.6
14.3
14.1
12.7
SSW-KIN
7.9
8.7
6.6
32.4
33.5
30
10.7
11.3
8.6
NYA-KIN
9.2
7.7
7.1
35
32.2
31
12.5
10.7
9.5
YOR-LUG
3.6
3.9
3.3
25.2
24.9
24.6
4.9
5.6
5.2
XHO-ZUL
12.9
12.1
11.1
45.8
44.9
42.8
23.2
22.3
20.1
XHO-AFR
20.7
19.8
17.2
46.7
45.3
42.7
24.8
23.3
20.5
ZUL--AFR
22.4
18.8
17.4
48.3
44.3
42.4
26.3
22.2
20.3
TSO-ZUL
10.5
9.4
8.4
42.7
41
39.1
20.3
18.1
16.3
AFR-KIN
11.5
8.8
38.9
35.2
34.9
15.9
12.2
12
HAU-SWA
20.9
17.1
15.2
47.4
42.7
40.9
24.2
20.3
18.3
ORM-SWA
10.5
7.2
6.3
34.7
28.8
26.5
12.2
8.6
7.3
TSO-AFR
17.1
16
14
42.2
41
38.4
20.9
19.3
17.1
LUG-KIN
2.9
6.3
5.3
19.9
29
27.5
4.2
8.5
7.1
ZUL-KIN
10.9
9.2
7.4
37.9
34.5
31.9
14.8
12.4
10
SSW-ZUL
11.1
10.2
9.1
43
42.2
40.4
20.5
19.2
17.5
XHO-KIN
10.9
9.8
8.2
37.1
35.3
32.4
14.4
12.9
10.5
LUG-AMH
3.5
2.6
2.4
18.1
17.2
16.5
10.7
10
9.1
SSW-AFR
16
15.6
13.2
41.3
40.4
37.7
19.7
18.6
NYA-AFR
16.4
14.7
13.1
42.4
39.8
37.7
20.7
18.2
16.4
SWA-TSO
14.8
12.1
13.5
41.5
37
39.5
17.7
13.9
15.4
TSO-SSW
6.6
5.9
5.2
36.8
35.4
33.4
13.1
12.2
10.6
KIN-AMH
6
4.1
25.7
21.9
22
16.4
13.5
13.4
NYA-TSO
11.7
10.7
10.7
37
36.1
36.3
14.5
13.5
13.3
NSO-TSO
12.8
13.9
14
39.4
39.7
39.8
15.3
16.5
16.3
KIN-NSO
14.6
12.1
12.3
39.1
36.7
36.7
16.2
14.1
13.9
IBO-YOR
2.3
2.9
2.5
17.5
18
17.4
3.8
5.2
4.2
SWA-SSW
5,9
4.3
4.7
36.2
31.9
34.1
12.4
9.5
10.5
SWA-NYA
12.4
9.4
10
43.6
38.6
39.3
16.5
12.7
13.3
SWA-YOR
2.7
3.7
3.1
18.4
18.8
18.5
3.8
6.6
4.5
NSO-SSW
7.1
6.4
6.1
37.2
35.8
35.1
13
12.4
12
NYA-SSW
4.7
4.6
4.7
33
33
32.6
10.4
10.6
10
SWA-AFR
25.3
20
19.9
51.8
46.2
45,9
29.1
23.5
23.2
TSO-XHO
9,2
8.6
8
40.1
39.1
38.1
16.9
15.6
15.1

---

BLEU
CHRF+
spBLEU
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
Baseline
L-Fine
F-Fine
NYA-LUG
3.2
4.6
4.6
24.2
27.4
26.9
4.5
6.9
6.6
AFR-AMH
9.3
8.2
33.1
30
30.7
23.1
21.1
21.9
NSO-LUG
3.4
5.3
5.3
24.6
29.3
28.5
4.4
7.6
7.4
AFR-NSO
18.7
14.9
15.4
44.8
41.1
20.7
17.3
17.8
KIN-HAU
14.9
12.9
12.3
39.6
36.4
36.3
17.5
15.3
14.6
SWA-IBO
14.3
11.8
12.7
38.3
34.9
36.4
17.2
14.9
15.7
ZUL-AMH
7.7
6.5
5.7
30.3
27.6
25.5
20.7
18.7
16.8
SWA-LUG
4.5
4.2
4.8
29.1
27.3
27.8
6.1
6.2
6.6
IBO-LUG
3.1
4.1
3.9
24.6
26.8
26.4
4.3
6.1
6
ZUL-NNO
18.8
16.7
15.8
44.5
41.6
41.1
20.8
18.3
17.5
SWA- -UU
13.1
9.8
10.1
47.3
42.3
42.7
24
18.7
18.9
SWA -HO
11.1
7.9
9.3
44.5
39
40.8
20.2
15
16.6
XHO-LUG
4
5.7
5
26.7
29.5
27.6
5.4
7.9
6.6
NSO-XHO
10.3
9.2
8.7
41.9
40.2
38.8
17.4
16.5
15.8
NYA-ZUL
8.8
7.8
7.2
40.6
38.6
37.1
17.8
16.2
14.8
SWA-KAM
2.7
2.8
2.8
19.5
20.7
20
4
4.3
4.1
WYA-XHO
7.7
6.7
6.4
38.7
36.4
35.6
15.7
13.7
13.2
KIN-TSO
13.1
10.7
10.9
39.3
35.8
36.2
15.9
12.9
12.8
NYA-NSO
1244
12.1
12.5
36.6
36.6
37.4
14.2
14.1
14.6
AFR-LUG
4.8
4.7
4.7
28.8
28.5
28.9
6.3
6.7
7.1
ORM-AMH
3,8
2.8
2.2
21.1
18.4
16.4
11.8
10
8.4
SWA-AMH
8.5
5.5
7
31.9
26.4
28.5
21.8
17.5
19.1
KIN-SWA
19.6
15.6
13.8
46.2
41.6
39.1
23.1
19
16.7
ZUL-LUG
3.6
5.3
4.7
26.1
28.7
27.2
5
7.6
6.5
SWA-NSO
17.4
14.4
15.1
43.1
39.4
40.4
19.1
16
16.7
SSW-XHO
8.9
8.9
7.8
39.8
39.7
37.5
16.3
16.5
14.8
KIN-SSW
5.1
4.6
3.9
34.1
31.7
31.4
11.3
9.8
KIN-NYA
10.4
8.4
39,9
37.2
35.2
14
12.1
10.9
LUG-YOR
2.7
3.1
2.8
16.1
17.2
16.5
4.8
5.8
ZUL-XHO
12
10.4
10
45.1
42.9
41.5
21.3
19.2
18.1
A AHO
11.1
9.7
9.5
44.6
42.4
42.1
20.5
18.3
18.1
AFR-ZUL
13
11.4
10.7
47.2
45.1
44.6
23.9
21.6
20.8
ZUL-TSO
14.3
14.1
14
41.9
40
40.3
17.4
16.9
16.7
KIN-AFR
17.2
15.3
14
42.3
39.7
37.8
20.4
17.8
16.2
SWA-HAU
19.5
14.8
16.5
45.8
38.9
41.6
22.3
17.2
19
SWA-ORM
1.I
0.7
0.6
18.3
15
14.6
2.3
1.1
0.9
AFR-TSO
15.4
13.5
13.8
43.2
40.1
41.2
18.9
16.2
16.6
KIN-LUG
1.9
4.1
4.1
19.3
26.8
26.5
3.4
6.2
5.8
KIN-ZUL
9.8
8.2
7.3
41.5
39.1
37.2
18.6
16.2
14.4
ZUL-SSW
7.3
7.2
6.6
39.3
38.2
37.1
14.9
15.1
13.7
KIN-XHO
8.6
6,9
6.8
39.1
36.6
35.7
15.7
13.5
12.6
AMH-LUG
2.6
3.2
3
24.7
25.8
25.4
3.6
5
4.6
AFR-SSW
6.3
5.4
5
37.7
36.1
35.9
13.6
12.5
11.9
AFR-NYA
12.3
11
10.3
43.1
41.1
40
16.4
14.6
13.9
Table B.5: Results of all language pairs FLORES200 devtest


Reviews:
[{"review_id": "esqy3EeZZQ", "reviewer": "Reviewer_BgTR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Presentation of LIMIT could have been more detailed, and additional experimentation on the method could have strengthened their findings\n\nThank you for your comments! Below, we share additional experimental results (extension of Table 7) as per your request. We trained 2 additional stronger baselines: \n- `multi`: a large fastText multilingual model with all 350+ languages\n- `group`: a traditional hierarchical model that predicts language family first, followed by language \n\nHighlights:\n- The traditional hierarchical model underforms the multilingual model on both MCS-350 and FLORES-200. \n- Our proposed method, LIMIT, outperforms the large multilingual model and the group-first baselines across domains. \n- Please see the last row of the table below for aggregate results.\n\nWe will include these details to strengthen the evidence in favor of our proposed technique and the presentation! Thanks again for your feedback and scores.\n\t\n\n|          | MCS-350 |              |              |              | FLORES  |              |       |              |\n| -------- | ------- | ------------ | ------------ | ------------ | ------------ | ------------ | ----- | ------------ |\n| Language | root    | multi        | group        | LIMIT        | root         | multi        | group | LIMIT        |\n| guj      | 0.49    | 0.44         | 0.58         | **0.63** | 1            | 0.81         | 0.99  | 0.99         |\n| kfr      |         | 0.78         | **0.83** | 0.8          |              |              |       |              |\n| bhil     |         | **0.63** | 0.03         | 0.28         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| amh      | 0.2     | 0.81         | 0.78         | **0.83** | 0.6          | 0.95         | 0.93  | **0.99** |\n| tir      | 0.56    | 0.93         | **0.94** | 0.85         | **0.99** | 0.96         | 0.95  | 0.95         |\n| stv      |         | 0            | 0.00         | 0            |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| ben      | 0.47    | 0.82         | 0.05         | **0.85** | 1            | 0.94         | 0.96  | 0.99         |\n| asm      |         | 0.63         | 0.89         | **0.89** | 0            | **0.98** | 0.96  | 0.66         |\n| cdz      |         | 0.57         | 0.87         | **0.93** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| zho      | 0.61    | 0            | 0.00         | **0.68** | 0.99         | 0            | 0.01  | 0.99         |\n| yue      |         | 0            | 0.02         | **0.14** | 0            | 0            | 0     | 0            |\n|          |         |              |              |              |              |              |       |              |\n| tel      | 0.92    | 0.75         | 0.80         | **0.94** | 1            | 0.83         | 0.91  | 1            |\n| kfc      |         | 0.66         | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kan      | 0.7     | 0.77         | 0.78         | **0.81** | 1            | 0.96         | 0.93  | 0.99         |\n| kfa      |         | 0.66         | **0.68** | 0.52         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| tso      | 0.49    | 0.53         | 0.34         | **0.67** | 0.97         | 0.79         | 0.72  | 0.97         |\n| tsc      |         | 0.74         | 0.52         | **0.77** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| dagaare  | 0.83    | 0.48         | 0.54         | **0.87** |              |              |       |              |\n| mzm      |         | 0.69         | **0.88** | 0.82         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kat      | 0.66    | 0.46         | 0.42         | **0.8**  | 1            | 0.72         | 0.7   | 1            |\n| bbl      |         | **0.82** | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| **Average**  | 0.29    | 0.56         | 0.47         | **0.68** | 0.77         | 0.70         | 0.73  | **0.86** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces MCS-350, a dataset of 50,000+ parallel children's stories in 350+ languages, offering a valuable resource for future research.\n\n-  The LIMIT model proposed in the paper utilizes limited data effectively to improve language identification, providing a novel and practical approach.", "reasons_to_reject": "- The authors' presentation of the Language (Mis)identification LIMIT could have been more detailed, and additional experimentation on the method could have strengthened their findings\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper constructs MCS-350, a dataset of 50,000+ parallel children's stories from the African Storybooks Initiative and Storyweaver, in 350+ languages. \n\n- In this paper, the authors present LIMIT, a misidentification-based hierarchical model designed to enhance the identification of low-resource languages using limited data\n\n- This paper presents a machine translation benchmark that allows evaluation in more than 1400 new translation directions.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CPp3U7saPQ", "reviewer": "Reviewer_Ri3q", "summary": "", "questions": "Q A: As the main failing from the part of the off-the-shelf language identifiers seems to be their coverage of the languages, have you taken a look at the identifier with the record number of languages (1,400)? It is available at https://sourceforge.net/projects/la-strings/ (http://www.cs.cmu.edu/~ralf/langid.html)", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! Below, we address each of your questions/concerns one by one:\n\n> The reasoning to use LIMIT is not on solid ground. What is the reason to \"avoid training large multilingual models\"? \n\n- To clarify, by “large multilingual models”, we refer to LID systems, not multilingual LLMs. \n- Low-resource languages simply do not have enough data to warrant training of large multilingual LID models (Section 2). Instead of a 1-level k-way classification with large k, hierarchical models break down the problem into a multi-level classification task. Previous work has shown this to be successful for identifying closely-related languages and for low-resource setups (lines 505-524). \n- Large models like AfroLID train on >4000 sentences/lang, while we are often working with <=1000 (lines 351-354). We mention in the paper that large multilingual models are resource-hungry, data-hungry, and inaccessible to train (591-599).\n\nThanks for highlighting this point. We will incorporate this into the paper’s Introduction section around line 97 to better motivate our proposed model. \n\n---\t\n> In the LIMIT method, the models for the missing languages are calculated anyway and the method adds complexity on top of an existing system. \n\nModels need to be calculated for missing languages and some complexity must be added to extend a system *when retraining is not an option* (due to lack of data, closed-source code, etc). LIMIT’s “complexity” is an intuitive hierarchical model that achieves improved LID quality across domains with very little data. \n---\t\n> For the existing generative language identification systems, you need only to add the models for the new languages and there is no need to re-train all the languages.\n\nThe premise of our paper is that when faced with low-resource settings, existing generative models are unfit. Such low-quality or domain-specific “coverage” creates a vicious cycle (293-305). Similarly, not needing to retrain all languages doesn’t imply good performance. See, for instance, the difference between's Franc’s performance on supported languages on FLORES and the children's stories domain (Table 7). We will include this clarification in Section 4’s motivation.\n---\t\n> At least it would be nice to see how the end results compare with another system with all the same languages. Currently, there is no meaningful comparison for the LIMIT language identification results.\n\n- With a large number of classes, low-resource languages suffer the most due to class imbalance in large multilingual models (lines 474-478). \n- Based on your request, we have attached an extended Table 7 demonstrating empirically that our model fares much better than a large multilingual model (trained on all the languages and training data). In addition, we also trained a traditional hierarchical model that predicts family first, followed by language. \n- We will include these details and additional strong baselines to strengthen the evidence in favor of our proposed technique (Section 4 and Table 7)! \n\t\nHighlights:\n- The traditional hierarchical approach (language family first, then language) underperforms the large multilingual model on both MCS-350 and FLORES-200. \n- Our hierarchical model, LIMIT, outperforms the large multilingual and the group-first models across domains. Aggregate results are available on the last line of the table below.\n\n|          | MCS-350 |              |              |              | FLORES  |              |       |              |\n| -------- | ------- | ------------ | ------------ | ------------ | ------------ | ------------ | ----- | ------------ |\n| Language | root    | multi        | group        | LIMIT        | root         | multi        | group | LIMIT        |\n| guj      | 0.49    | 0.44         | 0.58         | **0.63** | 1            | 0.81         | 0.99  | 0.99         |\n| kfr      |         | 0.78         | **0.83** | 0.8          |              |              |       |              |\n| bhil     |         | **0.63** | 0.03         | 0.28         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| amh      | 0.2     | 0.81         | 0.78         | **0.83** | 0.6          | 0.95         | 0.93  | **0.99** |\n| tir      | 0.56    | 0.93         | **0.94** | 0.85         | **0.99** | 0.96         | 0.95  | 0.95         |\n| stv      |         | 0            | 0.00         | 0            |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| ben      | 0.47    | 0.82         | 0.05         | **0.85** | 1            | 0.94         | 0.96  | 0.99         |\n| asm      |         | 0.63         | 0.89         | **0.89** | 0            | **0.98** | 0.96  | 0.66         |\n| cdz      |         | 0.57         | 0.87         | **0.93** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| zho      | 0.61    | 0            | 0.00         | **0.68** | 0.99         | 0            | 0.01  | 0.99         |\n| yue      |         | 0            | 0.02         | **0.14** | 0            | 0            | 0     | 0            |\n|          |         |              |              |              |              |              |       |              |\n| tel      | 0.92    | 0.75         | 0.80         | **0.94** | 1            | 0.83         | 0.91  | 1            |\n| kfc      |         | 0.66         | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kan      | 0.7     | 0.77         | 0.78         | **0.81** | 1            | 0.96         | 0.93  | 0.99         |\n| kfa      |         | 0.66         | **0.68** | 0.52         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| tso      | 0.49    | 0.53         | 0.34         | **0.67** | 0.97         | 0.79         | 0.72  | 0.97         |\n| tsc      |         | 0.74         | 0.52         | **0.77** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| dagaare  | 0.83    | 0.48         | 0.54         | **0.87** |              |              |       |              |\n| mzm      |         | 0.69         | **0.88** | 0.82         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kat      | 0.66    | 0.46         | 0.42         | **0.8**  | 1            | 0.72         | 0.7   | 1            |\n| bbl      |         | **0.82** | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| **Average**  | 0.29    | 0.56         | 0.47         | **0.68** | 0.77         | 0.70         | 0.73  | **0.86** |\n\n---\t\n\n> The method would have to be very bad for it not to outperform a system missing many of the languages in the test set (referring to Table 7).\n\nFor the root model, LIMIT also outperforms on the languages that were *not missing* across domains (Table 7), ex. Gujarati, Amharic, Bengali etc. Our method also outperforms a system that is trained on all the languages in the test set. See group-first and large multilingual models in the extended Table 7 above. We will highlight this in the paper when we analyze our system (Section 4.3-4.4)!\n\n---\t\n\n> The evaluation setting for language identification is in-domain.\n\nOur submission already includes evaluation in a different domain i.e FLORES-200. Please refer to lines 20-21 (Abstract), 103, 336-339, 421-425, 544-546 for details. For analysis, refer to lines 427-439\n\n---\t\n\n> Have you taken a look at the identifier with the record number of languages (1,400)?\n\nWe assume that you’re referring to the 2014 paper (that trains richer embeddings with non-linear mappings). We have looked at it and we didn’t use it because:\n- No experiments with the technique in low-resource training setups (current training data is 2.5 million bytes per language) \n- There is no analysis of cross-domain impact of the mapping techniques \n- Newer embedding methods trained on larger corpora like fastText have since been released\n\nWe will cite this paper in the Related Work section under “Language Identification” and mention our reasoning for not using the mapping technique on fastText embeddings. Thank you for pointing this out.\n\n---\t\n\n> Reproducibility\n\n- We have attached all code and data in a ZIP folder. Footnote on Page 1 indicates that “data, code, and models will be shared on GitHub publicly under permissive licenses.” \n- Instructions to reproduce all experiments and data collection (219-220, 256-257) are already provided. In Appendix A (1027-1070), we have outlined the relevant directories/scripts as well. We will expand Appendix A to include more details about the code and data statistics to encourage reproducibility!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The compiled dataset seems useful and should be published.\n- Further evaluation of existing off-the-shelf language identifiers on new challenging datasets and tasks is important.\n- The research in each of the areas presented in the paper shows promise and continuing with it is worthwhile.", "reasons_to_reject": "- The reasoning to use LIMIT is not on solid ground. What is the reason to \"avoid training large multilingual models\"? In the LIMIT method, the models for the missing languages are calculated anyway and the method adds complexity on top of an existing system. For the existing generative language identification systems, you need only to add the models for the new languages and there is no need to re-train all the languages. At least it would be nice to see how the end results compare with another system with all the same languages. Currently, there is no meaningful comparison for the LIMIT language identification results. The method would have to be very bad for it not to outperform a system missing many of the languages in the test set (referring to Table 7).\n- The evaluation setting for language identification is in-domain.", "questions_to_authors": "Q A: As the main failing from the part of the off-the-shelf language identifiers seems to be their coverage of the languages, have you taken a look at the identifier with the record number of languages (1,400)? It is available at https://sourceforge.net/projects/la-strings/ (http://www.cs.cmu.edu/~ralf/langid.html)", "paper_topic_and_main_contributions": "This very ambitious paper presents a corpus for 350+ languages, evaluates several off-the-shelf language identification methods on a test set they created, proposes a hierarchical language identification system added on top of an existing language identifier, and train new machine translation models and evaluate them on an existing dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Could you use commas in at least Table 1 to mark the thousands? The caption of Table 1 says \"with with\".\n\nThe heading \"Parallel Datasets\" on line 441 is not good as not all introduced datasets are parallel?"}}, {"review_id": "oilidM0Ud4", "reviewer": "Reviewer_g5ek", "summary": "", "questions": "A) How do you check that the corpus has the correct language labels?\n\nB) In table 4, which baseline is being used and how many language pairs are being averaged in each column?\n\nC) How did you choose the hyperparameters in section 4.1, footnote 8?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to genuinely thank you for your detailed comments. All the concerns you raised are important and we have provided clarifications, statistics, and new experimental results below. We address all the points one by one:\n\n> A) How do you check that the corpus has the correct language labels? \n\n- Both African Storybooks Initiative and Pratham Storyweaver verify stories and language.\n- For *unverified* stories, we implemented English text detection (since translators often copy English text into their story first and then translate it sentence by sentence) and empty story deletion (abandoned translation projects).\n- We manually verified all extracted monolingual text after using string matching on bilingual stories. \n- We provide metadata for Verified and Unverified stories at the source level, and versions of the dataset at each stage of pre-processing in the dataset. \n\nThank you for raising this point! We care about the data’s quality and fully agree that ‘representation washing’ must be avoided. Details about the data preprocessing, verification and quality inspection are present on the GitHub but we will incorporate them into the main text as well (Section 2, Appendix). \n\n---\n\n> B) In table 4, which baseline is being used? \n\n“Baseline” in Table 4 refers to DeltaLM model finetuned on 26 languages without adapters (line 255). L-Fine and F-Fine models are also built on top of DeltaLM but include adapters based on language or phylogeny (lines 247-254)\nWe will rewrite the relevant lines to avoid confusion. Thank you for flagging it.\n\n---\n\n> Table 4 column heading: X->Eng typo\n\nYou’re right. The sixth column’s header contains a typo and should be Eng->X (with column values 13.20, 17.46, 25.26). We can see that it is harder to translate out of English than into English. We will correct this typo in the camera-ready and double check all other tables. Thank you!\n\n---\n\n> In Table 4, how many language pairs are being averaged in each column?\n\nHere are the statistics for Table 4 for language pairs. Note that it doesn't include all the language pairs from MCS-350 by design and focuses on those that are common with FLORES-200 (for fair out-of-domain evaluation).\n- `All`: 176 (88 unique pairs)\n- `Afri->Afri`: 116 (58 unique pairs)\n- `X->Eng`: 16 \n- `Eng->X`: 16 \n- `Y->Fra`: 14 \n- `Fra->Y`: 14 \n\nWe will redesign Table 4 to include these counts and the standard deviations (below) so the data can be properly interpreted. Thank you for highlighting this!\n\n---\n\n> In Table 4, there is no indication on how scores vary across languages (e.g. standard deviation)\n\nWe share standard deviations for Table 4 below. We will add this to the Appendix as well.\n\n|          | All   | Afr->Afr | X->Eng | Eng->X | Y->Fra | Fra->Y |\n| -------- | ----- | -------- | ------ | ------ | ------ | ------ |\n| Baseline | 6.31  | 5.06     | 7.75   | 8.19   | 5.22   | 5.81   |\n| L-Fine   | 10.33 | 10.06    | 13.63  | 8.46   | 4.87   | 6.86   |\n| F-Fine   | 11.74 | 11.31    | 14.36  | 13.72  | 6.00   | 7.32   |\n\n---\n\n> Section 3 -  which pairs of languages have parallel data and how much is there for each \n\n- Due to space constraints, we omit the Afr->Afr matrix here, but we share training data sizes for English/French and African languages in both FLORES and MCS-350. Test set sizes are 1000, except for kam and orm, where total data < 1000, and test set size is 500 (line 235)\n- Statistics about each language pair in MT experiments are available in the data folder and on GitHub. But, we will certainly incorporate them into the Appendix to provide a better picture of the dataset! \n\n| Lang | w/ eng | w/ fra |\n| ---- | ------ | ------ |\n| kin  | 1334   | 379    |\n| kam  | 97     | \\-     |\n| zul  | 1828   | 392    |\n| ssw  | 984    | 115    |\n| afr  | 1746   | 769    |\n| swa  | 5379   | 2914   |\n| ibo  | 363    | 280    |\n| nya  | 917    | 598    |\n| orm  | 444    | \\-     |\n| nso  | 1104   | 102    |\n| xho  | 1873   | 790    |\n| tso  | 782    | 476    |\n| lug  | 1478   | 779    |\n| yor  | 319    | 132    |\n| hau  | 42     | 135    |\n| amh  | 1401   | 514    |\n\n---\n\n> C) How did you choose the hyperparameters in section 4.1, footnote 8?\n\nWe explored several starting learning rates and settled on lr=0.5  due to the performance on dev set. FastText’s lrUpdate parameter, by default, reduces the starting learning rate gradually over iterations. We don’t optimize for embedding dimension and use fastText’s default, 100, for supervised training. We will edit lines 347-358 to make these choices more clear.\n\n---\n\n> In Table 7, under FLORES-200 (root), you include 0.00 scores for some languages which are not covered.\n\nasm (Assamese) and yue (Yue Chinese) are both covered by FLORES-200. The root model received 0 F1 score on the FLORES test sentences for these 2 languages. We’ll include this in the Table’s caption to avoid confusion on seeing zeros.\n\n---\n\n> Stronger baseline (training fastText model on new data) required\n\n- Based on your request, we have attached an extended Table 7 demonstrating empirically that our model fares much better than a multilingual fastText model (`multi`). In addition, we also trained a traditional hierarchical model (`group`) that predicts language family first, followed by language. \n- We will include these details and additional strong baselines to strengthen the evidence in favor of our proposed technique (Section 4 and Table 7). Thanks for pointing this out, we agree that a stronger baseline helps the argument!\n\t\n**Highlights:**\n- The traditional hierarchical approach (family-then-language) underperforms the multilingual model on both MCS-350 and FLORES-200. \n- Our proposed LID method, LIMIT, outperforms the large multilingual model and the traditional hierarchical model across domains. \n- Aggregate results are present on the last line of the extended table below.\n\n|          | MCS-350 |              |              |              | FLORES  |              |       |              |\n| -------- | ------- | ------------ | ------------ | ------------ | ------------ | ------------ | ----- | ------------ |\n| Language | root    | multi        | group        | LIMIT        | root         | multi        | group | LIMIT        |\n| guj      | 0.49    | 0.44         | 0.58         | **0.63** | 1            | 0.81         | 0.99  | 0.99         |\n| kfr      |         | 0.78         | **0.83** | 0.8          |              |              |       |              |\n| bhil     |         | **0.63** | 0.03         | 0.28         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| amh      | 0.2     | 0.81         | 0.78         | **0.83** | 0.6          | 0.95         | 0.93  | **0.99** |\n| tir      | 0.56    | 0.93         | **0.94** | 0.85         | **0.99** | 0.96         | 0.95  | 0.95         |\n| stv      |         | 0            | 0.00         | 0            |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| ben      | 0.47    | 0.82         | 0.05         | **0.85** | 1            | 0.94         | 0.96  | 0.99         |\n| asm      |         | 0.63         | 0.89         | **0.89** | 0            | **0.98** | 0.96  | 0.66         |\n| cdz      |         | 0.57         | 0.87         | **0.93** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| zho      | 0.61    | 0            | 0.00         | **0.68** | 0.99         | 0            | 0.01  | 0.99         |\n| yue      |         | 0            | 0.02         | **0.14** | 0            | 0            | 0     | 0            |\n|          |         |              |              |              |              |              |       |              |\n| tel      | 0.92    | 0.75         | 0.80         | **0.94** | 1            | 0.83         | 0.91  | 1            |\n| kfc      |         | 0.66         | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kan      | 0.7     | 0.77         | 0.78         | **0.81** | 1            | 0.96         | 0.93  | 0.99         |\n| kfa      |         | 0.66         | **0.68** | 0.52         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| tso      | 0.49    | 0.53         | 0.34         | **0.67** | 0.97         | 0.79         | 0.72  | 0.97         |\n| tsc      |         | 0.74         | 0.52         | **0.77** |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| dagaare  | 0.83    | 0.48         | 0.54         | **0.87** |              |              |       |              |\n| mzm      |         | 0.69         | **0.88** | 0.82         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| kat      | 0.66    | 0.46         | 0.42         | **0.8**  | 1            | 0.72         | 0.7   | 1            |\n| bbl      |         | **0.82** | 0.66         | 0.66         |              |              |       |              |\n|          |         |              |              |              |              |              |       |              |\n| **Average**  | 0.29    | 0.56         | 0.47         | **0.68** | 0.77         | 0.70         | 0.73  | **0.86** |\n\n---\n\n> Comparison in Table 6 - coverage vs. high F1\n\nWe share an updated Table 6 below with the language counts, where we can see that Franc wins on both coverage and performance (caption, Table 6). This certainly provides more transparency to the comparison, so we thank you for your comment. We will include this updated table in the camera-ready to provide more information on the root model’s selection.\n- \"Common\" means languages in common with the LangID model and our new MCS-350 dataset. \n- \"Coverage with LIMIT\" indicates the number of languages that the root model and our model, together, will cover\n\n| Model     | F1           | Supported | Common      | Coverage with LIMIT |\n| --------- | ------------ | --------- | ----------- | -------------- |\n| CLD3      | 0.11         | 101       | 81          | 376             |\n| langid.py | 0.09         | 97        | 73          | 380             |\n| Franc     | **0.18** | **369** | **116** | **609**    |\n| fastText  | 0.1          | 176       | 117         | 415             |\n| HeLI-OTS  | 0.13         | 200       | 81          | 475            |\n\nWe hope our responses and additional experiments address your soundness concerns (as reflected by the soundness score).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The MCS-350 corpus has the potential to be a valuable resource for the NLP community because it covers a large number of previously under-supported languages. I particularly like that they have included language varieties (though I hope that the mappings to non-standard codes are explained carefully in the repository). In addition, children's stories are a novel domain for low-resource language data, since most existing data is either religious text or Wikipedia.\n\n2) Their hierarchical approach to language identification with learned confusion patterns is a novel and interesting idea.\n\n3) The machine translation benchmark could be a useful addition to research into low-resource machine translation", "reasons_to_reject": "1) My main concern with this paper is the lack of information on how the authors ensured the data was in the correct language. They mention in section 2.1 that they remove empty stories or those which have majority English text (according to langdetect), which to me implies that the other language labels may not be faithful. If this work is to be used as a benchmark for work on low-resource languages, then it is particularly important to check that it is in the right language to avoid \"representation washing\" (see section 6, https://aclanthology.org/2022.tacl-1.4.pdf). This is especially the case for the lowest-resource languages mentioned in section 2.2: the string similarity matching technique mentioned is not perfect, and the authors should say what they did to ensure the labels were correct. \n\n2) Section 3 on the machine translation benchmark is currently lacking in some key details. In particular I would like more detail about the dataset, including how much data there is for each language and how parallel it actually is (i.e. which pairs of languages have parallel data and how much is there for each). With respect to the analysis, it is not clear which baseline is being used: in section 3.2 they say they use the DeltaLM model with fine-tuning on 26 languages, but then on line 254 they mention an additional baseline without adaptors. In addition, the results in table 4 lack information needed to understand the performance of the model: for example, it is not clear how many language pairs' scores are being averaged in each column, two of the columns have the same heading (Avg_{X -> ENG}, and there is no indication on how scores vary across languages (e.g. standard deviation). Overall, I would like more information about the dataset and the cross-language performance so I can understand the quality of the dataset.\n\n3) In section 4, there is no comparison with existing work, only with the authors' `root` model which does not support all the languages. A strong baseline would strengthen the analysis (e.g. train a fasttext model on the new dataset), and even if the baseline outperformed the model, there is still a contribution as the authors' method can be used to extend existing models. In a related point, the comparison of other models in table 6 is not fair, since the models all cover different languages so higher F1 score is likely a function of higher coverage. The authors should note how many languages each LID model covers.", "questions_to_authors": "A) How do you check that the corpus has the correct language labels?\n\nB) In table 4, which baseline is being used and how many language pairs are being averaged in each column?\n\nC) How did you choose the hyperparameters in section 4.1, footnote 8?", "paper_topic_and_main_contributions": "This paper deals with improving language identification for low-resource languages. Its main contributions are the following:\n\n1) A new corpus, MCS-350, which is derived from two collections of original-language and human-translated children's stories. The stories are covered under permissive licenses so they can be used easily in future work.\n\n2) A machine translation benchmark based on the parallel natures of the above dataset. They train a baseline MT system based on the highest-performing publicly-available model from WMT 2022, and show they can improve MT performance for African languages by fine-tuning adaptor units in a hierarchy based on language family. \n\n3) A new hierarchical modelling approach based on learned confusion patterns in an existing model. It works by observing which languages are most often confused by an existing model, then training new subunits to distinguish between these languages. They present some analysis showing their method improves F1 scores for previously unsupported and confused languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 61: missing an 's' in 'translation systems\"\n- Could you explain what phylogeny means in the text? I don't think it's common enough a term to leave it unexplained (or just use 'language family').\n- The captions for tables 4 and 5 do not explain the contents of the tables enough (e.g. in table 5, it's not clear if these are results from FLORES or MCS-350). The captions should explain what's in the table without having to refer to the text.\n- In Table 7, under FLORES-200 (root), you include 0.00 scores for some languages which are not covered."}}]

Author rebuttal:

Meta-review: Summary (adapted from reviewer Ri3q): This paper presents a corpus for 350+ languages, evaluates several off-the-shelf language identification methods on a test set they created, proposes a hierarchical language identification system added on top of an existing language identifier, and trains new machine translation models and evaluates them on an existing dataset.

All reviewers were positive regarding the value of the new corpus developed in this paper.

Reviewers raised some questions regarding the details of the experiments, the vast majority of which seem to be addressed well in the authors’ response. While the paper should be updated to reflect all the additional information provided during the discussion period, these are mostly clarifications and the addition of some useful but not necessarily essential baselines. The paper submitted for review was still sound, but these additions will enhance its soundness.

The submitted paper also did not clearly justify why the specific hierarchical model needed to be used, in comparison to other models. This was clarified in the authors’ response as well.

Overall, the corpus and the models represent significant contributions for the field of language ID for less-resourced languages.
Decision: Accept-Main