On the use of linguistic similarities to improve Neural Machine Translation for African Languages
Authors: Pascal Junior Tikeng Notsawo
NANDA ASSOBJIO Brice Yvan
James Assiene
Venue:  (2021)
Keywords: Machine Translation
Multilingualism
Linguistic similarity
Dataset
African languages
Multi-task learning

Abstract: In recent years, there has been a resurgence in research on empirical methods for machine translation. Most of this research has been focused on high-resource, European languages. Despite the fact that around 30% of all languages spoken worldwide are African, the latter have been heavily under investigated and this, partly due to the lack of public parallel corpora online. Furthermore, despite their large number (more than 2,000) and the similarities between them, there is currently no publicly available study on how to use this multilingualism (and associated similarities) to improve machine translation systems performance on African languages. So as to address these issues: 
We propose a new dataset for African languages that provides parallel data for vernaculars not present in commonly used dataset like JW300 [1]. To exploit multilingualism, we first use a historical approach based on historical origins of these languages, their morphologies, their geographical and cultural distributions as well as migrations of population to identify similar vernaculars.
We also  propose a new metric to automatically evaluate similarities between languages. This new metric does not require word level parallelism like traditional methods but only paragraph level parallelism.
We then show that performing Masked Language Modelling and Translation Language Modeling in addition to multi-task learning on a cluster of similar languages leads to a strong boost of performance in translating individual pairs inside this cluster.
In particular, we record an improvement of 29 BLEU on the pair Bafia-Ewondo using our approaches compared to previous work methods that did not exploit multilingualism in any way.

[1] http://opus.nlpl.eu/JW300.php

On the use of linguistic similarities to improve Neural Machine Translation for African Languages
Pascal Tikeng"
Brice Nanda Department of Computer Science NASEY
Department of Computer Science
NASEY pascalnotsaxo@gmii.ccom
bricenanda21@gmaii com
James Assiene assiene, assiene. jameseejmmmm@mmccoomm
Abstract
In recent years, there has been a resurgence in research on empirical methods for machine translation. Most of this research has been focused high-resource, European languages. Despite the fact that around 30% of all languages spoken worldwide are African, the latter have been heavily under investigated and this, partly due to the lack of public parallel corpora online. Furthermore, despite their large number (more than 2,000) and the similarities between them, there is currently no publicly available study on how to use this multilingualism (and associated similarities) improve machine translation systems performance on African languages. So as to address these issues, we propose a new dataset (from source that allows us to use and release) for African languages that provides parallel data for vernaculars not present in commonly used dataset like JW300. To exploit multilingualism, we first use a historical approach based on migrations of population to identify similar vernaculars. We also propose new metric to automatically evaluate similarities between languages. This new metric does not require word level parallelism like traditional methods but only paragraph level parallelism. We then show that performing Masked Language Modelling and Translation Language Modeling in addition to multi-task learning on a cluster of similar languages leads to a strong boost of performance in translating individual pairs inside this cluster. In particular, the pair awaond using our approaches compared to previous work methods that did not exploit multilingualism in any way. Finally, release the dataset and code of this work to ensure reproducibility and accelerate research in this domain.
record improvement of 29 BLEU on
Introduction
Machine Translation (MT) of African languages is challenging problem because of multiple reasons. As pointed out by Martinus and Abbott (201)). the main ones are:
Morphological complexity and diversity of African languages Africa is home to around 2144 languages out of nowadays 7111 living languages (they thus make 30.15% of all living languages) with often different alphabets (Eberhard al.) 2020).
Lack/Absence of large parallel datasets for most language pairs.
'NASEY National Advanced School Engincering Yaounde, Cameroon MMLLA-Quebe Artificial Intelligence Institute
5th Black AI Workshop @ NeurIPS

---

Discoverability: The existing resources for African languages are often hard to find.
Reproducibility: Data and code of existing research are rarely shared, making difficult for other researchers to reproduce the results properly.
Lack of benchmarks: Due to the low discoverability and the lack of research the field, there are publicly available benchmark or leaderboard to compare new machine translation techniques to.
However, despite the strong multilingualism of the continent, previous research focused solely on translating individual pairs of language without taking advantage of the potential similarities between all of them. The main purpose of this work is to exploit this multilingualism, starting from a few languages spoken in Central and West Africa, to produce better machine translation systems.
Our contributions can be summarised as follows:
We provide new parallel corpora extracted from the Bible for several pairs of African vernacular that were not available until now as well the code used to perform this extraction.
2. We present method for aggregating languages together based their historical origins, their morphologies, their geographical and cultural distributions etc... We also propose of new metric evaluate similarity between languages this metric, based on language models, doesn't 't require word level parallelism (contrary to traditional like (Levenshtein) 1965) and (Swadesh [952]) but only paragraph level parallelism. It also takes into account the lack of translation of words present in (Swadesh) 1952) but not in African vernaculars (like "snow").
3. Using the language clusters created using the previous similarities, we show that Translation Language Modelling (Lample and Conneau 2019) and multi-task learning generally improve the performance on individual pairs inside these clusters.
Our code, data and benchmark are publicly available at wethttts:///gt /ttp::///hthub. Catiub..oomiiiiee com/Tikkuuss//mttat XLM
The rest of the paper is organised as follows: In section we discuss the motivation of this work i.e. problems faced by African communities that could be solved with machine translation systems.
Related work is described in Bld relevant background is outlined in 4 In section 5 we describe
dataset and provide more details the language aggregation methods. In section 6 we present
the experimental protocol and the results of our methodology on machine translation of 3 African
(Cameroonian) vernaculars. Lastly, we summarise our work and conclude in 7
2
Motivation
Africa, because of its multilingual nature, faces a lot of communication problems. particular, most African countries have colonial languages (French, English, Portuguese, etc...) official languages (with sometimes one or two African dialects). The latter are those taught in schools and used in administrations and workplaces (which often located in urban areas). In contrast, rural and remote areas citizens mainly speak African dialects (especially the older ones and the less educated ones). This situation creates the following fairness and ethical concerns (amongst others):
Remote area population often have difficulties communicating during medical consultations since they mainly speak the local languages, contrary to the doctors who mostly speak French, English, etc.... Similar situations arise when NGOs try to intervene in rural regions.
This discrepancy between spoken languages across different regions of individual African countries make the spread of misinformation easier (especially in rural regions and especially during election periods).
Young Africans (in particular those living in urban areas) struggle learn their vernaculars. Lack of documentation and schools dealing with African languages as well as the scarcity of translators and trainers make the situation even more complicated.
For all of the reasons above, is essential set up translation systems for these languages.
https://en.. wwt.ipedia..rg/viiiee....e httpp://oaaaiillllaaaoo///////rs

---

3 Related work
Several scholarships have been conducted on machine translation for African languages, Wilken jet al. (2012) performed statistical pbrase-based translation for English to Setswana translation. This research used Aiguistically-motivaan
pre and post processing of the corpus in order to improve the translations. The system was trained on the Autshumato dataset (Groenewald and Fourie 2009) and also used an additional monolingual dataset. They reached a BLEU score of 28.8. MccKellar (2014) used statistical machine translation for English to Xitsonga translation. The models were trained on the Autshumato data (Groenewald and Fourie| 2009), as well as large monolingual corpus, A factored machine translation system was used, making use of a combination of lemmas and part of speech tags. It obtained a BLEU score of 37.31. Nickerk (2014) used unsupervised word segmentation with phrase-based statistical machine translation models. These models translate from English to Afrikaans, N. Sotho, Xitsonga and isiZulu. The parallel corpora were created by crawling online sources and official government data and aligning these sentences using the HunAlign software package. Large monolingual datasets were also used. They reached a BLEU score of 71 on English to Afrikaans and 37.9 on English to N.Sotho. Wolff and Kotze (2014) performed word translation for English to isiZulu. The translation system was trained on a combination of Autshumato, Bible, and data obtained from the South African Constitution. All of the isiZulu text was syllabified prior to the training of the word translation system.
Abbott and Martinus (2018) trained Transformer models for English to Setswana on the parallel Autshumato dataset (Groenewald and Fourie (2099 and obtained BLEU score of 33.53. Data was not cleaned nor was any additional data used. This is the rare study reviewed that released datasets and code. Martinus and Abbott (2019) used Google Translate to perform English to Afrikaans translation and achieved BLEU score of 41.18 To the best of our knowledge, |Msakhaaee (2020) are the only ones to have carried out research on wide variety of African languages. They used the JW300 dataset (Agié and Vulié, 2019) from opus collection and trained Transformer models on it. However, they did not do any preliminary linguistic study of their languages of interest. Lastly, Emezue and Dossou (2020) provided a benchmark for Fon to French translation using the same JW300 dataset (Agié and Vulié 2 They used GRU (Chung et al. 2014) sequence sequence model with attention (Bahdanau et al. (015)) and obtained a BLEU score of 30.55, These last two studies are ones of the very few to have released datasets and code.
Regarding the use of multilingualism in machine translation, (Johnson et al. 2016) applied multi-task learning to a single mecoer-ddcooer LSTM model to translate between English, French and German. The best gain of performance of their many-to- model was 0.43 BLEU on (French, English)) compared to a single-task model trained on that pair. However, its worse decrease of performance was 2.75 BLEU on (English, German)
Ha et al. 2016) also worked on English, French and German. Their best gain of performance compared to a single task model was 2.64 BLEU on (English, German) in an under-resourced scenario. They used an original approach called mix-source where the pair (German, German) was added the translation model training data. The goal was help the model obtaining a better representation of the source side and learn the translation relationships better. They argued that including monolingual data in this way might also improve the translation of some rare word types such as named entities.
4 Background
4.1 Neural Machine Translation (NMT)
Machine Translation (MT) refers to fully automated software that can translate source content into target languages. Neural Machine Translation (NMT) is an approach to MT that uses an artificial neural network to predict the likelihood of sequence of words, typically modeling entire sentences in single integrated model. NMT aims to translate an input sequence from a source language to a target language. An NMT model usually consists of an encoder to map an input sequence to hidden representations, and a decoder to decode hidden representations to generate sentence in the target language.
http::/opus. hwapp//300.ppppm
3

---

NMT has made rapid progress in recent years. The attention module is first introduced by Bahdanau al. (2013), which used better align source words and target words. The encoder and decoder can be specialized as LSTM (Hochreiter and Schmidhuber 1997; Sutskever et al., 2014 Wu et al. 2016), CNN (Gehring et al. 2017) and Transformer (Vaswani et 2017). consists of three sublayers, self-attention layer that processes sequential data taking the context of each timestep into consideration, an optional encoder-decoder attention layer that bridges the input sequence and target sequence which exists in decoder only, and a feed-forward layer for non-linear transformation. Transformer achieves the state-of-the-ar results for NMT (Barrault et al. 2019).
Transformer layer
4.2 Cross-lingual Language Model (XLM) Pretraining
Recently, pre-training techniques, like EL (Peters et al. 2018), GPT/GPT-2 (Radford et al. 2018)
BERT (Devlin et al. 2019), cross- cross-lingual language model (briefly, XLM) (Lample and Conneau 2019), XLNet (Yang et al. 2019) and RoBERTa (Liu et al. 2019) have attracted more and more attention in machine learning and natural language processing communities. The models are first pre-trained on large amount of unlabeled data to capture rich representations of the input, and then applied to the downstream tasks by either providing context-award embeddings of an imput sequence (Peters al., 2218)), initializing the parameters of the downstream model (Devlin et al. 2019) for fine- tuning. Such pre- pre-training approaches lead to significant improvements on natural language understanding tasks.
Cross-lingual language model (Lample and Conneau 2019) is used to obtain a better initialization of sentence encoders for zero-shot cross-lingual classification, better initialization of supervised and unsupervised neural machine translation systems, a language models for low-resource languages and unsupervised cross-lingual word embeddings. This done through four objectives:
Shared sub-word vocabulary: XLM processes all languages with the same shared vocabu- lary created through Byte Pair Encoding (BPE) (Sennrich et al. 2015)
Causal Language Modeling (CLM): XLM causal language modeling (CLM) task consists a Transformer language model trained to model the probability of a word given the previous words in sentence P(wr (WI, ... W), 0).
Masked Language Modeling (MLM) XLM consider the masked language modeling (MLM) objective of Devlin et al. 2019). also known as the Cloze task (Taylor 1953). Following Devlin et al. (2019). XLM sample randomly 15% of the BPE tokens from the text streams, replace them by a [MASK] token 80% of the time, by a random token 10% of the time, and keep them unchanged 10% of the time. Differences between XLM approach
and the MLM Devlin et al. (2019) include the use of text streams of an arbitrary number of sentences (truncated at 256 tokens) instead of pairs of sentences.
Translation Language Modeling (TLM) : Both the CLM and MLM objectives are un- supervised, only require monolingual data, and cannot be used to leverage parallel data when it is available. XLM introduce a new translation language modeling (TLM) objective for improving cross-ling cross-lingual pretraining TLM objective is an extension of MLM, where instead of considering monolingual text streams, parallel sentences are concatenated. XLM randomly mask words in both the source and target sentences. To predict word masked in an source sentence, the model can either attend to surrounding source words or to the target translation, encouraging the model to align the source and target representations. In particular, the model can leverage the target context if the source one is not sufficient to infer the masked source words. XLM also reset the positions of target sentences to facilitate the alignment.
In this work, we used the XLM framework to train and evaluate our models.

---

Methodology
5.1 Data
You provides biblical content in 1,756 languages among which many (local) African ones. We have extracted this content in the desired languages to build our dataset. In particular, for some countries like Cameroon, Version provide more languages (22 for the latter) than JW300 (Agić and Vulić 2019) (18 for Cameroon) that is the most used data source for work on African languages (e.g Masakhane 2020). We release our adaptable extraction script along with this publication. The number of sentences per language is given in figure 1
English
French
Bafia
MASSANA
Bulu
Dii
Doyayo
Dun
30901
31296
7950
31128
28895
28895
7900
7916
Ejagham
Fulfulde
Ghomala
Guidar
Guiziga
Gbaya
Kapsiki
Limbum
7890
30840
7942
7915
31084
31092
31095
7919
Ewondo
Mofa
Igiemboon
Mofu
Peere
Samba
Tupurri
Vute
7944
7945
7929
7941
7905
7905
31268
7909
Figure I: Number of sentences per language. The number of sentences for language pair (L1, L2) is equal to min(|4il,14a) where [Li] represents the number of sentences of the language L;
The Bible is one of the rare publicly available online corpora for low-resource languages such as those spoken Africa. Its structure (made of books, chapters and verse numbers) allows to automatically and unambiguously identify parallel paragraphs (verses) across all languages. However, it suffers from several flaws for machine translation:
general, You Version only provides the New Testament for African languages, consisting of about 7958 verses only. In the case of languages for which both New and Old Testament are available, one can retrieve around 170 verses.
2. The Bible contains a lot of redundant sentences ("Truly, say to you'l tell you" for example), verses made up of characters and places names etc... Making the translation task casier than if closer to real world dataset like WMT was used. Its antiquated writing style is also far from contemporary style.
3. The alignment information is limited to verses rather than sentences unlike the JRC-Acquis corpus (Steinberger el al. 2006 for instance.
Christodouloupoullos and Steedman 2015 gives more details pros and cons of using the Bible as machine translation corpus.
Following/Lample and Conneau 2019), we process the languages with the same shared vocabulary created using Byte Pair Encoding (BPE) (Sennrich al.) (205). Due to the lack of specifie tokenizers for A African languages, we used an english one on all of them (we performed tokenization before applying the BPE algorithm).
5.2 Languages similarities and aggregations
5.2.1 Historical approach for languages aggregation
From historical point of view, just like African borders, African languages have undergone the movements of the early inhabitants of the continent. These migrations have shaped and modified the languages spoken in multiple regions and led resemblances between of them. For these reasons, we examined and used languages aggreations proposed by Eberhard et al.) (2020) that are based their origins and evolutions.
The intersection between our dataset and Eberhard et al. (2020) analysis led the following languages clusters:
https://www bible. wttp://wwwww.....e///o/oo///ocee heaaee.c////e//aguaaeeee (October 22, 2021)
5

---

CCuustr 1 : (Bafia, Ewondo, Bulu). These languages are all part of the Niger-Congo, worthwes Narrow Bantu family and spoken in the central and southern regions of Cameroon.
Figure 2: Historical approach for lan- guages aggregation
Bantu family central southern
Cluster 2: (Ngiemboon, Ghomala, Limbum). They are part of the Niger-Congo. Mbam-Nkam family and spoken in the West Cameroon.
Biu-Mandara family Northem Cameroon
Cameroon
Guiziga
Mofa
Guidar
Bulu
Bafla
Cluster 3 (Dii, Doyago, Samba Leko, Peere). These languages are spoken in the Northern region of Cameroon and belong to the Niger-Congo. Adamawa family
Mucu-Gudur
Kapsiki
Ewondo
Doyago
Peere
Nyiemboon
Samba Leko
Dii
Limbum Ngomala
Cluster 4 (Guiziga, Mofa, Mofu-Gudur, Kapsiki, Guidar). They are also spoken in Northern Cameroon but have been classfied as Afro-Asiatic languages. particular, they belong to the Biu-Mandara 10 family.
Adamawa family Northern Camercon
Mbam-Nkam fam
West Cameroon
In the rest of this work used the first cluster mentioned above for the experiments it is the smallest of the 4 and is made of some of the most spoken languages in Cameroon (compared to the others listed above)
5.2.2 Language model based approach for lingustic similarities
Despite providing a historically grounded methodology to identify languages families, methods such as Eberhard et al. (2020) are time-consuming and require a lot of background knowledge on specific population migration throughout history. Such in-depth knowledge might not be available for all languages. Besides, they don't 't provide a quantitative measure for languages similarity, making it difficult for one to assess if language L, is closer language L, than to language Lę- Quantitative metrics are necessary for well known clustering algorithms like K-Means or Hierarchical Clustering.
Multiple linguistic similarity metrics have been proposed in the past. The most popular are the ones used in lexicostatistics among which Swadesh lists (Swadesh 1952) and Levenshtein distance (Levenshtein) 1965). Nonetheless, the latter require data parallelism at word level and dictionaries might not be freely available online for all languages pairs. Additionally, Swadesh lists (Swadesh) 1952) are based on a small number of words that might not even exist in all languages. For example, the word "snow" (present in the Swadesh 207 list) doesn't 't have translation in Bafia (likely because it never snows in this region).
To alleviate these problems, we propose language model based metric for linguistic similarity that requires paragraph level parallelism only. This approach formalises (using language models) the following human intuition child whose native language is LA may think that a language L2 is similar to L1 if words (possibly with the same meaning) in L2 "sounds" ""oooks" like words in L1.
For example, the english word "bilingual" is translated in french by "bilingue" and in spanish by "bilingüee" but its polish translation is "dwujezyczny". A french speaker can easily think that the spanish "bilingüe" is a french word both at the spelling level because of common syllables ("bi", "lin" and "gu""/"gü" and at the pronunciation level. Nevertheless, it is unlikely for the polish "dwujezyczny" to be seen as french word. Following the success of recent language models like GPT-2 Radford al. (2019), the previous french speaker can be seen as a very powerful french language model fm If we evaluate it on both "bilingüe" and "dwujezyczny" it should give a higher probabilty to the spanish "bilingüe" than to the polish "dwujezyezny" because of the french "bilingue". More formally, we expect the BPE encoding to capture syllables like "bi". "lin" and "gu"/"gü"" gü' so that if Pim,. (hilingue) = ,. (bi) * Pimm,.(1ia (lin(bi) * Pom (gu(bi, lin) Pimy..C (e (i, lin, gu) is relatively high then Pmm,. (bilingüe) = Pimma (bi) Pimy, (lin(bi) Pimyy (gūl bi, lin) Pony. (elbi, lin gü) should be greater than Pum, (dwujezyczny)
-kam
"I" can become "u" during lemmatization

---

At a paragraph level, we expect that given two parallel paragraphs C sponish and Cpolish, Imfr gives a higher probability to Opanish i.c. Pimm (C (pannssh) (
Given a trained language model LMo in language To and two parallel corpora Co in Lo and C1 in language L1, we define the LM-similarity between Lo and L1 according to LMoo and Co by
LMSLMM.CO (Lo. L1) PPLLM, (C1) PPPLMP
PPLLM, (Cu)
Where PPLLM, (C²) is the perplexity of the language model LMo on the corpus C.. It is important to precise that Accuracy (percentage of tokens correctly predicted by the language model) can be used in lieu of Perplexity (this may lead to different similarities between languages).
Despite the fact that this metric is not symmetric, it can be used build clusters using hierarchical clustering approach. We start by a set of languages G1 = {L1, L2], the language L that will be added G1 can be determined using the following formula (inspired by the unweighted average linkage clustering criteria):
L arg min (LMSLM,,C. Li) + LEDA
(L2, L.))
In other words, L* the language in the set possible languages L (but we exclude G1 {L1, L2)) that minimises the average "distance" (LM-similarity) from Lj and L2. By recursively applying this formula, one can build a cluster of any size.
The proposed similarity metric should be interpreted more dissimilarity, hence the equation normalizing perplexity difference according base perplexity, to allow be zero on the base language and the base corpus. The word "similarity" here is only an abuse of language, to signify the fact that obenuous languages are similar the sense of this metric. This method mainly motivated by the fact that the resources needed to apply other methods such as those mentioned above are not available. M. Eberhard et al. (2019) has been easy replicate in that it directly provides groups of similar languages: we have just made an intersection between these groups and the languages in our dataset. On the other hand, a method like (Swadesh) (952) requires parallelism data at the word level, which we did not have and could not find: this is one of the limitations of this work..
6 Experiments and Results
For each ordered language pair in the cluster of interest (Bafia, Ewondo, Bulu), we examine the machine translation performance (using BLEU score (Papineni al. 2002) on that pair we add third language to the pair fo form a cluster. The model will thus become multi-task machine translation model capable of translating 3 languages rather than 2. The language (among the 22 in dataset) to add is chosen in 3 ways in work:
1. Using historical approach like Eberhard et al. (2020). For the pair (Bafia, Ewondo) we will add Bulu. It corresponds the column Historical in Table
Table 1: Languages used respectively for the Ran- dom, Historical and LM columns of Table The first column represents the reference pairs, the columns Random, Historical and LM contain re- spectively the languages that have been added to form the clusters that gave the results of the corre- sponding columns in Table
2. Using the LM-similarity (as described in 5.2.2p. For the pair (Bulu, Ewondo), it will be Mofa. It corresponds the column LM in Table 2
3. Purely at random it will serve baseline to make sure that the difference in performance is due the particular choices of our method- ology rather than multi-task learning solely. It corresponds to the column Random Table
Pair
Random Historical LM
Bafia-Bulu Bafia-Ewondo Bulu-Ewondo
Du_ n
Ewondo
Mofa Bulu
Guidar
Bulu
Dii
Bafia
Mofa
Having identified our clusters and to further exploit multilingualism, we follow the approach of Lample and Conneau (2019) by pre-training the model's encoder with MLM and TLM objectives
7

---

(without CLM) on the cluster languages, then tune the model on these same languages. The MLM pee-traimm pre-training was used as the languages did not have the same number of sentences : for example for the Bafia-Bulu pair, we have 7950 parallel sentences (representing the number of sentences of the Bafia language) but 28895 sentences for Bulu (i.e. 28895 7950- = 20945 non-parallel sentences). The TLM is naturally motivated by XLM encourage the model to align source and target sentence representations (especially by taking advantage of the target context if the source context is not sufficient to infer hidden source words).
All our models are tagged models, and the tag for given language is its id in the list of languages used. Indeed. in addition to positional and token embedding, XLM does language embedding. This motivated by several works. Caswell et al. (2019) add a specific token (the <BT> tag) at the beginning of the synthetical sentences produced using back fack-translation to distinguish them from the available non-synthetical sentences. Johnson et al. (2017) do multilingual MT by adding a special token at the beginning of each sentence of the same language indicating the source language, and do not change anything about the architecture of classical MT' models. Yamagishi et al. (2016) add tag specific to the voice (passive, active) of the target sentence in the source sentence that the sentences produced by the model keep the voice of the source sentence. Kuczmarski and Johnson (2018) do the same, but more for gender (male and female gender).
We use the average BLEU score on the validation sets across all the languages as stopping criterion. Due to the multi-task nature of the model, this criterion is preferred to validation BLEU score on the pair of interest in some cases, the model can first have low, stagnating performance on that pair for a large number of epochs (greater than the patience, threshold to stop the training) but at the same time have good performance on the other pairs (leading to an increase of average BLEU score). At the end of a training with average BLEU score as stopping criterion, the pair of interest "catches up" with the other pairs and obtain high scores.
We also compare the performance with model trained: 1) On the pair of interest without any form of MLM/TLM pre-training (Column None in Table 2). 2) On the pair of interest with MLM+TLM pre- pre-training (Column Pair in Table 2)
6.1 Training details
We used the default settings mentioned in the documentation of the XLM framework: : Transformer architectures with 1024 hidden units, 8 heads, GELU activations (Hendryeks and Gimpel| (016))
Dropout rate of 0.1 and learned positional embeddings, Adam optimizer (Kingma and Ba. 2014).
linear warmup (Vaswani al., (017) and learning rates of 0.0001 for both XLM pre- pre-training (MLM
and TLM, without CLM) and machine translation. For the MLM and TLM pre-training, we used the averaged perplexity over all the languages as a measure of performance and stopping criterion for
training, and for machine translation, we used the average BLEU score across all the languages pairs of the models. The patience was set to 20 for all models.
The models were trained on one 48GB NVIDIA Quadro RTX 8000 GPU for Random, Historical and LM and on a 11GB NVIDIA RTX 2080 Ti for None and Pair. In all our experiments, we used 80% of the dataset as training data, 10% validation data and 10% as test data.
6.2 Results, discussion and limitations
Table 2 describes the results of the baselines mentioned in and our models. We notice that using our historical approach systematically leads to strong boost of performance and always outperforms the None, Pair and Random basclines. As the aggregation approach that led to the best results, we evaluated its impact on translating African vernaculars to European languages. We chose french it is the most spoken language in Cameroon resulting in the cluster (Bafia, Bulu, Ewondo, French) on which MLM TLM pretraining and multi-task learning were done and compared to None and Pair baselines. Again, the historical approach outperformed all the baselines with notably gain of 22 BLEU on the pair French-Ewondo.
Our LM-similarity approach systematically performs better than the None and Pair baselines, sug- gesting that adding a language using this metric and do multi-task learning on the resulting cluster is a good heuristic even if one only cares about a specific pair of languages. It outperforms the baseline Random for 4 out of the 6 pairs of languages. According to this similarity, the closest language to
8

---

the pair (Bafia. Ewondo) is Bulu, thus reconstituting the historical cluster identified b.5..21 without any human supervision. Randomly choosing this particular language would have happened with 1 5% probability (20 = 22 2, there are 22 languages in our dataset and we exclude 2 languages, 20
Bafia and Ewondo, since we want to add another language to this pair).
The fact that the Random baseline performs worse than models derived from our methodology 10 times out of 12 suggests that the performance boost is not solely due to the multi-task nature of the model but mostly because of the good choices of languages to add. It is worth noticing that despite the MLM and TLM pre- rre-training, the Random baseline performs worse than the baseline Pair 4 times out of 6. This suggests that randomly adding languages may often hurt the performance compared to a simple model trained on the pair of interest.
Table Machine Translation BLEU scores. The rows correspond to the pairs of interest on which BLEU scores reported. The column None is baseline it represents the BLEU score of model trained on the pair without any MLM or TLM pre-training The column Pair is a baseline : represents the BLEU score of model trained column Random is also a baseline: it is the BLEU score of a 3 languages multi-task model where the language added was chosen purely at random. The column Historical refers the BLEU score of our 3 languages multi-task model where the language added was chosen using clusters identified in 5.2.1 The column LM describes the BLEU score of our 3 languages, multi-task model where the language added was chosen using the LM similarity ad described in 5.2.2 (Ba = Bafia, Bu = Bulu,
the pair with MLM and TLM pre-trame tr-iiinniin The
Ew Ewondo, Fr French)
Pair
None
Pair
Rand
Hist
LM
Pair
None
Pair
Hist
Ba-Bu
9.19
12.58
23.52
28.81
13.03
Fr-Bu
19.91
23.47
25.06
Bu-Ba
13.5
15.15
24.76
32.83
13.91
Bu-Fr
17.49
22.44
23.68
Ba-Ew
9.30
11.28
8.28
38.90
38.90
Fr-Ba
14.48
15.35
30.65
Ew-Ba 13.99
16.07
10.26
35.84
35.84
Ba-Fr
8.59
11.17
24.49
Bu-Ew 10.27
12.11
11.82
39.12
34.86
Fr-Ew
11.51
13.93
35.50
Ew-Bu 11.62
14.42
12.27
34.91
30.98
Ew-Fr
10.60
13.77
27.34
We believe that the difference between Random and Pair is not due to a disparity in data. First of all, they come from the same domain, the biblical domain. With regard to the size of the dataset:
For Bafia-Bulu, the three languages added are Du_n, Ewondo and Mofa for Random (R), Historical (H) and LMS (Language Model-based approach for linguistic Similarity) respectively this makes three clusters of almost the same size in terms of data, since only Bulu (present in all three clusters) among all these languages has 28895 sentences, the others have an average of 7939 (average of the sentences of the languages present only in the new testament).
Same thing for the Bulu-Ewondo pair with Dii, Bafia and Mofa added languages for R, H and LMS respectively.
The pair which is an exception to this rule is Bafia-Ewondo, for which the languages added are respectively Guidar, Bulu and Bulu for R, H and LMS H and LMS thus have an advantage over with Bulu, but the gain quite as good as for the other clusters.
This difference may come from the natural similarity in the Historical clusters. In Cameroon you will have difficulty, if you are not very good at speaking these languages, in differentiating between the languages of the same historical cluster (both alphabetically and orally). On the other hand, you can very easily, even if you are not familiar with these languages, make the difference between two languages of two different clusters.
We hypothesize that multi-task machine translation model (potentially with MLM and TLM pre- training) is capable of exploiting similarities between similar languages. A similar phenomenon can be observed with human beings: it is easier for someone that speaks French and Spanish learn Italian (since they are all Latin languages) rather than polish (that a Lechitic language).
Limitations

---

Out Of Distribution (OOD) generalization impossible, as well as practical use of our method, because of the use of data from the biblical domain
Due to the lack of specific tokenizers for African languages, we used english. The development spacialized tokenizers like the one proposed by Dossou and Emezue (202T)
is left perspective to this work.
The method used in this work to create the language clusters is only one agglomerative method among many others. We did not find it necessary to very elaborate methods since the cluster was of size (the historical cluster has three languages).
The authors do not foresee negative social impacts of this work specif- ically. Our theoretical work aims at showing the advantage of combining similar languages for machine translation, and the practical implications of this work (subject to the development of more appropriate dataset) are mainly aimed at solving the problems related to language barriers in Africa
Negative Social Impact:
Conclusion
In this work, we studied the impact of using multilingualism to improve machine translation of African languages. Our first contribution is the creation and public release of a new dataset (as well as the code used to create it) that increases the number African languages for which parallel machine translation corpora is available. This study focused on pairs derived from a cluster of 3 of the most spoken languages in Cameroon. Our second contribution is the proposal of a new metric to evaluate similarity between languages : this metric, based on language models, doesn't 't require word level parallelism (contrary to traditional like (Levenshtein) 1965) and (Swadesh) 1952) ) but only paragraph level parallelism. It also takes into account the lack translation of words present in (Swadesh) 1952) but not in African languages (like "snow"). Translation Language Modeling and multi-task learning were used to exploit multilingualism. Our third contribution is the set of empirical evidences that doing multi-task learning multiple similar languages generally improves the performances on individual pairs. The choice of the similarity is crucial since in our work, 4 times out of 6 a random choice leads to a decrease of performance. On the other hand, using our historical approach (derived from Eberhard et al. (2020)) systematically led to a boost of performance on this dataset and our LM similarity approach improved them 4 times out of 6.
References
Jade Z. Abbott and Laura Martinus. Towards neural machine translation for african languages. Computing Research Repository, arXiv:1811.0544 2018. UtL.
054677
Zeljko Agié and Ivan Vulić. JW300: wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204-3210, Florence, Italy, July 2019. Association for Computational Linguistics. doi:
100655//1//19131/0 URL http:: http://wwwwwwwww...///./ aclweb. howpboorg/aaae wwrgganthology/P// PPi-loggyyyP111331
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0479 2015.
pttttp::/aarxv.orrgte /ttp//ff///999 1409
0473v7
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (wmt19). In In Proceedings of the Fourth Conference Machine Translation, volume Volume 2: Shared Task Papers, Day 1, pages 1-61, Association for Computational Linguisties, August 2019. URL http://////ww acwweb. org/ http//anhooooo//N1-55550
Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. CoRR, abs//b00.0000000 1906.06442, 2019. URL http:/ttt//ai...orrrerre/ org/ abs/ wttp://gaaa/99.00000000 19421 06442
Christos Christodouloupoulos and Mark Steedman. A massively parallel corpus: the bible in 100 languages. Lang Resources Evaluation 49, 375-395 (2015) https://doi.or//10. 10079 9287-y, 2015.
10

---

Junyoung Chung. Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jacob Devlin, Ming-War Wei Chang, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019. URL https:::/aaww..orr/////////////s
04805
pdf
Bonaventure F. P. Dossou and Chris C. Emezue. Crowdsourced phrase-based tokenization for low-resourced neural machine translation: The case of fon language, 2021.
Eberhard, David M., Gary F. Simons, and Charles D. Fennig. Ethnologue: Languages of the world. SIL Internationnal, Dallas, Texas, 2020.
Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou. FFR v1.1: Fon-French neural
machine translation. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pages 83-87, Seattle, USA, July 2020. Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning, Journal of Machine Learning Research, 70: 1243-125222 2017.
Hendrick Groenewald and WIldrich Fourie. Introducing the autshumato integrated translation environment. In In Proceedings of the 13th Annual Conference of the EAMT, Barcelona, pages 190-196, 2009.
Lh Ha, Jan Niehues, and Alexander H. Waibel. Toward multilingual neural machine translation with universal encoder and decoder. CoRR, 0ess/6111.47799 2016. URL http: org/
Abs/1611.04799
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08411 2016.
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997. doi: 10.1162/neco. 1997.9.8. 1...7.9..7735......... 1735.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's multilingual neural machine translation system: Enabling zero-shot translation. CoRR, abs/161 1. 04558, 2016. URL ltttp://axxvv. www.:/xwivvvrrob////////aa/ /tt////ss../..../../.. 04558
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 'sg multilingual neural machine translation system: Enabling zero-shot translation. 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kuczmarski and M. Johnson. Gender-aware natural language translation. 2018.
Guillaume Lample and Alexis Conneau. Cross-l language model pretraining. (Tt::1901.0799)) 2019. https:ts:://///tt arxiv. org/ wwwp///...//../.
Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Dokl. Akad. Nauk SSSR, 1965.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 1rXiv:1907.11692, 2019. URL https://arxii.. org. atgg/7bb/1997.. 11692
David M. Eberhard, Gary F. Simons, and Charles D. Fennig. Languages of the world. 2019. URL
htttps:/www.eethnol.. wtwp:hhwwwwoo.....oeeeeeeooooooo rewraa/afriiiiaaaaaaaaa Africal
Laura Martinus and Jade Abbott. A focus on neural machine translation for african languages.
arXiv preprint arXiv:1906 000.:::990000666000 05685, 2019.

---

Masakhane. Machine translation for africa. arXiv:2020. 11529, 2020. URL pdf/2003. 11529, pdf
org/
Cindy A. McKellar. An english to xitsonga statistical machine translation system for the government domain. In In Proceedings the 2014 PRASA, RobMech and AfLaT International Joint Symposium, pages 229-233, 2014.
Daniel Van Niekerk. Exploring unsupervised word segmentation for machine translation in the south african context. In In Proceedings of the 2014 PRASA, RobMech and AfLaT International Joint Symposium, pages 202-206,, 2014.
Kishore Papineni, Salim Roukos. Todd Ward, and Wei jing Zhu. Bleu: a method for automatic evaluation of machine translation. pages 31 311-318, 2002.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. urXiv:1802.05365, 2018. URL
http://arxiv. wwwpg///////////aa//ooo//aaaeer/ /ww.///s//. 1802. 05365
Alec Radford, Karthik Narasimhman, Tim Salimans, and Πlya Sutskever. Improving lan-
guage understanding by generative pre-training. 2018. URL htttps://ss--ss-esssts
amazonaws. cot.://wwwwwccccaoooooommmmo wwwp///pppalllassseeeeeeeaaaaaaa wtt.:///aweeeeerrcooooooaaaaaaae untpgervienseeeiieedeee
language_ htn.aaggg_understtadnnnnnng httpsaaaainnnnpapppppppap pdf
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715-1725, 2015.
Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Erjavec, Dan Tufiş, and Dániel Varga. The JRC-Acquis: multilingual aligned parallel corpus with 20+ languages. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), Genoa, Italy, May 2006. European Language Resources Association (ELRA). URL pdf
htttp:/www..n Irec-conf ottp://pceeiiigg/eeeee/////////p
Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In In Advances in Neural Information Processing Systems, pages 3104-3112, 2014.
Morris Swadesh. sexico-statisticl dating of prehistoric ethniccontacts: With special reference to north american indians and eskimos. In Proceedings of the American Philosophical Society, volume 96, pages 452-463., 1952.
Wilson L. Taylor. Cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4): 415-433, 1953.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, and Aidan N Gomez. Attention is all you need, In In Advances in Neural Information Processing Systems, pages 5998-6008. 2017.
llaaa Wilken, Marissa Griesel, and Cindy McKellar. Developing and improving statistical machine translation system for english to setswana: A linguistically approach. In In 23rd Annual Symposium of the Puttern Recognition Association of South Africa, page 114, 2012.
Friedel Wolff and Gideon Kotze. Experiments with syllable- zulu-english machine translation. In Proceedings of the 2014 PRASA, RobMech and AfL International Joint Symposium, pages 217-222, 2014.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, and al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv: 1609.08 08144, 2016. URL http:/ /tatp///rxiv.. org/ abs/ 1609, 08144
12

---

Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. Controlling the voice of sentence in Japanese-to- torsss--tt-E neural machine translation. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 203-210, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.
Zhiling Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe Le. Generalized autoregressive pretraining for language understanding. arXiv:1906.082377 2019. URL https://arxiv..rrt wttp://www.rrgb///b/obb///// 08237
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] The details of the dataset are given in Section 5.1 the proposed metric for evaluating linguistic similarity is detailed in Section 5.2.2 and the
results obtained are presented in Table 2
(b) Did you describe the limitations of your work? [Yes] OOD and practical use impossible, use of an inappropriate tokenizer
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Our theo- retical work aims at showing the advantage of combining similar languages for machine translation, and the practical implications of this work (subject to the development of a more appropriate dataset) are mainly aimed solving the problems related to language barriers in Africa
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We did not have enough computational resources to run the experiments multiple times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] S.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The code used in this work is based on XLM the historical approach for languages aggregation is based on the work of Eberhard et al. (2020), and the data are from Youversion]
(b) Did you mention the license of the assets? [Yes] These assets are subject to their initial licenses (XLM, YouVersion)
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No] The data used was directly extracted from Youversion, under their Privacy Policy
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The data used directly extracted from Youversion, under their Privacy Policy)
sstpa://my.bible com/
https::/my. bible wwt.:/ccoppppaa...
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
A Appendix
Additional details regarding motivations In relation point 2 of the motivation section (this discrepancy between spoken languages across different regions of individual African countries make the spread of misinformation easier (especially in rural regions and especially during election periods)), we take an example relating to the election period. During this period, the government sends people explain to people rural areas what the vote is going to be used for and how they should vote. But it is common for them to do this in way that favours certain political parties instead of simply translating the law, for example. Let's imagine that the text says English
... You must therefore choose one and only one candidate between Candidate A and Candidate B." (... vous devez donc choisir un et un seul candidat entre candidat et le candidat B) and the translator says in French (to favour party A): "... vous devez donc choisir le candidat A et non le candidat B" (You must therefore choose candidate and not candidate B). On the other hand, the information that is broadcast on television radio is in French or English. If the villagers had access to tool allowing them to translate the legal texts (which are in French or English) into their mother tongue, this could be avoided.
Someone may ask "Why learning a different unrelated language should hurt other languages know?" In our case, language learning takes place simultaneously. and we believe that learning several languages that are not similar (in the sense the two similarities mentioned above) simultane- ously makes the learning task more difficuul. We also believe that the gain in performance at the level of the historical clusters comes from a natural similarity in the Historical clusters. In Cameroon you very good at speaking these languages, in differentiating between
will have difficulty, if you the languages of the same historical cluster (both alphabetically and orally). On the other hand, you very easily, even if you not familiar with these languages, make the difference between two languages two different clusters
14


Reviews:
[{"review_id": "qB9zWsnF5pP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper motivates clearly the need for research in machine translation of underresourced (and thus underresearched) African languages, and proposes ways to aid the training of MT systems using data from related languages. The main contribution is that two ways (and a random baseline) to select a langauge to add to the training of an MT model are evaluated (one based on linguists' clustering, one based on probabilities assigned through a language model of the target langauge) on pairs drawn from a set of three langauges, choosing new langauges from a set of 13.\n\n#### Strengths/what I loved:\n- The paper performs a very clearly motivated and well set-up experiment with appropriate analysis.\n- I loved to see a focus on datasets for underresourced langauges and the paper's repeated focus on whether or not previous work was reproducible and how this paper tries to be as reproducible as possible (even if that is not possible with Bible data for legal reasons).\n- I always like to see attention to detail and explanation as i the \"Due to the multi-task nature...\"-paragraph on page 7!\n- The observation that adding weakly related languages *hurt* was very interesting to see, thank you for including it!\n\n#### Criticism/weaknesses:\n- While the overall setup (choose a language to add using either linguistic knowledge or automatic metrics) is solid and interesting, the setup and contributions promised on pg. 2 make it sound especially like you are proposing a way to infer linguistic relations between languages (\"a method for aggregating languages based on [a number of features]\").\n- The related work section is rather unfocused: it might help to make clear at the beginning of each paragraph how all the work cited in that paragraph relates to the present paper (e.g., \"There are a number of papers on translation of African langauges, covering English to Setswana (Abbott & Martinus (2018); Transformers on the Autshumato dataset (Groenewald & Fourie (2009) reaching BLEU 33.53), English to Afrikaans...\" or something along these lines. The division between the first and the second paragraph isn't quite clear to me and I am not sure what the third paragraph is on: transfer in MT?\n- Not only training, but unfortunately also validation is performed only on Bible data and only on a cluster of 3 languages. The first is an issue because while I understand that Bibles are the easiest data to come by and that for training one takes what one can get, any results that stay within this Bible domain are unlikely to transfer well to real-world settings, making me doubt the practical use of the paper as-is. The latter exacerbates the issue, because essentially as a paper motivated by poor MT performance or nonexistence on many African languages it fails to convince me that it is indeed solving that task and not one of Bible translation, and as a paper that talks about linguistic features in general, the extremely narrow focus on a cluster of literally only 3 languages seems unnecessarily restrictive if we are already settling for Bible data. (Note that I think it is fair to say that these limitations were necessary because individual experiments are too costly to run more than this very barebones proof-of-concept, but that claim should be made and substantiated.)\n- Experiments are not tested for statistical significance or otherwise qualified through a sense of the variance that's inherent in these results. This is especially unfortunate for the \"Random\" baseline, which given the small set of candidates will produce rather unreliable estimates when only tested on 1 sample (please let me know if I misunderstood!)\n\n#### Questions:\n- Point 2 of the Motivation section claims that language barriers aid the spread of misinformation... that's not quite clear to me: shouldn't any barrier impede information flow and in fact easy machine translation help the spread? Unless it is correcting counter-information that you want to advocate for...\n- Page 5 says you used an English tokenizer for all languages. Just to clarify: this is tokenizing before applying BPE, right? Did you try to go without tokenization at all, i.e., using BPE as a standalone tokenizer?\n- What is the point of the last two paragraphs of section 5, i.e., talking about clustering? From my understanding you don't use that for your experiments, as there you only choose the 1 nearest neighbor---and if you did want to build larger clusters, the greedy approach sketched in here would be a needlessly restricted choice: there are a number of better agglomerative clustering algorithms out there!\n- The paper does cite the Masakhane project preprint from March 2020 (though with a broken arXiv identifier in the references!), but I would be very curious to see how it relates to the newer October 2020 preprint that contains a lot more content. (Of course, I know that this paper is far too recent to expect or even ask you to refer to it in the preprint! I would just personally be curious and think it might be a nice thing to add for a camera-ready.)\n\n\n#### Typos and other small things:\n- Probably due to my unfamiliarity, I wasn't 100% clear on what you meant by ``vernacular'' throughout, it might be nice to give a quick definition.\n- Some small grammar errors throughout, for example \"as follow\" instead of \"as follows\" in a few places.\n- Citations should be in parentheses when they are not constituents in a sentence and bare when they are (as in the beginning of the second paragraph on page 3). The citation of Lample and Conneau on page 2 somehow ended up with double parentheses...\n- I would have liked to see references for section 2, but I suppose that is not really the scope or point of the papers. I also guess that the third point implies that colonial languages are learned instead of vernaculars? That might get expanded a little, too.\n- The JW300 dataset should be properly cited (Agić and Vulić, 2019: https://www.aclweb.org/anthology/P19-1310/ ).\n- The description of MLM on page 4 says \"we keep them unchanged 10%\"---this sounds like this is a deviation you are introducing.\n- The entire first half of page 6 could be condensed into a much smaller and more concise paragraph, namely a sentence like the one that then starts with \"At a paragraph level\", or to more clearly separate motivation from actual definition (which is nice and simple)\n- I would separate the top and bottom half of Table 1 more clearly, or better yet, make them separate tables altogether, given how they don't even both have the same set of columns.\n- I fear the example given in the first paragraph of page 7 is a poor example: I see no reason a priori why learning a different unrelated language should *hurt* other languages I know. Then again, as a non-native speaker of English, my own native langauge certainly has taken the back seat in my head... either way it might be good to elaborate a little on this idea or just leave it out.\n\nLooking forward to discuss!", "extra_scores": {}, "extra_text": {}}, {"review_id": "oqnbZgzfteY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the problem of machine translation for African languages. The authors show that trilingual models outperform bilingual models for a set of languages spoken in Cameroon, and explore two strategies to find the best auxiliary language for a given translation pair: a manual one based on their linguistic relationship, and an automatic one based on language modelling.\n\nI agree with the authors that African languages have been underrepresented in NLP research in general and machine translation in particular. In that regard, I think that their motivation is truly admirable, and I would like to encourage them to keep working on this direction.\n\nUnfortunately, I think that the work itself does not meet the standards of the conference in its current form. While the focus on African languages is new and interesting, I find that the paper has little substance beyond that. According to the authors the paper makes 3 main contributions, which I feel are too narrow or otherwise questionable as discussed next:\n\n1) \"Our first contribution is the creation and public release of a new dataset.\" The dataset in question is a parallel version of the Bible. There have already been some efforts to extract parallel corpora from the Bible, which are not even discussed in the paper. For instance, Mayer & Cysouw (2014) [1] report covering 830 languages, which are likely to include the ones explored in the paper. Moreover, the authors extracted the corpus from an existing website with Bible versions in >1000 languages, so this was just a scraping effort. In connection to that, I am not sure if the authors considered potential copyright restrictions when releasing their corpus.\n\n2) \"Our second contribution is the proposal of a new metric to evaluate similarity between languages\". The proposed method measures the similarity between two languages, L1 and L2, by applying a language model trained on L1 to L2. This looks like a rather simplistic approach and, in the lack of any other baseline, it is hard to get a sense of how good it is. Also, I am not familiar with this topic, but there is certainly some work on automatically identifying similar languages. For instance, [2] shows that the degree of overlap in the BPE vocabulary of different languages is already a good indicator of the their linguistic proximity.\n\n3) \"Our third contribution is the set of empirical evidences that doing multi-task learning on multiple similar languages generally improves the performances on individual pairs.\" There were already in-depth studies showing that multilingual training is helpful for low-resource machine translation, see for instance [3]. Unfortunately, this is not even cited in the paper.\n\nIn connection to the last point, the authors limit themselves to trilingual models and the main focus of their work is on identifying the most suitable auxiliary language. An obvious baseline that is missing in the paper is to train a multilingual model on the combination of all languages.\n\nIn addition, the models used by the author are usually trained at a large scale, and might not work well with very small datasets like the Bible. Even a phrase-based statistical machine translation system might work better in this setup, and I overall miss better baselines.\n\n[1] http://www.lrec-conf.org/proceedings/lrec2014/pdf/220_Paper.pdf\n[2] https://aiide.org/ojs/index.php/AAAI/article/view/4677\n[3] https://arxiv.org/abs/1907.05019", "extra_scores": {}, "extra_text": {}}, {"review_id": "g7IV5s8uQAO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes three contributions: (1) a new multilingual parallel corpus that covers 1,477 languages, with text from the Bible -- for the region of interest in this paper (Cameroon) this means 22 languages instead of 18 with JW300; (2) a method for determining similarity of languages based on LM scores; (3) a series of experiments that evaluate the utility of adding a third language to a pair of interest based on either the similarity metric proposed by (2) or based on historical data, for use in a multilingual MT system. The authors show large improvements by adding a third language based on historical language similarity (always outperforms random) or their language-model-derived similarity (often outperforms random).\n\nWorking on MT for African languages is important, and I’d definitely like to see more of it. Unfortunately, I think this paper suffers substantially from insufficient replication details. Even if those details were present, I don’t think it is necessarily sufficiently novel for ICLR.\n\nRegarding replication, the paper relies on three technologies: language modeling for the language similarity calculation, CLM and MLM pretraining, and multilingual MT. Crucial details are missing for all three of these. For the language modeling and for CLM and MLM pretraining, it isn’t clear what the monolingual data source is. Without being specified, it’s implied that it’s just one side of the parallel corpora - but why would pretraining help in that setting if it’s not adding data? In general, the data should be discussed with more precision. It would be really useful to talk about sizes of datasets (if the different languages have differently sized datasets). \nFor multilingual MT, it isn’t clear what flavor of multilingual MT is being used: is it a tagged model? Where are the tags (source versus target versus embedding)? I understand that the authors have provided code, but the paper should stand alone, and I wasn’t able to easily answer these questions after spending 15 minutes with the code.\n\nRegarding novelty, I think the paper would need to flesh out either contributions (2) or (3) to really find a home at a venue like ICLR. (2) is an interesting idea, but right now we have one extrinsic test of the idea - I think a fully fleshed out version would need to compare to baselines and also provide intrinsic evaluations that compare to other “ground truth” definitions of similarity, such as the historical definition used in this paper. (3) would require looking into why we’re seeing such huge jumps just from adding another language. Are there disparities in dataset sizes that aren’t mentioned in the paper? If adding another language at most doubles the amount of parallel data, I don’t see why that should reflect a boost from 12.6 to 28.8 BLEU. There might be something really interesting going on here, but it would need to be explored much more carefully.\n\nSmaller points:\n\nWhen citing Eberhard, I would include “M.” with the first name, not the last name.\n\nThe proposed similarity metric is based on an equation normalizing a perplexity difference according to a base perplexity. But I don’t think this can be treated as a similarity, as lower is better in perplexity. So if I have language A that receives a perplexity of 10 according to LM0, and another language B that receives a perplexity of 3 (and thus much higher average probability per token), and LM0 assigns its own language a perplexity of 2, then we’ll get scores of (10-2)/2 = 4 for A and (3-2)/2 = 0.5 for B, making A look more similar. I think you probably intended to use average probability or average log probability here, or have the equation calculate a difference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NWyBOq57d6U", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers translation between African languages. Overall, I think this is a great effort, I think it's great that the authors are tackling this important problem.\n\nHowever, the field of multilingual machine translation is a very well-researched field, and it seems that the authors have developed their methodology largely independent of the literature in this field. In fact, there are already existing well-researched methods on many of the topics presented in this paper. To give just a few examples:\n\n* *Pre-training for low-resource translation:* Liu, Yinhan, et al. \"Multilingual denoising pre-training for neural machine translation.\" arXiv preprint arXiv:2001.08210 (2020).\n* *Leveraging linguistic similarities:* Lin, Yu-Hsiang, et al. \"Choosing transfer languages for cross-lingual learning.\" arXiv preprint arXiv:1905.12688 (2019).\n* *Translation between low-resource language pairs:* Chen, Yun, et al. \"A teacher-student framework for zero-resource neural machine translation.\" arXiv preprint arXiv:1705.00753 (2017).\n\nI would suggest that the authors read these papers, and maybe other papers that cite them. Also, perhaps read papers on the ACL Anthology (https://www.aclweb.org/anthology/) from prominent conferences such as ACL, EMNLP, NAACL that contain the keywords \"multilingual\" and \"translation\" to get a better idea of the state of the art in the field. There are lots of methods that people have developed, and I think that they could be effectively applied to the very important problem at hand here!", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new multilingual parallel Bible dataset for African languages, a new method for determining similarities between languages, and a collection of experiments to evaluate methods for choosing an additional language based on (a) similarity and (b) language history to include in a multilingual MT system. Results show that strategic inclusion of an additional language can substantially improve BLEU. Reviewers universally agree that progress on MT for African languages is a very important goal. However, reviewers pointed to several major concerns with the current draft: (1) lack of sufficient detail for replicating experiments, (2) missing analysis to interpret why experimental gains are so large, and (3) missing discussion and comparison with already existing methods in multilingual MT (e.g. multilingual training for low-resource languages). I agree with reviewers that the paper is not ready for acceptance in its current form, but encourage re-submission, possibly at an NLP conference. 

Decision: Reject