SlowGuess/NAACL_2022_OCR huggingface dataset:
model has seen during pre-training, we follow past works (Madaan et al., 2020; Cahyawijaya et al., 2021; Lee et al., 2022) in selecting another closely-related language that is represented in the pretrained model. For convenience, we make use of Swahili (sw) as the target language when an African language is not represented since Swahili is represented in all the pre-trained models. The only exception is Nigerian-Pidgin, where we make use of French (fr) since it is closely related to English. When a language is represented in the pre-trained model like M2M-100 has seen Yorubá (yo), we make use of the correct language code.
To train AfriMT5 and ByT5, we start with MT5 and ByT5. We pre-train with the learning rate  $1e - 4$ , 10,000 warm up steps and a batch size of 2048 for one epoch. For mBART50, we pretrain with learning rate of  $5e - 5$  for 50,000 steps
Model Name	HuggingFace Model name	Remark
AfriMT5	masakhane/afri-mt5-base	mT5-base adaptation to 17 African languages, English, French and Arabic.
AfriByT5	masakhane/afri-byt5-base	ByT5-base adaptation to 17 African languages, English, French and Arabic.
AfriMBART	masakhane/afri-mbart50	mBART50 adaptation to 17 African languages, English, French and Arabic.
NEWS (MT5)	masakhane/mt5_{{src}}_{{tgt}}_{{news}}	MT5 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (AfriMT5)	masakhane/afrint5_{{src}}_{{tgt}}_{{news}}	AfriMT5 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (ByT5)	masakhane/byt5_{{src}}_{{tgt}}_{{news}}	ByT5 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (AfriByT5)	masakhane/afribyt5_{{src}}_{{tgt}}_{{news}}	AfriByT5 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (mBART50)	masakhane/mbart50_{{src}}_{{tgt}}_{{news}}	mBART50 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (AfriByT5)	masakhane/afrimbart_{{src}}_{{tgt}}_{{news}}	AfriMBART fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (M2M-100)	masakhane/m2m100_418M_{{src}}_{{tgt}}_{{news}}	M2M-100 fine-tuned on {src}-{tgt} direction using parallel NEWS corpus.
NEWS (M2M-100-EN)	masakhane/m2m100_418M-EN-NEWS	M2M-100 fine-tuned on NEWS data that are English-centric i.e en-{hau, ibo, lug, luo, pcm, swa, tsn, twi, yor, zul}
NEWS (M2M-100-FR)	masakhane/m2m100_418M-FR-NEWS	M2M-100 fine-tuned on NEWS data that are French-centric i.e fr-{bam, bbj, ewe, fon, mos, wol}.
REL	masakhane/m2m100_418M_{{src}}_{{tgt}}_{{rel}}	M2M-100 fine-tuned on {src}-{tgt} direction using parallel REL corpus.
REL+NEWS	masakhane/m2m100_418M_{{src}}_{{tgt}}_{{rel\_news}}	M2M-100 fine-tuned on {src}-{tgt} direction using parallel REL+NEWS corpus.
REL→NEWS	masakhane/m2m100_418M_{{src}}_{{tgt}}_{{rel\_ft}}	M2M-100 fine-tuned on {src}-{tgt} direction using parallel REL corpus and additional fine-tuning on NEWS
REL+NEWS→NEWS	masakhane/m2m100_418M_{{src}}_{{tgt}}_{{rel\_news\_ft}}	M2M-100 fine-tuned on {src}-{tgt} direction using parallel REL+NEWS and additional fine-tuning on NEWS
using Fairseq (Ott et al., 2019) without modifying the mBART50 vocabulary. Table 11 has the names of all the models that are publicly available on HuggingFace Model Hub  $^{13}$ . In total, we have 357 models from  $22 \times 16$  bilingual models, two English/French-centric models, and three adapted models to African languages (i.e AfriMT5, AfriByT5, and AfriMBART).
E BLEU vs spBLEU
Table 12 and Table 13 compares BLEU and spBLEU metric for the domain transfer experiments. We observe that spBLEU gives higher scores than BLEU especially in the direction of en/fr-xx, which shows that it may be better for evaluating African languages. Although, further analysis and human evaluation are still needed to show that spBLEU is generally better. On the other hand, in the xx-en/fr, there is no much difference in the scores between BLEU and spBLEU.
The following examples from the Fon-to-French translations of the test set illustrate the advantage of multilingual modeling and its limitations:
- Source (fon): Louis Guy Alimanyidokpo kpódlissa Etchlekoun kpo o, sin azan mokpán dye o, ye do wuve sè wε tawun do agbaza mε, có ye ka tuun fi é azon nε leε gosin e oă.
- Reference (fr): Les faits Louis Guy Alimagnidokpo et Issa Etchlekoun se plaaignent
Table 11: Model names on HuggingFace Model Hub. For bilingual models, supply the correct src or tgt language. English/French make use of a 2-letter language code i.e en or fr, while all the African languages make us of 3-letter language codes e.g yor.
Evaluation Domain	Tuned on NEWS	hau	ibo	lug	luo	swa	wol	yor	zul
en/fr-xx
FLORES	✘	2.4	2.0	0.9	-	19.6	0.4	1.0	1.9
FLORES	✓	2.9	12.3	4.9	8.8	22.5	4.2	5.1	8.4
REL	✘	2.5	1.8	0.0	-	14.6	0.0	1.4	2.1
REL	✓	6.7	9.4	1.1	2.4	17.4	2.7	8.2	8.3
NEWS	✘	0.4	2.4	1.8	-	20.1	1.3	2.1	5.6
NEWS	✓	14.4	20.3	13.0	10.8	27.0	11.1	12.8	16.5
xx-en/fr
FLORES	✘	6.6	6.0	2.6	-	26.2	2.1	2.7	10.5
FLORES	✓	5.4	11.8	6.9	10.3	25.4	6.6	7.9	18.1
REL	✘	9.7	5.9	0.5	-	22.3	0.3	1.8	7.8
REL	✓	7.7	10.7	1.8	2.6	20.5	1.7	8.8	12.9
NEWS	✘	2.2	6.4	4.8	-	25.2	0.8	3.0	13.8
NEWS	✓	17.2	18.5	19.4	12.8	29.9	9.5	16.0	36.6
Table 12: BLEU on Wikipedia domain (FLORES), REL, and NEWS for M2M-100 before  $(X)$  and after  $(\checkmark)$  fine-tuning on NEWS.
depuis quelques jours de multiples douleurs,  
ignorant l'origine réelle de leurs maux.
- Bilingual Transformer (REL+NEWS, fon→fr): on ne peut pas avoir une trentaine d'années ni un jeune homme ni un jeune homme d'âge pour un jeune homme qui soit 12 ans.
- M2M-100 (REL+NEWS→NEWS, fon→fr): Louis Guy Alimanyion et Issa Etchlekoun ont depuis plusieurs jours souffert d'une maladie grave malgré les conséquences de cette maladie qu'ils ne connaissent pas.
M2M-100 (REL+NEWS  $\rightarrow$  NEWS, fr  $\rightarrow$  fon): Sin azan yoyweywe de dye qokpoo we nuekan Louis Guy Alimagnidokpo kpodó Issa Etchekén kpan ddo xo do wuve gewe, ye ka tuun nu e wu wve yeton de 3a.
The translation of the bilingual Transformer model
<sup>13</sup>https://huggingface.co/masakhane
Evaluation Domain	Tuned on NEWS	hau	ibo	lug	luo	swa	wol	yor	zul
en/fr-xx
FLORES	X	2.6	2.8	0.8	-	20.9	0.6	1.5	3.3
FLORES	✓	4.0	19.9	7.6	13.7	27.1	8.2	13.4	19.2
REL	X	1.2	1.0	0.0	-	11.0	0.0	0.4	1.6
REL	✓	3.7	10.3	3.3	5.4	14.6	6.7	10.6	13.0
NEWS	X	0.6	4.1	2.3	-	21.4	1.2	2.4	5.6
NEWS	✓	20.2	31.6	22.6	16.4	31.4	19.9	25.5	27.6
xx-en/fr
FLORES	X	8.0	7.2	3.7	-	26.9	3.0	3.8	11.9
FLORES	✓	16.3	12.0	7.7	11.8	25.8	7.5	9.3	19.2
REL	X	6.4	3.7	0.5	-	15.4	0.4	0.9	8.5
REL	✓	3.8	6.0	1.7	2.5	13.9	1.7	5.7	12.5
NEWS	X	2.6	9.1	7.2	-	27.8	1.0	3.9	15.7
NEWS	✓	17.6	22.8	24.4	15.8	32.0	12.3	17.5	39.0
Table 13: spBLEU on Wikipedia domain (FLORES), REL, and NEWS for M2M-100 before  $(X)$  and after  $(\checkmark)$  fine-tuning on NEWS.
is very poor and far from the Fon source, highlighting how poorly the model generalized from the few thousand training sentences. The M2M-100 model gives a more meaningful and adequate translation. M2M-100 makes a surprising but beautiful move, switching se plaignent depuis quelques jours de multiples douleurs (sín azán mokpán qye o, ye do wüvε sè wε tawun dò agbaza mε) to ont depuis plusieurs jours souffert d'une maladie grave. The BLEU score here might be low but the meaning is conserved and even more detailed than the French reference. In fact, in this source context, wüvε means souffrir; souffrance (suffer, suffering): the French reference made use of se plaignent (complaining) which makes less sense than souffert used in the M2M-100 prediction. M2M-100 also learned the style of the sentence: co ye ká tuun fi é azon nε lεv gosin (but they do know the origin of their sufferings) é oă (NOT) - this last part is crucial for the meaning of the entire sentence. Given the structural and morphological differences between Fon and French, we expected it to be more complicated to predict. However, this translation is structurally wrong even though any French native speaker would understand the conveyed message quickly and easily. In the M2M-100 translation, the word malgre is at the wrong place, corrupting syntax and logic of the second clause. A perfect translation (in the idea to be expressed) would be: "Louis Guy Alimanyion et Issa Etchlekoun ont depuis plusieurs jours souffert d'une maladie grave malgre (dont) ils ne connaissent pas les eonséquences (causes/raisons) de cette maladie qu'ils ne connaissent pas."
In the opposite translation direction,  $\text{fr} \rightarrow \text{fon}$ , M2M-100 (REL+NEWS→NEWS) still preserved some sense of logical reasoning and predicted the last part right ye ká tuun nu è wú wúvé yeton (they
do know why they are suffering) dé 3 à (NOT). However, the model had some limitations: the names which are part of the translation are not spelled correctly. Some expressions are incomplete: For instance sín azán + number means since xxx days but yεywε is not a number, and do not have any meaning in this context.
G Limitations and Risks
Despite the promising results, our work has the following limitations:
1. Translation quality: Even the best model scores low BLEU on some of the reported languages (bbj, mos, zul), in particular when translating into them.
2. Evaluation: Our evaluation is focused on BLEU. We report ChrF results as well, but without a deeper human evaluation, we cannot make claims about the absolute quality of the translations. Manual inspections of translations like the example discussed in Section F gave us the impression that translations are surprisingly fluent and make good use of language-specific expressions when translating into English or French, but that errors in grammar and logic can be easily overlooked. Automatic reference-based metrics like BLEU and ChrF might not be able to capture the semantic relatedness to the reference sufficiently, as well potentially being tricked by word matches in incoherent phrases.
3. Language bias: We have shown that even when not included in pre-training, and without large out-of-domain data, significant gains in translation quality can be achieved. However, language-specific biases, in terms of resourcedness, morphology, standardization, inclusion in pre-trained models and available corpora, or relatedness to other languages, still affect the relative quality of translations, and require more efforts to be overcome.
4. Domain limitations: While we showed a rapid adaptation to the news domain and the auxiliary benefit of the religious domain, our study also revealed how automatically estimated translation quality drops when the test domain is narrow. Therefore, future work should aim to expand the study to multiple test domains and develop systematic methods
for distilling knowledge from multiple narrow domains.
5. Language coverage: Africa has thousands of other languages that are not covered in our study but deserve the same attention. We hope that our work is encouraging enough to inspire native speakers of those languages not covered here to collect translations, run our code, and report their findings to the NLP research community, so that we can make joint progress in developing language technology for more people.
We believe that our translation models carry similar risks of causing harm by inaccurate and biased translations as the underlying large pre-trained models. M2M-100 is trained on large collections of texts crawled from the web, and the quality for most of the languages studied here is questionable (Kreutzer et al., 2021). Our fine-tuning successes show that some obvious biases can be overcome when the quality of the fine-tuning set is controlled (see the examples in Section 6.3), but we cannot guarantee that biases prevailing in the pre-training corpus or more subtle biases will not occur with other inputs. Together with a careful human evaluation, this should be the main concern for future work on the produced models. The methodology of rapid fine-tuning might also be misused to tune the models towards harmful content or purposes that harm the speakers of the languages presented here.