SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Evaluation scores on the recently introduced NTREX test set (depicted as <bleu> / <chrf>) for the MT models and language models described in Section 4.1 and Section 4.2 compared against unsupervised baselines [9]. Note that LM-8B is evaluated on a  $50\%$  split of the NTREX data and is not comparable to the MT-model evaluations.
Baziotis et al. [9]	MT-3B	MT-7.2B	MT-10.7B	LM-8B
0-shot	1-shot	5-shot	10-shot
xx2en	- / -	30.6 / 54.5	32.7 / 56.2	33.6 / 57.6	3.2 / 17.3	20.4 / 43.8	23.8 / 48.2	24.4 / 49.0
en2xx	- / -	16.5 / 39.6	17.6 / 41.9	17.9 / 41.9	0.8 / 7.3	11.7 / 31.2	12.6 / 32.4	12.3 / 32.3
Average	19.8 / -	23.5 / 47.0	25.1 / 49.0	25.7 / 49.7	2.0 / 12.3	16.0 / 37.4	18.1 / 40.2	18.3 / 40.6
parameter model is competitive with the significantly larger NLLB-54B model [46] on WMT. For the recent NTREX dataset, the only published results are small-scale results by Baziotis et al. [9].
In Table 5 we find that on the Flores-200 dataset, our model is within 3.8 chrf of the 54B parameter NLLB model, while on xxxx pairs the 10.7B models is behind by 6.5 chrf. This is likely due to a combination of factors, including using a significantly smaller model (5x smaller), domain differences [9, 8], and back-translated data [55]. Similarly, in Table 7, we find that the 10.7B parameter model is within 5.7 chrf of the scores reported by Bapna et al. [8]. Again, it is very difficult to compare their results to ours; their two largest advantages are 1) iterative back-translation, and 2) access to a much larger in-house text data. We note that the results we present are merely baselines to demonstrate the utility of MADLAD-400, and hope that future work builds upon these experiments by applying improved modeling techniques.
Across all evaluation datasets, we find that while results on few-shot translation using the 8B language model increase with an increasing number of demonstrations, these results are still significantly weaker than the results of models trained on supervised data. We present per-language pair results on all datasets in Appendix A.10.
Table 7: Evaluation scores on the GATONES test set used by Bapna et al. [8] (depicted as <bleu> / <chrf>) for the MT models and language models described in Section 4.1 and Section 4.2.
NTL (Bapna et al. [8])	MT-3B	MT-7.2B	MT-10.7B	LM-8B
1.6B	6.4B	0-shot	1-shot	5-shot	10-shot
xx2en	- / 37.2	- / 41.2	13.3 / 34.6	15.5 / 36.5	14.8 / 36.0	0.3 / 6.5	6.6 / 25.4	8.3 / 28.1	8.4 / 28.4
en2xx	- / 28.5	- / 33.1	4.5 / 23.9	5.5 / 26.4	5.4 / 26.2	0.2 / 4.2	1.7 / 10.5	1.7 / 9.9	1.8 / 9.4
Average	- / 32.9	- / 37.2	8.9 / 29.3	10.5 / 31.5	10.1 / 31.1	0.3 / 5.4	4.2 / 18.0	5.0 / 19.0	5.1 / 18.9
Extensive work has been done to mine general purpose datasets for multilingual machine translation and language modeling. Xue et al. [63] introduce mC4, a general web domain corpus on 101 languages to train mT5, a pretrained language model for downstream NLP tasks. Similarly, Conneau et al. [16] introduce CC-100, later extended to CC100-XL by Lin et al. [42]. The OSCAR corpus [2] is also a mined dataset that supports 166 languages and the ROOTS corpus is a compiled dataset that contains 46 natural languages. Glot500-C [28] covers 511 languages: however, it is not clear how many of these languages comprise solely of religious texts. Bapna et al. [8] create an internal dataset on  $1500+$  languages, while NLLBTeam et al. [46] mine a dataset from CommonCrawl and ParaCrawl [19]. Recently, Leong et al. [40] created a  $350+$  language dataset from children's books.
In addition, there have been efforts to get better represented corpora and models for languages often underrepresented in general multilingual corpora: Serengeti [3] introduces a dataset and associated model trained on 517 African languages and language varieties, while IndicTrans2 [23] introduces a machine translated model for the 22 scheduled languages in India.
While we used thorough self-audits to guide the creation of MADLAD-400, we note that most audits were conducted by non-speakers of the languages in MADLAD-400; as a result, many types of noise,
like machine-generated or disfluent content, could not be detected. Moreover, toxicity detectors, classifiers and filters that work reliably for all the 419 languages in MADLAD-400 do not exist, limiting the extent to which we can clean and document [18, 7] the dataset. It is possible that issues still remain, so we encourage users to report issues that will be listed on the project github page. This paucity extends to the availability of multilingual evaluation sets for these languages - we could only evaluate our models on 204 of the languages in MADLAD-400. Additionally, even though decoder-only models are often evaluated on NLP tasks that are not necessarily machine translation [27, 6, 5], we did not conduct such evaluations - most available benchmarks cover only 30-50 languages of which most are not tail languages (which forms the focus of MADLAD-400). We instead leave this to future work. Finally, during our self-audit we noted the skew of data on the long tail towards specific domains such as religious texts. We hope that these limitations motivate the creation of more language-specific corpora not captured by web crawls, and the development of language-specific data cleaning tools and practices.
Through MADLAD-400, we introduce a highly multilingual, general web-domain, document-level text dataset. We perform a self-audit of this dataset for quality on samples of all 498 languages, develop filters, and remove spurious datasets, for a total of 419 languages in the release. We carefully describe the dataset creation process, laying out the iterations of audits and improvements upon the preliminary dataset along with observations that guided our decisions. We hope that this encourages creators of large-scale pretraining datasets both to put in their due diligence for manually inspecting and dealing with data, and also to describe and publicize the process in a level of detail that is reproducible and insightful for downstream users. This increased visibility into the dataset creation cycle can in turn improve model development and enable responsible data use [53]. Using MADLAD-400, we train and release large machine translation and general NLP models and evaluate them thoroughly. We hope that this further motivates work towards language technologies that are more inclusive of the rich language diversity housed by humanity.
8 Ethics Statement
Innovation in NLP technologies in English has been accelerated by training large scale deep learning models [17, 11] on massive web corpora [13, 68, 52]. However, on the long tail of written languages in the world there is a lack of high quality general data sources [32] that impede the progress of NLP tools for other languages. We hope that making an audited and cleaned corpus such as MADLAD-400 available mitigates this issue. While we extensively cleaned MADLAD-400, the extent to which we can preprocess this data is limited by how not all languages have available tools for removing problematic content such as porn, toxic content, PII, copyrighted content or noise. We urge practitioners to carefully consider their target usecase before using MADLAD-400.
We would like to thank Wolfgang Macherey, Zoubin Ghahramani and Orevaoghene Ahia for their helpful comments on the draft. We would also like to thank Subramanian Venkateswaran for debugging the virama rendering issues, and Ali Dabirmoghaddam for his insight on data samples of various languages in MADLAD-400.
[1] StatMT. https://www.statmt.org/. Accessed: 2022-05-03.
[2] J. Abadji, P. O. Suarez, L. Romary, and B. Sagot. Towards a cleaner document-oriented multilingual crawled corpus. arXiv preprint arXiv:2201.06642, 2022.
[3] I. Adebara, A. Elmadany, M. Abdul-Mageed, and A. A. Inciarte. Serengeti: Massively multilingual language models for africa. arXiv preprint arXiv:2212.10785, 2022.
[4] Z. Agic and I. Vulic. Jw300: A wide-coverage parallel corpus for low-resource languages. Association for Computational Linguistics, 2019.