Lanfrica Talks #16 | AfroLID: A Neural Language Identification Tool for African Languages
Channel: Lanfrica
Date: 22/03/2023

If you liked this talk, check out more talks at Lanfrica Talks: https://lanfrica.com/blog/lanfrica-talks

Listen to this episode in our podcast: https://podcasters.spotify.com/pod/show/lanfrica-talks-podcast/episodes/Lanfrica-Talks-16--AfroLID-A-Neural-Language-Identification-Tool-for-African-Languages-e20srro

Learn more about AfroLID
- https://lanfrica.com/record/afrolid-a-neural-language-identification-tool-for-african-languages
- https://demos.dlnlp.ai/afrolid/
- https://github.com/UBC-NLP/afrolid
Abstract
Language identification (LID) is a crucial precursor for NLP, especially for mining web data. Problematically, most of the world’s 7000+ languages today are not covered by LID technologies. We address this pressing issue for Africa by introducing AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. When evaluated on our blind Test set, AfroLID achieves 95.89 F_1-score. We also compare AfroLID to five existing LID tools that each cover a small number of African languages, finding it to outperform them on most languages. We further show the utility of AfroLID in the wild by testing it on the acutely under-served Twitter domain. Finally, we offer a number of controlled case studies and perform a linguistically-motivated error analysis that allow us to both showcase AfroLID’s powerful capabilities and limitations.