IndT5: A Text-to-Text Transformer for 10 Indigenous Languages
![drawing](https://huggingface.co/UBC-NLP/IndT5/raw/main/IND_langs_large7.png)
IndT5
We train an Indigenous language model adopting the unified and flexible text-to-text transfer Transformer (T5) approach . T5 treats every text-based language task as a “text-to-text" problem, taking text format as input and producing new text format as output. T5 is essentially an encoder-decoder Transformer , with the encoder and decoder similar in configuration and size to a BERTBase but with some architectural modifications. Modifications include applying a normalization layer before a sub-block and adding a pre-norm (i.e., initial input to the sub-block output).
IndCourpus
We build IndCorpus, a collection of 10 Indigeous languages and Spanish comprising 1.17GB of text, from both Wikipedia and the Bible.
Demographic information of 10 Indigenous languages
Language | Language Code | Main Location | Number of Speakers |
---|---|---|---|
Aymara | aym | Bolivia | 1,677,100 |
Asháninka | cni | Peru | 35,200 |
Bribri | bzd | Costa Rica | 7,000 |
Guarani | gn | Paraguay | 6,652,790 |
Hñähñu | oto | Mexico | 88,500 |
Nahuatl | nah | Mexico | 410,000 |
Quechua | quy | Peru | 7,384,920 |
Rarámuri | tar | Mexico | 9,230 |
Shipibo-Konibo | shp | Peru | 22,500 |
Wixarika | hch | Mexico | 52,500 |
Data size and number of sentences in monolingual dataset (collected from Wikipedia and Bible)
Target Language | Wiki Size (MB) | Wiki #Sentences | Bible Size (MB) | Bible #Sentences |
---|---|---|---|---|
Hñähñu | - | - | 1.4 | 7.5K |
Wixarika | - | - | 1.3 | 7.5K |
Nahuatl | 5.8 | 61.1K | 1.5 | 7.5K |
Guarani | 3.7 | 28.2K | 1.3 | 7.5K |
Bribri | - | - | 1.5 | 7.5K |
Rarámuri | - | - | 1.9 | 7.5K |
Quechua | 5.9 | 97.3K | 4.9 | 31.1K |
Aymara | 1.7 | 32.9K | 5 | 30.7K |
Shipibo-Konibo | - | - | 1 | 7.9K |
Asháninka | - | - | 1.4 | 7.8K |
Spanish | 1.13K | 5M | - | - |
Total | 1.15K | 5.22M | 19.8 | 125.3K |
Github
More details about our model can be found here: https://github.com/UBC-NLP/IndT5
BibTex
title={IndT5: A Text-to-Text Transformer for 10 Indigenous Languages},
author={Chen, Wei-Rui and Abdul-Mageed, Muhammad and Cavusoglu, Hasan and others},
booktitle={Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas},
pages={265--271},
year={2021}
}