EL Moatez Billah Nagoudi commited on
Commit
34ff392
1 Parent(s): 2ebea58

update readme

Browse files
Files changed (1) hide show
  1. Readme.md +77 -1
Readme.md CHANGED
@@ -1 +1,77 @@
1
- Will add a readme here.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # IndT5: A Text-to-Text Transformer for 10 Indigenous Languages
2
+ <img src="IND_langs_large7.png" alt="drawing" width="45%" height="45%" align="right"/>
3
+ In this work, we introduce IndT5, the first Transformer language model for Indigenous languages. To train IndT5, we build IndCorpu, a new corpus for 10 Indigenous languages and Spanish. We also present the application of IndT5 to machine translation by investigating different approaches to translate between Spanish and the Indigenous languages as part of our contribution to theAmericasNLP 2021 Shared Task on OpenMachine Translation.
4
+
5
+ &nbsp;
6
+
7
+ # IndT5
8
+ We train an Indigenous language model adopting the unified and flexible
9
+ text-to-text transfer Transformer (T5) approach . T5 treats every
10
+ text-based language task as a “text-to-text" problem, taking text format
11
+ as input and producing new text format as output. T5 is essentially an
12
+ encoder-decoder Transformer , with the encoder and decoder similar in
13
+ configuration and size to a BERT<sub>Base</sub> but with some
14
+ architectural modifications. Modifications include applying a
15
+ normalization layer before a sub-block and adding a pre-norm (i.e.,
16
+ initial input to the sub-block output).
17
+
18
+ # IndCourpus
19
+ We build IndCorpus, a collection of 10 Indigeous languages and Spanish comprising 1.17GB of text, from both Wikipedia and the Bible.
20
+
21
+ ### Demographic information of 10 Indigenous languages
22
+ | **Language** | **Language Code** | **Main Location** | **Number of Speakers** |
23
+ |------------------|-------------------|-------------------|------------------------|
24
+ | Aymara | aym | Bolivia | 1,677,100 |
25
+ | Asháninka | cni | Peru | 35,200 |
26
+ | Bribri | bzd | Costa Rica | 7,000 |
27
+ | Guarani | gn | Paraguay | 6,652,790 |
28
+ | Hñähñu | oto | Mexico | 88,500 |
29
+ | Nahuatl | nah | Mexico | 410,000 |
30
+ | Quechua | quy | Peru | 7,384,920 |
31
+ | Rarámuri | tar | Mexico | 9,230 |
32
+ | Shipibo-Konibo | shp | Peru | 22,500 |
33
+ | Wixarika | hch | Mexico | 52,500 |
34
+
35
+ ### Data size and number of sentences in monolingual dataset (collected from Wikipedia and Bible)
36
+ | **Target Language** | **Wiki Size (MB)** | **Wiki #Sentences** | **Bible Size (MB)** | **Bible #Sentences**|
37
+ |-------------------|------------------|-------------------|------------------------|-|
38
+ |Hñähñu | - | - | 1.4 | 7.5K |
39
+ |Wixarika | - | - | 1.3 | 7.5K|
40
+ |Nahuatl | 5.8 | 61.1K | 1.5 | 7.5K|
41
+ |Guarani | 3.7 | 28.2K | 1.3 | 7.5K |
42
+ |Bribri | - | - | 1.5 | 7.5K |
43
+ |Rarámuri | - | - | 1.9 | 7.5K |
44
+ |Quechua | 5.9 | 97.3K | 4.9 | 31.1K |
45
+ |Aymara | 1.7 | 32.9K | 5 | 30.7K|
46
+ |Shipibo-Konibo | - | - | 1 | 7.9K |
47
+ |Asháninka | - | - | 1.4 | 7.8K |
48
+ |Spanish | 1.13K | 5M | - | - |
49
+ |Total | 1.15K | 5.22M | 19.8 | 125.3K|
50
+
51
+
52
+ # Parallel datasets for machine translation
53
+ The datasets are provided by AmericasNLP 2021 Shared Task on Open Machine Translation (https://github.com/AmericasNLP/americasnlp2021).
54
+ ### Number of sentences in parallel dataset (train, dev and test set)
55
+ | **Language Pair** | **Train** | **Dev** | **Test** |
56
+ |-------------------|------------------|-------------------|------------------------|
57
+ |es-aym | 6,531 | 996 | 1,003 |
58
+ |es-cni | 3,883 | 883 | 1,003 |
59
+ |es-bzd | 7,506 | 996 | 1,003 |
60
+ |es-gn | 26,032 | 995 | 1,003 |
61
+ |es-oto | 4,889 | 599 | 1,003 |
62
+ |es-nah | 16,145 | 672 | 1,003 |
63
+ |es-quy | 125,008 | 996 | 1,003 |
64
+ |es-tar | 14,720 | 995 | 1,003 |
65
+ |es-shp | 14,592 | 996 | 1,003 |
66
+ |es-hch | 8,966 | 994 | 1,003 |
67
+
68
+
69
+ # BibTex
70
+
71
+ ```@inproceedings{chen2021indt5,
72
+ title={IndT5: A Text-to-Text Transformer for 10 Indigenous Languages},
73
+ author={Chen, Wei-Rui and Abdul-Mageed, Muhammad and Cavusoglu, Hasan and others},
74
+ booktitle={Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas},
75
+ pages={265--271},
76
+ year={2021}
77
+ }