Crowd-sourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The case of Fon Language
Authors: Bonaventure F. P. Dossou
Chris Chinenye Emezue
Venue:  (2021)
Keywords: nmt
nlp
neural machine translation
natural language processing
deep learning
machine learning
machine translation
mt

Abstract: Building effective neural machine translation (NMT) models for very low-resourced and morphologically rich African indigenous languages is an open challenge. Besides the issue of finding available resources for them, a lot of work is put into preprocessing and tokenization. Recent studies have shown that standard tokenization methods do not always adequately deal with the grammatical, diacritical, and tonal properties of some African languages. That, coupled with the extremely low availability of training samples, hinders the production of reliable NMT models. In this paper, using Fon language as a case study, we revisit standard tokenization methods and introduce Word-Expressions-Based (WEB) tokenization, a human-involved super-words tokenization strategy to create a better representative vocabulary for training.

Under review conference paper at ICLR 2021
CROWDSOURCED PHRASE-BASED TOKENIZATION FOR LOW-RESOURCED NEURAL MACHINE TRANSLATION: THE CASE OF FON LANGUAGE
Anonymous authors Paper under double-blind review
ABSTRACT
Building effective neural machine translation (NMT) models for very low-resoureed and morphologically rich African indigenous languages is open challenge. Besides the issue of finding available resources for them, a lot of work is put into preprocessing and tokenization. Recent studies have shown that standard tokenization methods do not always adequately deal with the grammatical, diacritical, and tonal properties of some African languages. That, coupled with the extremely low availability of training samples, hinders the production of reliable NMT models. In this paper, using Fon language as case study, we revisit standard tokenization methods and introduce Wordd-xpreesiooss Baasedds--BBedd (WEB) tokenization, numan-involvvd super-words tokenization strategy to create better representative vocabulary for training. Furthermore, we compare our tokenization strategy to others the Fon-French and French- Fon translation tasks.
INTRODUCTION
We would like to start by sharing with you this Fon sentence: metà metà zinwó hen aligbo me
How would you tokenize this? What happens we implement the standard method of splitting the sentence into its word elements (either using the space delimiter or using subword units)?
metà metà zinwó hen aligbo
metà
metà
we
zinwó
hen
wa
aligbo
me
Well we did that and discovered that translation (to French) model, trained on sentences split this way, gave literal translation of <chaque singe est entré dans vie avec sa tête, son destin
(English: each monkey entered the stage of life with its head, its destiny)> for the above Fon sentence. But we are not talking about monkey here @ It is metaphor and so some of the words should be taken collectively as phrases. Using a phrase-based tokenizer, we got the following:
metà metà we zinwó hen aligbo me
metà
metà we
zinwó
hen wa aligbo me

---

Under review conference paper at ICLR 2021
A native speaker looking at some of these grouped phrases will quickly point out that some of the grouped phrases are wrong. Probably the phrase-base based model could not effectively learn the phrases due to the low data it was trained on? Also, we got a translation of æsinge chaque vient monde dans vie avec tête et destin (English: monkey each comes into world in life with head and fate)s However, this translation is still not correct. The expression actually means <Every human being born with chances> Another interpretation would be that we must be open to changes, and constantly learning to take advantages of each situation in life @
One illustrative example, which we encourage the reader to try, is to go to Google Translate and try translating <it costs an arm and leg to your language (native language language you understand). For the 20 languages we tried, all the translation results were wrong: literal and not fully conveying the true (some would say phrasal) expression or meaning. The expression <it costs an arm and legs, just means <it is expensive>. Now imagine a language with sentence structure largely made up of such expressions that is Fon @
Tokenization is generally viewed as a solved problem. Yet, in practice, we often encounter difficulties in using standard tokenizers for NMT tasks, shown above with Fon. This may be because of special tokenization needs for particular domains (like medicine (He & Kayaalp 2006 Cruz Díaz & Maña López 20155). or languages. Fon, one the five classes of the Gbe language clusters (Aja, Ewe, Fon, Gen, and Phla-Phera according to (010), is spoken by approximately 1.7 million people located in southwestern Nigeria, Benin, Togo, and southeastern Ghana. There exists approximately 53 different dialects of Fon spoken throughout Benin. Fon has complex grammar and syntax, is very tonal and diacritics are highly influential (Dossou& & Emezue 2020).. Despite being spoken by 1.7 million speakers, Joshi al. (2020) have categorized Fon <left behind> or kunderstudied> in NLP. This poses challenge when using standard tokenization methods.
Given that most Fon sentences (and by extension most African languages) are like the sentence example given above (or the combination of such expressions), there is need to re-visit tokenization of such languages. In this paper, using Fon in our experiment, we examine standard tokenization methods, and introduce the Word-Expred Bordd-xxressions-De Based (WEB) tokenization. Furthermore, we test our tokenization strategy on the Fon- French and French- Fon translation tasks. Our main contributions the dataset, our analysis and the proposal of WEB for extremely low-resourced African languages (ALRLs). The dataset, models and codes will be open-sourced on Github page.
2 BACKGROUND AND MOTIVATION
Modern NMT models usually require large amount of parallel data in order effectively learn the representa- tions of morphologically rich source and target languages. While proposed solutions, such as transfer-learming
from a high-resource language (HRL) to the low-resource language (LRL) (Gu et al. 2018 Renduchintala
et al.
2018, Karakanta et al. 2018). and using monolingual data (Sennrich et al.) 201 6a
Zhang & Zong
2016 Burlot & Yvon 2018; Hoang al. 2018). have proved effective, they are still not able to produce better translation results for most ALRLs. Standard tokenization methods, like Subword Units (SU) (Sennrich jet al. 2015), inspired by the byte-pair-encoding
(BPE) (Gage 1994), have greatly improved current NMT systems. However, studies have shown that BPE does not always boost performance of NMT systems for analytical LinguagesAAbbou
& Martinus 2018). Ngo et al. (2019) show that when morphological differences exist between source and target languages, SU does significantly improve results. Therefore, there is a great need to revisit NMT with a focus on low-resourced, morphologically complex languages like Fon. This may involve taking look at how to adapt standard NMT strategies to these languages.
2

---

Under review conference paper at ICLR 2021
3 TOKENIZATION STRATEGIES AND THEIR CHALLENGES FOR FON
this section, we briefly discuss the standard tokenization strategies employed in NMT, as well as challenges faced while applying them to Fon.
Word-Based tokenization (WB) consists splitting sentences into words, according to delimiter. We'll show the limits of this method using this Fon expression: eun d'o ganjis sun> on its own is an interjection, to express an emotion of surprise or astonishment. But <un dow already means "I am", "I am at", or "I have", depending on the context in which it used. The whole expression, sun do ganji could mean am fine" or am okay".
Phrase-Baas Based tokenization (PhB) encodes phrases (group of words) atomic units, instead of words. As a result, models trained on PhB have the ability to learn and interpret language-specificc phrases (noun, verbal and prepositional phrases). making it better than WB for Fon language. However, due to the low- resourcedness of the language and the randomness of PhB alignments, some extracted pairs are not always contextually faultless. For example, the computer alignment gave respectively [zen, une (a, an, one)] and
[azon, la (the)] instead of [zen, une marmite (a pot) and [azon, la maladie (the disease)]
Encoding with SU has made great headway in NMT, especially due to its ability to effectively encode rare out-of-vocabulary words (Sennrich al. 2016b). Machácek et al. (2018), in analyzing the word segmentation for NMT, reported that the common property of BPE and SU relies on the distribution of character sequences, but disregards any morphological properties of the languages in question. Apart from rule-based tokenization, there are machine learning approaches to tokenization require a substantial amount of training samples (both original and tokenized versions of the same texts) (Riley 1989 Mikheev 22000; Jurish & Würzner, 2013). To the best of our knowledge, there is no known language-specific tokenization proposed for ALRLs, although there have been a number of works on adapting NMT specifically to them (like (Orife et al.
well, which unfortunately
2020 van Biljon et al. 2020
Vaswani et al. 2017). to mention
but a few).
4 WORD-EXPRESSION--ASS -ASEDD TOKENIZATION (WEB)
WEB involves aligning and extracting meaningful expressions based on linguistic components of Fon (phonemes, morphemes, lexemes, syntax, and context). This requires the assistance of Fon-French native speakers. Some examples of good alignments are
noncé
maman (mum)
kud'o jighézän
joyeux anniversaire (Happy Birthday)
noncé vivr
maman chérie (dear mum)
do jid'de do wutu cé nonvi cé
It is important note that WEB not human-in-the-te process, because it doesn't require human intervention to run. The human intervention occurs while cleaning and preprocessing the dataset. Although not perfect yet, deseribe our algorithm recursive search algorithm, which looks and finds the most
3

---

Under review conference paper at ICLR 2021
optimal combination of words and expressions which will produce better translation for source sentence. The following algorithm was designed to encode and decode sentences using the established vocabularies:
1. Run through the vocabulary and output a list of all possible word combinations of words and expressions appearing in the sentence S.
2. Important principle in Fon: higher word orders = more precise and meaningful expressions. Using this principle. for each element (word or expression), E L.
(a) Check if there exists a higher word order, v E I,, such that w & v.
(b) Irfalis true, discard w, else keep w.
3. The output is list L of optimal expressions that from the initial L, making up the initial sentence S.
4. Add <start> and <end> taggers respectively at the beginning and the end of every element á (word or expression)
E L.
5. Encode every vi (word or expression) E L
We argue that WEB scales well because it does not require any linguistic annotations but knowledge and intuitions from bilinguals, meaning, we can crowdsource those phrases. We want to state clearly, in order avoid any confusion, that WEB is another version of PhB, involving human evaluation. For our study, it took a group of people, all bilinguals speaking Fon and French, and 30 days to align and extract meaningful sentences manually. No preliminary trainings have been done with the annotators, given the fact that they are majority linguists and natives of the Fon language. This made the step of sentences splitting into expressions, more natural, reliable and faster.
5 THE FON-FRENCH DATASET: DATA COLLECTION, CLEANING AND EXPANSION PROCESSES
As our goal is to create reliable translation system to be used by the modern Fon-speaking seessippking community, we set out to gather more data on daily conversations domain for this study. Thanks to many collaborations with Fon-French bilinguals, journalists and linguists, we gathered daily citations, proverbs and sentences with their French translations. After the collection's 's stage, we obtained a dataset of 8074 pairs of Fon-French sentences.
The cleaning process, which involved the Fon-French bilinguals, mainly consisted of analyzing the contextual meanings of the Fon sentences, and checking the quality of the French translations. In many cases, where the French translations were really bad, we made sigmificant corrections.
Another major observation was the presence of many long and complex sentences. That's where the idea of expanding the dataset came from: we proceeded to split, when possible, Fon sentences into short, independent, and meaningful expressions (expression of 1-6 words), and accordingly add their respective French translations. At the end of these processes, we obtained our final dataset of 25,383 pairs of Fon-French sentences. The experiments, described in this paper, were conducted using the final dataset.
We strongly believe that involving the Fon-French bilinguals into the cleaning process, greatly improved the quality of the dataset. In fact, many initial translation errors were disregarded by standard, rule-based tokenization (like WB, PhB and SU) and cleaning techniques However, with the help of the intuitive or natural language knowledge of the Fon-French bilinguals, bunch of those errors have been fixed. This highlights, the importance of having native speakers of ALRLs to clean and review the dataset, during the initial stages of its compilation.
'ssing Python Regex and String packages ing library (htps://www.nllk.org))
together with NLTK preprocess-
4

---

Under review conference paper at ICLR 2021
6 METHODOLOGY, RESULTS AND CONCLUSION
In this section, we describe the implementation of WB, PhB, SU, WEB and we compare the results of our NMT model trained on them for our analysis.
6.1 CREATION OF VOCABULARIES FOR WB, PHB, SU AND WEB
For WB, we split the sentences according the standard 'space' delimiter, using the TensorFlow- text tokenizer getting vocabulary of 7,845 and 8,756 Fon and French tokens (words) respectively.
For PhB, we use the IBMI model from wiikktransltte... api module align and extract all possible pairs of sentences. Our main observation was that, some pairs generated were either not meaningful or not maching, but we didn't try rearrange them in order to see how well the generated pairs, without human intervention, would affect the translation quality. In so doing, we got vocabulary of 10,576 and 11,724 Fon and French tokens respectively (word and expressions).
To implement SU we used TensorFlow's SubwordTextEncodee and built a vocabulary of 7,952 and 8, 16 Fon and French tokens (words and subwords) respectively.
To implement WEB, we considered unique expressions as atomic units. Using the steps highlighted for WEB section 4 we encoded those atomic units and obtained a vocabulary of 18,759 and 19,785 Fon and French tokens (word and expressions) used for the model training.
6.2 DATASET SPLITTING, MODEL'S S ARCHITECTURE AND TRAINING.
From the dataset, we carefully selected 155 long and complex sentences i.c. sentences made of 5 or more expressions, as test data; sentences that we believe, would test the model's ability to correctly translate higher word order expressions in Fon. 10% of the training data, was set aside for validation.
For training, we used an encoder-decoder- based architecture (Sutskever et al., (014), made up of 128- dimensional gated rectified units (GRUs) recurrent layers(Cho et al. [2014). with a word embedding layer of dimension 256 and 10-dimensional attention model (Bahdanau al.] 2015).
We trained with batch size of 100, learning rate of 0.001 and 500 epochs, using validation loss to track model performance. The training took all the 500 epochs, with the loss reducing from one epoch another. We would like to emphasize that up only at 500 epochs, with the given hyperparameters, we obtained significant and meaningful translations.
All training processes took 14 days on 16GB Tesla K80 GPU. We evaluated our NMT models performances using BLEU (Papineni et al. 2002). METEOR (Banerjee & Lavie 2005). CharacTER (TER) (Wang et al. 2016). and GLEU (Wu et al. 2016) metrics.
6.3 RESULTS AND CONCLUSION:
Table and Table 2 show that our baseline model performs better with PhB, and best with WEB, in terms of metric and translation quality. is important to note that while BLEU scores of PhB and WEB, reduced on the Fr-+Fon task, BLEU scores of WB and SU improved on it. We speculate that this might be because WB and SU enhanced the model's understanding of French expressions over Fon, confirming the findings of (Abbott & Martinus 2018) Ngo et al. (2019). This corroborates our argument that in order to help NMT
hhtps:/wwww.eessoffoow.orr
hhtp://www.eensollll.o
5

---

Under review conference paper at ICLR 2021
Translation
Tokenization
BLEU + METEOR 1
TER + GLEU1
Fon - Fr
WB
6.8
12.2
86.2
9
Fon + Fr
SU
7.6
13.6
87.4
10
Fon + Fr
PhB
38.9
53.7
43.9
42
Fon + Fr
WEB
66.6
77.77
24.2
67
Fr Fon
WB
15.65
8
Fr + Fon
SU
25.68
9
Fr Fon
PhB
38.74
25.74
Fr Fon WEB
49.37
43
Table
Experimental results of our model trained on WB, SU, PhB and WEB.
systems translate ALRLs better, paramount create adequate tokenization processes that can better represent and encode their structure and morphology.
This is pilot project and there is headroom be explored with improving WEB, We are also working on combining WEB with SU, to get the best of both worlds. To promote research and reproducibility in this direction, the dataset and model will be made publicly available on Github after the review. Simultaneously, we are working on releasing platforms for the translation service to be used. We believe that it would be good way to gather more data and keep constantly improving the model's 's performance.
6

---

Under review conference paper at ICLR 2021
Sentences: Fon French and English Translations
Source
a do jid'ide do wutu cé à nonvi cé a do jid'ide do wutu cé à nanvi cé
Tokenization output
Target
est-ce que tu me fais confiance mon frère? (my brother, do you trust in me?)
WB
confiance mon oncle (trust my uncle)
PhB
tu me fais confiance? (do you trust me?)
SU
aies la foi (have faith)
WEB
mon frère, est-ce que tu me fais confiance? (my brother do you trust in me?)
Source
d'é man yon nùmi à, bo yi doto hwé
Tokenization output
d'é man yon nùmi à na be yi doto hwé
Target
j'irai à l'hopitål que je ne me sens pas bien(Sine) am not feeling well, I will go to hospital)
WB
être malade et se rendre à l'hopitål (to be sick and to go to hospi- tal)
PhB
je me rends à l'hopitål parce que je ne me sens pas bien ]col- oreng(I am going to hospital because I am not feeling well)
SU
rends l'hopitål, je suis malade (Go to hospital, am sick)
WEB
je me rendrai à l'hopital vu que je ne me sens pas bien (I will go to hospital since I am not feeling well)
Table 2: Model translations with WB, PhB, SU and WEB
7

---

Under review conference paper at ICLR 2021
REFERENCES
Jade Z. Abbott and Laura Martinus. Towards neural machine translation for african languages. CoRR, 2018.
05467
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http::tttttiiiiiee wwwp:rgg./...oooo... 140900//33
Satanjeev Banerjee and Alon Lavie. METHOR: An automatic metric for MT evaluation with improved correla- tion with human judgments. Proceedings the the ACL Workshop on Intrinsic and Extrinsic Evaluation Mea- sures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Associ- ation for Computational Linguistics. URL htttt:::: www. ac web. orq htw.://...........ooogoo///eeee/
Franck Burlot and François Yvon. Using monolingual data in neural machine translation: a systematic study, In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 144-155, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 1011665//1//186660 URL
https: www ac http://wwww.rrg/o/h/oooeece /wwleggyy///// -315
Hounkpati B.C. Capo. A Comparative Phonology of Gbe. De Gruyter Mouton, Berlin, Boston, 2010. ISBN 978-3-11-087053-4.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, htt//1406.10733 2014. URL http:// /tatt:://ariivve http:://wwwwwx.o../aaaaaaa////aa wtwp/////a///////eeae
Noa Cruz Díaz and Manuel Maña López. An analysis of biomedical tokenization: Problems and strategies. In Proceedings of the Sixth International Workshop Health Text Mining and Information Analysis, pp. 40-49, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/ W15-2605. URL (tLhhttss:// www.wwwwwwww........wwwwppwwwwww wwwwwb.www....... org http://attooooooooloa/5555
Bonaventure F. P. Dossou and Chris C. Emezue. Ffr v1.0: Fon-french neural machine translation. In Proceedings of the AfricanNLP Workshop, International Conference on Learning Representations, 2XXivarXXiv2200312211.. 2020.
Philip Gage. A new algorithm for data compression. C Users J., 12(2):23-38, February 1994. ISSN 0898-9788.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 344-354, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi:
10.18653/v1/N18- 1032, [tL.ttts::: //ww: www.waaclllooo web. org. aaaaanhoooooooooo oooologggll N1 818--11 1-1322
Ying He and Mehmet Kayaalp. Comparison of 13 Tokenizers on MEDLINE. 12 2006. doi: 10.13140/2.1. 1133.1206.
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back-translation for neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 18-24, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/W 10665/111/118 11-22773. URL wwww ac web. org/ /tteg//aatttiae htt.ch//oooooyyytotooooo 0atpp//ww1--23133
Pratik M. Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. ArXiv, abs/2004.09095, 2020.
8

---

Under review conference paper at ICLR 2021
Bryan Jurish and Kay-Michacl Würzner. Word and sentence tokenization with hidden markov models. J. Lang. Technol. Comput. Linguistics, 28:61-83. 83, 2013.
Alina Karakanta, Jon Dehdari, and Josef Genabith. Neural machine translation for low-resource languages without parallel corpora. Machine Translation, 32(1 1211-1))167-189, June 2018. ISSN 0922-6567. doi: 1111007/s1090-01---00
/ttt:s/:/ool..org/00..001/////// 017 9203-5
Dominik Machácek, Jonás Vidra, and Ondrej Bojar. Morphological and langugooggotoggooooooooic word segmentation for NMT. CoRR, abs/1 806.05482, 2018. /t..httt:///arrxi.orr//aaa
05482
Andrei Mikheev. Tagging sentence boundaries. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, NAACL 2000, pp. 264-271 USA, 2000.
Association for Computational Linguistics.
Thi-Vinh Ngo. Lh Ha, Phuong-Thar Thai Nguyen, and Le-Minh Nguyen. Overcoming the rare word problem for low-resource language pairs in neural machine translation. Proceedings of the 6th Workshop on Asian Translation, pp. 24, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D11- -228..
URL hattps://wett acl web. orq wwt./nnhooogg///aa//aoooooo DD19- 5228
Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp Öktem, Wole Akin, Ghollah Kioko, Kevin Degila, Herman Kamper, Bonaventure Dossou, Chris Emezue, Kelechi Ogueji, and Abdallah Bashir. Masakhane machine translation for africa, 2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 31 1-318, Philadclphia, Pennsylvania, USA, July 2002. Association for Computational URL https:/ www. org/ anthology/
Linguistics. doi: 10.3115/1073383.107111
P02 1040
Adithya Renduchintala, Pamela Shapiro. Kevin Duh, and Philipp Koehn. Character-aware decoder for neural machine translation. CoRR, hts/1809.02223, 2018. http:ttpp//arxx...otot /22/ 02223
Michael D. Riley. Some applications of tree-based modelling to speech and language. In Proceedings of the Workshop on Speech and Natural Language, HLT '89, pp. 339-352, USA, 1989. Association for Computational Linguistics. ISBN 1558601 120.
doi: 10.3 10.31|5/10054444177735
URL https
7
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909, 2015. URL htth::/:///artt arwiv..ivvvvvvv///iv//aa./aaavv. org/ arggabs/////..oo..o//////////.bp 111011111100 0 07009
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with mono- lingual data. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86-96, Berlin, Germany, August 2016a. Association for Computational Lin- guistics. doi: 10. 1118633/11/P1/ 1009. Uttsttts::/ wwwppp:://www........... www ac http:///./o..otooo/o//l-//-----/
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016b. Association for Computational Linguis- tics. doi: 10.18653/v1/P1--11 1162. URL httpp::/ wwww awwweb. wwwpeee...ggaggaaa.aaaee httpg//atoooooooooooooeee
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. CoRR, 100.31409, 2014.

---

Under review conference paper at ICLR 2021
Elan van Biljon, Arnu Pretorius, and Julia Kreutzer. On optimal transformer depth for low-resource language translation. In Proceedings of the International Conference on Learning Representations, https//aariv.orr/ab/2204...41 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http:
7/arxiv. wttp://oggaaabaa/aaooooaa 03762
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. CharacTTer: Translation edit rate on character level. Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 505-510. 10, Berlin, Germany, August 2016. Association for Computational Linguisties. doi:
10.18653/v 1/W 16-2222.
URL httthttt:://// www. wcw.eb. htt.weewww.../atoooggo/////o///e
httpe////tthootoyooggooo WW1 2342
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Eukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08 184444,844444444 2016. URL http://arxiv
org / abs / wbs. 08144
Jiajun Zhang and Chengqing Zong. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 20/6 Conference on Empirical Methods in Natural Language Processing, pp. 1535, Austin, Texas, nov 2016. Association for Computational Linguistics. doi: 10. 11885//1/D16-11 1/D16-1160. URL
ht https: www. ac web. webb.org//mmthhooog//011
10


Reviews:
[{"review_id": "XjVDidBdZoL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes another variant of phrase-based MT for African languages,\ninvolving native speakers for manual annotations.\nInstead of just using subwords or statistical phrase identification, the \nauthors propose to use the intuition of native speakers for translating African\nFon languages into French (and vice versa).\nAccording to their experiments, BLEU and other indexes significantly improved\nover standard IBM-1 phrase-based machine translation.\n\nHowever, from the description and examples in this paper, I have a little doubt\nfor this improvement:\n\n- For creating the aligned corpus, the authors say that they chose only short\nexpressions, namely 1-6 words. According to the results shown in Table 1,\nthis essentially amounts to simply memorizing frequent idiomatic phrases.\nTherefore, improvements with this kind of human intervention over such an\neasy sentences is basically trivial. Of course, the paper says that the test\ndata comprises of long and complex sentences; but the examples are not, thus\nI cannot identify whether the problem is really difficult or not.\n\n- Even if the proposed human annotation is effective, that does not seem to\nleverage characteristic property of African languages. In Section 3, \"un d'o\nganji\" has an ambiguities about \"un\", but this kind of ambiguity of a word is \nshared by almost all the other languages (imagine translating \"given\" in a\nconditional proposition). The property of African Fon languages, such as\ndiacritics and affixation, are not used here.\n\nFinally, the proposed annotation algorithm in page 3 seems to quite vague to\nme. Where v came from? If w is a word, what is the meaning of \"w \\subseteq v\"?\nAlso, this algorithm seems to use a simple longest match: however, in many\ncases the usage of a word is only clear using succeeding words; i.e. some\nforward-backward algorithm is necessary for correct identification of a phrase.\n\nThat being said, I strongly agree with the authors that neural machine\ntranslation of African low-resourced language is important. I hope that the\nauthors would add more persuasive results and analysis to realize a practical\ntranslation of Fon languages.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TtTVGfJl-6G", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit after seeing others reviews -- I think I gave this paper a MUCH higher score than the other reviewers, simply because it is very novel with Fon language. I agree with all of your points about what is lacking, but in my mind, the novelty was enough to still give a 7. Now I definitely think that is too high. I think this paper can reasonably be rejected, but I'd like to give actionable of constructive criticism, since I do think the work on this low resource language is important for the NLP community. With such low resources, we cannot expect the same type of work as we would for other languages.  \n\nOverview: This paper discusses the problems of common tokenization strategies for low resource african languages, and proposes a new tokenization method to overcome these problems. They train low resource NMTs using 4 different tokenization strategies, to show that their proposed tokenization method leads to the best NMT results by several metrics.\n\nContribution: The authors contribute a new tokenization method, code, and a dataset. \n\nThe good: Very interesting and important work! Many people will be excited to use this data. Paper is mostly clearly written, and easy to read. The paper flows well. Someone with this paper could reproduce the work, more or less. \n\nThe bad:\n* Figure 1 is difficult to read and messy. First, by \"Input\" you actually mean \"Source\". The input would be the source sentence with its appropriate tokenization, no? Also, I think putting the english translation in a different font or color would be greatly helpful to our eyes. I really think this must be fixed! Figure 1 is presently not pleasant to look at, even though it has interesting results`! \n* Section 4 - I think you really need to re-state that the algorithm has a human-in-the-loop for clarity. Before describing your algorithm, humans are only mentioned once in the algorithm. Indeed, at first, the words \"The following algorithm\" confused me, because I thought it was more a \"methodology\", since Step 2 is where the humans are in the loop, unless you have a Fon POS tagger and I am misunderstanding? But then at the end, I saw you include Encode as step 4, so it is the machine...  The fact that I flustered a bit with my understanding here, was confused, and had to spend a few minutes thinking about it, means it needs a bit of tweaking. Maybe add a comment saying Step 2 is the human-in-the-loop step of the algorithm?\n\nSuggested additions:\n* I think more specific linguistic details about Fon are missing. For example, if you could give us one or two sentences of Fon in the beginning of the paper, that demonstrate some of the difficulties of the language, I think this would greatly strengthen the motivation. You *tell* us that Fon is \"a language with special tokenization needs\" and that \"standard tokenization methods do not alwaysadequately deal with the grammatical, diacritical, and tonal properties of some African language\", and you cite the relevant papers. But I would still like to be *shown*. I think just including two sentences that have some of these features, and that gets the point accross of \"how would we tokenize this?\" would really help the motivation. Its not that I/readers dont believe you when we are *told*, but being *shown* makes it much more interesting and give people an appreciation for Fon tokenization challenges! \n* Can we get any information about how the annotators were trained? I think this is standard for such papers.  \n\nOther smaller suggested fixes: \n* Section 5, near the end - Little grammatical mistake. \"... bunch of those errors has\" should be \"errors have\". \n* Section 6.3 - Please change \"The results from Table 2 and Table 1\" to say \"Table 1 and Table 2\". It does not make sense to list them out of order. I also think it makes sense to switch Figure 1 and Figure 2 entirely. I.e., Figure 1 should be your results table, and figure 2 should be the examples for us to see. \n* Section 6.3 - Slightly confusing wording. The second sentence is confusing to me, and I am a native English speaker. \"It is important to note that while BLEU of other methods reduced on the Fr→Fon task, WB improved on it.\"  To me saying \"BLUE reduced for the other methods\" means that you have some other baseline you are comparing to. Am I missing something? Are you comparing against Fon--> Fr? \n\nQuestions:\n* Section 6.2 - Does it really take all 500 epochs to run, or do you have early stopping at some point when the loss flatlines?\n* Because BPE is such a standard baseline, why do you not include it as a baseline? I know you cite the Abbott & Martinus, 2018 paper, stating that BPE is bad for analytical languages, but I still think it would prove a point to show BPE performing badly for your data. \n\nOverall: Very interesting work, and can't wait to see this data be used :-) I think the paper could be greatly strengthened by taking some time to include an example that demonstrates the linguistic and typological features of Fon that makes it difficult. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KjKVwahcz-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate different tokenization methods for the translation between French and Fon (an African low-resource language). This means that they compare different ways to construct the input and output vocabularies of a neural machine translation (NMT) system. They further propose their own way to create those units, based on phrases, which is called WEB.\n\nThe NMT system the authors use follows Bahdanau et al. (2015): it is a GRU sequence-to-sequence model with attention. The dataset they use has been created and cleaned by bilingual speakers and consists of roughly 25k examples (this is a really small dataset for NMT, so the authors are taking on a really hard task!). \n\nWEB works in the following way: after phrases have been found automatically, bilingual speakers analyze what the longest phrases which correspond to translated phrases in the other language are. Only the longest phrases for each example are kept for the final vocabulary. The authors show that WEB improves the performance in both translation directions by a lot on all metrics, clearly showing that the work they invest into creating the vocabulary pays out. Thus, I think this work is important to be able to provide speakers of Fon with a functioning translation system.\n\nHowever, I am unsure if this work is suitable for a machine learning conference. While the overall goal of this work is to create an NMT system, the main contribution is the manual cleaning of the dataset and semi-manual creation of the vocabularies. I would recommend to the authors to submit this paper to a conference with a stronger focus on NLP and NLP resources (maybe LREC)? I further want to emphasize that I think work like this paper is incredibly important and the authors shouldn't feel discouraged. Importantly, the manual labor needed for WEB has been a lot and it's obvious that it helps for NMT. I just don't think that this paper is a good fit for ICLR.\n\nMinor point: has the creation of WEB access to the test data? If so, the authors should change that (or collect new test data?) to ensure a fair evaluation. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors investigate different tokenization methods for the translation between French and Fon (an African low-resource language). Low-resource machine translation is a very important topic and it is great to see work on African languages - we need more of this!

Unfortunately, the reviewers unanimously agree that this work might be better suited for a different conference, for example LREC, since the machine learning contributions are small. The AC encourages the authors to consider submitting this work to LREC or a similar conference.
Decision: Reject