Towards Supervised and Unsupervised Neural Machine Translation Baselines for Nigerian Pidgin

Abstract: Nigerian Pidgin is arguably the most widely spoken language in Nigeria. Variants of this language are also spoken across West and Central Africa, making it a very important language. This work aims to establish supervised and unsupervised neural machine translation (NMT) baselines between English and Nigerian Pidgin. We implement and compare NMT models with different tokenization methods, creating a solid foundation for future works.

Over 500 languages are spoken in Nigeria, but Nigerian Pidgin is the uniting language in the country. Between three and five million people are estimated to use this language as a first language in performing their daily activities. Nigerian Pidgin is also considered a second language to up to 75 million people in Nigeria, accounting for about half of the country's population according to BIBREF0.
The language is considered an informal lingua franca and offers several benefits to the country. In 2020, 65% of Nigeria's population is estimated to have access to the internet according to BIBREF1. However, over 58.4% of the internet's content is in English language, while Nigerian languages, such as Igbo, Yoruba and Hausa, account for less than 0.1% of internet content according to BIBREF2. For Nigerians to truly harness the advantages the internet offers, it is imperative that English content is able to be translated to Nigerian languages, and vice versa.
This work is a first attempt towards using contemporary neural machine translation (NMT) techniques to perform machine translation for Nigerian Pidgin, establishing solid baselines that will ease and spur future work. We evaluate the performance of supervised and unsupervised neural machine translation models using word-level and the subword-level tokenization of BIBREF3.
Some work has been done on developing neural machine translation baselines for African languages. BIBREF4 implemented a transformer model which significantly outperformed existing statistical machine translation architectures from English to South-African Setswana. Also, BIBREF5 went further, to train neural machine translation models from English to five South African languages using two different architectures - convolutional sequence-to-sequence and transformer. Their results showed that neural machine translation models are very promising for African languages.
The only known natural language processing work done on any variant of Pidgin English is by BIBREF6. The authors provided the largest known Nigerian Pidgin English corpus and trained the first ever translation models between both languages via unsupervised neural machine translation due to the absence of parallel training data at the time.
All baseline models were trained using the Transformer architecture of BIBREF7. We experiment with both word-level and Byte Pair Encoding (BPE) subword-level tokenization methods for the supervised models. We learned 4000 byte pair encoding tokens, following the findings of BIBREF5. For the unuspervised model, we experiment with only word-level tokenization.
The dataset used for the supervised was obtained from the JW300 large-scale, parallel corpus for Machine Translation (MT) by BIBREF8. The train set contained 20214 sentence pairs, while the validation contained 1000 sentence pairs. Both the supervised and unsupervised models were evaluated on a test set of 2101 sentences preprocessed by the Masakhane group. The model with the highest test BLEU score is selected as the best.
Unsupervised model training followed BIBREF6 which used a Transformer of 4 encoder and 4 decoder layers with 10 attention heads. Embedding dimension was set to 300.
Supervised model training was performed with the open-source machine translation toolkit JoeyNMT by BIBREF9. For the byte pair encoding, embedding dimension was set to 256, while the embedding dimension was set to 300 for the word-level tokenization. The Transformer used for the byte pair encoding model had 6 encoder and 6 decoder layers, with 4 attention heads. For word-level, the encoder and decoder each had 4 layers with 10 attention heads for fair comparison to the unsupervised model. The models were each trained for 200 epochs on an Amazon EC2 p3.2xlarge instance.
English to Pidgin:
Pidgin to English:
For the word-level tokenization English to Pidgin models, the supervised model outperforms the unsupervised model, achieving a BLEU score of 17.73 in comparison to the BLEU score of 5.18 achieved by the unsupervised model. The supervised model trained with byte pair encoding tokenization outperforms both word-level tokenization models, achieving a BLEU score of 24.29.
Taking a look at the results from the word-level tokenization Pidgin to English models, the supervised model outperforms the unsupervised model, achieving a BLEU score of 24.67 in comparison to the BLEU score of 7.93 achieved by the unsupervised model. The supervised model trained with byte pair encoding tokenization achieved a BLEU score of 13.00. One thing that is worthy of note is that word-level tokenization methods seem to perform better on Pidgin to English translation models, in comparison to English to Pidgin translation models.
When analyzed by L1 speakers, the translation qualities were rated very well. In particular, the unsupervised model makes many translations that did not exactly match the reference translation, but conveyed the same meaning. More analysis and translation examples are in the Appendix.
There is an increasing need to use neural machine translation techniques for African languages. Due to the low-resourced nature of these languages, these techniques can help build useful translation models that could hopefully help with the preservation and discoverability of these languages.
Future works include establishing qualitative metrics and the use of pre-trained models to bolster these translation models.
Code, data, trained models and result translations are available here - https://github.com/orevaoghene/pidgin-baseline
Special thanks to the Masakhane group for catalysing this work.
Unsupervised (Word-Level):
Supervised (Word-Level):
Supervised (Byte Pair Encoding):
The following insights can be drawn from the example translations shown in the tables above:
The unsupervised model performed poorly at some simple translation examples, such as the first translation example.
For all translation models, the model makes hypothesis that are grammatically and qualitatively correct, but do not exactly match the reference translation, such as the second translation example.
Surprisingly, the unsupervised model performs better at some relatively simple translation examples than both supervised models. The third example is a typical such case.
The supervised translation models seem to perform better at longer example translations than the unsupervised example.
Unsupervised (Word-Level):
Supervised (Word-Level):
Supervised (Byte Pair Encoding):

Figure and table captions:
Table 3: Unsupervised (Word-Level) Results from English to Nigerian Pidgin
Table 4: Supervised (Word-Level) Results from English to Nigerian Pidgin
Table 5: Supervised (Byte Pair Encoding) Results from English to Nigerian Pidgin
Table 6: Unsupervised (Word-Level) Results from English to Nigerian Pidgin
Table 7: Supervised (Word-Level) Results from English to Nigerian Pidgin
Table 8: Supervised (Byte Pair Encoding) Results from English to Nigerian Pidgin

Questions about this paper:
How long is their dataset?
What is the best performing system?

Answers:
Data used has total of 23315 sentences.
In English to Pidgin best was byte pair encoding tokenization superised model, while in Pidgin to English word-level tokenization supervised model was the best.