Introduction
Over 500 languages are spoken in Nigeria, but Nigerian Pidgin is the uniting language in the country. Between three and five million people are estimated to use this language as a first language in performing their daily activities. Nigerian Pidgin is also considered a second language to up to 75 million people in Nigeria, accounting for about half of the country's population according to BIBREF0.
The language is considered an informal lingua franca and offers several benefits to the country. In 2020, 65% of Nigeria's population is estimated to have access to the internet according to BIBREF1. However, over 58.4% of the internet's content is in English language, while Nigerian languages, such as Igbo, Yoruba and Hausa, account for less than 0.1% of internet content according to BIBREF2. For Nigerians to truly harness the advantages the internet offers, it is imperative that English content is able to be translated to Nigerian languages, and vice versa.
This work is a first attempt towards using contemporary neural machine translation (NMT) techniques to perform machine translation for Nigerian Pidgin, establishing solid baselines that will ease and spur future work. We evaluate the performance of supervised and unsupervised neural machine translation models using word-level and the subword-level tokenization of BIBREF3.
Related Work
Some work has been done on developing neural machine translation baselines for African languages. BIBREF4 implemented a transformer model which significantly outperformed existing statistical machine translation architectures from English to South-African Setswana. Also, BIBREF5 went further, to train neural machine translation models from English to five South African languages using two different architectures - convolutional sequence-to-sequence and transformer. Their results showed that neural machine translation models are very promising for African languages.
The only known natural language processing work done on any variant of Pidgin English is by BIBREF6. The authors provided the largest known Nigerian Pidgin English corpus and trained the first ever translation models between both languages via unsupervised neural machine translation due to the absence of parallel training data at the time.
Methodology
All baseline models were trained using the Transformer architecture of BIBREF7. We experiment with both word-level and Byte Pair Encoding (BPE) subword-level tokenization methods for the supervised models. We learned 4000 byte pair encoding tokens, following the findings of BIBREF5. For the unuspervised model, we experiment with only word-level tokenization.
Methodology ::: Dataset
The dataset used for the supervised was obtained from the JW300 large-scale, parallel corpus for Machine Translation (MT) by BIBREF8. The train set contained 20214 sentence pairs, while the validation contained 1000 sentence pairs. Both the supervised and unsupervised models were evaluated on a test set of 2101 sentences preprocessed by the Masakhane group. The model with the highest test BLEU score is selected as the best.
Methodology ::: Models
Unsupervised model training followed BIBREF6 which used a Transformer of 4 encoder and 4 decoder layers with 10 attention heads. Embedding dimension was set to 300.
Supervised model training was performed with the open-source machine translation toolkit JoeyNMT by BIBREF9. For the byte pair encoding, embedding dimension was set to 256, while the embedding dimension was set to 300 for the word-level tokenization. The Transformer used for the byte pair encoding model had 6 encoder and 6 decoder layers, with 4 attention heads. For word-level, the encoder and decoder each had 4 layers with 10 attention heads for fair comparison to the unsupervised model. The models were each trained for 200 epochs on an Amazon EC2 p3.2xlarge instance.
Results ::: Quantitative
English to Pidgin:
Pidgin to English:
For the word-level tokenization English to Pidgin models, the supervised model outperforms the unsupervised model, achieving a BLEU score of 17.73 in comparison to the BLEU score of 5.18 achieved by the unsupervised model. The supervised model trained with byte pair encoding tokenization outperforms both word-level tokenization models, achieving a BLEU score of 24.29.
Taking a look at the results from the word-level tokenization Pidgin to English models, the supervised model outperforms the unsupervised model, achieving a BLEU score of 24.67 in comparison to the BLEU score of 7.93 achieved by the unsupervised model. The supervised model trained with byte pair encoding tokenization achieved a BLEU score of 13.00. One thing that is worthy of note is that word-level tokenization methods seem to perform better on Pidgin to English translation models, in comparison to English to Pidgin translation models.
Results ::: Qualitative
When analyzed by L1 speakers, the translation qualities were rated very well. In particular, the unsupervised model makes many translations that did not exactly match the reference translation, but conveyed the same meaning. More analysis and translation examples are in the Appendix.
Conclusion
There is an increasing need to use neural machine translation techniques for African languages. Due to the low-resourced nature of these languages, these techniques can help build useful translation models that could hopefully help with the preservation and discoverability of these languages.
Future works include establishing qualitative metrics and the use of pre-trained models to bolster these translation models.
Code, data, trained models and result translations are available here - https://github.com/orevaoghene/pidgin-baseline
Conclusion ::: Acknowledgments
Special thanks to the Masakhane group for catalysing this work.
Appendix ::: English to Pidgin translations
Unsupervised (Word-Level):
Supervised (Word-Level):
Supervised (Byte Pair Encoding):
Appendix ::: English to Pidgin translations ::: Discussions:
The following insights can be drawn from the example translations shown in the tables above:
The unsupervised model performed poorly at some simple translation examples, such as the first translation example.
For all translation models, the model makes hypothesis that are grammatically and qualitatively correct, but do not exactly match the reference translation, such as the second translation example.
Surprisingly, the unsupervised model performs better at some relatively simple translation examples than both supervised models. The third example is a typical such case.
The supervised translation models seem to perform better at longer example translations than the unsupervised example.
Appendix ::: Pidgin to English translations
Unsupervised (Word-Level):
Supervised (Word-Level):
Supervised (Byte Pair Encoding):

Question: How long is their dataset?
Answer: 21214

Data used has total of 23315 sentences.

Question: What metrics are used?
Answer: BLEU score

BLEU

Question: What is the best performing system?
Answer: A supervised model with byte pair encoding was the best for English to Pidgin, while a supervised model with word-level encoding was the best for Pidgin to English.

In English to Pidgin best was byte pair encoding tokenization superised model, while in Pidgin to English word-level tokenization supervised model was the best.

Question: What tokenization methods are used?
Answer: word-level , subword-level

word-level, Byte Pair Encoding (BPE) subword-level

Question: What baselines do they propose?
Answer: Transformer architecture of BIBREF7

supervised translation models