SlowGuess/NAACL_2022_OCR huggingface dataset:
$^{7}$ https://scikit-learn.org/stable/modules/generated/
84 core Intel i7-6700K CPU @ 4.00GHz with 64GB of RAM.
912 core Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz, GV100GL
10Pre-trained bilingual and monolingual embeddings are available for researchers on request. Experimental details, model implementations, and trained language models are available for researchers, all bound by the Kaitiakitanga license: https://github.com/MaoriEnglish-Codeswitch/ MaoriEnglish-CodeSwitch-Detection
Figure 2: Pie Chart of the languages detected by Google (left) and Azure (right) at word level for the test set of the Hansard Database. The gold-standard label for all the words used here is 'Māori'.
7.1 Task 1: Language Detection
7.1.1 Cloud-based Online Tools
To analyse the effectiveness of using Google Translate and Azure services to detect Māori (and English), we experimented with the test set of the Hansard database where the sentences are either monolingual (Māori or English) or code-switched. Google Translate detected  $99.7\%$  of the English words, and Azure detected  $97.8\%$  of the English words correctly. Figure 2 presents pie charts of the resulting language detection for 'Māori' word (i.e. the gold-standard labels for the words is 'Māori'). For Māori words, Google Translate detected with an accuracy of  $65.2\%$ , and Azure detected with an accuracy of  $52\%$ . Although the accuracy of Google Translate was better than Azure, the error rate of both services are too high for Māori language detection. In addition, apart from wrongly detecting Māori words as English, around  $14 - 21\%$  of the words were classified as various other languages by both cloud services. We acknowledge that cloud-based services such as Google and Azure are multilingual and hence low-resource languages such as Māori are dominated by the resource-rich languages during training. This will inevitably influence the accuracy of LD of Māori using Google and Azure's cloud services. However, given there is no other system available to detect Māori, it was still vital to evaluate the outcome of the above mentioned cloud-based services.
7.1.2 Baseline Classifiers
LD task using the Hansard database is a multi-class classification problem at the sentence level (classes: Māori, English or Code-Switched sentence). The LD task using MLT corpus is a binary classification problem of relevant/irrelevant tweets based on the usage of the Māori loanwords. Table 4 presents
Model	Data	Results
Multi-class	Macro-F1
Multinomial NB (BOW)	Hansard	0.887
LR (BOW)	Hansard	0.913
LR (Eng300)	Hansard	0.831
LR (Māori-Eng-300)	Hansard	0.853
LR (Māori-Eng-300SG)	Hansard	0.859
Binary	F1-score
LR (Eng300)	MLT corpus	0.833
LR (Māori-300SG)	MLT corpus	0.812
LR (Māori-Eng-300)	MLT corpus	0.849
LR (Māori-Eng-300SG)	MLT corpus	0.846
Table 4: Macro-F1 scores and F1-scores for the test set of Hansard database and labelled MLT corpus respectively, where BOW or sentence level features are used to represent text. Bold: best results for each task.
overall macro-F1 and F1 scores for the LD task using Hansard database and MLT corpus, respectively, where BOW and static word embeddings at the sentence level (or tweet level) are used to represent the text. We obtain embeddings for each sentence by computing the vector sum of the embeddings for each word in the sentence. This vector sum is then normalised to have length one, to ensure that sentences of different lengths have representations of similar magnitudes. The bilingual embeddings perform better than monolingual embeddings for both Hansard and MLT corpus. However, BOW outperforms static embeddings feature representation for LR.
7.1.3 Neural Networks
After evaluating the performance of baseline classifiers, we further proceed with LD task using neural networks. As the size of the labelled MLT corpus is small, it is insufficient for training and evaluating neural networks. Table 5 presents macro-F1 and weighted-F1 scores obtained using the test set of the Hansard database for performance comparison
Model	Macro-F1	Weighted-F1
Monolingual Embeddings
CNN (E300)	0.946	0.985
CNN (Māori-300)	0.905	0.986
CNN (Māori-300SG)	0.914	0.990
BiLSTM (E300)	0.943	0.996
BiLSTM (Māori-300)	0.926	0.995
BiLSTM (Māori-300SG)	0.940	0.995
Bilingual Embeddings
CNN (Māori-Eng-300)	0.963	0.995
CNN (Māori-Eng-300SG)	0.969	0.996
BiLSTM (Māori-Eng-300)	0.984	0.997
BiLSTM (Māori-Eng-300SG)	0.989	0.997
Contextual Embeddings
BERT-base	0.931	0.988
mBERT-base	0.946	0.991
Table 5: Comparison of results for the Hansard database (test set) with various models. Bold: best results.
Figure 3: Critical difference plots identifying statistical differences between models presented in Tables 4 & 5.
Model	Training data	Testing data	Accuracy (Māori)
Google	Wikipedia	RMT	68.2%
BiLSTM (E300)	Hansard	RMT	56.6%
BiLSTM (Māori-Eng-300)	Hansard	RMT	85.4%
BiLSTM (Māori-Eng-300SG)	Hansard	RMT	85.6%
Table 6: Accuracy of Māori words detection in RMT corpus using Hansard-based trained models (Table 5).
across language models. The macro-F1 score is an unweighted average score of all the classes. In comparison, weighted-F1 scores are higher than macro-F1 scores across the models. The imbalanced distribution in the data, where labels are predominantly English, is reflected in the scores where the minority classes penalise the macro-F1 scores. Bilingual embeddings (Māori-Eng-300) consistently perform better than monolingual embeddings. BiLSTM with Māori-Eng-300SG embeddings are the best across all models, including BERT-base and mBERT-base. Skip-gram models are better than CBOW. In comparison, English-only embeddings E300 outperform Māori-only monolingual embeddings. One possible explanation for this is the lack of training data for Māori-only embeddings compared to E300.
Figure 3 presents critical difference plots across the models presented in Table 5 and BOW repre
sensation presented in Table 4. BiLSTM (Māori-Eng-300SG) has the lowest rank, and multinomial naive Bayes (BOW) has the highest rank with no bold line connecting the two, indicating the difference in average ranking is statistically significant. Bold lines are connecting BiLSTM (Māori-Eng-300SG) with mBERT and BERT-base in the CD-plot, indicating that the difference in average ranking is not statistically significant. A  $4 - 6\%$  improvement was observed between BERT/mBERT and BiLSTM (Māori-Eng-300SG).
To further evaluate the language models, we used the models trained with the Hansard data to detect Māori words in the RMT corpus. Table 6 presents the accuracy of the detection. We also present the accuracy of Māori language detection using Google Translate for comparison. Evidently, BiLSTM with Māori-Eng-300SG embeddings model trained on the training set of the Hansard database has the best accuracy. As observed with other databases, the accuracy of the bilingual embeddings is higher than the monolingual embeddings. However, the accuracy of BiLSTM with E300 embeddings is considerably lower than other models, including Google. One possible reason is the lack of vocabulary in E300 for the informal language used in RMT data (Tweets).
7.1.4 In Summary
The results suggest that the bilingual embeddings perform better than monolingual embeddings (both the downloaded Eng300 and Māori only models) for the LD task. This finding was verified across the Hansard database (Tables 4, 5) and the MLT corpus (Table 4). Further evidence is provided in Māori words detection using RMT corpus (Table 6). We also observed that the bilingual embeddings outperformed the contextual embeddings. One possible reason for this finding is the lack of vocabulary in BERT models, as no further training was performed using Māori data. This research only fine tunes the BERT models for down streaming tasks. As emphasised before, the Māori data availability is the biggest limitation to this research. Among the experimented models for LD task, BiLSTM with Māori-Eng-300SG performed the best.
7.2 Task 2: Code-Switch Detection
For evaluation of the code-switch detection between Māori-English pair, we require word-level labels and hence, only the Hansard database was used for this task. We use selected trained models pre
Figure 4: F1-scores for Māori and English calculated at the word level for the Hansard database.
Model	CS: Accuracy
CNN (E300)	35%
BiLSTM (E300)	83%
BiLSTM (Māori-Eng-300)	67%
BiLSTM (Māori-Eng-300SG)	87%
Table 7: Accuracy of code-switch detection in the Hansard data (bilingual sentences of the test set) using the trained models, as shown in Figure 1.
sented in Section 7.1, and identify the code-switch points (see Figure 1). Figure 4 presents word-level F1 scores of Māori and English for CS task. For English words, all systems perform equally well. However, for Māori, cloud-based multilingual systems perform poorly, and BiLSTM with bilingual embeddings shows a substantial improvement in F1 score, as observed before. It is vital to point out, cloud-based services such as Google and Azure are multilingual models and these systems have to classify between large number of languages. Hence, the poor performance with detecting Māori is not surprising, especially when compared to models which only have to classify between English and Māori. However, we include the results of large scale models here to emphasise the fact that the only existing tool that can detect Māori have limitations. Furthermore, Table 7 presents the accuracy of detecting the code-switch points of the test set of the Hansard database. Among the reported results, CNN with E300 performed poorly, and BiLSTM with Māori-Eng-300SG outperformed the other models.
8 Discussion and Conclusions
This research is the first attempt to use advances in NLP in two tasks - Māori (a low-resourced language) language detection, and Māori-English code-switch detection. Our experiments show that the accuracy of existing cloud-based systems to detect Māori is very low. Hence, there is the need to have more specialised systems for detecting Māori.
We collected data in collaboration with Māori researchers for training and evaluations. Experiments obtained across tasks using three databases show that our bilingual embeddings outperformed downloaded, pre-trained English-only embeddings trained on large databases. Among the models tested, BiLSTM with bilingual embeddings trained using the Skip-gram model is the best for both tasks. We provide evidence to show BERT-base used on the down-streaming task of language detection –where Māori is under-represented or unseen by the model vocabulary– is not always the best solution (as also observed by (Wu and Dredze, 2020; Wang et al., 2020)). For most low-resourced languages, including Māori, the Wikipedia data is significantly smaller than English, resulting in a reduced vocabulary. Due to limited resources, continuous training or training from scratch of models such as BERT-base is not possible.
For future work, it is a possibility to use ideas such as Extend M-BERT (Wang et al., 2020) and explore more efficient pre-training techniques to improve the accuracy of BERT like models for language detection of low-resource languages such as Maori. In addition, hybrid models using handcrafted rules based on the phonotactic differences between the languages and deep learning-based methods are a promising pathway for future work.
The availability of digitised Māori and bilingual data is limited, which restricts the ability to train large language models. In addition, considering this is the first deep learning-based research in this area, comparison with published work is not possible. We overcome these limitations by respecting the available data and data sovereignty for this research. We provide experimental results using cloud services such as Google and Azure, as these are the only available systems that can detect Māori. The study reported here is a much-needed contribution to Māori language technology development. Word embeddings developed in this research are available to other researchers on request, bound by the Kaitiakitanga license.
Thanks to all researchers who shared their data collections with us, and to the University of Auckland Faculty Research Development Fund (FRDF) 2020 for support. We also thank Human Aspects of Software Engineering Lab (HASEL) at the University of Auckland for the computing facilities.
Wafia Adouane, Jean-Philippe Bernardy, and Simon Dobnik. 2018. Improving neural network performance by injecting background knowledge: Detecting code-switching and borrowing in algerian texts. In Proc. of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 20-28.
Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In Proc. of the first workshop on computational approaches to code switching, pages 13-23.
Winifred Bauer, William Parker, and Te Kareongawai Evans. 1993. In Māori. London: Routledge.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proc. ACL Conference.
Alexis Conneau, Guillaume Lample, Rudy Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating Cross-lingual Sentence Representations. In Proc. EMNLP, pages 2475–2485.
David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2):215-232.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Yoav Goldberg. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1-309.
Google-AI-Blog. Google AI Blog: Recent advances in Google Translate. https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html, accessed Dec 15 2021.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proc. of the International Conference on Language Resources and Evaluation, pages 3483-3487.
Ray Harlow. 2007. Maori: A linguistic introduction. Cambridge University Press.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.
Jesin James, Isabella Shields, Rebekah Berriman, Peter J. Keegan, and Catherine I. Watson. 2020. Developing Resources for Te Reo Māori Text To Speech Synthesis System. In Proc. Sojka P., Kopeček I., Pala K., Horák A. (eds) Text, Speech, and Dialogue, Lecture Notes in Computer Science, pages 294-302.
George H John and Pat Langley. 1995. Estimating continuous distributions in bayesian classifiers. In Proc. Conference on Uncertainty in Artificial Intelligence, pages 338-345.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proc. of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431.
Te Taka Adrian Gregory Keegan. 2017. Machine translation for te reo Māori. He Whare Hangarau Māori - Language, culture & technology, Editors: Whaanga, H., Keegan, T. T., Apperley, M.. 23-28. Te Pua Wananga ki te Ao / Faculty of Māori and Indigenous Studies, Te Whare Wananga o Waikato / University of Waikato, pages 23-28.
Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020. Don't use english dev: On the zero-shot cross-lingual evaluation of contextual embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 549-554.
Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proc. of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575-3585.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR), pages 1-13.
Yash Kumar Lal, Vaibhav Kumar, Mrinal Dhar, Manish Shrivastava, and Philipp Koehn. 2019. De-mixing sentiment from code-mixed text. In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 371-377.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Mahsa Mohagheh, Michael McCauley, and Mehdi Mohammadi. 2014. Maori-English machine translation. in Proc. NZCSRSC New Zealand Computer Science Research Student Conference, Canterbury University. Unitec Research Bank.
Siddhartha Mukherjee, Vinuthkumar Prasan, Anish Nediyanchath, Manan Shah, and Nikhil Kumar. 2019. Robust deep learning based sentiment classification of code-mixed text. In Proc. of the 16th International Conference on Natural Language Processing, pages 124-129.
Benjamin Muller, Antonios Anastasopoulos, Benoit Sagot, and Djame Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proc. of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448-462.
Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. In Proc. of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1971-1982.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by backpropagating errors. Nature, 323(6088):533-536.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958.
N. Z. Stats. 2020. Ngā tikanga paihere: a framework guiding ethical and culturally appropriate data use. Guidelines, page 8. https://data.govt.nz/toolkit/data-ethics/nga-tikanga-paihere/.
Te-Hiku-Media. Kaitiakitanga license. https://github.com/TeHikuMedia/Kaitiakitanga-License, accessed 10 Dec 2021.
Te-Hiku-Media. Te Hiku Media. https://tehiku.nz/te-hiku-tech/papa-reo/.
Mads Toftrup, Søren Asger Sørensen, Manuel R. Ciosici, and Ira Assent. 2021. A reproduction of apple's bi-directional LSTM models for language identification in short strings. In Proc. of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 36-42.
David Trye, Andreea S Calude, Felipe Bravo-Marquez, and Te Taka Adrian Gregory Keegan. 2019. Māori loanwords: a corpus of New Zealand English tweets. In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 136-142.
David Trye, Te Taka Keegan, Paora Mato, and Mark Apperley. 2022. Harnessing indigenous tweets: The Reo Maori Twitter corpus. *Language resources and evaluation*, pages 1-40.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998-6008.
Zihan Wang, K Karthikeyan, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual bert to low-resource languages. In Proc. of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2649-2656.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proc. of the 5th Workshop on Representation Learning for NLP, pages 120-130.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer, Andreas Stolcke, Dong Yu, and Geoffrey Zweig. 2017. The microsoft 2016 conversational speech recognition system. In Proc. Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 5255-5259.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep learning requires re-thinking generalization. In *Proc. International Conference on Learning Representations* 2017, pages 1-15.