SlowGuess/NAACL_2022_OCR huggingface dataset:
Language Models for Code-switch Detection of te reo Maori and English in a Low-resource Setting
Jesin James<sup>1</sup>, Vithya Yogarajan<sup>2</sup>, Isabella Shields<sup>1</sup>, Catherine Watson<sup>1</sup>, Peter J Keegan<sup>3</sup>, Peter-Lucas Jones<sup>4</sup> and Keoni Mahelona<sup>4</sup>
$^{1}$ Dept. of Electrical, Computer, and Software Engineering, The University of Auckland  
 $^{2}$ Strong AI Lab, School of Computer Science, The University of Auckland  
 $^{3}$ Te Puna Wānanga, The University of Auckland  
 $^{4}$ Te Hiku Media, New Zealand
Te reo Māori, New Zealand's only indigenous language, is code-switched with English. Māori speakers are at least bilingual, and the use of Māori is increasing in New Zealand English. Unfortunately, due to the minimal availability of resources, including digital data, Māori is under-represented in technological advances. Cloud-based multilingual systems such as Google and Microsoft Azure support Māori language detection. However, we provide experimental evidence to show that the accuracy of such systems is low when detecting Māori. Hence, with the support of Māori community, we collect Māori and bilingual data to use natural language processing (NLP) to improve Māori language detection. We train bilingual sub-word embeddings and provide evidence to show that our bilingual embeddings improve overall accuracy compared to the publicly-available monolingual embeddings. This improvement has been verified for various NLP tasks using three bilingual databases containing formal transcripts and informal social media data. We also show that BiLSTM with pretrained Māori-English sub-word embeddings outperforms large-scale contextual language models such as BERT on down streaming tasks of detecting Māori language. However, this research uses large models 'as is' for transfer learning, where no further training was done on Māori-English data. The best accuracy of  $87\%$  was obtained using BiLSTM with bilingual embeddings to detect Māori-English code-switching points.
Te reo Māori (referred to as Māori) is New Zealand's only indigenous language, spoken by  $4.5\%$  of the total population of 5 million. Māori speakers are bilingual, and code-switching between Māori and English is expected. Māori revitalisation efforts have increased Māori use in the otherwise English-speaking country. Detecting Māori
language and code-switch instances is a prerequisite to analysing language data. Māori and English both use the Roman script (specifically, Māori uses a modified Roman script). Currently, annotations are done manually, making the process time-consuming and slowing down research and technology development. Consider the following sentences:
(a) Perā anō i ngā mate kua hinga atu i te motu.
(b) I want to give no offence to my mate Willie Jackson, but once a week hardly qualifies as the significant Māori voice.
where green indicates Māori, red is used to indicate that the word has same spelling in Māori and English, and the remaining are English. Based on expert knowledge, we know the word mate is Māori in sentence (a) and English in sentence (b).
In this research, we focus on two primary tasks:
Task 1: Language Detection (LD) - detecting Maori language words from input text.
Task 2: Code-switch Detection (CS) - detecting Māori to English or English to Māori code-switch points from input text.
There is limited Māori-only and Māori-English bilingual data available. We collected data in collaboration with the Māori researchers, Māori technology developers and Māori community, where data-sharing is based on trust. As researchers, we remain guardians of the data, ensuring data sovereignty (Stats, 2020). Hence, all the resources shared from this study are bound by the Kaitiakitanga license (Te-Hiku-Media). This paper presents some of the first research to use advances in NLP to detect Māori language and code-switching. No existing models are using NLP techniques for Māori-English code-switch detection. Google and Microsoft Azure's cloud-based services are the only options available for language detection, which is the primary reason for using such large-scale multilingual cloud-based services for comparison in this paper.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 650 - 660 July 10-15, 2022 ©2022 Association for Computational Linguistics
This paper's contributions are:
1. Evaluation of detecting Māori using multilingual models, including the cloud-based services such as Google and Azure, and large scale language models such as Bidirectional Encoder Representations from Transformers (BERT).
2. Pre-training Māori-English bilingual, and Māori-only monolingual sub-word embeddings using collected data. Experiments using three different bilingual data for various NLP tasks show that bilingual embeddings outperform monolingual embeddings.
3. Large scale language models such as BERT –without further training on Māori-English data– fine-tuned on down streaming tasks of detecting Māori are outperformed by BiLSTM with fastText pre-trained sub-word bilingual embeddings for low-resourced language such as Māori.
4. Providing baseline results for detecting low-resourced Māori and code-switch between Māori-English language pair.
2 Te reo Māori (The Māori Language)
Māori is a Polynesian language belonging to the Austronesian family. Phonologically, Māori has ten consonants /p t k m n η f r w h/. The Māori vowel system is described by five short vowels /i e a o u/ (Bauer et al., 1993). Orthographically, there is mostly a one-to-one mapping of a Māori phoneme to a grapheme, except for two digraphs, 'wh', which is /f/, and 'ng' which is /η/. In modern orthography, long vowels are denoted with a macron (e.g. ā). In older texts, they are sometimes expressed as double vowels (e.g. aa), with an umlaut (e.g. ā), or ignored completely (that is, ā is written a). In addition, there is some regional variation in the way words are spelt (e.g. Aorangi vs Aoraki). English, in contrast, has a highly non-phonemic orthography. The Māori syllable structure consists of a nucleus, which may be occupied by a vowel (or a diphthong), and an optional onset (syllable start) occupied by a single consonant (consonant clusters are not present in Māori) (Harlow, 2007).
Research using NLP for tasks relating to Māori is relatively young. Examples include statistical machine translation for Māori-English pair (Mohagheh et al., 2014) and the inclusion of Māori
language detection and translation using cloud services Google and Azure (Keegan, 2017). (Keegan, 2017) indicates that although the growth of cloud services for Māori translations is welcoming, due to the minimal availability of digitised Māori data, the resulting output is inaccurate. Google also acknowledges that for low-resource languages, the quality of language detection and automatic machine translation is far from perfect (Google-AI-Blog).
We present the first research that uses deep learning techniques to detect a code-switch between Māori and English. Hence, except for the Google and Azure cloud services (more details in Section 5.1), we are limited by the availability of systems for Māori language detection and Māori-English code-switch detection for comparison. We use approaches that were inspired by the literature on other language pairs. Examples include XNLI (Cross-lingual Natural Language Inference) cross-lingual classification benchmark (Conneau et al., 2018) where the bidirectional long short-term memory (BiLSTM) model was used across several low resource languages, including Swahili and Urdu; and code-switch detection using BiLSTM and Character-LSTM for language pair English-Hindi (Lal et al., 2019; Mukherjee et al., 2019). XNLI benchmark uses fastText common-crawl embeddings (denoted as E300 in this paper) and aligns it with the MUSE library. Comparison among deep learning models shows that adding background information through sub-word pre-trained embeddings trained using fastText and in the form of lexicons improves the overall performance of deep neural networks on databases of low-resource languages (Adouane et al., 2018).
Transformers such as BERT is the state-of-the-art in many NLP tasks, including language detection, named entity recognition, and machine translation (Devlin et al., 2019; Conneau et al., 2020). There are many large scale multilingual models, such as XLM-R (Conneau et al., 2020) and multilingual BERT (mBERT) (Devlin et al., 2019) trained in more than 100 languages. Research shows that for languages that are under-sampled during training, the effectiveness of large scale multilingual models such as mBERT are sub-optimal (Wu and Dredze, 2020; Wang et al., 2020). In comparison to the contextual representations such as BERT, embeddings with sub-word representation are more data-efficient when data availability is limited (Wu
Data	# Sentences	# Words	Text	Labels	Task
Hansard data0	2,021,261	36,757,230	formal	word-level & sentence level language labels	LD, CS
MLT corpus (Trye et al., 2019)	2,500	50,000	informal	tweet level labels: relevance/irrelevance	LD
RMT corpus (Trye et al., 2022)	79,018	1,000,000	informal	Māori words are identified and labelled	LD
and Dredze, 2020). Furthermore, (Muller et al., 2021) provide evidence to show that many undersampled or unseen languages during training –such as Maltese or Narabizi– code-mixed with French perform worse when using mBERT compared to an RNN with non-contextual dependency parsing baseline. It has been shown that for such unseen or under-sampled languages, there is a need to further train or fine-tune directly with available raw data in the unseen target languages (Muller et al., 2021).
4 Databases
Due to the low-resource nature of the Māori language, extensive databases are currently are unavailable. We collected text data from different sources to form the Māori-English Words (MEW) database, as summarised in Table 2. MEW database contains legal context, stories, social media posts and newspaper articles. The unlabelled MEW database is used to pre-train bilingual and Māori-only monolingual embeddings. We use three labelled databases for experiments: Hansard database, MLT corpus, and RMT corpus. Details of these databases are provided in Table 1.
Hansard database contains the New Zealand Parliament debates from 2003 onwards. Together with experts in Māori (Te-Hiku-Media), we have labelled the Hansard database, where English or Māori labels are assigned using linguistic rules and manual checking. Each sentence in the databases is marked as Māori, English or bilingual. Each word of each sentence is labelled as Māori or English. The resulting data includes 102,559 bilingual, 1,909,876 English-only and 8,826 Māori-only sentences.
The Māori Loanword Twitter (MLT) corpus is a small database, where each tweet is labelled as 'relevant' and 'irrelevant', based on the presence of a pre-determined set of Māori loanwords in a given tweet. Given detecting Māori language in tweets is a prerequisite to this task, we consider this task also as a Māori language detection task.
Table 1: Databases used for experimental evaluations. LD: Language Detection, CS: Code-Switch Detection.
Name and Database	#Words
Māori only
D1: Te Taka Database*1	9,862,131
D2: Nga Mahi corpus (James et al., 2020)	81,036
D3: Māori Wikipedia	431,280
D4a: LMC Corpus2	5,486,328
Total size of Māori-only database = 92 MB
Māori and English
D4b: LMC Corpus	7,197,059
D5: Niupepa (Māori Newspapers)3	5,050,988
D6: Twitter Corpus*(Trye et al., 2019)	48,289,375
Total size of bilingual data = 0.4 GB
Table 2: Māori-English Words (MEW) database.‘*’ indicates private collections of data.
Reo Māori Twitter (RMT) corpus contains tweets, where at least  $80\%$  of text is in Māori. RMT corpus provides a list of 879,000 Māori words across the tweets. We use this corpus also for the language detection task where the aim is to detect the Māori words identified in the RMT corpus.
5 Language Models and Classifiers
This section provides details of the language models and classifiers we used. We evaluate the performance of cloud-based language detection systems from Google and Azure for Māori. We represent text as bag-of-words and sub-word embeddings using fastText. We use logistic regression and multinomial naive Bayes as baseline classifiers for language detection. We also use neural networks such as RNNs and CNNs to train and evaluate language detection and code-switch detection tasks. Furthermore, we fine-tune transformer models, BERT and mBERT, for the down streaming task of language detection.
0https://www.parliament.nz/en/pb/hansard-debates/rhr/
<sup>1</sup>Private collection of te reo Māori text data, Te Taka Keegan, The University of Waikato, New Zealand, 2021
<sup>2</sup>http://nzetc.victoria.ac.nz/tm/scholarly/tei-legalMaoriCorpus.html.
3http://www.nzdl.org/cgi-bin/library.cgi?a=p&p=about&c=niupepa.
5.1 Cloud-based Online Tools
Google Translate $^{4}$  and Microsoft Azure Cognitive Services language detection $^{5}$  are two popular cloud-based online tools that can detect multiple languages. Google supports 108 languages, including New Zealand English and Maori. Google's RNN-based GNMT (Google Neural Machine Translation) model (Wu et al., 2016) showed significant improvements in enabling translations to cover many languages, including low-resourced languages. Google recently replaced the GNMT model with a hybrid model (transformer encoder and RNN decoder). This model has shown significant improvements compared to the other machine translation systems. Azure's cognitive services can translate  $100+$  languages, including Maori. Azure's early-stage neural network model (Xiong et al., 2017) included a CNN-BiLSTM architecture. Recently, Azure has combined several machine learning algorithms and neural networks to provide various cognitive services.
5.2 Bag of words
Bag of words (BOW) is an effective method (Goldberg, 2017; Joulin et al., 2017) to represent text as a sparse vector, where the order of words in a document is not considered. The number of occurrences of a word or a binary value indicating that the word is present in the document is stored.
5.3 Word Embeddings
For language processing tasks, continuous word representations such as word embeddings trained on large unlabelled databases facilitate effective representation learning (Bojanowski et al., 2017; Joulin et al., 2016). Here, we use fastText (Bojanowski et al., 2017) to learn word embeddings, as novel words not present during training can also be represented using fastText-based embeddings. This can be beneficial for a low-resource setting. FastText supports two word embeddings models: continuous bag-of-words (CBOW) and Skip-grams (Mikolov et al., 2013). The CBOW predicts the specific word from the source context. Skip-gram predicts the source context from the specific word. The embeddings in this research are trained to the specifications of Wikipedia and common crawl fastText models (Grave et al., 2018) (re
Embeddings Model	Data	Size	# Unique Words
Monolingual Embeddings
E300 (Grave et al., 2018)	downloaded	7GB	2,000,000
Māori-300/300SG	D1 - D4a	3GB	49,315
Bilingual Embeddings
Model-Māori-Eng-300 (and 300SG)	D1 - D6	3GB	303,505
Table 3: Outline of fastText pre-trained 300 dimensional embeddings. The MEW database (Table 2) was used for training. 'SG': Skipgram model, otherwise it is CBOw.
ferred to as E300) for both CBOW and Skip-gram $^{6}$ . E300 uses the CBOW method, character n-grams of length 5, window of size 5, and 10 negative samples per positive sample with 300 dimensions. The learning rate is 0.05. Table 3 provides details of our bilingual embeddings, which are available to on request, including E300 details for comparison.
5.4 Baseline Classifiers
We used multinomial naive Bayes (John and Langley, 1995) and logistic regression (LR) (Cox, 1958) to classify text features represented by BOW and static word embeddings. LR is a statistical model used to analyse databases where independent variables determine an outcome. Naive Bayes (John and Langley, 1995) is an easy to build supervised learning algorithm, which applies Bayes' theorem with the "naive" assumption of independence.
5.5 Convolutional Neural Network (CNN)
CNN for text (Kim, 2014) combines one-dimensional convolutions with a max-over-time pooling layer and a fully connected layer. If  $x_{i:i+j}$  is a concatenation of words from a sentence, each word,  $x_i$ ,  $x_{i+1}$ , ... is mapped to its  $k$ -dimensional embeddings using word embeddings. A new feature is produced using convolution. Max-over-time pooling is applied over the feature map to capture the most important feature value. The final prediction is made by computing a weighted combination of the pooled values and applying Softmax activation function.
5.6 Recurrent Neural Networks (RNN)
RNNs (Rumelhart et al., 1986) are designed to handle sequential data, such as text, where the
Embeddings trained on a 4 core Intel i7-6700K CPU @ 4.00GHz with 64GB of RAM. Average time: <30 minutes.
<sup>4</sup>https://translate.google.com/
<sup>5</sup>https://azure.microsoft.com/en-us/
services/cognitive-services/Translator/
data contains complex temporal dependencies and hidden information. Long Short Term Memory networks (LSTM) (Hochreiter and Schmidhuber, 1997) are modified RNNs designed to overcome the issue of vanishing gradient with RNNs. LSTM has a gating mechanism consisting of input gate, forget gate, and output gate, ensuring a constant error flow and avoiding long-term dependency problems. The memory in LSTM is stored in an internal state, and the three gates play a vital role in deciding which information is to be included, added or removed from the memory. Over time, the memory cells learn which information is essential based on the weights. Bidirectional RNNs are widely used extensions where the input sequence is fed from beginning to end and from end to beginning. For BiLSTM (Grave et al., 2018), given there are two LSTM layers, the hidden layer output is split into two - for forward and backwards passes over the input.
5.7 Transformers
BERT (Devlin et al., 2019) is one of the early transformer models that apply bidirectional training of encoders (Vaswani et al., 2017) to language modelling. The 12-layer BERT-base model with a hidden size of 768, 12 self-attention heads, 110M parameter neural network architecture was pre-trained from scratch on BookCorpus and English Wikipedia. The mBERT-base (Devlin et al., 2019) model uses the same pre-training objective as BERT-base and is pre-trained with Wikipedia text of 104 languages with most articles. In this research, we use BERT and mBERT to refer to BERT-base and mBERT-base. It is vital to point out that this research does not pre-train BERT models (both BERT-base and mBERT-base) from scratch or continuously on the very limited available Māori language data. Instead, this research only performs fine tuning on down streaming task. There are evidence on mBERT performance of zero-shot tasks (Keung et al., 2020), and hence the decision to limit this study to only fine-tuning. Pre-training BERT models is out of scope of this current research.
We experiment with various language models and classifiers for two main tasks: language detection (LD) and code-switch detection (CS). Our ultimate goal is to find a combination of language modelling
and NLP techniques to improve the overall accuracy of LD and CS tasks. We use three databases to evaluate these tasks with details provided in Table 1. We use the Hansard database sentences as the primary data for training and testing. All three datasets were pre-processed by lower-casing and using regular expressions to remove punctuation using Python 3.9 library with Pandas data frame. All experimental results are obtained from a random seeds training-testing scheme;  $70\%$  of the shuffled data is used for training, with  $10\%$  for validation and  $20\%$  for testing, and averaged over three runs. The variation of these three independent runs is within a range of  $\pm 0.015$ .
To represent text we use both fastText pre-trained embeddings (see Table 3) and sparse vectors obtained from BOW representations. An overview of code-switch detection using trained models such as BiLSTM and CNN is presented in Figure 1. This diagram is an example to demonstrate the system we used for end-to-end code-switch detection using neural networks. Step 1 includes training and evaluating a neural network. We use the training set of the Hansard database to train the model and use validation loss as the stopping condition to avoid over-fitting. In step 2, we load the trained model and detect languages at the word level on testing data. Once the language detection is done, the points in the sentence where the language labels switch from Maori to English or from English to Maori are marked as code-switch points.
Neural network models presented in this research are implemented using Keras/Tensorflow. Adam (Kingma and Ba, 2015), an adaptive learning rate optimisation algorithm, is used as the optimiser for neural networks. Softmax activation function is used in the output layer of the network. We use a combination of dropout (Srivastava et al., 2014), with a rate of 0.5, and early stopping (Zhang et al., 2017) to avoid over-fitting. We use a maximum length of 250 tokens (or words) for BiLSTM and CNN, and padding for sentences with less than the maximum length. The term tokens and words is used interchangeably in this paper. The embeddings layer is with a dimension of 300. The hidden units of BiLSTM are 128, and the hidden units of one-dimensional convolutions are 128. For both CNN and BiLSTM, categorical cross-entropy is used as the loss function.
We also fine tune pre-trained transformers, BERT and mBERT on the down streaming task
Figure 1: Code-switch detection using neural networks. Example shows 'English' words {Everyone, who, spoke, at, those, meetings, did, so, with} are detected as 'English' and 'aroha' detected as 'Māori'.
of language detection. We use batch size of 16, maximum sequence length of 256 and learning rate of 1e-5. For both BERT and mBERT, the loss and accuracy were reported at each epoch. For both BERT and mBERT, the model converges fast, needing an average of 5 epochs per run.
All evaluations were done using Sklearn metrics<sup>7</sup>. Evaluations using baseline classifiers such as multilingual naive Bayes and LR with BOW and static features from embeddings require CPU only<sup>8</sup> machines and are very quick to train and evaluate. Neural networks require GPU devices<sup>9</sup> for efficient training and testing. The average training time for CNN was 150-180 minutes, and BiLSTM was 300-360 minutes, while BERT and mBERT required 240 minutes per epoch being trained for an average of 5 epochs. The testing time for trained deep learning models is rapid, requiring a few minutes. The code used in this research is made available<sup>10</sup>.
We present overall macro-F1 score and weighted-
F1 score to provide different insights (Toftrup et al., 2021; Khanuja et al., 2020). We also provide F1-scores of each label where appropriate. Macro-F1 provides average per-language results and is equally important to all languages. The weighted-F1 score considers the popularity of the languages in the data set.
The Nemenyi posthoc test (95% confidence level) identifies statistical differences between learning methods. Critical Difference (CD) plots show the average ranking of individual F1 scores obtained using various language models. The lower the rank, the better the model is. The difference in average ranking is statistically significant if there is no bold line connecting the two settings.
The results are presented for the language detection (LD) tasks and code-switch detection (CS) tasks. The language detection task is a crucial first step for detecting code-switching (Rijhwani et al., 2017; Barman et al., 2014). First, we present the results of the language detection tasks using the three databases (Table 1), followed by the results of the code switch task using the Hansard database. As indicated in the experimental setup, all experimental results are obtained from a random seeds training-testing scheme and averaged over three runs. The variation of these three independent runs is within a range of  $\pm 0.015$ .