SlowGuess/NAACL_2022_OCR huggingface dataset:
Gati L Martin $^{1}$ , Medard E Mswahili $^{2}$ , Young-Seob Jeong $^{2*}$ , Jiyoung Woo $^{1}$
<sup>1</sup>Soonchunhyang University, Asan-si, Korea
$^{2}$ Chungbuk National University, Cheongju-si, Korea
{gatimartin, jywoo}@sch.ac.kr
{medardedmund25, ysjay}@chungbuk.ac.kr
The rapid development of social networks, electronic commerce, mobile Internet, and other technologies has influenced the growth of Web data. Social media and Internet forums are valuable sources of citizens' opinions, which can be analyzed for community development and user behavior analysis. Unfortunately, the scarcity of resources (i.e., datasets or language models) has become a barrier to the development of natural language processing applications in low-resource languages. Thanks to the recent growth of online forums and news platforms of Swahili, we introduce two datasets of Swahili in this paper: a pre-training dataset of approximately 105MB with 16M words and an annotated dataset of 13K instances for the emotion classification task. The emotion classification dataset is manually annotated by two native Swahili speakers. We pre-trained a new monolingual language model for Swahili, namely SwahBERT, using our collected pre-training data, and tested it with four downstream tasks including emotion classification. We found that SwahBERT outperforms multilingual BERT, a well-known existing language model, in almost all downstream tasks.
Nowadays, online social networking has revolutionized interpersonal communication. The influence of social media in our everyday lives, at both a personal and professional level, has led recent studies to language analysis in social media (Zeng et al., 2010). Especially, natural language processing (NLP) tools are often used to analyze textual data for various real-world applications; mining social media for information about health (De Gennaro et al., 2020), diseases analysis (e.g., COVID-19 (Gao et al., 2020), Ebola (Tran and Lee, 2016)), identifying sentiment and emotion toward products and services, and developing dialog systems (Zhou et al., 2020). Language models have recently
drawing much attention as they are known to be effective in many NLP tasks (e.g., text classification, entailment, sequence labeling), but they commonly require a huge amount of data for pre-training and fine-tuning; some models are designed for few-shot learning that does not require much labeled data for fine-tuning, though they still require plenty of pre-training data. As it is expensive and difficult to get the labeled and unlabeled data, the majority of the data are in high-resource languages (HRLs) (e.g., English, Spanish). Unfortunately, other than about 20 HRLs languages, approximately 7,000 low-resource languages (LRLs) in the world are left behind, where most of LRLs are spoken and little written (Magueresse et al., 2020). Africa and India are the main hosts of LRLs, where some languages are spoken by more than 20 million people (e.g., Hausa, Oromo, Zulu, and Swahili). As more data on social media in LRLs, qualified datasets, and publicly available language models will bring many advantages in various fields, such as education (Obiria, 2019), healthcare (de Las Heras-Pedrosa et al., 2020), entertainment (Ahn et al., 2013), and business.
Swahili, a Bantu language, is one of the two official languages (the other being English) of the East African countries such as Tanzania (Petzell, 2012), Kenya, and Uganda. It has been widely spread in African countries not only as a lingua franca but also as a second or third language across the African continent and broadly in education, administration, and media. With the rapid development of social networks, electronic commerce, mobile Internet, and other technologies, Swahili is also spreading in online places that result in the growth of Web data. For example, JamiiForum is a popular online platform in Tanzania, and it provides a place to discuss different issues, including political, business, educational, and lifestyle; this means more collected textual data of Swahili is available.
By making use of the online textual data of
Human Language Technologies, pages 314 - 324
Swahili, there are several studies for different tasks (e.g., sentiment classification (Obiria, 2019; Noor and Turan, 2019; Seif, 2016), news classification (David, 2020)). Recently, language models have drawn much attention from the industry and academic world, as the language models brought much better performance (e.g., accuracy) than other existing models. There are few studies that employed the language models for Swahili: named entity recognition (NER) (Adelani et al., 2021) and sentiment classification (Martin et al., 2021). Although these studies have shown successful results, they are limited in that they just borrow the language models (e.g., multilingual Bidirectional Encoder Representations from Transformer (mBERT) (Devlin et al., 2019), Cross-lingual Model-RoBERTA (XLM-R) (Conneau et al., 2020)) pre-trained with other resources (i.e., other languages); in other words, their language models are pre-trained for multiple languages but not dedicated for Swahili. Although such multilingual language models have shown great generalization power across multiple languages, several studies (Bhattacharjee et al., 2021; Tanvir et al., 2021; Vilares et al., 2021) reported that the monolingual models often outperform these multilingual models. There was no study that proposed a monolingual language model for Swahili (i.e., Swahili-specific language model), and the main reason is that Swahili is one of the LRLs, so the existing studies commonly suffered from lack of available data.
In this paper, we focus on the Swahili language. To the best of our knowledge, this is the first study that collects a pre-training dataset and uses it for pre-training of the Swahili-specific language model. We also provide a manually annotated dataset for the emotion classification task. The contributions are summarized as follows.
- Pre-training dataset: we collected Web data from different sources (news sites and social discussion forums) for pre-training the Swahili language model.
- Emotion dataset: we introduce a new Swahili dataset for multi-label emotion classification with six Ekman's emotions: happy, surprise, sadness, fear, anger, and disgust.
- Swahili language model: we pre-trained the Swahili language model and compared its performance with other language models on several downstream tasks (e.g., emotion classifi
cation, news classification, and named entity recognition (NER)).
Most African countries have minority languages that are used by specific ethnic groups (approx. 158 in Tanzania<sup>1</sup>). However they speak different national and official languages of their countries, including native and colonial that can be used in public services such as education, politics, and the media. Swahili is a Bantu language widely spoken in sub-Saharan Africa and acts as the common tongue for most East African (Lodhi, 1993; Amidu, 1995). Many Swahili vocabularies are derived from loanwords, the vast majority from Arabic, but also English, Hindi, Portuguese, and other Bantu languages<sup>2</sup>. As the language grows, new formal and informal vocabularies emerge. The formal vocabularies are used in official documents, whereas the informal vocabularies are mostly used by young adults and on social media platforms (Momanyi, 2009).
Structurally, it is considered an agglutinative language with polysemous features. Its morphology depends on prefixes and suffixes which are syllables (Shikali et al., 2019). A single word is generated with morphemes (i.e., stem, prefixes, and affixes) that will have corresponding inflectional forms. Nouns are divided into classes on the basis of their singular and plural prefixes. Despite its popularity, a limited amount of textual data is available and it is one of the low-resource languages (LRLs). Although there have been a few studies that illustrate the value of NLP (Martin et al., 2021; Obiria, 2019; Gelas et al., 2012), they commonly suffered from the lack of available data.
2.1 Existing dataset of Swahili
To overcome the problem of limited language resource, they have been few datasets for different tasks: new classification dataset, NER dataset, sentiment classification dataset, and emotion classification dataset.
2.1.1 News classification dataset
This dataset  $^{3}$  is created and shared by a data science competition platform Zindi (David, 2020). It contains a total of 23,266 instances collected from different news websites in Tanzania. There are 6
1 https://www.tanzania.go tz/home/pages/228  
2 https://en.wikipedia.org/wiki/Swahili_language  
3 https://zenodo.org/record/4300294?ref=hackernoon.com
categories of news: kitaifa (national), kimataifa (International), biashara (finance), michezo (sports), afya (health), and burudani (entertainment). The amount of each category is 10,242 (national), 1,905 (International), 2,028 (finance), 859 (health), 6,003 (sports), and 2,229 (entertainment); so this dataset is highly imbalanced. Kastanos and Martin (2021) applied a deep learning model called Text Graph Convolutional Network (Text GCN) on this dataset and achieved an F1 score of  $75.67\%$  for the news classification task.
2.1.2 Sentiment classification dataset
Obiria (2019) collected 886 posts from Twitter and Facebook to analyze the student opinion in Kenyan universities and achieved an accuracy of  $83\%$  on binary classification task using support vector machine (SVM) (Hearst et al., 1998). Noor and Turan (2019) extracted 1,087 Twitter texts about demonetization in Kenya and performed ternary sentiment classification with Naive Bayes. They applied various feature extraction methods and obtained an accuracy of  $70.8\%$ . Recently, Martin et al. (2021) used a cross-lingual model, mBERT, to perform binary sentiment classification on a social media dataset that they manually annotated, and achieved an accuracy of  $87.59\%$ . None of the above datasets are publicly available.
2.1.3 Named entity recognition dataset
This dataset, namely MasakhaNER  $^{4}$  (Adelani et al., 2021), is created for ten African languages, including Swahili. The news texts are collected from local news sources and annotated using ELISA tool (Lin et al., 2018) by native speakers of each language. The dataset contains a total of 3,006 instances and covers four entities: personal name (PER), location (LOC), organization (ORG), and date & time (DATE) as inspired by the English CoNLL-2003 corpus (Tjong Kim Sang and De Meulder, 2003). The number of entities of each type is 1,702 (PER), 2,842 (LOC), 960 (ORG), and 940 (DATE). They compared three models (e.g., CNN-BiLSTM-CRF, mBERT, and XLM-R) on the NER task, and mBERT and XLM-R achieved  $89.36\%$  and  $89.46\%$  of an F1 score, respectively.
Over the years, models for word representation have been developed and have shown that they are capable of capturing the semantics and syntactic dependencies between words: Word2Vec (Mikolov et al., 2013), Glove (Pennington et al., 2014), and FastText (Bojanowski et al., 2017). As these models do not incorporate context of words, many context-aware language models based on Transformer (Vaswani et al., 2017) were introduced (e.g., Bidirectional Encoder Representations from Transformer (BERT) (Devlin et al., 2019) and XLM-R (Conneau et al., 2020)). These language models are trainable with a monolingual or multilingual dataset. For example, multilingual BERT (mBERT) is trained with a dataset of 104 languages and a shared vocabulary.
Although mBERT has shown its potential in some previous work, several studies reported some limitations of mBERT especially to LRLs: (1) the limited scale of pre-training data (only Wikipedia was used) (Conneau et al., 2020); (2) the small vocabulary size for specific language (Wang et al., 2019). To overcome that, XLM-RoBERTA modifies mBERT by increasing the amount of pretraining data, which increases the shared vocabulary between different languages. It provides a strong improvement over mBERT, however, it is outperformed by monolingual models (Tanvir et al., 2021; Bhattacharjee et al., 2021) due to better representation of morphological language such as Swahili. This is a good improvement since the model can learn morphological information. Another limitation is the nature of pre-training corpora. Most available corpora are extracted from Wikipedia, Bookcorpus, or news blogs which may not be compatible with the task that covers multi-domain such as social media data. In this work, we collect our data from different sources across several domains for our new language model.
In this section we describe our collected dataset for pre-training and the downstream task of emotion classification. We open our datasets for future use in various studies<sup>5</sup>.
3.1 Pre-training dataset
The existing available corpora for Swahili are very small, for example, the Open Super-large
<sup>5</sup>https://sites.google.com/view/swahbert/home
4https://github.com/masakhane-i/o/masakhane-ner/tree/main/data
Crawled Aggregated coRpus (OSCAR) database contains about 25 megabytes of the corpus. Using crawler tools, we scraped our own data from different sources such as news Web sites, forums, and Wikipedia. The news Web sites include UN news  $^{6}$ , Voice of America (VoA)  $^{7}$ , Deutsche Welle (DW)  $^{8}$  and taifaleo  $^{9}$ . We collected data from JamiiForums, which is one of the most popular social media websites in Tanzania founded in 2006. The forum has provided a discussion platform for the public to discuss different issues, including political, business, educational, and lifestyle. Since JamiiForums is a discussion platform, most of its contents are either passages of information or short comments. We collected the passages with more than four logically connected sentences. We removed URL links, usernames, non-textual content (e.g., HTML tags) and filtered out non-Swahili characters (e.g., Latin, Chinese). The size of dataset is about 105MB with 16M words, where a sentence has an average of 27 subword tokens. Most of these platforms contain data that range from 5 to 10 years. The contribution (in percentage) of each source was taifaleo (39.4), UN news (28.6), JamiiForum (10.2), Wikipedia (9.5), VoA (7.2), and DW (5.1).
3.2 Emotion classification dataset
Existing non-Swahili datasets typically use annotation schemes based on Ekman (Ekman, 1992), Plutchik (Plutchik, 1980) or with multiple categories (Demszky et al., 2020). For example, there are English datasets with multiple emotion categories: Affective Text (Strapparava and Mihalcea, 2007) with 11 categories, CrowdFlower with 14 categories, GoEmotions (Demszky et al., 2020) with 27 categories and others (Oberlander and Klinger, 2018). In this paper, to construct a new Swahili dataset for emotion classification, we chose to use 6 emotion categories from Ekman's (Ekman, 1992) scheme: anger (hasira), surprise (mshangao), disgust (machukizo), joy (furaha), fear (woga), and sadness (huzuni). Our dataset is collected from two source types: social media platforms of Swahili and existing emotion datasets of English. The social media platforms include YouTube, JamiiForum  $^{10}$  and Twitter The conversations and comments on these platforms cover different topics,
Item	Value
# of examples	12,976
# of labels	7 (including ‘neutral’)
# of text per labels	joy: 2,439
disgust: 3,227
anger: 1,772
sadness: 2,339
fear: 1,116
surprise: 2,305
Neutral: 863
# of labels per examples	1: 92.09%
2: 7.45%
3: 0.46%
Ratio of source	Dailydialog: 13.26%
Emotion cause: 13.08%
ISEAR: 2.26%
Social media: 71.40%
Appx. ratio of taxonomy	Politics: 15%
social issues: 80%
pandemic: 5%
Table 1: Statistics of Emotion classification dataset.
such as politics, disease outbreaks, and aspects of daily life. We reviewed the dataset and removed profanity towards a specific person or ethnic group. For example, in the sentence '[name] is very stupid, she doesn't act like a leader at all,' we replaced the target name with pronoun. We also selected three existing English datasets with relevant topic coverage and converted them into Swahili using Google translator. The three datasets include: (1) Dailydialog (Li et al., 2017) that reflects daily communication and covers various topics about our daily life, (2) Emotion Cause (Ghazi et al., 2015), and (3) ISEAR (Scherer and Wallbott, 1994) collected from participants from varying cultural backgrounds who complete questionnaires about their experiences and reactions. The Swahili emotion texts obtained from the Google translator were checked and corrected thoroughly by a native Swahili speaker.
For the dataset collected from forums of Swahili, two native Swahili speakers were assigned to annotate the emotion labels. These speakers agreed to the consent of serving as annotators and were given an instruction of annotation. They were asked to select one or multiple suitable emotion labels that were expressed in the text, and labeled 'neutral' for unsure texts. The statistics summary is presented in Table 1. We also calculated annotator agreement using Cohen's kappa metric, which computes a score of agreement level between two annota
<sup>6</sup>https://news.un.org/
<sup>7</sup>https://www.voaswahili.com/
<sup>8</sup>https://www.dw.com/sw/idhaa-ya-kiswahili/s-11588
<sup>9</sup>https://taifaleo.nation.co ke/
<sup>10</sup>https://www.jamiiforums.com/
tors who each classify N items into C mutually exclusive categories. The scores for each label are joy (0.835), disgust (0.845), anger (0.763), sadness (0.733), fear (0.694), and surprise (0.806). Figure 1 is a heatmap that shows the degree of relationship between emotions. The emotion pair with high intensity (e.g. hasira (anger) and machukizo (disgust)) has a positive correlation in multi-label emotion.
Figure 1: Pearson correlation matrix for the multi-emotion.
4 SwahBERT
With the collected dataset, we pre-trained the monolingual BERT for Swahili, namely SwahBERT  $^{11}$ . The SwahBERT basically has the same architecture as the original BERT. This section describes the process of pre-training and fine-tuning of SwahBERT.
4.1 Tokenizer
In mBERT, not all languages have equal content size (Wu and Dredze, 2020), and some languages are dominated; for example, Swahili is only less than  $1\%$  of the approximately 120K vocabulary of mBERT. Although it might benefit from high resource languages as Swahili has the same typology (word order) and many loanwords, it would definitely be better to generate a Swahili-specific tokenizer. That is, the multilingual tokenizer often splits the words without considering morphological boundaries (e.g., stem, prefixes, and suffixes), like the sentence in Table 2, so the individual subword units do not have a clear semantic meaning. Swahili is morphologically rich language and polysynthetic language; for example, a word alimpikia (cooked
for) has a lexical morph  $\{-pika\}$ , four grammatical morphs  $\{a-, -li-, -m-, -i-\}$  and two in the verb skeletal morphological frame which has the root  $\{-pik-\}$ , and bantu end vowel  $\{-a\}$  (Choge, 2018). In this paper, to incorporate such linguistic complexity, we try monolingual tokenizers for Swahili with different vocabulary sizes (e.g., 32K, 50K, and 70K) using the WordPiece algorithm  $^{12}$ .
SwahBERT has 12 encoder blocks and 768 hidden units. We employ two unsupervised pre-training tasks: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) as described in (Devlin et al., 2019). We conduct experiments by varying the vocabulary size and number of training and warm steps. Following the pre-training process of (Devlin et al., 2019), we pre-trained SwahBERT in two-phases: uncased was firstly trained for 600K steps using an input length of 128, and then further trained for an additional 200K steps using an input length of 512. Cased models were trained for 600K and 900K steps initially, and an additional 200K and 100K steps in the second phase. The batch size is 32 and 6 for the two-phases, respectively, and the parameters were optimized using the Adam optimizer (Kingma and Ba, 2014) with a warmup over the first  $1\%$  of the steps to a peak learning rate of 1e-4.
Table 3 gives the results of pre-training, where it took around 105 hours to complete all phases using two GeForce GTX 1080 Ti GPUs. The best result was obtained with a vocabulary size of  $50\mathrm{K}$  for uncased models, while a vocabulary size of  $32\mathrm{K}$  was the best for cased models. Compared to the mBERT that has a vocabulary size of  $119\mathrm{K}$ , the best vocabulary size of SwahBERT seems small. This is consistent with the vocabulary sizes of other monolingual BERT models; for example,  $32\mathrm{K}$  for English,  $50\mathrm{K}$  for Estonian, and  $30\mathrm{K}$  for Dutch. With the pre-trained models, we put an additional layer on top of the models and fine-tuned them in a supervised way with the labeled datasets for downstream tasks.
We tested our model on downstream tasks and compared with other models. We put an additional linear layer and an output layer on top of the pretrained language models, where all models are im
<sup>11</sup>https://sites.google.com/view/swahbert/home
$^{12}$ https://github.com/kwonmha/bert-vocab-builder
Vocabulary	Tokenization
mBERT	wa ##nan ##chi wa ##nata ##raj ##ia fur ##sa ke ##dek ##ede
SwahBERT(32K)	wananchi wanatarajia fursa ke ##de #ke #de
SwahBERT(50K)	wananchi wanatarajia fursa kede #ke #de
SwahBERT(70K)	wananchi wanatarajia fursa kedekede
Table 2: Tokenization of the sentence 'Wananchi wanatarajia fursa kedekede' (Citizens expects more opportunity) by using mBERT and SwahBERT tokenziers.
Steps	vocab size	MLM acc	NSP acc	loss
800K	32K (uncased)	73.37	99.50	1.1822
800K	50K (uncased)	76.54	99.67	1.0667
800K	70K (uncased)	73.38	100.0	1.2131
800K	32K (cased)	76.94	99.33	1.0562
1M	32K (cased)	73.81	98.17	1.2732
Table 3: Accuracy and loss of pre-training.
tasks	Total	Train	Development	Test
Emotion	12,976	9,732	1,297	1,947
News	23,266	18,612	2,327	2,327
Sentiment	7,107	5,330	710	1,067
NER	3,006	2,104	300	602
pleted with HuggingFace PyTorch library. During the fine-tuning, the parameters are optimized using the Adam optimizer (Kingma and Ba, 2014) with an initial learning rate of  $5\mathrm{e} - 5$  and  $\epsilon$  parameter of 1e-8. The batch size was set to 32. Table 5 summarizes the averaged F1 scores of language models for different downstream tasks, where all language models are with uncased vocabularies except for SwahBERT<sup>cased</sup>. Except for the NER task, the SwahBERT outperformed the mBERT for all tasks. The statistic of the datasets is summarized in Table 4, where the emotion classification dataset is introduced in this paper. Among the tasks, all models achieved much better performance in the news classification task, and this might be explained by the fact that the data source of this task is online news documents that may have similar characteristics to the pre-training dataset that is collected from online forums. In the following subsections, detailed results of each task will be described, where the best scores were obtained from three independent experiments.
5.1 Emotion Classification
We use our new dataset for this task and split the dataset into training  $(75\%)$ , development  $(10\%)$ ,
Table 4: The number of instances of the datasets of downstream tasks.
tasks	SwahBERT	SwahBERTcased	mBERT
Emotion	64.46	64.77	60.52
News	90.90	89.90	89.73
Sentiment	70.94	71.12	67.20
NER	88.50	88.60	89.36
Table 5: F1 scores  $(\%)$  of language models on downstream tasks, where NER indicates named entity recognition.