SlowGuess/NAACL_2022_OCR huggingface dataset:
and test (15%) sets. As shown in Table 5, there is an improvement of  $3.94\%$  F1 score from SwahBERT (64.46) compared to mBERT (60.52). The model exhibits the best performance on emotions like joy (0.80), sadness (0.71), and surprise (0.68), as exhibited in Table 6; this is consistent with the fact that these emotions have a lower correlation with other emotions, allowing the models to more easily classify them. For the neutral (0.25) case, we found that there were many instances of incomplete or uncertain expressions, and this caused confusion with other emotions. This is reasonable as 'neutral' might not even exist because people are always feeling something (Gasper et al., 2019). For example, 'Hivi aliyekudanganya hivyo nani?' (who lied to you that anyway?) was predicted as disgust, while 'Nani aliyekudanganya?' (who lied to you?) was classified as neutral. As mentioned in (Ohman et al., 2020), such uncertain texts are usually not self-contained since they are reactions to other posts; that is, the emotion will be different whether we consider its context information or not.
Post: Vifo vya Corona kila kukicha (Corona cases increases everyday) [sadness], <sadness>
Com1: Acha tuu (Yeah...) [sadness], <neutral>
Com2: Kila mtu anaongea lake.. (Everyone is quick to talk what they wish) [disgust], <neutral>
Com3: Popote, mimi na barakoa yangu (anywhere, with my mask) [fear], <neutral>
labels	SwahBERT	mBERT
P	R	F1	P	R	F1
joy	0.88	0.73	0.80	0.74	0.71	0.72
anger	0.61	0.43	0.51	0.70	0.32	0.44
sadness	0.68	0.74	0.71	0.69	0.65	0.67
disgust	0.61	0.61	0.61	0.54	0.56	0.55
surprise	0.73	0.64	0.68	0.74	0.66	0.70
fear	0.65	0.61	0.63	0.65	0.56	0.60
neutral	0.30	0.22	0.25	0.33	0.11	0.16
Table 6: Results of emotion classification, where P, R, and F1 indicate precision, recall, and F1 score, respectively.
The annotators made the labels based on the context, whereas the language models predicted labels without the context, and this caused the performance degradation. The example in the box demonstrates how emotions can be affected by contextual information, where emotion with context is represented in [emotion] and emotion of non-context is <emotion>.
5.2 News Classification
We used the existing news classification dataset, and it is split into three sets with a ratio of  $80\%:10\%:10\%$  which is equivalent to 18,612:2,327:2,327 instances. As this dataset has six news categories, this task is a classification on six classes: kitaifa (national), kimataifa (International), biashara (finance), michezo (sports), afya (health), and burudani (entertainment). Table 7 shows the results of SwahBERT and mBERT models. Compared to the existing study of (Kastanos and Martin, 2021) that achieved  $75.56\%$  of F1 score using graph convolutional networks (GCN), we observed the improvement of  $14.06\%$  and  $15.23\%$  F1 scores with mBERT and SwahBERT, respectively. The performance for the 'health' class is relatively lower than others, and the reason might be the data imbalance; the 'health' class has a much smaller amount of instances than other classes, as described in subsection 2.1.1.
5.3 Sentiment Classification
As there is no publicly available dataset for this task, we used our emotion dataset by converting some emotion categories into three sentiment classes: positive, negative, and neutral, where we mapped 'joy' to positive, 'disgust' to negative, and 'neutral' was unchanged. For the neutral class, we extracted additional instances from 'surprise' emo
labels	SwahBERT	mBERT
P	R	F1	P	R	F1
national	0.91	0.92	0.92	0.91	0.91	0.91
sports	0.96	0.97	0.97	0.94	0.98	0.96
entert.	0.89	0.94	0.91	0.85	0.93	0.89
business	0.94	0.85	0.89	0.91	0.82	0.86
Internat.	0.90	0.89	0.90	0.91	0.84	0.88
health	0.50	0.41	0.45	0.47	0.44	0.45
Table 7: News classification results, where P, R, F1 indicate precision, recall and F1-score.
labels	SwahBERT	mBERT
P	R	F1	P	R	F1
negative	0.70	0.70	0.70	0.65	0.69	0.67
positive	0.82	0.83	0.82	0.75	0.82	0.79
neutral	0.59	0.60	0.59	0.58	0.48	0.53
Table 8: Sentiment classification results, where P, R, F1 indicate precision, recall, and F1 score.
tion because 'surprise' can be mapped mid-way of negative and positive (Marmolejo-Ramos et al., 2017). We split the dataset into three sets with ratio  $75\% : 10\% : 15\%$  equivalent to 5,330:710:1,067. Results are presented in Table 8. We found that SwahBERT outperformed the mBERT with a gap of  $3 - 6\%$  of F1 scores. As we observed in the results of the emotion classification task, the overall performance of sentiment classification task for the 'neutral' class was much lower than the other classes.
5.4 Named Entity Recognition
We used MasakhaNER (Adelani et al., 2021) dataset for this task, and it has  $70\%:10\%:20\%$  ratio for training, development, and test set. As shown in Table 5, we did not observe performance improvement of SwahBERT against the mBERT of (Adelani et al., 2021). The biggest reason of this will be the small size of the dataset compared to other downstream tasks, as shown in Table 4. That is, the small NER dataset was not enough for SwahBERT to learn the underlying patterns for NER task, so there was no performance improvement compared to the multilingual language model. We believe that the NER performance of SwahBERT will increase as we keep gathering more NER data.
We constructed two datasets for the low-resource Swahili language: for the downstream task of emotion classification and pre-training of the new
Figure 2: Histogram of downstream task datasets, where the x-axis represents vocabulary words of SwahBERT.
Swahili-specific BERT model. For pre-training purposes, we managed to collect a corpus of about 105 MB. Although the size of our corpus is quite smaller than that of rich-resource languages (e.g., English), the pre-trained SwahBERT has shown great improvement on the downstream tasks. This result is consistent with other previous studies. For example, Micheli et al. (2020) found that well-performing language models can be obtained with a little size of corpora of 100MB. Similarly, in (Martin et al., 2020), experimental results with the language models pre-trained with the 4GB dataset were comparable to those pre-trained with 138GB dataset. However, we believe that plenty of qualified datasets will help to increase the power of language models.
As demonstrated in the experimental results, SwahBERT is generally superior to mBERT in almost all downstream tasks. We believe that our tokenizer with Swahili vocabulary has the biggest contribution to the results. The tokenizer of mBERT works by sharing vocabulary over multiple languages, and this tokenizer tends to split the words without taking into account morphological boundaries (e.g., stem, prefix, and postfix), as shown in Table 2, even though Swahili is a morphologically rich language. The tokenizer of SwahBERT accommodates most single words (e.g., wanatarajia (excepts), fursa (opportunity)) as one and thus helps the model to get better representation.
We examined the characteristics of the datasets by frequency histograms of vocabulary words in the same order as depicted in Figure 2. The emotion
Tasks	Cosine similarity
News	98.616
NER	52.465
Emotion	84.445
Sentiment	81.543
Table 9: Similarity scores between pre-training dataset and datasets of downstream tasks.
classification dataset and the sentiment classification dataset have a similar curve of histogram, and the NER dataset and the news classification dataset seem similar to each other. This explains that the language models (e.g., SwahBERT, mBERT) achieved similar performance (e.g.,  $88.5\%$  to  $90.9\%$  F1 scores) for the news classification and NER tasks, and similar performance (e.g.,  $60.52\%$  to  $71.12\%$  F1 scores) for the emotion classification and sentiment classification tasks. Another interesting point is that SwahBERT showed no improvement in the NER task compared to mBERT. The main reason for this, of course, is the small amount of NER dataset, but we further examined more details by similarity scores between the pre-training dataset and downstream task datasets, as shown in Table 9. The similarity score is computed using a cosine similarity function on word frequencies in datasets. Note that the NER dataset has a much lower score than others, which implies that language models have a smaller chance to learn linguistic patterns for the NER task. This can be resolved if we collect more data for the NER task. We also believe that collecting more qualified data
for different tasks in low-resource languages (LRL) will significantly contribute to various future NLP applications (e.g., social network services, news recommendations, etc.).
In this study, we introduced our pretraining corpus and annotated dataset for the emotion classification task. The emotion classification dataset contains 7 emotion classes, including neutral, and has approximately 13K instances. We also performed pretraining of monolingual BERT for Swahili, namely SwahBERT, and experimentally compared it with the multilingual BERT (mBERT). The SwahBERT outperformed the mBERT in almost all downstream tasks, where the downstream tasks include emotion classification, news classification, sentiment classification, and NER. Although SwahBERT exhibited superior performance with a relatively smaller pre-training corpus, a more qualified pre-training corpus will definitely contribute to the development of better language models. Therefore, with the growth of the digital platforms for Swahili, we will continue to use the available sources, including native Swahili speakers as annotators, and collect more data from different domains. We hope that this study will facilitate the development of other methodologies and pre-trained language models (e.g., XLM-R) and also aid in social services (e.g., user emotion analysis on forum texts).
The two native Swahili annotators are the authors of this paper, and they receive fair compensation from the project fund. We collected data from a few sources, and all annotated datasets are in Swahili language. We anonymized all texts and confirm that our datasets are allowed to be disclosed for academic purpose according to policies and laws (e.g., UN news $^{13}$ , Voice of America $^{14}$ , Deutsche Welle $^{15}$ , and taifaleo $^{16}$ ). We gratefully acknowledge a favor of every institute or company that approved of a generous data policy for academic purpose.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(NRF-2020R1I1A3053015). This work was supported by Chungbuk National University BK21 program(2022).
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobias Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021. MasakhaNER: Named Entity Recognition for African Languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
JoongHo Ahn, Sehwan Oh, and Hyunjung Kim. 2013. Korean pop takes off! social media strategy of korean entertainment industry. In 2013 10th International Conference on Service Systems and Service Management, pages 774-777.
Assibi Apatewon Amidu. 1995. Kiswahili: People, language, literature and lingua franca. Nordic Journal of African Studies, 4(1):104-123.
Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin, Md Saiful Islam, M. Sohel Rahman, Anindya Iqbal, and Rifat Shahriyar. 2021. Banglabert: Combating embedding barrier in multilingual models for low-resource language understanding. CoRR, abs/2101.00204.
Susan C Choge. 2018. A morphological classification of kiswahili. Kiswahili, 80(1).
<sup>13</sup>https://www.un.org/en/about-us/terms-of-use
$^{14}$ https://www.voanews.com/p/5338.html
15https://www.gesetze-im
internet.de/englisch_urhg/english_urhg.html
<sup>16</sup>https://www.nationmedia.com/terms-of-use/
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451.
Davis David. 2020. Swahili: News classification dataset.
Mauro De Gennaro, Eva G Krumhuber, and Gale Lucas. 2020. Effectiveness of an empathic chatbot in combating adverse effects of social exclusion on mood. Frontiers in psychology, 10:3061.
Carlos de Las Heras-Pedrosa, Pablo Sánchez-Núñez, and José Ignacio Peláez. 2020. Sentiment analysis and emotion understanding during the COVID-19 pandemic in Spain and its impact on digital ecosystems. International Journal of Environmental Research and Public Health, 17(15):5542.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. "GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-4054. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169-200.
Junling Gao, Pinpin Zheng, Yingnan Jia, Hao Chen, Yimeng Mao, Suhong Chen, Yi Wang, Hua Fu, and Junming Dai. 2020. Mental health problems and social media exposure during COVID-19 outbreak. Plos one, 15(4):e0231924.
Karen Gasper, Lauren A Spencer, and Danfei Hu. 2019. Does neutral affect exist? how challenging three beliefs about neutral affect can advance affective research. Frontiers in Psychology, 10:2476.
Hadrien Gelas, Laurent Besacier, and François Pellegrino. 2012. Developments of swahili resources for an automatic speech recognition system. In Spoken Language Technologies for Under-Resourced Languages.
Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In CICling (2), pages 152-165.
Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18-28.
Alexandros Kastanos and Tyler Martin. 2021. Graph convolutional network for swahili news classification. arXiv preprint arXiv:2103.09325.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Ying Lin, Cash Costello, Boliang Zhang, Di Lu, Heng Ji, James Mayfield, and Paul McNamee. 2018. Platforms for non-speakers annotating names in any language. In Proceedings of ACL 2018, System Demonstrations, pages 1-6. Association for Computational Linguistics.
Abdulaziz Y Lodhi. 1993. The language situation in africa today. Nordic Journal of African Studies, 2(1):11-11.
Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. *ArXiv*, abs/2006.07264.
Fernando Marmolejo-Ramos, Juan C Correa, Gopal Sakarkar, Giang Ngo, Susana Ruiz-Fernandez, Natalie Butcher, and Yuki Yamada. 2017. Placing joy, surprise and sadness in space: a cross-linguistic study. Psychological Research, 81(4):750-763.
Gati L Martin, Medard E Mswahili, and Young-Seob Jeong. 2021. Sentiment classification in swahili language using multilingual bert. arXiv preprint arXiv:2104.09006.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219. Association for Computational Linguistics.
Vincent Micheli, Martin d'Hoffschmidt, and François Fleuret. 2020. On the importance of pre-training data volume for compact language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7853-7858, Online. Association for Computational Linguistics.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Clara Momanyi. 2009. The effects of 'sheng' in the teaching of kiswahili in Kenyan schools. Journal of Pan African Studies.
Ibrahim Moge Noor and Metin Turan. 2019. Sentiment analysis using twitter dataset. *IJID (International Journal on Informatics for Development)*, pages 84-94.
Laura Ana Maria Oberländer and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2104-2119.
Peter Obiria. 2019. Swahili text classification using support vector machine and feature selection to enhance opinion analysis in kenyan universities. PAC University Journal of Arts and Social Sciences, 2(2).
Emily Ohman, Marc Pamies, Kaisla Kajava, and Jörg Tiedemann. 2020. Xed: A multilingual dataset for sentiment analysis and emotion detection. In The 28th International Conference on Computational Linguistics (COLING 2020).
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Malin Petzell. 2012. The linguistic situation in tanzania. Moderna språk, 106(1):136-144.
Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In *Theories of emotion*, pages 3-33. Elsevier.
Klaus R Scherer and Harald G Wallbott. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2):310.
Hassan Seif. 2016. Naïve bayes and j48 classification algorithms on swahili tweets: Performance evaluation. International Journal of Computer Science and Information Security, 14(1):1.
Casper S Shikali, Zhou Sijie, Liu Qihe, and Refuoe Mokhosi. 2019. Better word representation vectors using syllabic alphabet: a case study of swahili. Applied Sciences, 9(18):3648.
Carlo Strapparava and Rada Mihalcea. 2007. Semeval-2007 task 14: Affective text. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 70-74. Association for Computational Linguistics.
Hasan Tanvir, Claudia Kittask, Sandra Eiche, and Kairit Sirts. 2021. EstBERT: A pretrained language-specific BERT for Estonian. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 11-19. Linkoping University Electronic Press, Sweden.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Thanh Tran and Kyumin Lee. 2016. Understanding citizen reactions and ebola-related information propagation on social media. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 106-111. IEEE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, volume 30, pages 5998-6008.
David Vilares, Marcos Garcia, and Carlos Gomez-Rodriguez. 2021. Bertinho: Galician bert representations. CoRR, abs/2103.13799.
Hai Wang, Dian Yu, Kai Sun, Janshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 316-327, Hong Kong, China. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130. Association for Computational Linguistics.
Daniel Zeng, Hsinchun Chen, Robert Lusch, and Shu-Hsing Li. 2010. Social media analytics and intelligence. IEEE Intelligent Systems, 25(6):13-16.
Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of XiaoIce, an empathetic social chatbot. Computational Linguistics, 46(1):53-93.