RideKE: Leveraging Low-Resource, User-Generated Twitter Content for Sentiment and Emotion Detection in Kenyaǹ Code-Switched Dataset (2024)

Abstract: Social media has become a crucial open-access platform for individuals to express opinions and share experiences. However, leveraging lowresource language data from Twitter is challenging due to scarce, poor-quality content and the major variations in language use, such as slang and code-switching. Identifying tweets in these languages can be difficult as Twitter primarily supports high-resource languages. We analyze Kenyan code-switched data and evaluate four state-of-the-art (SOTA) transformerbased pretrained models for sentiment and emotion classification, using supervised and semisupervised methods. We detail the methodology behind data collection and annotation, and the challenges encountered during the data curation phase. Our results show that XLM-R outperforms other models; for sentiment analysis, XLM-R supervised model achieves the highest accuracy (69.2%) and F1 score (66.1%), XLMR semi-supervised (67.2% accuracy, 64.1% F1 score). In emotion analysis, DistilBERT supervised leads in accuracy (59.8%) and F1 score (31%), mBERT semi-supervised (accuracy (59% and F1 score 26.5%). AfriBERTa models show the lowest accuracy and F1 scores. All models tend to predict neutral sentiment, with Afri-BERT showing the highest bias and unique sensitivity to empathy emotion. 1

heading: 1 Introduction
text: Kenya, reflecting Africa’s extensive multilingual diversity, offers a unique insight into the continent’s rich linguistic heritage, standing as a focal point of language contact, expansion, and diversity. It is home to many languages that bridge its vibrant storytelling, poetry, song, and literature and exemplifies Africa’s linguistic wealth, albeit on a more localized scale. With over 40 languages grouped into Bantu, Nilotic, and Cushitic, Kenya’s linguistic
1https://github.com/NEtori21/Ride_hailing_ project
landscape is diverse and dynamic (Dwivedi, 2014; Carter-Black, 2007; Banks-Wallace, 2002).
Central to linguistic diversity is the co-official language status of English and Kiswahili, with the latter spoken by the majority and enjoying nearequal prominence with English. However, the linguistic equilibrium faces challenges from Sheng, a language that blends English, Kiswahili, and words from other ethnic languages that initially were used in Nairobi Eastlands slums. Sheng emerged as a sociolect among urban youth in the city’s workingclass neighborhoods and has since spread across various social and age groups. Hence, it is an inte-
234
gral part of Kenyan culture, influencing the traditional dominance of English and Kiswahili (Barasa, 2016; Momanyi, 2009; Mazrui, 1995).
In recent years, language diversity has also been mirrored in the urban transportation sector, primarily due to the growth of Ride-Hailing Services (RHS) such as Uber, Bolt, and Little Cab. These services have rapidly transformed from urban novelties to essential components of daily mobility for many Kenyans, connecting remote areas with vibrant urban cities. However, with the entry of global giants like Uber in 2015, followed by Bolt and the local contender Little Cab, this transformation is not just physical; it extends into digital and social media platforms such as Twitter.
Since many languages are spoken across Kenya, each population has its own dialect. Hence, codeswitching is common in these new forms of communication, where speakers alternate between two or more languages in one conversation (Kanana Erastus and Kebeya, 2018; Santy et al., 2021; Angel et al., 2020; Thara and Poornachandran, 2018). Analyzing sentiment and emotions in code-switched language context is critical in the broad natural language processing (NLP) field, for example, creating systems that can predict emotional states from text to speech which can be applied in various use cases, such as measuring consumer satisfaction (Ren and Quan, 2012), natural disasters (Vo and Collier, 2013), marketing strategy (Zamani et al., 2016), e-learning (Ortigosa et al., 2014), e-
commerce(Jabbar et al., 2019) and psychological states (Aytuğ, 2018). However, despite this linguistic richness, African languages remain significantly underrepresented in NLP research (Muhammad et al., 2023a). Although NLP research has made extensive progress and demonstrated broad utility over the past two decades, the focus on African languages has been limited. This disparity is often attributed to the scarcity of high-quality, annotated datasets for these languages.
Recently, researchers (Muhammad et al., 2023a)2 have focused on addressing this challenge by introducing a comprehensive benchmark with over 110,000 tweets across 14 African languages, Swahili among them, and introduced the first Africentric SemEval Shared task (Muhammad et al., 2023b). Various studies have evaluated the performance of state-of-the-art (SOTA) transformer models on African languages, highlighting unique challenges and opportunities (Aryal et al., 2023).
However, research on social media NLP analysis for RHS datasets mainly targets high-resource languages. NLP for low-resource languages is constrained by factors like NLP research’s geographical and language diversity (Joshi et al., 2020). Using pre-trained transformer models, we introduce RideKE, a sentiment and emotion analysis dataset for African-accented English code switched with Swahili and Sheng.
2https://github.com/afrisenti-semeval/ afrisent-semeval-2023
Our dataset contains over 29,000 tweets, each sentiment classified as either positive, negative, or neutral, and emotions classified as frustration, happy, angry, sad, empathy, fear, love, and surprise. The dataset represents one location, Kenya, as shown in Table 1. Our goal is to advance research in low-resource languages.
The experiments in this paper are designed to allow us to answer the following specific questions:
1. How do pretrained language models enhance the detection and representation of Kenyan low-resource languages and accents in modern NLP tools?
2. How does the performance of sentiment and emotion detection varies across different pretrained transformer-based models?
3. How effective are different transformer-based models in performing sentiment and emotion detection on the low-resource (RideKE) dataset using semi-supervised learning?
Our paper makes the following contributions as we address these questions:
• We use semi-supervised learning to classify sentiments and emotions. We compare four SOTA transformer-based models and provide a detailed model performance analysis.
• We contribute a partially curated humanannotated labeled public dataset with over 29,000 tweets from the RHS domain. This is Kenya’s first-ever code-switched sentiment and emotion dataset in the RHS domain. It contributes resources to low-resource areas, which can be used for other analyses.

heading: 2 Literature Review

heading: 2.1 Sentiment Analysis on Social Media
text: Sentiment analysis (SA) emerged as a significant field early in the 2000s (Das and Chen, 2001; Na-
sukawa and Yi, 2003). SA (Dave et al., 2003; Pang et al., 2008) aims to determine the attitudes, opinions, or emotions expressed in text on specific topics or entities (Liu, 2022) and has become an increasingly popular research area. Due to higher user-generated content available on social media, understanding sentiment in text cannot be overstated (Naseem and Musial, 2019).
Diverse strategies to accurately interpret and classify user sentiments have been employed. For example, lexicon-based approaches, like SENTIWORDNET (Baccianella et al., 2010) and AFINN (Nielsen, 2011), used predefined word lists to classify text sentiment. While effective in some applications, these methods often struggled with context and nuance. Rule-based systems (Suttles and Ide, 2013) further enhanced this method by applying contextual rules to detect sentiment nuances, including handling negations (Taboada et al., 2011).
Advancements in Machine learning (ML) (Pang et al., 2002), such as supervised techniques trained on large amounts of labeled sentiment datasets, offer another powerful avenue for SA. Hence, the exploration of semi-supervised methods in SA could leverage unlabelled data to address the challenge of data annotation and labeling (Vo and Zhang, 2015; Hwang and Lee, 2021). Deep learning approaches such as Convolutional Neural Networks (CNN) (Chen, 2015) have significantly advanced SA capabilities. However, SA on social media poses unique challenges compared to more traditional domains due to the informal and conversational nature of the text (Medhat et al., 2014; Naseem and Musial, 2019).

heading: 2.2 Code-Switching on Low-resource
text: Code-switching, the practice of alternating between two or more languages or dialects within a conversation, is particularly prevalent in multilingual communities and has become increasingly visible
on social media platforms (Poplack, 2000; Scotton, 1993; Danet and Herring, 2007). It presents unique challenges and opportunities for NLP (Barman et al., 2014). Most NLP research traditionally focuses on high-resource languages like English, leaving low-resource languages underrepresented (Strassel and Tracey, 2016; Adelani et al., 2021). This gap is more pronounced in African and code-switched languages due to linguistic variability (Adelani et al., 2021). Therefore, highresource language techniques may underperform on low-resource language data (Lewis, 2014). The study in (Lee and Wang, 2015) emphasizes the importance of analyzing emotions in code-switching data. The use of Generative Pre-trained Transformers (GPT) to generate synthetic code-switched data has been proposed to address data scarcity (Terblanche et al., 2024). A recent survey (Winata et al., 2022) revealed that until October 2022, only a few papers from the ACL Anthology and ISCA Proceedings focused on code-switching research in African languages. For South African languages (Niesler et al., 2018; Niesler and De Wet, 2008) the first dataset was presented in 2018. Even though Swahili-English code-switching has been studied in a few papers (Piergallini et al., 2016; Otundo and Grice, 2022), no datasets are available.

heading: 2.3 Transformer-based Pretrained Models
text: Transformer-based architectures (Vaswani et al., 2017), such as BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2018), have gained popularity owing to their effectiveness in learning general representations using large unlabelled datasets (Matthew, 2018) that can further be fine-tuned for downstream tasks (Gururangan et al., 2020; Bhattacharjee et al., 2020). Hence, it has become the foundation for many NLP tasks (Bhattacharjee et al., 2020).
Pretrained language models are trained on large, diverse datasets (Raffel et al., 2020). For example, RoBERTa (Liu et al., 2019) was pretrained on over 160GB of uncompressed text, from BOOKCORPUS (Zhu et al., 2015) and CommonCrawl English dataset (Nagel, 2018). These models learn representations that perform well across various tasks, handling datasets of different sizes from diverse sources while remaining easily understandable (Wang et al., 2019). Examples of a few applications in low-resource include improving speech recognition accuracy (ASR) (Olatunji et al., 2023), machine translation (MT) (Wang et al., 2024) and
SA (Muhammad et al., 2023a).

heading: 3 Methods and Datasets

heading: 3.1 Overview of RideKE Dataset
text: RideKE dataset. as shown in Table 1 and 2, includes a blend of Kenyan-accented English, approx. (70%), with a minority mix of Swahili and Sheng (30%). The dataset includes a total of 29,623 entries across 12 distinct columns. See Table 13 in the Appendix.

heading: 3.2 Data Collection
text: We used a systematic scraping process using the snscrape python library 3 which allows for querying and retrieving tweets based on specified criteria. We targeted three keyword search terms—#UBERKenya, #BOLT-kenya, and #LITTLECAB, from January 2017 to April 2023, capturing not only the tweet texts but also other essential metadata such as user engagement metrics (likes, retweets, replies), user account details (followers, following, tweet counts), and relational markers (hashtags, user mentions). Initially, the data was in a dictionary format but it was later converted to DataFrame using pandas and preserved in a CSV format to ensure reproducibility.

heading: 3.2.1 Geo-based data collection
text: The tweet’s location metadata was crucial in determining the regional focus of our study. We referenced Kenya’s location as shown in Table 3. To ensure uniformity, we used a simple yet effective keyword filtering normalization technique to address location inconsistencies as shown by the diverse representations of Nairobi in the dataset
3https://pypi.org/project/snscrape/l
shown in Table 3. To isolate the relevant tweets, we applied a filter on the user_location field to include only locations mentioning Kenya and discard entries with missing data and all those with no location. We assessed the frequency distribution of different locations using value count function.

heading: 3.3 Language Detection
text: We used langdetect 4 Python library to detect languages within text. It revealed diverse languages, English being the most prevalent, then Indonesian, Swahili and others as shown in Table 10. For the Sheng language, native speakers manually detected the language. We only kept English (code-switched) for our analysis.

heading: 3.4 Data Preprocessing
text: Tweets often feature slang, abbreviations, and nonalphanumeric characters such as hashtags and emojis, contributing to the data’s unstructured nature (Adebara and Abdul-Mageed, 2022). We implemented a refined text preprocessing pipeline to enhance data consistency and accurate analysis. The pipeline standardizes data by converting text to strings, trimming whitespace, lowering case, and expanding contractions to preserve semantic integrity. The text is then normalized by reducing repeated characters, removing punctuation, newlines, and tabs, and then tokenizing.

heading: 3.5 Data Annotation
text: Inspired by (Raffel et al., 2020) established guidelines, we created a set of annotation guidelines for emotion annotations to ensure a standardized and high-quality approach in our labeling efforts, as shown in Table 12. We added a ’frustration’ label and used ’happy’ instead of ’joy.’ For the sentiment
4https://pypi.org/project/langdetect/l
annotation, we adhered to the established annotation framework detailed by (Mohammad, 2016). However, human annotation is time-consuming and costly. We employed two Kenyan volunteer annotators fluent in English, Swahili, and Sheng. One holds a bachelor’s degree in political science and the other in computer science. They received a small token of appreciation for their efforts. We ensured the annotator’s comprehension of the task. Two annotators labeled the same dataset entries to enhance quality. Each labeled 1,554 tweets with sentiment labels (positive, negative, neutral) and emotion labels (sadness, happy, love, anger, fear, surprise, frustration, and neutral).

heading: 3.5.1 Annotation Quality Control
text: We used Cohen’s Kappa (Artstein, 2017)5 as our primary metric for assessing the level of interannotator agreement between the two annotators. It is perfect for categorical items, such as sentiment and emotion labels. Cohen’s Kappa provides a means to compute an inter-rater agreement score that accounts for the probability of random agreement:
κ = Po − Pe 1− Pe
(1)
where Po is the observed agreement, and Pe is the expected agreement by chance.
To assign the final sentiment and emotion label to each tweet, we employed a majority voting method (Davani et al., 2022) to determine the final label of the tweet (Mohammad, 2022). Instances of complete disagreement among annotators were resolved by involving a lead annotator and applying a majority rule rather than omitting them from the dataset. We found a Cohen’s Kappa coefficient of 0.60 for sentiment classification tasks. Cohen’s Kappa score for the emotion annotations is approximately 0.67, which indicates a substantial level of agreement beyond chance and suggests a good degree of consistency in their annotations.

heading: 3.5.2 Data Splits
text: The dataset was split into three sets (A, B, and C) as shown in the dataset division Table 4. We used ChatGPT (Brown et al., 2020) for automatic labeling to augment the training dataset and increase training labels since we had only two human annotators. Set A provided Ground truth labels for initial supervised training. Set B is the test dataset
5https://github.com/zyocum/cohens_kappa
that is manually annotated by human annotators. Set C represented the unlabelled dataset Used in a semi-supervised training loop, with empty rows and duplicates removed, labels standardized and encoded.

heading: 3.6 Semi-supervised Learning Phase
text: Semi-supervised learning (SSL) offers a framework for utilizing large amounts of unlabelled data when obtaining labels is expensive (Chapelle et al., 2006; Learning, 2006) as applied to our case. Research shows SSL improves performance on different machine learning tasks such as text classification and machine translation (Najafi et al., 2019). SSL connects supervised and unsupervised learning by utilizing a small fraction of labelled data alongside a larger pool of unlabeled data to improve learning accuracy. SSL has been widely studied to show effectiveness for a wide range of low-resource applications, such as in text-to-speech synthesis (TTS) (Saeki et al., 2023), speech recognition (Du et al., 2023; Thomas et al., 2013), machine translation(Pham et al., 2023; Singh and Singh, 2022), POS-Taggers (Garrette et al., 2013), and sentiment classification (Gupta et al., 2018). Our work extends the application of SSL to sentiment and emotion classification tasks. We seek to mitigate this limitation by leveraging labeled and unlabeled data to train pretrained models. We used accuracy, precision, recall, and F1 scores to evaluate the models’ performance.

heading: 4 Experiments

heading: 4.1 Models and Architecture
text: We evaluate four transformer-based models in our experiments: DistilBERT (Sanh et al., 2019), a smaller and faster version of BERT; mBERT (Devlin et al., 2018), a multilingual version of BERT trained on 104 languages; XLM-RoBERTa (Conneau et al., 2019), a multilingual model trained on 100 languages with improved performance; and AfriBERTa large (Ogueji et al., 2021), a model specifically designed for African languages to address the unique linguistic challenges in this re-
gion. Each model was trained on supervised and semi-supervised learning on sentiment and emotion classification tasks. The initial supervised training and subsequent semi-supervised fine-tuning were conducted separately for each model.

heading: 4.2 Experimental Setup

heading: 4.2.1 Supervised Learning Phase
text: In supervised training, we utilized the humanannotated, well-curated labeled dataset. We used batches ranging from 16 to 64 depending on the model sizes, optimizing for computational efficiency. A combined categorical cross-entropy loss shown in Figure 3 function, with equal weighting for sentiment and emotion tasks, guided the model toward effective multitasking. We applied a dropout rate of 0.1 for each model to prevent overfitting and enhance generalization. We employed the Adam optimizer, with a learning rate 1e− 5 through 10 epochs of training and monitoring. Initially, the four transformer-based models were fine-tuned on a dataset with 1,189 labeled tweets. We then evaluated the model.

heading: 4.2.2 Semi-supervised Learning Phase
text: Our goal in using SSL is to leverage the vast, unlabeled datasets to mitigate the high cost of human annotations. Following an initial supervised learning phase, each transformer-based model underwent a semi-supervised training loop. In this loop, the models dynamically labeled the unlabeled dataset based on their predictions, generating a pseudolabeled dataset. We employed a dynamic threshold, set at the 75th percentile of the models’ probability predictions across all classes for each batch, to ensure only high-confidence predictions were used for labeling. Samples with predictions below this threshold were excluded to minimize the inclusion of erroneous labels in the training data.
We extended the semi-supervised training loop over 4 epochs, a duration we empirically selected to refine the models’ generalization capabilities without causing performance degradation due to overtraining, as indicated by either worsening or plateauing loss. We carefully chose the hyperparameters to ensure optimal training dynamics and model performance.
We set the learning rate at 1e-5 and dynamically adjusted it using a learning rate scheduler during training to optimize generalization and reduce overfitting. The batch size varied between 16 and 64, depending on the specific transformer model, to en-
sure computational efficiency. We used a combined loss function shown in Figure 3 for sentiment and emotion analysis and applied a dropout rate of 0.1 to prevent overfitting. We employed the Adam optimizer with a learning rate of 1e-5 and no weight decay.

heading: 5 Results and Discussions

heading: 5.1 Sentiment Analysis
text: Table 5 summarizes the performance of all models on sentiment analysis. XLM-R supervised achieves the highest overall performance with an accuracy of 62.5% and an F1-score of 66.7%. This is followed closely with semi-supervised XLM-R, which has an accuracy of 62.1% and an F1-score of 68.3%. However, DistilBERT supervised performance falls behind with an accuracy of 57.8% and an F1-score of 54.6%. On the other hand, mBERT models show consistency between supervised and semisupervised training, maintaining average F1-scores of 59.8% and 59.6%, respectively. AfriBERTa models struggled, with the supervised learning achieving an F1-score of 35.8%, and overall poorest performance across all metrics.
The detailed performance metrics for negative, neutral, and positive sentiment classification are
presented in Table 6. For the negative sentiment, the supervised XLM-R achieves a high F1-score of 69.9%, unlike the semi-supervised AfriBERTa, which has the worst F1-score of 17.4%. In neutral sentiment classification, the supervised XLM-R again excels with an F1-score of 52.6%. For the positive sentiment, the semi-supervised XLM-R stands out with an exceptional F1-score of 77.1%, and the semi-supervised AfriBERTa shows robust performance with an F1-score of 66.3%.

heading: 5.2 Emotion Analysis
text: Table 5 summarizes the performance of all models on emotion analysis. The models generally show lower performance than sentiment analysis. The su-
pervised DistilBERT achieves the highest F1-score of 31%, followed by mBERT semi-supervised, with an F1-score of 29.7%.
Table 7 shows performance for emotion classification across neutral, frustration, and happy. DistilBERT supervised leads in frustration with an F1-score of 40%. All models perform poorly on happy emotion classification. In Table 8, XLM-R supervised leads for anger and love emotions with F1-scores of 69.7% and 48.9%, respectively, but all models struggle with fear emotion. Table 9 shows low performance for sadness and surprise but outstanding performance for empathy with XLM-R supervised, leading with an F1-score of 76.8%.

heading: 5.3 Pretrained Models performance
text: As shown in Figure 4, XLM-R, particularly in its supervised form, consistently outperforms other models across sentiment and emotion analysis tasks. mBERT also performs reliably well in sentiment analysis and some emotion classifications. DistilBERT, while efficient, has limitations in handling a range of emotions. AfriBERTa shows lower performance across most metrics than other models. Despite being tailored to African languages, AfriBERTa models do not perform as well in sentiment and even worse in emotion analysis.

heading: 5.4 Semi-Supervised Performance Analysis
text: The detailed analysis of SSL models reveals mixed outcomes, with clear performance enhancements in certain models and tasks, particularly in sentiment analysis. For example, mBERT’s semisupervised version slightly improved sentiment analysis with an F1-score of 59.8% compared to 59.6% for supervised version. In emotion analysis, mBERT’s semi-supervised version outperformed its supervised counterpart with an F1-score of 29.7% versus 26.5%. The semi-supervised AfriBERTa achieved an F1-score of 36.6% in sentiment analysis, marginally higher than the supervised version’s 35.8%, and scored 15.7% compared to 14.2% in emotion task.

heading: 6 Limitations
text: We acknowledge the subjective nature of sentiment and emotion analysis, which can be influenced by label bias, leading to inconsistencies in labeled
data. We will publicly share our dataset to address this issue and facilitate further study on label bias and annotator disagreement. Secondly, the cost of obtaining labeled datasets, particularly from native speakers, can be challenging. Transformer models, SOTA for sentiment and emotion analysis, require large data and computational resources, which is still challenging in low-resource setting. Lastly, We recognize the ethical considerations of LLM use.

heading: 7 Conclusions and Future Work
text: We presented RideKE, a code-switched dataset from Twitter, with sentiment and emotion labels partially annotated for Kenyan-accented English mixed with Swahili and Sheng. Our semisupervised learning shows mixed results, with clear performance enhancements in certain models and tasks, particularly in sentiment analysis, suggesting its potential to generally enhance model performance. We highlight the benefits of semisupervised learning in improving model performance and reducing data annotation costs.
In the future, we aim to further enhance model performance by expanding the pool of humanlabeled datasets, use other semi-supervised approaches, utilizing techniques like few-shot learning, and experimenting with different model architectures and hyperparameters tuning.

heading: Acknowledgments
text: We thank the volunteer annotators who dedicated their time and expertise to this project, which would not have succeeded without their commitment.

heading: A Appendix
text: A.1 Language Detection
A.2 Tweets Per Location
A.3 Sheng-to-English Sample Sentences
A.4 Annotation Guidelines
A.5 Sample dataset structure

heading: Future Work
text: In the future, we aim to further enhance model performance by expanding the pool of humanlabeled datasets, use other semi-supervised approaches, utilizing techniques like few-shot learning, and experimenting with different model architectures and hyperparameters tuning