Foul at SemEval-2023 Task 12: MARBERT Language model and lexical filtering for sentiments analysis of tweets in Algerian Arabic
Year: 2023

Abstract:
This paper describes the system we designed for our participation in SemEval-2023 Task 12 Track 6 about Algerian dialect sentiment analysis. We propose a transformer language model approach combined with a lexicon mixing terms and emojis which is used in a postprocessing filtering stage. The Algerian sentiment lexicons were extracted manually from tweets. We report on our experiments on the Algerian dialect, where we compare the performance of MARBERT to the one of ArabicBERT and CAMeLBERT on the training and development datasets of Task 12. We also analyze the contribution of our post-processing lexical filtering for sentiment analysis. Our system obtained an F1 score equal to 70%, ranking 9th among 30 participants.

heading: 1 Introduction
text: AfriSenti-SemEval shared task (Task 12) (Muhammad et al., 2023b) aims at sentiment analysis on African dialectal languages of different African countries. It covers three main tasks and twelve languages with a corpus made of messages collected from the micro-blogging service Twitter (Muhammad et al., 2023a). Our participation is for the first task (Task A): monolingual opinion detection, in the specific case of the Algerian dialect. In case a tweet conveys both positive and negative sentiment expression, the strongest one must be selected for the whole message.
Algerian dialect is considered a less-resourced language for sentiment analysis as few works have addressed the task in the past despite the relatively large number of speakers of the language1 unlike other languages like English (Belbachir and
1Algerian Arabic is spoken by the majority of Algerian population, counting around 32M speakers of the dialect. Source: https://www.visualcapitalist. com/100-most-spoken-languages/
Boughanem, 2018; Pak and Paroubek, 2010). Various works on opinion detection have addressed Arabic in the past (Mohammad et al., 2018), but most of them concern classical Arabic language the exception of the work of Touileb and Barnes (2021). However everyday life communication uses the dialect. Each Arabic country is characterized by its dialect different from the others by a large number of aspects: phonology, orthography, morphology, lexicon,...(Saâdane et al., 2018). It can be written in different ways as shown as follow Table 1:
The importance of social media in everyday communication and the fact that Algerian Arabic is used by a majority of the population warrants our interest in developing an opinion-mining algorithm able to handle the dialect. To do so we need to solve the problems specific to the dialect, augmented by the peculiarities of the form communication takes in a microblogging service like Twitter for instance:
1. the mixing of words coming from different languages: Arabic, Berber, Turkish, French, and Spanish;
2. the liberty that tweets take with spelling, syntax, and language in general;
3. the relatively important presence of emojis for iconic communication;
4. the lack of translation resources and annotated corpora and the impossibility to use an-
389
notated material in other languages. Even the resources in standard Arabic are of little help for handling dialectal variants.
For our approach, we chose transformers (Devlin et al., 2019) because of their ability to leverage context for semantic analysis and also because they have already been investigated for Arabic dialect sentiment analysis (Fsih et al., 2022). But the lack of Algerian dialect-specific resources needed by this type of approach for training leads us to consider an approach mixing transformers and specific lexicons. Since the pioneering work of Pak (2012), many have provided proof of the utility of emoticons for tweets sentiment analysis in Arabic Refaee and Rieser (2014) and other languages: Felbo et al. (2017), Chen et al. (2018), Choudhary et al. (2018) and Weissman (2022). This is why we complemented our lexicon of Algerian-specific subjective terms (relatively rare but with a strong subjective value) with a second lexicon based on emoticons.
This paper is organized as follows: Section 2 presents methods existing for sentiment analysis and describes the corpus data. Section 3 describes our system and explains our different experiments. Sections 4 and 5 present our experiments and results. Section 6 discusses negative results.

heading: 2 Background

heading: 2.1 State-of-the-Art Method
text: Several approaches have been proposed to address sentiment analysis (positive, negative, and neutral) in Arabic.
Some works are based on a subjective lexicon constructed manually, this lexicon can also be weighted, and each term has a sentiment score (Abdul-Mageed and Diab, 2012; Mataoui et al., 2016). The problem with these approaches is the difficulty of finding the subjective lexicon available. Other works focus on building their lexicon (automatically), they rely on machine translation (Mohammad et al., 2016; Guellil and Azouaou, 2016) they use English sentiment resources like the lexicon2 of Hu and Liu (2004) or like the semantic network SentiWordNet (Baccianella et al., 2010) and then translate the subjective words into Arabic. The problem with these approaches is the difficulty to obtain a correct translation. There are also works that represent a document as a features vector with different representations such as
2http://www.cs.uic.edu/˜liub/FBS/ opinion-lexicon-English.rar
Tf-idf, bag-of-words, or Wod2Vec, and then use different classifiers to return the polarity of the document (El Mahdaouy et al., 2017; Mikolov et al., 2013). Recent deep learning algorithms such as convolutional neural network (Kalchbrenner et al., 2014; Safaya et al., 2020), long short-term memory (Cheng et al., 2016), bidirectional LSTM (Sujana et al., 2020) improved performances for opinion mining tasks because of their ability to better take into account the sentiment word context. The transformer-based BERT approach showed particularly good performances for a wide variety of natural language understanding tasks (Devlin et al., 2019).
All these approaches require large amounts of training data and up to now have been essentially applied to the classical Arabic language, whose linguistic characteristics are more stabilized than the ones of the dialectal variants like the Algerian dialect. Furthermore, special genres like social media (Twitter) suffer from a higher variability of all language aspects than other media. Two points that we had to address when building a solution of opinion mining of tweets in the Algerian dialect, despite the existence of a few works for other dialects (Alharbi et al., 2018) and contributions for Algerian Saâdane et al. (2018), Touileb and Barnes (2021).

heading: 2.2 The Corpus
text: The shared Task corpus comprises tweet text. Three sets of the corpus were accessible for participants: (1) training, (2) development and (3) testing sets. Some statistics of the corpora are provided in Table 2 and Table 3. Table 2 represents the number of tweets in the three corpus and the obtained polarity (positive (pos), negative (neg), and neutral(neu)). Using three types of Emoticons corpus, Table 3 shows the obtained positive (pos), negative (neg), and neutral (neu) polarity.

heading: 3 System Description
text: To detect positive, negative, or neutral sentiments, we developed an approach that uses the contextual language model called MARBERT (AbdulMageed et al., 2021). This model is Arabic-focuses Language Model (LM) exploiting large and massive diverse data. Note that it has been adapted for Arabic dialect de-
tection task (AlKhamissi et al., 2021) on short utterances but not for language understanding. To complement the performance of the language model which we found during our experiments to be under-performing on some Algerian dialect sentiment terms, we constructed two lexicons: one for subjective terms and another for emoticons, both regrouping highly opinionated lexical items identified by hand. The lexicon was used in a post-processing stage of the language model as described in Figure 1. In this section, we detail the three modules of our pipeline: a) filtering, b) language Model, and c) lexical post-processing.

heading: 3.1 Filtering
text: The pre-processing phase impact on the performance of sentiment analysis. The Arabic language is known by the property to have multiple forms of a given letter, for instance, @ , @ , @ , @ , @ are sev-
eral forms for the letter @ alif and also for ð , ð waw
and ya ø, ø . Classic Arabic imposes contextual rules for selections of these letters, while dialectal Arabic variants consider three equivalences classes, the characters being undistinguished inside each class. For our letter normalization, we select an arbitrary letter as a representative of each equivalence class. We Pre-process the data tweet and remove all the @user and RT. We choose to not remove stopwords cause we consider they provide important information for sentiment analysis. For example in: é KBñ»ñ I. k@ áË A K @ ( I don’t like chocolate),
the form áË could be considered as a stop word, but it changes the meaning of the sentence from positive to negative.

heading: 3.2 The MARBERT cased Model
text: We use MARBERT model transformer (AbdulMageed et al., 2021) to detect polarity tweets. We chose this model because it considers a dimension of the Arabic dialect language. The pre-training model’s data includes 6B of Arabic tweets. The data set makes up to 1 GB of text which means 15.6 B tokens. The model use’s the same network architecture as BERTbase with 160 M parameters, 17 steps, and 36 epochs.

heading: 3.3 Lexical post-processing
text: We build our lexicon of subjective terms, according to their respective frequency in the collection. Figure 2 and Figure 3 shows the top 50 positive and negative terms used in tweets. Our hypothesis is to determine the least frequent terms, therefore poorly represented by the model, but carrying a strong opinion. We manually filter the terms to obtain a few opinionated words. For example, words are considered as negatives è XQ k, é «PA ¯, Q £, ñË@ð, èC«ð i J£, AÓ, é¢ÓA ,ù ÒÊª K , and words are considered as
positives . é ÓA K. , é K Q Ó , ñ Ê jÖÏ @ èC«ð , á º Ó èC«ð We proceed in the same way to build the emoticon lexicon. Figure 4 and Figure 5 show the top 20 positive and negative emoticons used in tweets.
Only in the case where MARBERT tags a document as ”neutral”, we then apply our lexicons on the document in a post-processing step and change the tag of the document if the lexical filtering yields results different from the one of MARBERT .

heading: 4 Experiments

heading: 4.1 Training and Validation Data
text: Since we were limited to 100 submissions in total for the validation phase, we decided to create our internal validation set by splitting the original training and development set and conducting several experiments. Thus, our internal training set consists of 1858 annotated tweets, and the internal development set included 207 tweets. The labeled test collection was not provided. all our experiments were done by the train and dev collection; For the standard hyper-parameter setting, we based it on the work of (Alamro et al., 2021). We test only some variations shown in the following table 4.

heading: 4.2 The features influencing sentiment analysis
text: In our study, we test the impact of different language models on opinion detection. We complement our transformer model with a post-processing stage with lexical filtering on the part of the data
that were labeled as neutral by the transformer. We ask ourselves three main questions:
• What is the best language model that represents the opinion using the Algerian dialect?
• Does our two manual subjective lexicons (Algerian terms and emoticons) have a significant impact on opinion detection for tweets?
We answer these questions in the next section.

heading: 5 Results on Official Sets
text: For our system, we obtain an F1 score equal to 0.70 on test data and ranking 9th among 30 participants. Table 5 summarizes the results.
We analyze in more detail the behavior of our model on the positives, negatives, and neutrals tweets classes. For this, we use the train and development collection which is annotated for all our experiments as explained in subsection 4.1.
Table 6 shows the results of the MarBet model. We can observe that our model determines the negative and positive documents better than the neutral ones. This is probably due to the distribution of the number of tweets in the collection.

heading: 5.1 What about Other Language Models ?
text: Regarding the first question, we compare during, the development phase, MARBERT (AbdulMageed et al., 2021) against Arabic-BERT(Safaya et al., 2020), a pre-trained language model for standard Arabic. After the test phase, we confirmed our choice of language model by comparing this time MARBERT against CAMeLBERT(Inoue et al., 2021), a language model for Arabic pre-trained on a dataset that contains a part of dialectal Arabic. For the last question, we look at the impact that the lexicon post-processing has on sentiment detection performance. Interpreting the results tables 8, 7, we can notice that the two models yield more than 72% of the F1 score for the positive and negative classes but produce only around 45% of the F1 score for the neutral documents. Both models perform worse than the MARBERT .

heading: 5.2 What about our lexicons ?
text: Regarding the second question, we introduce our lexicons related to Algerian terms and emoticons; and analyze their impact on sentiment analysis. The goal of this experimentation is to see the role of our lexicons in sentiment analysis.
Table 9 represents the comparison of MARBERT lexicons and MARBERT regarding precision, recall, and F1. We can see that there is an improvement of 0.4%, 0.5%, and 0.4% respectively in precision, recall, and F1 on sentiment detection with
MARBERT lexicons. We can probably improve this score by further enriching our lexicons but we can say that taking the lexicon into account plays a role in sentiment analysis.

heading: 6 Negative Results
text: We use different methods to determine sentiment tweets which did not yield better results than the pre-training language models on Arabic. We used the SVM classifier with a vectorization based on TF-IDF. This reports 2 points drop on the F1 measure. We use the long-short-term memory model (LSTM) which yielded an accuracy of around 60%. We can conclude that the language models trained in Arabic more specifically in dialectal Arabic report better results.

heading: 7 Conclusion
text: In this article, we describe our approach based on a language model and sentiment lexicons that we build manually to detect positive, negative, and neutral documents. Our approach ranks among the top 10 for the Algerian dialect sentiment analysis task. We show that the MARBERT language model performs better than Arabic-BERTor CAMeLBERT. We prove that the introduction of lexical filtering with both Algerian dialectal terms and emoticons as a post-processing step to analysis with MARBERT increases the performance of our system.
We believe that this system can be improved by adding new terms or emoticons to lexicons used in complementing analysis with a BERT-based transformer and by using other combination strategies between the different models. For future work, we plan to improve our system by using other subjective lexicons of different languages.

heading: Acknowledgments
text: I would like to thank Patrick Paroubek, research engineer at LISN lab of CNRS-U. Paris Saclay, for his help. Without his support and valuable contributions, this work would not have been possible.

heading: Future Work
text: For future work, we plan to improve our system by using other subjective lexicons of different languages