FIT BUT at SemEval-2023 Task 12: Sentiment Without Borders - Multilingual Domain Adaptation for Low-Resource Sentiment Classification
Year: 2023

Abstract:
This paper presents our proposed method for SemEval-2023 Task 12, which focuses on sentiment analysis for low-resource African languages. Our method utilizes a language-centric domain adaptation approach which is based on adversarial training, where a small version of Afro-XLM-Roberta serves as a generator model and a feed-forward network as a discriminator. We participated in all three subtasks: monolingual (12 tracks), multilingual (1 track), and zero-shot (2 tracks). Our results show an improvement in weighted F1 for 13 out of 15 tracks with a maximum increase of 4.3 points for Moroccan Arabic compared to the baseline. We observed that using language family-based labels along with sequence-level input representations for the discriminator model improves the quality of the cross-lingual sentiment analysis for the languages unseen during the training. Additionally, our experimental results suggest that training the system on languages that are close in a language families tree enhances the quality of sentiment analysis for low-resource languages. Lastly, the computational complexity of the prediction step was kept at the same level which makes the approach to be interesting from a practical perspective. The code of the approach can be found in our repository 1.

heading: 1 Introduction
text: The goal of sentiment analysis is to determine the attitude or emotion expressed by the writer or speaker. The objective of sentiment analysis is to extract subjective information from text data, such as opinions, emotions, and intentions (Mukherjee and Bhattacharyya, 2013). Sentiment classification is a subtask of sentiment analysis that involves classifying the sentiment of a given text as either positive, negative, or neutral. The analysis of sentiment can be on document, sentence, phrase, or token level.
1https://github.com/KNOT-FIT-BUT/ sentiment-without-borders
The sentiment analysis of low-resource languages is challenging due to the limited availability of annotated data and the scarcity of language-specific resources. SemEval-2023 Task 12 (Muhammad et al., 2023b) and the Aftrisenti dataset (Muhammad et al., 2023a) alleviates this limitation for African languages.
The problem of dataset shift or domain shift can be mitigated by domain adaptation, which is a technique that deals with the fact that datasets may contain samples from different distributions. This issue can arise when certain domains have limited labeling or data, which results in a model failing to generalize well on those domains. Moreover, domain adaptation can improve out-of-distribution generalization and enable targeting of unknown domains (Volpi et al., 2018).
This paper describes an approach that is based on a language-centric domain adaptation with a small version of Afro-XLM-Roberta (Alabi et al., 2022) as a generator model and a feed-forward network as a discriminator. Our results show an improvement in weighted F1 for 13 out of 15 tracks with a maximum increase of 4.3 points for Moroccan Arabic compared to the baseline. In addition, it was observed that the quality of cross-lingual setup improves for unseen languages when language familybased labels are used in adversarial training. We have experimented with token and sequence-level discriminator inputs and found the latter one to be more favorable. Moreover, based on our experimental results, it can be suggested that training
1518
the model on languages that are closely related in the language family tree can enhance the quality of sentiment analysis for low-resource languages using language-centric domain adaptation.

heading: 2 Task Description
text: The goal of the SemEval-2023 Task 12: Sentiment Analysis for African Languages is to enable sentiment analysis research in African languages. The shared task is based on the Afrisenti dataset (Muhammad et al., 2023a) that consists of tweets in 14 African languages (Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Algerian Arabic dialect, Moroccan Arabic/Darija, Swahili, Kinyarwanda, Twi, Mozambique Portuguese, Xitsonga, Tigrinya, Oromo) from 4 branches of language families (Afro-Asiatic, English Creole, Indo-European, Niger-Congo). Each tweet has a positive, neutral, or negative label. The SemEval-2023 Task 12 has 3 subtasks:
1. Subtask A: Monolingual Sentiment Classification (all languages except Tigrinya and Oromo)
2. Subtask B: Multilingual Sentiment Classification (all languages except Tigrinya and Oromo)
3. Subtask C: Zero-Shot Sentiment Classification (Tigrinya and Oromo).
The submissions are ranked by weighted F1 score. Our team participated in all three subtasks with a single model trained on the Subtask B data.

heading: 3 Related Work
text: Previous works in sentiment analysis, including multilingual problems, relied on lexicon-based approaches (Nielsen, 2011), classical ML models with hand-crafted features (Dashtipour et al., 2016), or elder NN architectures (Kim, 2014). Recent approaches include the usage of pre-trained Transformer-based LMs (Sun et al., 2019; Yang et al., 2019).
Previous studies on domain adaptation for sentiment analysis mitigate the problem of domain shift aiming at building a model which is generalized to multiple domains (Ruder et al., 2017; ToledoRonen et al., 2022; Ganin et al., 2016; Guo et al., 2020).
In our work, we leverage a Transformer-based model pre-trained on African languages and adopt
the same perspective in terms of generalization but from a different angle, treating languages and language families as domains. A similar usage of domain adaptation was proposed in Lample et al. (2018) for a two-language machine translation problem. In our study, we extend this languagecentric approach to a multilingual setup and experiment with different options for discriminator input and domain labels.

heading: 4 System Overview
text: In this section, we first describe the main model which is also called the generator model. Next, we discuss variations of domain adaptation applications and options for discriminator input. A highlevel schema of the final system used for submission is provided in Figure 1.

heading: 4.1 Generator Model
text: We utilized the Afro-XLM-Roberta (Alabi et al., 2022) as the generator model. It was created by reducing the vocabulary of the base version of XLM-Roberta (Conneau et al., 2020) from 250K to 70k tokens. This was followed by the multilingual adaptive finetuning adaptation process. The model has 12 transformer layers with a hidden size of 768. It was trained on 17 African languages. However, 6 languages and dialects from the Afrisenti dataset were not seen during the training phase: Algerian Arabic, Moroccan Arabic, Twi, Mozambican Portuguese, Xitsonga (Subtasks A and B), and Tigrinya (Subtask C). In order to experiment faster and obtain ablation study results for all the modifications we considered, a small version of the model was used.

heading: 4.2 Domain Adaptation
text: There are different ways to categorize domain adaptation, such as supervised (source and target labels are available) and unsupervised (unlabeled target data), and model-centric, data-centric, and hybrid approaches (Ramponi and Plank, 2020).
Our work focuses on model-centric domain adaptation with adversarial training, where we modified the loss function of the generator model by adding an adversarial component. Languages and language families were concerned as domains for adaptation and were used as target labels by the discriminator model.

heading: 4.2.1 Adversarial Loss
text: At a high level, each training step consists of two phases. In the first phase, the discriminator model learns to predict the correct domain class based on the latent text representations produced by the generator model. In the second phase, the generator model produces latent text representations which are used to predict sentiment class and to fool the discriminator at the same time. The more discriminator is uncertain in its predictions, the better.
The inspiration to use adversarial training came from Lample et al. (2018). The goal of the approach is to restrict the generator to output text representations from the same feature space for different domains. Then, the classification head responsible for sentiment classification would be able to operate with those representations regardless of the domain. As a domain, we considered two options: language and language family. In addition, a language script could be used as a domain, but we did not experiment in that direction.
Discriminator classifies a single vector x, x ∈ Rgen. hidden size at a time, which can be either (i) one of the latent text representations output by the generator or (ii) an average of the output sequence of latent text representations. Then, it outputs domain class probabilities p(l|x), where l, l ∈ [0, . . . , L] is a domain class. The discriminator is trained to predict the correct domain class l by minimizing the cross-entropy loss:
Ldisc = − 1
N
N∑
i=1
log p(ltrue|xi), (1)
where N is a length of an input sequence. Then, the generator is trained to predict sentiment and to fool the discriminator at the same time. Lample et al. (2018) approach uses a binary classification problem setup and forces the discriminator
to predict an opposite label. The same setup can not be directly applied to multi-class classification. Intuitively, the goal of multi-class adversarial training is to make the discriminator to be equally uncertain in all the classes. Therefore, we consider the discriminator as fooled if produced class probabilities are close to 1L which is used as target probabilities. The adversarial loss would have the following form:
Ladv = − 1
N
N∑
i=1
L∑
j=i
1 L log p(lj |xi) (2)
The objective function of the main model is:
Lgen = Lcls + λLadv, (3) where Lcls is a cross-entropy loss of sentiment classification and λ is an adversarial loss scaling factor.

heading: 4.2.2 Discriminator Input
text: In our paper, we explored the effects of applying domain adaptation at different levels of discriminator input, specifically on the token and sequence levels. Intuitively, our goal is for the generator to produce text representations that are languageagnostic, meaning they are consistent across different input languages. Domain adaptation on the token level should lead to better alignment in the generator latent text representations space. Conversely, applying domain adaptation at the sequence level may indirectly result in an improved generalization in a space of semantic meaning across languages. To investigate these hypotheses, we experimented with both approaches. In order to represent sequence-level we use a vector obtained as an average of token-level latent representations produced by the generator.

heading: 5 Experimental Setup
text: Our experiments focused on Subtasks B and C and used the available multilingual training data from Subtask B. The training, validation, and test datasets were provided by the task organizers and were used as is, without any additional preprocessing. Texts were tokenized by Afro-XLMRoberta pre-trained SentencePiece tokenizer (Kudo and Richardson, 2018)
First, we performed hyperparameter tuning for the generator and discriminator models separately. We tuned the hyperparameters for the generator model using a randomly sampled 25% of the training data while keeping the validation dataset as is. We then repeated the same process for the discriminator model, using the same set of optimized hyperparameters for both models separately. A summary of the hyperparameters space and bestperforming set of parameters can be found in Table 1.
We also prepared a mapping of languages to language families, using an ad-hoc approach. To group languages by family, we went up the language families tree until the Kth level and formed a group of languages that shared a common family. The value of K was empirically set to 5. It is possible that a group can have only one language.
While experimenting with the adversarial training setup, we investigated the effects of domain adaptation on the token and sequence levels, using either language or language family as labels for the discriminator, and varying the discriminator loss scaling factors. We used the same stop criterion for all experiments: if there were no improvement greater than 2 points in weighted F1 during 5 subsequent steps, training was stopped.
In our implementation, we utilized a pre-trained Afro-XLM-Roberta model and tokenizer from the Huggingface (Wolf et al., 2020) repository. Our training pipeline was developed using the PyTorchlightning framework with PyTorch (Paszke et al., 2019) as a backend.

heading: 6 Results and Ablation Study
text: Results were submitted for the model trained with a set of hyperparameters that differ from the presented experimental results. During the preevaluation period, we experimented with generator warmup steps which resulted in better results for Subtask A, almost the same performance for Subtask B, and worse quality for Subtask C. Tak-
ing into account our focus on Subtasks B and C we decided not to use warmup steps in our postevaluation experiments. As a result, the baseline and experiments are based on the same set of hyperparameters, and the submission model was trained on a different set. Results for submission, baseline, and experiments can be found in Table 2.
The baseline and experimental models had a batch size of 32, a generator learning rate of 3e-5 with linear decay, weight decay of 0, and maximum training steps of 150000.
The submitted model had a generator learning rate of 1e-4 with 2000 warmup steps and linear decay. The adversarial loss scaling factor λ was set to 1. Other parameters were preserved the same.
The discriminator had the same hyperparameters in both options and had a hidden layer size of 768 and a learning rate of 3e-4 with linear decay.

heading: 6.1 Adversarial Training
text: Domain adaptation setup beats baseline in all cases except Amharic (am) and Oromo (or) languages. The scores reported in Table 2 for each option represent the mean and standard deviation of three separate runs.
In Subtask C, zero-shot classification, Oromo does not outperform the baseline in contrast to Tigrinya. Interestingly, the former language was seen during the pre-training phase, while the latter one was not. Training data contains the Amharic language which is relatively close to Tigrinya in the language families tree – both are Ethiopic languages from the Afro-Asiatic branch. We hypothesize that domain adaptation in cooperation with having a close language helps in a zero-shot task setup and not having such language leads to faster degradation of performance.

heading: 6.2 Adversarial Loss Scaling
text: Our experimental results suggest that a discriminator loss scale within the [0, 1] range yields the best performance. Conversely, a larger value of 10 often leads to a degradation in quality across most cases.

heading: 6.3 Discriminator Labels
text: We found that using language-level labels outperforms the approach based on language families in the case of supervised learning. However, language families-based labels surpass language-based labels in cross-lingual setup for unseen languages from Subtask C (Table 3). We should note that our
procedure for mapping languages and their families was done ad-hoc, which could affect the results. Further investigations with a more careful mapping and selection of languages for fine-tuning would be interesting for investigation.

heading: 6.4 Discriminator Input
text: Comparison of the systems with sequence and token level discriminator input show that a discriminator operating at the token level performs less favorably compared to one that operates at the sequence level. This observation leads us to formulate a hypothesis: a discriminator operating in a space of semantic meaning would help in achieving better performance.

heading: 6.5 Rankings Comparison
text: The average rank we achieved on all tracks is 20.4 with the highest rank of 9 for Amharic. The results were obtained with a single model trained on the Subtask B data with all the languages. A small version of Afro-XLM-Roberta was used as a model for fine-tuning. Other Afro-XLM-Roberta model sizes as well as other models adapted to African languages were not considered in our study.

heading: 7 Conclusion
text: In conclusion, our paper proposed a languagecentric domain adaptation approach for sentiment analysis of low-resource African languages in SemEval-2023 Task 12. Our method utilized adversarial training with a small version of Afro-XLMRoberta as the generator model, resulting in an improvement of up to 4.3 points in weighted F1 compared to the baseline while maintaining computational efficiency. Additionally, our findings suggested that using language family-based labels in adversarial training can enhance the quality of the cross-lingual setup for unseen languages. We also demonstrated that training the model on closely related languages in the language family tree can improve the quality of sentiment analysis for lowresource languages using language-centric domain adaptation. Our approach has the potential to improve sentiment analysis for African languages and other low-resource languages.

heading: 8 Limitations
text: In this section, we would like to emphasize several limitations of our study. First, the language mapping to language families was done ad-hoc. Other ways to map languages, as well as comparison of performance for different mappings, are kept for future experiments. Second, the discriminator used in our experiments was limited to a feed-forward network. As an alternative, other models such as RNN or transformer decoder could be explored in future studies. Third, our experiments were conducted using a single model, and we did not investi-
gate the behavior of other Afro-XLM-Roberta sizes or other models. Finally, language script was not taken into account in our study. Overall, the listed limitations suggest that further research is needed to fully understand the aspects of domain adaptation application to the domains that are based on language information.

heading: 9 Acknowledgement
text: We would like to express our gratitude to the organizers of SemEval-2023 Task 12 for providing us with the opportunity to participate in this exciting challenge. Their efforts have contributed to advancing research in the field of sentiment analysis for low-resource African languages.

heading: Future Work
text: Other ways to map languages, as well as comparison of performance for different mappings, are kept for future experiments. Second, the discriminator used in our experiments was limited to a feed-forward network. As an alternative, other models such as RNN or transformer decoder could be explored in future studies. Third, our experiments were conducted using a single model, and we did not investi-
gate the behavior of other Afro-XLM-Roberta sizes or other models. Finally, language script was not taken into account in our study. Overall, the listed limitations suggest that further research is needed to fully understand the aspects of domain adaptation application to the domains that are based on language information