RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related Work
We briefly review three subtopics that are pertinent to this work: (1) sentiment lexicons, (2) cross-lingual adaptation for sentiment analysis, and (3) sentiment analysis in low-resource languages.
Sentiment Lexicons
A sentiment lexicon is a curated collection of words and phrases that are classified as bearing positive or negative polarity. Such lexicons have applications in fields including NLP, cognitive science, psychology, and social science Kiritchenko et al. ( 2014 ); Mohammad ( 2018 ) . There are two broad approaches to creating a sentiment lexicon: (1) direct annotation Nielsen ( 2011 ); Baccianella et al. ( 2010 ) — have annotators assign sentiment scores to individual words on a rating scale, typically ranging from - 5 (indicating very negative) to + 5 (indicating very positive), or based on positive/negative categorical labels Liu et al. ( 2005 ) ; and (2) best–worst scaling (BWS: Kiritchenko et al. ( 2014 , 2016 ); Mohammad ( 2018 ) ) — have annotators select the most positive and least positive word from a collection of n words, and infer a sentiment score based on the global rank of all words in the collection. BWS is considered more reliable than direct annotation as it helps mitigate annotator bias when assigning sentiment scores to individual words.

These sentiment lexicons have been constructed predominantly for English, but they also exist for languages such as Indonesian Koto and Rahmaningtyas ( 2017 ) , Arabic Kiritchenko et al. ( 2016 ) , Persian Dashtipour et al. ( 2016 ) , Dutch Moors et al. ( 2013 ) , and Spanish Redondo et al. ( 2007 ) . In this work, we use NRC-VAD Mohammad ( 2018 ) , which is the largest English lexicon (19,965 words) and was built using the BWS method. Existing non-English lexicons are not just limited in size, they were also generally curated using a less reliable method (i.e., direct annotation). We instead use multilingual NRC-VAD lexicons in 108 languages, which is created by the original authors of NRC-VAD via Google Translate. 3 3 3 The approach aligns with the utilization of bilingual lexicons such as Panlex, as demonstrated in Wang et al. ( 2022 ) .
Cross-lingual Transfer in Sentiment Analysis
Most previous studies have primarily focused on cross-lingual adaptation in sentiment analysis by transferring models trained on English sentences x_{i} and sentiment labels y_{i} to other languages. Abdalla and Hirst ( 2017 ) developed a mapper function to convert non-English word2vec embeddings to the English embedding space Mikolov et al. ( 2013 ) . Zhou et al. ( 2016b , a ); Wan ( 2009 ); Lambert ( 2015 ) translated English datasets to several languages, such as Chinese and Spanish, and performed joint training to improve the multilingual capabilities of the model. Fei and Li ( 2020 ) combined sentiment networks with unsupervised machine translation Lample et al. ( 2018 ) , and Meng et al. ( 2012 ); Jain and Batra ( 2015 ) have used unlabeled parallel texts in two languages to learn multilingual sentiment embeddings.
In more recent work, Sun et al. ( 2021 ) used linguistic features such as language context, figurative language, and the lexification of emotional concepts to improve cross-lingual transfer, while Zhang et al. ( 2021 ) introduced a representation transformation technique from source to target languages which requires labeled English and non-English datasets.

Cross-lingual transfer in previous work relies on sentence-level labeled English datasets, and has been evaluated on high/medium-resource languages. In this work, we do not use sentence-level labeled datasets, but solely lexicons, and test our methods on low-resource languages. To the best of our knowledge, our work constitutes the first effort to perform massively multilingual sentiment pretraining using lexicons.
Sentiment Analysis in Low-resource Languages
Most work in sentiment analysis has been applied to high/medium-resource languages, such as English Nielsen ( 2011 ); Baccianella et al. ( 2010 ); Koto and Adriani ( 2015 ) , Chinese Zhou et al. ( 2016b , a ) , Japanese Bataa and Wu ( 2019 ) , and Indonesian Koto and Rahmaningtyas ( 2017 ); Koto et al. ( 2021 ) . There also exists a small body of work on sentiment analysis for low-resource languages. First, NusaX Winata et al. ( 2023 ) is a parallel sentiment analysis dataset that comprises 10 local Indonesian languages, along with
Indonesian and English translations. SemEval-2023 Muhammad et al. ( 2022 , 2023 ) released sentiment analysis datasets for 14 African languages.
In other work, Sirajzade et al. ( 2020 ) annotated Luxembourgish sentences with sentiment labels, and Ali et al. ( 2021 ) built a sentiment lexicon for Sindhi. In this study, we include the low-resource languages of NusaX and the 14 African languages from SemEval-2023 among our test sets.
Background and Problem Definition
Prior research (e.g., Zhou et al., 2016b ; Zhang et al., 2021 ) typically assumes access to sentence-level annotated data in a source language, often English, for zero-shot cross-lingual transfer to a target language. In this work, we define zero-shot as a setting where there is no sentence-level annotated data available in the source or target languages. Instead, we use the multilingual NRC-VAD lexicon Mohammad ( 2018 ) which comprises words \{w_{1},w_{2},..,w_{n}\} manually annotated with valence \{v_{1},v_{2},..,v_{n}\} , arousal \{a_{1},a_{2},..,a_{n}\} , and dominance \{d_{1},d_{2},..,d_{n}\} scores. In this work, we train only over the valence scores v_{i} , and normalize them from a range of [0,1] to [-5,5] .

Valence represents the degree of positiveness-negativeness/pleasure-displeasure and has been demonstrated to have a strong correlation with sentiment classification Mohammad ( 2018 ) . While the valence scores are suitable for regression, we also introduce valence classes \{s_{1},s_{2},..,s_{n}\} that are derived from the valence score v_{i} .
For 3-way classification we set the neutral class to [-1,1) , while we set 0 as the boundary between the positive and negative classes in the binary setting.

As illustrated in Figure 1 , we fine-tune multilingual models Devlin et al. ( 2019 ); Conneau et al. ( 2020 ); Liu et al. ( 2020 ); Xue et al. ( 2021 ) on the parallel NRC-VAD lexicon in 109 languages. We specifically use average pooling over all tokens prior to the regression or classification layer. During zero-shot inference, we used fine-tuned models to predict sentiment labels at the sentence level.
Extending the Lexicon
As shown in Table 2 , the original NRC-VAD lexicon Mohammad ( 2018 ) comprises 19,965 English words, and has been extended to 108 languages by the original author resulting in 2.1M parallel words/phrases. 4 4 4 https://saifmohammad.com/WebPages/nrc-vad.html

In Table 1 , we provide an overview of the languages and datasets used in this paper, categorized into: (1) high/medium-resource languages; (2) NusaX, covering local Indonesian languages (low resource); (3) African languages from SemEval 2023 (low resource); and (4) code-switching texts. The high/medium-resource languages and individual languages present in the code-switching texts are covered by all pretrained models and the NRC-VAD multilingual lexicon. However, for NusaX and the African languages, a considerable number of them are not covered.

Language coverage of the NRC-VAD multilingual lexicons remains limited in 109 languages. Therefore, we opt to extend the NRC-VAD lexicon using the Panlex lexicon, a “panlingual” lexicon containing translation edges between many languages. As shown in Table 1 , only mad and pt-MZ are not covered by Panlex. Specifically, we focus on 15 languages that are not covered by NRC-VAD, and project the sentiment scores from English. Given an English word and its valence score pair (w_{i}^{\text{en}},v_{i}) , we first obtain the translation of w_{i}^{\text{en}} in language L . For each translation word \{w_{i_{1}}^{L},w_{i_{2}}^{L},..,w_{i_{m}}^{L}\} we assign v_{i} as the corresponding sentiment score. In total, we add 20K low-resource lexemes from 15 languages, as detailed in the Appendix (Table 7 ).
Filtering Lexemes
Although translating lexemes is relatively easier and often more accurate than sentences, the sentiment score of the translated lexemes can be misleading because of word sense ambiguity. For example, the English word cottage refers to a small house, while the Indonesian equivalent gubuk “shack” from the English-Indonesian lexicon may have more negative sentiment than cottage .

To address the issue, we implement a filtering strategy illustrated in Figure 2 . Initially, we train the English NRC-VAD w_{i}^{\text{en}} with XLM-R Conneau et al. ( 2020 ) using a regression approach.
The training and validation data are split 80:20, with the model trained to predict the valence score v_{i} based on the input word w_{i}^{\text{en}} . Subsequently, the model is used to predict the valence scores v_{i} of additional lexemes (from the extended lexicons by Mohammad ( 2018 ) and Panlex).
As a result, each word w_{i_{j}}^{L} in the extended lexicon has two valence scores: the original score v_{i} and the XLM score \hat{v}_{i_{j}}^{L} . All lexemes w_{i_{j}}^{L} where the absolute difference |\hat{v}_{i_{j}}^{L}-v_{i}| falls below a specified threshold \alpha are added to the training and validation sets proportionally. This iterative process continues by training over the new extended lexicon until the number of additional words added to the training set becomes less than \beta . 5 5 5 We set the threshold \alpha to 2.5, and \beta to 1000.
Data
As shown in Table 1 , we use 34 languages in binary (positive, negative) and 3-way (positive, negative, neutral) classification scenarios. For binary classification, we simply remove sentences with neutral labels, resulting in a smaller dataset size. The 34 languages are grouped into 4 categories:

high/medium-resource languages , including English ( en : Socher et al. ( 2013 ) ), Arabic ( ar : Alturayeif et al. ( 2022 ) ), Spanish ( es : García-Vega et al. ( 2020 ) ), Russian ( ru : Loukachevitch et al. ( 2015 ) ), Indonesian ( id : Koto et al. ( 2020 ) ), and Japanese ( ja : Hayashibe ( 2020 ) ).

Low-resource languages from NusaX Winata et al. ( 2023 ) , consisting of 10 local Indonesian languages: Acehnese ( ace ), Balinese ( ban ), Batak Toba ( bbc ), Banjarase ( bjn ), Buginese ( bug ), Madurese ( mad ), Minangkabau ( min ), Javanese ( jv ), Ngaju ( nij ), and Sundanese ( su ).

Low-resource African languages , based on the 14 languages of SemEval-2023 Muhammad et al. ( 2022 , 2023 ) : Amharic ( am ), Algerian Arabic ( dz ), Hausa ( ha ), Igbo ( ig ), Kinyarwanda ( kr ), Darija ( ma ), Nigerian Pidgin ( pcm ), Mozambique Portuguese ( pt-MZ ), 6 6 6 Mozambican Portuguese dialect differs in both pronunciation and colloquial vocabulary from standard European Portuguese. Swahili ( sw ), Xitsonga ( ts ), Twi ( twi ), Yoruba ( yo ), Oromo ( or ), and Tigrinya ( tg ). We additionally include Tunisian Arabizi ( aeb ) from Fourati et al. ( 2021 ) .

Code-switching texts , involving English–Spanish ( en – es : Vilares et al. ( 2016 ) ), English–Malayalam ( en – ml : Chakravarthi et al. ( 2020a ) ), and English–Tamil ( en – ta : Chakravarthi et al. ( 2020b ) )
Set-Up
We perform comprehensive evaluation over six multilingual encoder and encoder–decoder pretrained language models: (1) mBERT {}_{\text{Base}} Devlin et al. ( 2019 ) ; (2) XLM-R {}_{\text{Base}} Conneau et al. ( 2020 ) ; (3) XLM-R {}_{\text{Large}} Conneau et al. ( 2020 ) ; (4) mBART {}_{\text{Large}} Liu et al. ( 2020 ) ; (5) mT5 {}_{\text{Base}} Xue et al. ( 2021 ) ; and (6) mT5 {}_{\text{Large}} Xue et al. ( 2021 ) . We evaluate different scenarios, including: (1) lexicon-based pretraining via regression vs. classification; and (2) binary vs. 3-way classification.
Lexicon-based pretraining
We conduct pretraining on the six multilingual pretrained language models using three combinations of multilingual lexicons: (1) NRC-VAD; (2) NRC-VAD + Panlex; and (3) NRC-VAD + Panlex + filtering.
For regression-based pretraining, we use mean square error (MSE) loss, while for binary and 3-way classification, we use cross-entropy loss. Please see the Appendix for detailed hyper-parameter settings and computational resources.
Full Training, Few-shot and Zero-shot
Following the lexicon-based pretraining, we examine its impact on sentence-level sentiment analysis across three scenarios: (1) full training, (2) few-shot (training with limited data), and (3) zero-shot. For the first setting, we fine-tune the model with the complete training and development set of sentence-level sentiment data for each language listed in Table 1 . For the second, we simulate few-shot training by randomly sampling 100 training and 50 development instances. To ensure robustness and account for variability, we repeat the experiment five times using different random seeds, and report the average performance. Please note that these first two settings are our preliminary experiments and we report the average scores of mBERT {}_{\text{Base}} across 34 languages. Our main experiment in this work is zero-shot setting, simulating real-world scenarios for low-resource languages where no sentence-level sentiment data is available. For each of the six models, we present the average score for each language group in Section 4.1 .
Baselines
In both full and few-shot training scenarios, the baseline consists of vanilla models without lexicon-based pretraining. For zero-shot setting, we compare our approaches with (1) models trained on SST datasets Socher et al. ( 2013 ) – a sentence-level English sentiment data; and (2) prompting via LLMs, including BLOOMZ (3B) Muennighoff et al. ( 2022 ) , XGLM (2.9B) Lin et al. ( 2021 ) , and GPT–3.5 (175B) Ouyang et al. ( 2022 ) . 7 7 7 We do not include Llama-2 Touvron et al. ( 2023 ) and Falcon Penedo et al. ( 2023 ) as they are English-centric models. The first baseline is zero-shot cross-lingual transfer, following prior work Abdalla and Hirst ( 2017 ); Zhang et al. ( 2021 ) that used English as the main training language. For robustness, we fine-tuned the models with five different seeds for the first baseline. For the LLMs, we average the results of six English prompts as detailed in the Appendix. We report weighted macro-F1 scores for all experiments.

Work discussed in Section 2 is not suitable as a baseline due to the absence of word embeddings and machine translation systems in low-resource languages. For instance, Abdalla and Hirst ( 2017 ) require word2vec embeddings in the target language, while Zhou et al. ( 2016b , a ); Wan ( 2009 ); Lambert ( 2015 ); Zhang et al. ( 2021 ) rely on sentence-level machine translation. Additionally, Meng et al. ( 2012 ); Jain and Batra ( 2015 ) require unlabeled parallel texts, which are not consistently available for low-resource languages.
Preliminary Results: Full and Few-shot Training
Table 3 shows the average performance of mBERT {}_{\texttt{Base}} when training with full and limited training data at the sentence-level. Here we compare the vanilla multilingual model against four lexicon-based pretraining models, and its extensions (Panlex and Filtering). For each language in Table 1 , we fine-tune the models and measure the macro-F1 score over the test set. For this preliminary experiment, we only use regression in lexicon-based pretraining for the binary and 3-way classification tasks. The results demonstrate that lexicon-based pretraining enhances performance, surpassing vanilla mBERT {}_{\texttt{Base}} in both binary and 3-way classification settings.
The proposed filtering method further slightly improves performance.

The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively. In the full training scenario, the increments are smaller, at only +1.4 and +1.1 . These findings motivate us to further investigate zero-shot settings using all six multilingual models.
Zero-shot Results in High/Medium-Resource Languages
Table 4 presents the averaged zero-shot performance of all models categorized by four language groups. The reported results use regression and classification in lexicon-based pretraining for binary and 3-way classification, respectively.
In the case of high/medium-resource languages (HM-R), English is excluded to ensure a fair comparison with models fine-tuned on the English SST dataset. Overall, we observe that multilingual models fine-tuned on SST tend to perform the best in high/medium-resource languages, with mBERT {}_{\text{Base}} being the exception. It is not surprising to see these multilingual models outperforming the LLMs (the first three rows), as they are specifically fine-tuned on a sentence-level dataset.

Interestingly, we also observe that most of the lexicon-based pretrained models substantially outperform the LLMs. For instance, XLM-R {}_{\text{Large}} outperforms GPT–3.5 and XGLM by +7 and +24.9 in binary classification. In 3-way classification, GPT–3.5 tends to perform better than lexicon-based pretraining, while XGLM and BLOOMZ tend to perform poorly. It’s important to note that our models are significantly smaller in size, and BLOOMZ has been fine-tuned on multilingual sentiment analysis datasets, such as Amazon reviews Muennighoff et al. ( 2022 ) .
Zero-shot Results in Low Resource Languages
For low-resource languages, models fine-tuned on SST (i.e., sentence-level English dataset) underperform lexicon-based pretraining by a wide margin in both binary and 3-way classification settings. Notably, despite its significantly smaller size, lexicon-based pretraining to outperform larger models like BLOOMZ and GPT–3.5. Among the models, mT5 {}_{\text{Large}} achieves the best performance in NusaX and African languages for both classification scenarios, with disparities ranging from +8.5 to +15 and +5 to +7 when compared to LLMs. The impact of incorporating Panlex and/or filtering varies across models, with notable improvements observed for XLM-R {}_{\text{Base}} and mT5 {}_{\text{Large}} .

Expanding the multilingual lexicon with Panlex tends to improve the zero-shot capability for 3-way classification. This can be attributed to the fact that NusaX and African languages have a relatively small number of new lexemes (9.5K and 8.5K, respectively). Moreover, Panlex has English as the primary source language, making it inadequate to capture the diversity of languages in our experiments.

Although adding Panlex with the filtering method showed improvements in the preliminary experiment (see the full training results in Table 3 ), it does not enhance the zero-shot performance in NusaX and African languages. To investigate this, we conducted a manual analysis of 100 randomly-selected samples from the 124K filtered lexemes. We compared the original sentiment scores of the corresponding English lexicon with the predicted scores generated by our filtering model. Upon back-translating the non-English words to English, we found that 63 of the original scores were either correct or better than the predicted scores, 25 predicted scores were better than the original scores, and 12 were incorrect for both. Additionally, we identified 75 unique languages among the 100 samples, indicating that our English-centric filtering might not be effective in improving low-resource languages.
Zero-shot Results in Code-switched Text
Extending the lexicon with Panlex and the filtering method yields the best performance for code-switched text, surpassing LLMs and models fine-tuned on SST. In binary classification, our method achieves an average F1-score that is +24.1 higher than the models fine-tuned on SST, while in 3-way classification, our method achieves F1-scores that are +10 to +20 higher than LLMs, even though the individual languages in our code-switched texts are high-resource (i.e., English, Spanish, Tamil, and Malayalam).
Regression vs. classification in lexicon-based pretraining
In Table 5 we present the average performance across the four language groups to compare the effectiveness of lexicon-based pretraining in regression and classification tasks for both binary and 3-way classification. Our findings indicate that regression performs better for binary classification, while classification leads to better results for 3-way classification. However, regression in 3-way classification presents a challenge when determining the neutral class boundary during inference. In the zero-shot setting, we lack specific data for hyper-parameter tuning, leading us to arbitrarily set the neutral class boundaries to - 1 and + 1. Although this setting works reasonably well for XLM-R, it yields poor performance for mT5. A manual analysis of mT5’s predictions revealed that they tend to cluster around zero.
Performance over unseen low-resource languages
We compute the average results for languages that are completely unseen by all models, including 7 NusaX languages ( ace , ban , bbc , bjn , bug , mad , nij ) and 4 African languages ( dz , ma , pcm , pt-MZ , aeb ). We exclude lexicon-based pretraining with the Panlex extension since its performance is poor for low-resource languages. As a comparison, we include code-switched text as seen languages for the models.
Figure 3 presents the performance of XLM-R {}_{\text{Large}} and mT5 {}_{\text{Large}} , which outperform the LLMs and models fine-tuned on SST. This suggests that the multilingual sentiment lexicon is effective at enhancing language generalization for low-resource languages.
Discussion
Given the positive results, we explore the potential applicability of our methodology to NLP tasks beyond sentiment analysis, offering valuable directions for future research. We examine if the lexicon-based pretraining yields benefits in other semantic tasks, including stance detection Li et al. ( 2021 ) , hate speech detection Vidgen et al. ( 2021 ) , and emotion classification Demszky et al. ( 2020 ) .

For each task, we take two datasets and perform experiments in few-shot training using mBERT {}_{\text{Base}} , following the setup described in Section 4.2 . Instead of using the multilingual lexicon, we use the English NRC-VAD lexicon since all the data is in English. For detailed information about the datasets and results, see the Appendix.
Table 6 shows the average F1 scores on each task, demonstrating that lexicon-based pretraining boosts the performance of vanilla mBERT {}_{\text{Base}} . Particularly noteworthy are the substantial improvements in the emotion classification task, with increments of +16.9 and +5.5 for the binary and original class settings, respectively. These findings highlight the potential of sentiment lexicons for various semantic tasks, particularly in the context of investigating their effectiveness in low-resource languages in future works.
Conclusion
We have demonstrated the efficacy of employing a multilingual sentiment lexicon for achieving multilingual generalization in language model pretraining. Without utilizing sentence-level datasets in any language, we provide compelling evidence of superior zero-shot performance in sentiment analysis tasks for low-resource languages, surpassing the performance of large language models. These findings open up new avenues for research in the realm of low-resource languages, not only for language understanding but also language generation tasks. Our results encourage further exploration and investigation of this exciting research direction.