RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Limitations
This research focuses on general sentiment analysis, and we acknowledge that aspect-based sentiment analysis is a more fine-grained and expressive way of capturing sentiment, that warrants further exploration. Unfortunately, due to the scarcity of relevant datasets in low-resource languages, and task complexity, we were unable to explore aspect-based sentiment analysis in this work.

Regarding the proposed technique, we acknowledge four notable limitations. Firstly, due to the distinct nature of training (i.e., lexicon-level) and inference (i.e., sentence-level), our model may lack sensitivity to semantic complexity at the sentence level, encompassing nuances such as negation and sentences conveying multiple sentiments. One way to address this is to expand the NRC-VAD lexicon to include phrases, metaphors, culturally relevant words, and syntehtic sentences derived from the lexicon. Secondly, our lexicon-based pretraining is solely based on valence scores, and there is an intriguing avenue to explore the inclusion of dominance and arousal scores. Thirdly, the use of machine translation systems for translating lexicons may introduce errors in both translation and sentiment scoring. While translating lexicons is arguably less complex than translating entire sentences, a comprehensive error analysis of the translated lexicons and Panlex words could offer valuable insights into the quality of the additional lexicons. Fourthly, our filtering method (Figure 2 ) proves less effective in certain scenarios due to its English-centric nature. This limitation arises because the initial filtering model is exclusively trained using English lexicons. To enhance this method, we propose that incorporating lexicons manually annotated in more diverse languages could significantly improve its efficacy.
Ethical Considerations
When conducting sentiment analysis in low-resource languages, there are several important considerations that warrant reflection. First, it is crucial to ensure that the work benefits the local community rather than solely exploiting the language. In the era of large language models, the lack of computing resources often hinders the deployment of such systems in regions or countries where the language is spoken. Secondly, sentiment analysis can be subject to cultural ambiguity. Relying solely on European-centric multilingual models for sentiment prediction may introduce biases and produce inappropriate model predictions in certain cultural contexts. Therefore, cultural sensitivity and awareness are essential factors to address when conducting sentiment analysis in low-resource languages, which we leave for future work.
Figure 1: Left : pretraining with a multilingual sentiment lexicon. Right : zero-shot inference using sentences or documents.; Left : pretraining with a multilingual sentiment lexicon. Right : zero-shot inference using sentences or documents.
Figure 2: Lexicon filtering pipeline.; Lexicon filtering pipeline.
Figure 3: Average zero-shot performance of seen and unseen languages in binary classification across different models.; Average zero-shot performance of seen and unseen languages in binary classification across different models.
Languages used in this paper. “✓” (green) and “✗” (red) mean that the language has and has not been seen by the models or language resources. CW indicates code-switching text. The language coverage for GPT-3.5 is derived from GPT-3 Brown et al. ( 2020 ) .
Statistics of the original NRC-VAD lexicon, translations, Panlex extension, and filtering.
Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Full zero-shot results. The underlined score indicates the highest performance within the respective group, while scores in bold indicate the best global performance. “HM-R” = high/medium-resource languages, excluding English, “CS” = code-switched text, and “ML Lex.” indicates the multilingual NRC-VAD lexicon. “SST (sentence-level data)” is cross-lingual zero-shot transfer that is trained on English sentence-level sentiment data.
Regression vs. classification in lexicon-based pretraining for zero-shot sentiment analysis.
Lexicon-based pretraining performance (macro-F1) over stance detection, hate speech detection, and emotion classification. The results are based on the limited training data scenario.
Total lexemes added from Panlex. For min we additionally extend the lexicon with a bilingual min--id lexicon from Koto and Koto ( 2020 ) .
With the exception of GPT–3.5 Ouyang et al. ( 2022 ) , all models used in this study were sourced from Huggingface Wolf et al. ( 2020 ) .
All binary classification results for high/medium-resource languages and code-switched texts.
All binary classification results for NusaX.
All binary classification results for the 14 African languages from SemEval 2023.
All 3-way classification results for high/medium-resource languages and code-switched texts.
All 3-way classification results for NusaX.
All 3-way classification results for the 14 African languages from SemEval 2023..
Datasets used in our additional experiments.
Lexicon-based pretraining performance (macro-F1) in six other English semantic tasks. The results are based on the limited training data scenario.