Introduction
When it comes to under-represented languages, multilingual language models Conneau et al. ( 2020 ); Xue et al. ( 2021 ); Devlin et al. ( 2019 ); Liu et al. ( 2020 ) are often considered the most viable option in the current era of pretraining and fine-tuning, primarily due to the scarcity of labeled and unlabeled training data. However, the limited language coverage of these models often results in poor cross-lingual transfer to under-represented languages Xia et al. ( 2021 ); Wang et al. ( 2022 ) .

Prior work has extended multilingual models Conneau et al. ( 2020 ); Xue et al. ( 2021 ) to other languages by language-adaptive pretraining (i.e., continuing to pretrain on monolingual text) (e.g., Wang et al., 2020 ; Chau et al., 2020 ) and leveraging adapters Pfeiffer et al. ( 2020 ) . However, these language adaptation techniques are not compatible with low-resource languages due to the unavailability of adequate unlabeled monolingual texts.

Lexicons are more readily accessible and offer broader language coverage than monolingual corpora like Wikipedia and the Bible, making them a promising resource for extending multilingual models to under-represented languages. This is because when studying a new language, a lexicon is generally the first resource that field linguists develop
to document its morpho-phonemics and basic vocabulary. Of the 7,000+ languages spoken worldwide, lexicons are available for approximately 70% of them, while mBERT, Wikipedia/CommonCrawl, and the Bible are available for only 1%, 4%, and 23%, respectively Wang et al. ( 2022 ) .

In prior work, Wang et al. ( 2022 ) proposed to use the Panlex translation lexicon Baldwin et al. ( 2010 ) , 2 2 2 https://panlex.org/ to extend the language coverage of multilingual BERT (mBERT: Devlin et al. ( 2019 ) ). They further pretrained mBERT using synthetic texts generated through word-to-word translation, resulting in improvements in named-entity recognition tasks. Drawing inspiration from their work, our study aims to reassess the utility of sentiment lexicons in sentiment analysis tasks, which were once a prominent feature in sentiment analysis prior to the advent of pre-trained language models. Specifically, we seek to answer the following questions: (1) To what extent do sentiment lexicons boost sentiment analysis using pretrained language models? ; and (2) Do multilingual sentiment lexicons improve the multilingual generalizability of sentiment analysis, particularly in low-resource languages?

We chose sentiment classification as the focus of our study for two reasons. First, there is a wealth of sentiment classification datasets across diverse languages, allowing us to conduct experiments across 34 languages, including 6 high/medium-resource languages, 25 low-resource languages, and 3 code-switching language pairs. Secondly, compared to other semantic tasks such as hate speech detection Schmidt and Wiegand ( 2017 ); Röttger et al. ( 2021 ) and emotion recognition Abdul-Mageed and Ungar ( 2017 ); Sosea and Caragea ( 2020 ) , sentiment lexicons have been studied extensively and are well-established in the field.

Cross-lingual transfer in sentiment classification is a prime case of low-resource NLP. However, existing research has predominantly focused on high/medium-resource languages (Gupta et al., 2021 ; Fei and Li, 2020 ; Lample et al., 2018 ) , relying on sentence-level sentiment datasets in English. In this paper, we showcase how models trained on English datasets are suboptimal for low-resource languages, and introduce lexicon-based pretraining that improves multilingual sentiment modeling. Our contributions can be summarized as follows:

Our approach is arguably cost-effective since it relies exclusively on sentiment lexicons, reducing the need for sentence-level sentiment annotation in any language, and sentence-level machine or human translation for low-resource languages, which can be challenging to access.

We continue model pretraining using sentiment lexicons across 109 languages (see Figure 1 ), and demonstrate strong zero-shot performance in low-resource languages, particularly in low-resource languages that are not covered by the multilingual lexicons, and in code-mixing texts that include high-resource languages. Our approach outperforms English models fine-tuned on sentence-level sentiment datasets, as well as large language models such as XGLM Lin et al. ( 2021 ) , BLOOMZ Muennighoff et al. ( 2022 ) , and GPT–3.5 Ouyang et al. ( 2022 ) .

We conduct
comprehensive experiments in two sentiment classification scenarios: binary and 3-way classification. For each scenario, we benchmark two pretraining strategies: regression and classification. Unlike regression, the classification-based approach eases the constraint of determining the neutral class boundary before performing inference in 3-way classification in zero-shot setting.