Sentiment and Emotion Classification in Low-resource Settings
Year: 2023

Abstract:
The popularity of sentiment and emotion analysis has lead to an explosion of datasets, approaches, and papers. However, these are often tested in optimal settings, where plentiful training and development data are available, and compared mainly with recent state-of-the-art models that have been similarly evaluated. In this paper, we instead present a systematic comparison of sentiment and emotion classification methods, ranging from ruleand dictionary-based methods to recently proposed few-shot and prompting methods with large language models. We test these methods indomain, out-of-domain, and in cross-lingual settings and find that in low-resource settings, ruleand dictionary-based methods perform as well or better than few-shot and prompting methods, especially for emotion classification. Zero-shot cross-lingual approaches, however, still outperform in-language dictionary induction.

heading: 1 Introduction
text: Affective computing, including sentiment and emotion classification, has been research focuses inside of the Natural Language Processing (NLP) community for many years (Mohammad, 2016; Poria et al., 2023). This has lead to an incredible number of research directions and papers published on these topics, ranging from rule-based and dictionary-based approaches Turney (2002); Lee et al. (2010); Taboada et al. (2011); Staiano and Guerini (2014), to supervised training of deep learning models (Xu et al., 2019; Barbieri et al., 2022; Samuel et al., 2022) and finally to few-shot and prompting of large language models (Brown et al., 2020; Min et al., 2022; Plaza-del Arco et al., 2022). This also means that a systematic comparison of the benefits and weaknesses of models has not been performed, as each often individual papers compare only against more recent state-of-the-art models, and do not take into account previous approaches.
Like many other research areas in NLP, sentiment and emotion classification are dependent on domain and language-specific training data for optimal performance and this high-quality task-specific data is always in short supply as we apply our models to a constantly evolving set of scenarios.
The objective of this paper is therefore to identify trends in sentiment and emotion classification, especially regarding low-resource settings. As such, we attempt to address the following research questions:
• RQ1: Given a limited number of examples per class (<100), what method currently performs best?
• RQ2: Do these methods suffer domain transfer equally?
• RQ3: How well do these results hold for languages other than English?
To address these questions, we perform experiments1 on 10 sentiment classification datasets and two emotion classification datasets in 8 languages with a number of low-resource approaches. Specifically, we compare dictionary-based methods, rulebased methods, few-shot methods and prompting methods on the English datasets. We simultaneously test the out-of-domain performance for each of the methods that demand training data. Finally, we also perform cross-lingual experiments.
We find that rule- and dictionary-based methods often perform on par with few-shot approaches in low-resource settings, especially on emotion classification and are more robust to domain changes, while prompting similarly provides promising results. Zero-shot cross-lingual approaches, however, still outperform in-language dictionary induction for languages other than English, suggesting that more work could be done in this area.
1Code to reproduce the experiments available at https: //github.com/jerbarnes/low_resource_sa_emo.
290

heading: 2 Related Work
text: Current state-of-the-art models for sentiment and emotion classification are dominated by language models that have been pretrained on large corpora and then fine-tuned for each specific task (Sharma et al., 2020; Barnes et al., 2022). Although ELMO (Peters et al., 2018), BERT (Devlin et al., 2019), and its variants were the first to provide evidence for the usefulness of language modeling as a transfer learning objective, there has since been an explosion and it is somewhat difficult to navigate which current models give the best performance on many datasets.
Besides the fully supervised setup, many larger language models also show signs of being able to learn a task with less data, allowing for non-trivial zero- or few-shot performance. The most common way to achieve this zero or few-shot ability is by prompting a model using a Natural Language Inference model, trained to determine whether a premise is true/false, given a hypothesis. This model can then be applied to new tasks by reformulating the input and labels (Schick and Schütze, 2021; Min et al., 2022).
For few-shot prompting, we can make use of the generative abilities of language models by providing demonstrations input/label pairs and asking for a final label (Brown et al., 2020; Lin et al., 2022). More recently, the results of models trained using instruction tuning suggests that these models generalize well to unseen tasks (Chung et al., 2022).
The same kinds of large language models trained on multilingual corpora also allow for zero-shot cross-lingual transfer, where a model is fine-tuned on a task in a high-resource source language and then tested on an under-resourced language (Pires et al., 2019; Conneau et al., 2020). However, these approaches have rarely been compared to previous dictionary-based methods.

heading: 2.1 Rule and Dictionary-based methods
text: Rule and dictionary-based methods are common for sentiment and emotion analysis, in part due to their simplicity and interpretability. Early work focused on automatically inferring polarity dictionaries for categorizing words (Hatzivassiloglou and McKeown, 1997) or texts (Turney and Littman, 2003; Kamps et al., 2004). Taboada et al. (2011) propose SoCal, one of the most popular rule-based methods for sentiment analysis, which uses a set of dictionaries with sentiment scores for certain parts
of speech (adjectives, adverbs, nouns, intensifiers, and verbs) plus rules for interacting with negation, irrealis, and other sentiment shifting phenomena.
For emotion classification, there has been a good deal of work on creating dictionaries. Mohammad and Kiritchenko (2015) use word-association measures with emotional hashtags to create a large emotion dictionary from social media text, while Mohammad (2018) instead use best-worst scaling to crowdsource an emotion intensity dictionary. Buechel et al. (2016) adapt affective lexicons to historical German texts and use these to characterize emotional trends in various genres of writing across several centuries. Buechel et al. (2020) furthermore develop methods for inducing emotion dictionaries for 91 languages, but do not make use of these dictionaries for emotion classification.
For dictionary induction, Hamilton et al. (2016) propose a method to automatically induce domainspecific dictionaries and show their effectiveness across a number of historical and modern text classification tasks. An et al. (2018) similarly propose a method to create a semantic axis, SemAxis, in an embedding space and successfully create dictionaries for tasks beyond sentiment analysis, despite having small amounts of data available. In this approach, we create an average vector for positive V + and negative V - sentiment by averaging the vectors for seed words from an embedding space, such as Word2Vec or FastText. We can then define the axis vector as the difference of the two:
Vaxis = V + − V -
To use the semantic axis that we have created, we can measure the cosine distance of another embedding and the semantic axis.
score(w)Vaxis = cosine dist(w, Vaxis)
If the score is positive, we can assume the word is positive and vice versa, and expand the positive and negative seed dictionaries to cover all lemmas in the test set, effectively creating a high-coverage dictionary. We then use this dictionary to generate the semantic orientation score of a text.
However, most of these techniques have not been recently compared to what are considered state-ofthe-art models under low-resource settings.

heading: 3 Data
text: In this section we describe the datasets that are used for experimentation. The statistics are shown in Table 1 (see Tables 6 and 7 in the Appendix for further details).
Sentiment datasets As we want to explore how well methods work across a number of domains and languages, we choose to explore binary sentiment classification. We use the binary version of the following datasets, where any strong positive/negative has been mapped to positive/negative and neutral has been removed. Using only binary sentiment classification allows for us to compare across a larger number of datasets and languages.
MPQA: Wiebe et al. (2005) annotate English news wire texts with a complex set of annotation types. We map the polarities to sentences and keep those sentences that contain a majority of one polarity, such that we have only positive and negative sentence-level annotations.
SemEval: The SemEval 2013 Shared Task 2 (Nakov et al., 2013) collected tweets and annotated them as positive, negative, or neutral. We keep only the positive and negative tweets.
OpeNER: Agerri et al. (2013) annotate English and Spanish (among others) hotel reviews for structured and aspect-based sentiment. We use the script from Barnes et al. (2018) to map these to sentencelevel binary sentiment classification. ES is the Spanish data from this dataset.
AR: Nabil et al. (2015) annotate Arabic (both Modern Standard Arabic and various dialects) tweets. We remove the neutral and mixed classes.
DZ: Touileb and Barnes (2021) annotate Northern African Arabizi social media posts for sentiment. In this case, we use the transliterated Arabic script version of the dataset and remove the neutral class.
MT: The data for Maltese (Dingli and Sant, 2016; Cortis and Davis, 2019) comes from the combination (Martínez-García et al., 2021) of two smaller datasets.
DE: The GermEval 2017 Shared Task (Wojatzki et al., 2017) released annotated data for several subtasks on German social media texts. We use the document-level data (task B) and remove mixed and neutral.
EU: Barnes et al. (2018) annotate Basque hotel reviews for structured sentiment. We map these to sentence-level binary sentiment classification, using the script provided with the data.
NO: Velldal et al. (2018) provide a collection of professional reviews from news outlets. We keep the binary document-level data.
Emotion datasets For emotion classification we use the SSEC (Schuff et al., 2017) and EnISEAR (Troiano et al., 2019) datasets. The SSEC dataset reannotates a stance and sentiment dataset of political tweets with crowd-sourced labels for eight emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust). The EnISEAR dataset, on the other hand, crowd sources descriptions of events tied to emotions (anger, disgust, fear, guilt, joy, sadness, shame), as well as how readers perceive these events.
For the SSEC, we separate 583 examples from the training set for development. For EnISEAR, we split the fully labeled data into train (70%), dev (10%), and test (20%). For EnISEAR, we use the crowd sourced annotations for emotion labels, rather than the prior emotion to align with SSEC. For both datasets, we take the view that any number of annotations is valid (the 0.0 strategy in SSEC) and accept any label that has been assigned to an example by at least one annotator.

heading: 4 Experimental Setup
text: In this section, we describe the approaches for three experimental setups (monolingual English sentiment classification, monolingual English emotion classification, and cross-lingual sentiment classification) from most resource intensive to least.

heading: 4.1 Sentiment classification
text: Supervised: To provide an upper-bound of fully supervised in-domain models, we use DistilBERT (Sanh et al., 2019), and RoBERTa base and large (Zhuang et al., 2021). To simulate low-resource scenarios, we train the same models with varying amounts of training data (200, 100, and 20 examples). We finetune these models for 5 epochs, with a learning rate of 2e-5, a weight decay of 0.01, and a batch size of 16 on a single Tesla T4 GPU. We take the best model on the development set for testing.
Few-shot: In this scenario, we assume we have a development set and a limited number of training examples (200, 100, 20). We train the same models in the same way as fully supervised training, but with the reduced training set size. We again take the best model on the development set for testing.
Prompting: In this scenario, we assume we have only a few training examples. We explore few-shot prompting (concretely 2-shot) using two OPT models (Zhang et al., 2022): namely, the 125 million and 1.3 billion parameter versions. We prompt these models by giving them 2 positive and negative examples with the following template (an example from the hotel domain):
(1) I didn’t like the hotel. Label: negative. We loved the hotel. Label: positive. {text}. Label:
We take the first predicted token as the predicted label.
Rule-based: In this scenario, we assume no training data whatsoever. We compare these models with the rule-based SoCal system (see details in Section 2.1). This approach requires a large initial effort to create the rules and dictionaries, but afterward can be applied to new data without retraining.
Dictionary-based: Finally, we also compare simpler dictionary-based approaches which do not include rules, and instead rely on a simpler scoring procedure for each text:
score(text, D) = 1 |D| 1 |text| ∑
w∈text se(w,D)
where D is a sentiment dictionary, either containing a list of words with positive orientation Dpos or negative Dneg, and se is a function that returns 1 if a word w is in D, otherwise 0. The score function therefore returns the average score of a text, normalized by the length of the text and by the length of the dictionary D. To predict the aggregate semantic orientation (positive or negative), we divide the positive score by the negative score
semantic orientation = score(text, Dpos) score(text, Dneg)
If this orientation is greater than a certain λ, we will assume that the orientation is positive and return 1, otherwise we will assume it is negative, and return 0.
We can then use available sentiment dictionaries to estimate the semantic orientation of a text. For all dictionary-based methods, we further preprocess the texts by tokenizing and lemmatizing the text using spaCy.2 For sentiment dictionaries, we use the available HuLiu dictionary (Hu and Liu, 2004), the NRC Hashtag sentiment dictionary (Mohammad et al., 2013), and the MPQA subjectivity and sentiment dictionary (Wiebe et al., 2005).
Dictionary induction: Finally, it is also possible to automatically create a sentiment or emotion dictionary from a small seed dictionary. In this case, we use the SemAxis method (An et al., 2018) with a small seed dictionary of 10 words per class. We limit the expansion of the dictionaries to tokens found in the test set and allow only words which have a cosine ≥ 0.15 to reduce likely noisy.
We compare the use of three embedding spaces to induce the new dictionaries: 200 dimensional GloVe embeddings trained on Twitter data (Pennington et al., 2014), 300 dimensional FastText embeddings trained on Wikipedia data (Bojanowski et al., 2017), and 300 dimensional FastText embeddings trained on Wikipedia and the GigaWord corpus3 (Fares et al., 2017).
2Found at https://spacy.io/. 3These can be found at http://vectors.nlpl.eu/
repository/20/22.zip

heading: 4.2 Emotion classification
text: Supervised and Few shot: Given that both the SSEC and EnISEAR datasets are multi-label, we train the models using a one-vs-all approach, effectively creating a binary version of the dataset for each emotion and training a binary classifier. Like the sentiment experiments, we use DistilBERT, RoBERTa-base, and RoBERTa-large. The training procedure is the same as with sentiment. We perform experiments with 200, 100, and 20 training examples for the few shot experiments.
Prompting: For prompting, we use the Flan T5 models (Chung et al., 2022) (base and large), which are instruction tuned models. We performed initial experiments with the same OPT models used for sentiment analysis, but found that the multi-label nature of emotion classification was better covered using the Flan T5 models. For prompting the SSEC dataset, we use the following template:
(2) What emotions are found in this text (Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, Trust, None)?: {text}
where text is the text to be classified. For EnISEAR, we replace the emotions with anger, disgust, fear, guilt, joy, sadness, and shame. We assume that any mention of these words in the generated text is a predicted label.
Dictionary-based: As emotion classification in the datasets we use is a multi-label task, we cannot use the semantic orientation score as is. Instead, we set a threshold value λ = 1 and predict any label where score(text, Demotion) > λ. This allows for our dictionary-based approach to predict multiple labels.
We use the NRC emotion dictionary as an emotion dictionary (Mohammad and Kiritchenko, 2015), which contains 16,862 entries with annotations for 8 emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust), which were compiled semi-automatically using word–emotion association scores with hashtags.
Dictionary induction: Similarly, we can use an adapted version of the SemAxis method to induce emotion dictionaries. In this setting, we create a semantic axis vector for each emotion we wish to propogate. For example, to create a semantic axis
for ’anger’ Vanger, we create the positive pole vector V +anger by averaging the vector representation of seed words for ’anger’ and the negative pole vector V -anger by averaging the vectors of all other seed words.
Once we have the semantic axis vectors for each emotion, we can expand the original seed dictionaries by taking any word whose vector representation has a positive cosine distance with the semantic axis. As with sentiment, we take a conservative estimate and allow only words which have a cosine > 0.15 to reduce noise.
We then use the same prediction procedure as with the dictionary-based approach.

heading: 4.3 Cross-lingual generalization
text: We also compare zero-shot cross-lingual performance of multilingual large language models (MLLMs), in this case XLM-RoBERTa base and large, to dictionary induction. For the MLLM experiments, we train on one of the three English corpora (MPQA, OpeNER, and SemEval) and test the best model on the English development data on all non-English corpora.
For the dictionary induction experiments, we use the SemAxis method with FastText embeddings (Bojanowski et al., 2017), as these have embeddings available for most languages. For NArabizi (DZ), we use the embeddings trained on modern standard Arabic as a proxy.

heading: 4.4 Evaluation
text: For both sentiment and emotion classification datasets, we evaluate using Macro F1, as the distribution of labels is unbalanced and we are interested in knowing how well the models perform on the less frequent labels as well.

heading: 5 Results
text: In this section we detail the results for sentiment classification, out of domain performance, emotion classification, and cross-lingual transfer.

heading: 5.1 Sentiment classification
text: Table 2 shows the Macro F1 of the sentiment classification approaches on the English datasets (MPQA, OpeNER, and SemEval), as well as the average of all results per each approach.
The fully supervised upper-bound achieves an average F1 of 91.2, showing strong performance for this binary classification task.
In the low-resource scenario, FewShot-200 is the best performing approach (80.9), followed closely by the rule-based SoCal (77.6). The dictionaryinduction methods (61.8) and dictionary-based methods (61.6) achieve quite similar results, followed by prompting (56.9) and the few-shot methods using 100 examples (56.5) and 20 (40.5).
In general the RoBERTa-large model suffers more in the few-shot scenarios, losing 3.4-20 percentage points (pp) compared to RoBERTa-base. For prompting, however, the opposite is true, as the 1.3 billion parameter model performs 21.7 pp better than the 125 million parameter model. This ties in well with research indicating that the size of the language model leads to better few-shot performance (Brown et al., 2020).
Surprisingly, dictionary-based methods perform better than FewShot-100 or prompting large language models. Even more surprising is that inducing a sentiment lexicon from as few as 10 labeled words can outperform careful hand-designing of these dictionaries.
Therefore, revisiting RQ1, we can say for binary sentiment classification, fine tuning a model on as few as 100 examples per class gives competitive indomain performance. For anything less, rule-based methods perform better.
5.2 Out of domain performance of sentiment classification
Unlike prompting and dictionary-based approaches, supervised and few-shot methods are tied heavily to the domain they are trained with. In order to quantify the loss in performance of supervised models, we measure domain transfer loss, which is defined in Equation 1:
TLx→y = Sx→x − Sx→y (1) where TLx→y is the difference of the Macro F1 score Sx→x of a model fine-tuned on domain x and tested in the same domain, and the score Sx→y of the model fine-tuned on x and tested on domain y.
As we have two test domains B = {bdomain1, bdomain2} for each training domain x, we average over these using Equation 2:
TLx→B = 1
NB
∑
i∈B i ̸=x
Sx→x − Sx→i (2)
Finally, we compute the average domain transfer loss for all models of a certain approach A = {supervised, few shot, . . .} by computing the average of the domain transfer losses TLx→B for all models in the approach:
TLA→B = 1
NA
∑ i∈A TLi→B (3)
Table 3 shows the average in-domain results (Avg. In), average out-of-domain results (Avg. Out) and average domain transfer loss (TL) for the supervised models (the full results table can be found in Table 8 in Appendix A). Models finetuned in a supervised fashion achieve the best in-domain (91.2) and out-of-domain (84.6), with the smallest transfer loss (13.3).
Although FewShot-200 achieves relatively good in-domain performance (80.9), it has the largest transfer loss (20.7), with the out-of-domain performance dropping to 70.6, 7 pp. below SoCal. This suggests that it is highly dependent on the few training examples seen being in-domain and that it cannot be safely applied out-of-domain.
Finally, both FewShot-100 and FewShot-20 have similar transfer losses (18.1/18.2), although the
already low in-domain performance (56.5/40.5) means that using these models either in-domain or out-of-domain is impractical.
In contrast, the prompting, rule-based, and dictionary-based approaches do not suffer from this and perform more consistently across domains.
Therefore, the answer to RQ2 is that rule-based methods perform better across domains that fewshot supervision methods.

heading: 5.3 Emotion classification
text: Table 4 shows the Average Macro F1 scores for all approaches on the two emotion classification datasets, as well as the averaged score per approach (results for each emotion can be found in Tables 9 and 10 in the Appendix).
Again, the fully supervised upper bound achieves the best F1 (67.6), where DistilBERT achieves much better performance than either RoBERTa model. RoBERTA-base achieves poor performance on EnISEAR, RoBERTa-large consistently performs quite poorly, suggesting that it requires either more data or more careful fine-tuning than was used here.
The best performing method in the low-resource setting is prompting (57.7), followed by dictionary induction (54.4) and Few-shot 200 (54.1). The dictionary-based method, as well as the FewShot100 and -20 approaches, perform quite poorly (49.3, 45.8, and 41.6 respectively), with the latter achieving worse performance than the 10 word per emotion seed dictionary (42.7).
In contrast to prompting OPT models for sentiment analysis, the FlanT5-large model does not consistently improve over the base model, achieving a quite low score on the SSEC dataset (47.6).
Similar to the sentiment experiment, the induced emotion dictionaries perform as well or better than previously compiled emotion dictionaries (NRC).
Returning to RQ1, for emotion analysis prompting or dictionary induction perform better than few shot approaches.

heading: 5.4 Cross-lingual sentiment classification
text: The results of the cross-lingual experiments can be seen in Table 5. In general, the XLM-RoBERTa models perform much better than the dictionary induction approaches (10-20 pp). However, this depends heavily on the source language corpus used to train, as several XLM-RoBERTa results are lower than their respective dictionary induction
approach (large trained on MPQA and tested on AR, DZ, or MT for example).
Curiously, the large version performs worse than the base version when trained on MPQA or SemEval. Like with the previous experiments, this may suggest that the larger models need more data or require more careful tuning than we performed in our experiments. In either case, it is important to note that simply increasing the size of the crosslingual model will not necessarily result in better results.
Finally, the results of all models are generally worse for Narabizi (DZ) and for Maltese (MT), which is unsurprising, as they have little or no pretraining data in XLM-RoBERTA. The one exception is the Flant5-large, which achieves very good results on both. It is unclear what exactly causes this difference in multilingual ability, especially for low-resource languages like Narabizi and Maltese, although larger models are known to memorize training data (de Wynter et al., 2023) and both of these datasets are available in text format. Therefore, we cannot rule out data contamination as the source of such a jump in performance.
Finally, the cross-lingual models achieve an average of 71.5, compared to 66.3 for prompting or 52.7 for dictionary induction. Thus, we can cautiously venture that for RQ3, cross-lingual methods allow for the best results, although prompting larger multi-lingual LLMs may also provide good results in the future.

heading: 6 Conclusion and future work
text: In this paper we have performed experiments on 10 sentiment datasets and two emotion classification datasets in 8 languages with a number of low-resource approaches (dictionary-based methods, rule-based methods, few-shot methods and prompting methods). The main experiments were performed on the English language datasets (3 sentiment and 2 emotion), while further experiments were performed in 7 additional languages.
These results confirm that under ideal circumstances, fully supervised models perform much better than low-resource approaches. However, in low-resource settings (lack of training data, domain shift), these same models quickly lose performance and rule-based and dictionary-based approaches perform on par or even better if there is a domain shift involved.
While prompting achieved impressive performance in our experiments, given that the models were not explicitly trained, this came at a price. Namely, such approaches for languages other than English are currently not available or not on par with English versions. This area will surely be explored in the near future, but this current gap is nonetheless a product of the over-reliance on English in NLP.
The strong cross-lingual performance of the XLM-RoBERTa models suggests that cross-lingual approaches, especially those designed for adapting to new languages, scripts (Pfeiffer et al., 2021), or generally enabling ever more multilingual pretraining (Lauscher et al., 2020; Pfeiffer et al., 2022).
We find conflicting evidence on the importance
of model size for low-resource performance. On the one hand, prompting the larger OPT model for sentiment classification gave consistently better results. On the other hand, RoBERTa-large suffered much more in out-of-domain classification and generally performed worse than RoBERTAbase on emotion classification in all data regimens. For prompting in emotion classification, FlanT5large did not lead to consistent gains over the base version and finally, XLM-RoBERTa-large similarly performed worse than the base version on crosslingual sentiment classification. This finding seems to indicate that some of the promised few-shot performance found in large language models is either lacking or requires careful tuning.
In the future, it would be interesting to expand this comparison to other dictionary induction methods, such as cross-lingual propagation (Buechel et al., 2020), or high-coverage expansion (Köper and Schulte im Walde, 2016). Given the promising results from the simple prompting approaches we used in our experiments, further research on how to expand these models to new languages and tasks would be of great use.
Finally, multi-lingual few-shot approaches (Lauscher et al., 2020) could also be compared, as it is often possible to use a few examples in the target language.

heading: 7 Limitations
text: In this paper, we only explore binary sentiment classification, as it is enables cross-lingual experiments to be somewhat comparable. However, this is a simplified task, which should be taken into account when interpreting the results. Our multilingual datasets also come from various domains and, although we try to control for this in English, this does lead to some effect in the results. Finally, for emotion detection, we only experiment in English.
We also chose only a few representative methods for each approach (few-shot, prompting, rulebased, etc). This was a necessary simplification given the large number of available models, and care was given to choose truly representative methods for each approach. However, some relevant methods may not be represented here.
Finally, we only report the results for a single run for the supervised models, rather than the average of 5-10 runs as is common. We compensate by averaging over results on several datasets and across several methods.

heading: Acknowledgements
text: This work has been partially supported by the HiTZ center and the Basque Government (Research group funding IT-1805-22).
We also acknowledge the funding from the following projects: DeepKnowledge (PID2021-127777OB-C21) project funded by MCIN/AEI/10.13039/501100011033 and by FEDER Una manera de hacer Europa.

heading: Future Work
text: This area will surely be explored in the near future, but this current gap is nonetheless a product of the over-reliance on English in NLP. The strong cross-lingual performance of the XLM-RoBERTa models suggests that cross-lingual approaches, especially those designed for adapting to new languages, scripts (Pfeiffer et al. , 2021), or generally enabling ever more multilingual pretraining (Lauscher et al. , 2020; Pfeiffer et al. , 2022). We find conflicting evidence on the importance
of model size for low-resource performance. On the one hand, prompting the larger OPT model for sentiment classification gave consistently better results. On the other hand, RoBERTa-large suffered much more in out-of-domain classification and generally performed worse than RoBERTAbase on emotion classification in all data regimens. For prompting in emotion classification, FlanT5large did not lead to consistent gains over the base version and finally, XLM-RoBERTa-large similarly performed worse than the base version on crosslingual sentiment classification. This finding seems to indicate that some of the promised few-shot performance found in large language models is either lacking or requires careful tuning. In the future, it would be interesting to expand this comparison to other dictionary induction methods, such as cross-lingual propagation (Buechel et al. , 2020), or high-coverage expansion (Köper and Schulte im Walde, 2016). Given the promising results from the simple prompting approaches we used in our experiments, further research on how to expand these models to new languages and tasks would be of great use. Finally, multi-lingual few-shot approaches (Lauscher et al. , 2020) could also be compared, as it is often possible to use a few examples in the target language