MultiPICo: Multilingual Perspectivist Irony Corpus (2024)

Abstract: Recently, several scholars have contributed to the growth of a new theoretical framework in NLP called perspectivism. This approach aims to leverage data annotated by different individuals to model diverse perspectives that affect their opinions on subjective phenomena such as irony. In this context, we propose MultiPICo, a multilingual perspectivist corpus of ironic short conversations in different languages and linguistic varieties extracted from Twitter and Reddit. The corpus includes sociodemographic information about its annotators. Our analysis of the annotated corpus shows how different demographic cohorts may significantly disagree on their annotation of irony and how certain cultural factors influence the perception of the phenomenon and the agreement on the annotation. Moreover, we show how disaggregated annotations and rich annotator metadata can be exploited to benchmark the ability of large language models to recognize irony, their positionality with respect to sociodemographic groups, and the efficacy of perspective-taking prompting for irony detection in multiple languages.

heading: 1 Introduction
text: The pervasiveness of AI-based technologies has renewed the interest in making Artificial Intelligence more inclusive and attentive to users’ needs. AIbased technology mirrors the quality and problems of data that feed it (Dignum, 2023). Therefore, the way corpora are created and the design biases that are — consciously or unconsciously — included in datasets and models are of central importance. Perspectivist corpora allow turning bias from an undesirable criticality into a measurable trait by reporting the perception of multiple individuals with different social and cultural traits on pragmatic phenomena (e.g., hate speech or irony).
S. Casola, S. Frenda, and S. M. Lo contributed equally to this work.
In this work, we present MultiPICo (Multilingual Perspectivist Irony Corpus), a multilingual corpus of short conversations (Post-Reply) extracted from Twitter and Reddit and annotated as ironic or not ironic by crowdsourcing workers with different social backgrounds1.
MultiPICo covers 9 languages and 25 varieties, ranging from high to low resources. Specifically, we include 5 varieties of English (Australian, British, Indian, Irish and US English), 5 varieties of Spanish (Argentinean, Castilian, Colombian, Mexican, and US Spanish), 5 varieties of Arabic (Egyptian, Iraqi, Moroccan, Saudi Arabian, and Yemen), 2 varieties of French (Canadian and French), 3 varieties of German (Austrian, German, and Swiss), 2 varieties of Portuguese (Brazilian and Portuguese), and Italian, Dutch, and Hindi. Selected annotators’ nationality is related to these linguistic varieties, the annotators’ gender is balanced, and a rich set of other sociodemographic information (age, ethnicity, student and employment status) is provided.
Starting from these metadata, we analyze the perception of irony in different sociodemographic groups and explore which demographic dimensions primarily explain the annotation. Furthermore, we also show how to exploit this multifaceted corpus as a benchmark for AI-based technologies, including Large Language Models (LLMs). These models are becoming pervasive, but the lack of information on their training data and methods makes their intrinsic bias hard to measure.
In short, our contributions are the following:
• We present MultiPICo, a disaggregated multilingual dataset annotated with the presence of irony in social media short conversations (Section 3).
• We analyze the impact of sociodemographic 1MultiPICo is available at https://huggingface.
co/datasets/Multilingual-Perspectivist-NLU/ MultiPICo with a CC-BY 4.0 license.
16008
dimensions and cultural background in the perception of irony in multiple languages (Section 4).
• We explore the performance of three LLMs on the task of irony detection (Section 5).
• We propose a framework to evaluate LLMs’ positionality and the efficacy of sociodemographic prompting (Section 5).

heading: 2 Related Work
text: The NLP community is increasingly questioning how annotators’ background can influence their choices and perceptions of pragmatic linguistic phenomena, such as irony or hate speech.
Bender and Friedman (2018) proposed a set of best practices for NLP technicians to avoid ethical issues such as exclusion and misrepresentation of users. Their recommendations include reporting annotators’ demographic data, as they can impact how annotators use and interpret language.
Numerous works have highlighted human annotation is subjective (Aroyo and Welty, 2015), especially for semantic and pragmatic phenomena — as the point of view might differ in relation to one’s social background, beliefs, and demographics. An increasing number of works emphasize abandoning a single ground truth and preserving the disagreement among annotators (Leonardelli et al., 2021; Uma et al., 2021a; Leonardelli et al., 2023). Dealing with human label variation has consequences on the whole Machine Learning pipeline (Plank, 2022), from data to modeling (Mostafazadeh Davani et al., 2022) and evaluation (Uma et al., 2021c; Basile et al., 2021).
The perspectivist approach aims at leveraging disagreement and modelling annotators’ perspectives (Basile et al., 2021). To do so, scholars have exploited individual annotators’ decisions (Mostafazadeh Davani et al., 2022) or grouped their annotations based on their beliefs (Akhtar et al., 2019) or demographic traits (Frenda et al., 2023b). To work in this direction, disaggregated corpora with demographic information become of fundamental importance (Cabitza et al., 2023). In fact, aggregated data tend to reflect a minority of perspectives, under-representing others (Prabhakaran et al., 2021; Frenda et al., 2023b).
Disaggregated datasets have become more widespread in recent years, as listed in the Per-
spectivist Data Manifesto2 and by Plank (2022)3. Some of these corpora account for annotators’ demographic. For example, Sachdeva et al. (2022) built a perspectivist corpus for hate speech detection, asking annotators to self-identify their race, gender, sexuality, religion, education, and income. Almanea and Poesio (2022) collected a dataset about misogyny and sexism in Arabic and had it annotated by three Muslim annotators (2 female, 1 male), who self-identified their religious belief as liberal, moderate, and conservative. They also provided annotations by a larger cohort on a small set of their dataset. The work showed a correlation between annotators’ beliefs and their perception of misogyny and sexism in the texts.
Similarly, Akhtar et al. (2021) developed a dataset for hate speech detection involving migrants as victims of offensive texts and demonstrating that members of the same group tend to agree more. Moreover, they implemented perspectiveaware models by creating a gold standard for each group, training two separate models, and finally ensembled by majority vote. Perspective-based ensembling methods have also been explored by Casola et al. (2023) for irony and hate speech detection in English. In their work, perspectives are extracted using sociodemographic information of annotators and mined from annotations.
The use of demographic data when working on highly subjective language phenomena has shown improvement in NLP research. Sap et al. (2022) investigated how annotators with different identities and beliefs perceive toxic content considering the characteristics of texts along three dimensions: racial prejudices, African American English vernacular and dialects, and vulgar words. The authors demonstrated that identities and beliefs impact annotators’ judgments. Sociodemographic information has also been used to build a disagreement predictor by Wan et al. (2023), who tried to quantify how controversial an utterance is. They show that adding demographic information to the input improves results and returns insights on which instances need a more diverse group of annotators. Finally, Santy et al. (2023) developed a framework to identify datasets and model design biases, highlighting their positionality. Results show available datasets strongly align with the young WEIRD population (Western, Educated, Industrialized, Rich,
2https://pdai.info/ 3www.github.com/mainlp/
awesome-human-label-variation
Democratic). Few disaggregated datasets for irony detection exist. Simpson et al. (2019) released a corpus about humor detection in English, used as a benchmark in the first edition of the Learning With Disagreement shared task (Uma et al., 2021b). No annotators’ metadata, however, are included. Frenda et al. (2023b) proposed a dataset for irony detection and investigated the influence of the annotators’ characteristics on their perception (Frenda et al., 2023a); the dataset, however, contains English text only.
To the best of our knowledge, no multilingual disaggregated dataset exists. We fill this gap by proposing a perspectivist dataset for irony in 9 languages and a total of 25 high and low-resourced varieties. We also provide demographic information on all the annotators involved.

heading: 3 MultiPICo
text: In this section, we describe MultiPICo, a corpus of 18, 778 short conversations collected from Reddit (8, 956) and Twitter (9, 822) in 9 languages, and a total of 25 varieties. Data has been collected reproducing the structure of short conversations. Annotators were asked to read a set of Post and Reply pairs and answer whether the text of the reply was ironic or not, given the context.
We collected Reddit comments (first comments in the vast majority of cases and second-level comments in a few cases) with their direct replies and conversation-starting messages. For Twitter, the Post could be both a conversation starter or a direct reply to it. Reddit data were retrieved using the Pushshift repository4 from January 2020 to June 2021. To collect data in several linguistic varieties, we picked 26 subreddits, as reported in Table 10. Thus, we inferred the linguistic variety of subreddit users from the subreddit. We filtered out pairs having at least one deleted or removed comment and further analyzed the target languages using the Python library for language identification LangID5. We collected Twitter data via Twitter Stream API, using the geolocation service and excluding quotes and retweets. Then, we retrieved the full conversation and retained tweets that directly replied to the starting ones. We tried to collect the same number of Post-Reply pairs from the two selected sources. However, there was not enough data on Reddit for some linguistic varieties. This was the case of
4https://redditsearch.io/ 5https://github.com/saffsd/langid.py
US-American Spanish, Swiss-German, Iraqi, and Yemen Arabic (see Table 10). The data collection resulted in 18, 778 instances, together with their metadata, consisting of Post-Reply original IDs, subreddits, and geolocation information.
The human annotation of the collected data was performed on the crowdsourcing platform Prolific6, through an integrated custom-built annotation interface designed to collect a diverse and balanced set of annotators. The interface mimicked a message conversation, having the Post as context and asking whether the Reply was Ironic or Not ironic7 . We prevented an instance from being annotated more than a predefined number of times, with a mean of 5.02 annotations per instance (see Table 1).
Annotators were selected based on three factors: i) their completion rate had to be at least 99%; ii) they had to be native speakers of the considered language; iii) the set of annotators needed to be balanced both across genders (230 Female, 274 Male, 1 Prefer not to say, 1 Null8) and nationality (we hired around 24 annotators for all linguistic varieties, except for the English dataset with around 15). Considering both annotators’ mother tongue and nationality, we inferred that annotators from a specific country speak the corresponding linguistic variety. Because of the lack of annotators, we were not able to preserve balance for the Arabic subset of the dataset9.
The quality of the annotation has been further assured using attention check questions in the form of “Please answer X to this question”. Annotators had 1% probability of receiving these special questions. We set a threshold of 50% correct answers; only 13 annotators failed the test, and their annotations
6https://www.prolific.com/ 7The instructions and an example of annotation is presented in Appendix A.1. 8This annotator did not share the information. 9The Arabic portion of MultiPICo has 46 self-assessed males, 21 self-assessed females, and 1 annotator who prefers not to reveal their gender. Since we did not find enough annotators from the 5 considered countries, we removed this filter, collecting annotations from all mother-tongue annotators.
were excluded from the final corpus, leading to a total of 506 annotators.
Together with the annotators’ gender and nationality, we collected other demographic information, specifically: Age Group (13 Baby Boomer, 66 GenX, 260 Gen-Y, 162 Gen-Z, 5 Null), Ethnicity (315 White, 64 Mixed, 44 Asian, 13 Black, 66 Other, 4 Null), Student status (260 No, 165 Yes, 81 Null), Employment status (178 Full-time, 74 Part-time, 74 Unemployed and job seeking, 24 Not in paid work, 11 Due to start a new job within the next month, 36 Other, 109 Null)10.

heading: 4 Annotators’ polarization
text: In this section, we present an exploratory analysis of MultiPICo. We employ primarily the polarization index (P-index) proposed by Akhtar et al. (2019), which measures the instance-based polarization for k groups. For each instance i, the Pindex is defined as:
P (i) = 1
k
∑
1≤w≤k a(Gw)(1− a(G)) (1)
where k is the number of different groups of annotators, a(Gw) indicates the internal agreement for group Gw, and a(G) indicates the overall agreement on the instance. We compute the instancelevel agreement as:
a(G) = 1− χ 2(G)
|M | (2)
where χ2(G) is the chi-square statistics and M is the complete set of annotators. A high P-index (close to 1) indicates that the annotation of an instance is highly divergent (or polarized) across groups, while each group is internally consistent.
10Details per language are in Appendix A.2. 11The values of Ethnicity for French and Dutch are uncertain because of the unbalance amount of annotators for each trait of this dimension (see Table 8 in Appendix A.2).
For instance, looking at table 3, Example #1 is annotated as ironic regardless of the annotators’ sociodemographic traits, and its P-index is 0; Example #2 has a P-index of 1 because it is annotated as ironic by annotators who self-identified as male and not ironic by the one belonging to the other group (female).
We applied the P-index for two analyses to understand the relevance of sociodemographic information in the interpretation of irony and to investigate its cultural basis:
1. analysis of dimensions: we compared the Pindices obtained by dividing the population based on gender, generation, nationality, ethnicity, student, and employment status (Section 4.1);
2. analysis of cultural basis: we examined the possible connection between the linguistic variety of the instances and the nationality of annotators, observing the mean of P-index scores of the instances belonging to the same linguistic variety when annotated as ironic by workers from the same country (Section 4.2).
To evaluate the significance of the obtained Pindex scores, we computed random P-index values for each instance using random partitions with the same number of annotators for dimension and trait (see Table 8 in Appendix A). Thus, we could compare the mean of real P-index scores per group of annotators (henceforth ‘real P-index’) and the mean of their corresponding random P-index values (henceforth simply ‘random P-index’). Finally, specifically for analyses 1 and 2, we also calculated the difference in percentage (∆) between real and random P-index.

heading: 4.1 Analysis of dimensions
text: As the first exploratory analysis of MultiPICo, we examined the general level of polarization of all the dimensions available in this dataset: gender, generation, nationality, ethnicity, student, and employment status. In this case, annotators have been separated into various groups according to their trait (for instance, Boomers), and the P-index is computed among all the groups of the same demographic dimension (thus, e.g., Boomers, GenX, GenY, and GexZ for generation)12.
Table 2 shows that all the differences between real and random P-index are positive (see column of %∆), except in very few cases (i.e., two cases in Dutch and one in Italian), likely due to the unbalance of information available about annotators. Moreover, for specific dimensions, such as generation and student status, %∆ is higher than 10% for the majority of languages. This observation suggests that age and student status are the most polarizing dimensions when detecting irony, i.e., people from the same age group tend to agree on the annotation while often disagreeing with those in another age range.
12In this case k depends on the number of traits per dimension (i.e., k = 2 in gender, k = 4 in generation, etc.) (Akhtar et al., 2019).

heading: 4.2 Analysis of cultural basis
text: In this second analysis, to examine if irony is affected by the cultural background, we investigated the connection between linguistic varieties and nationality, observing the P-index of instances annotated as ironic by the majority of contributors coming from the corresponding nationality. In this case, we used the P-index in a binary fashion, i.e., one-vs-all. For each nationality, we divided the annotators into two groups, i.e., those with that nationality vs. those with other nationalities, and computed the P-index with k = 2 accordingly.
In Table 4, the agreement in the interpretation of irony appears to be country-based for specific varieties. A clear example is in German, where Austrian and German texts report the highest P-index when the texts are considered ironic by annotators coming from Austria and Germany respectively.
The case of English is also interesting: the identification of irony in Indian texts polarizes specifically when the annotators come from India. Similar results are also observed, for instance, between American and British English and the annotators from their corresponding countries. The same tendency is visible in Spanish, where higher values of the P-index are reported when Mexican annotators detect irony in Mexican texts.
The situation in French and Portuguese is less crisp, showing specific varieties (French from France and Brazilian, respectively) that induce polarization regardless of the provenience of annotators. Looking at French texts, we noticed that most of the Post-Reply pairs annotated as ironic also by Canadians are humorous:
(3) [Post] Direction Amsterdam ?
[Reply] @user Salon de l’herbe?13
In general, results in Table 4 suggest that the interpretation of irony is similar in speakers of the same language. However, for specific varieties, the sensibility of annotators is affected by their cultural background.

heading: 5 MultiPICo as a benchmarking tool
text: Given its fine-grained metadata, both at the annotation and the annotator level, MultiPICo allows for detailed performance evaluation in a perspectivist setting. In this section, we will first analyze LLMs zero-shot performance for irony detection (Section 5.1) and then show some of the evaluation possibilities in exploring the LLMs’ positionality (Section B.3) and the effectiveness of socio-demographic prompting (Section 5.3).

heading: 5.1 Baseline performance
text: This section analyzes the irony detection performance of multilingual Large Language Models on
13[Post] Direction Amsterdam ? [Reply] Herbal salon?
MultiPICo. In particular, we will consider ChatGPT (gpt-3.5-turbo), PolyLM (Polyglot Large Language Model) (Wei et al., 2023)14 in its instructiontuned 13B version and mT0 (Scao et al., 2023) with 13B parameters15.
As a first baseline, we consider the models’ performance on the aggregated dataset16. For all languages, we prompt the models to classify examples as ironic or not ironic. All prompts are written by native speakers of the target language. An example of a prompt for English is in Appendix B.1.
Table 5 reports the f1-score for the positive class for all languages. Open models tend to perform lower than ChatGPT and are often close to random when considering irony. All models tend to output some non-standard labels, i.e., different than irony or not irony as requested in the prompt; for ChatGPT (always) and PolyLM (in the majority of cases); however, the labels are semantically meaningful and can be mapped to the correct class in a semi-automatic way. In contrast, mT0 tends to prepend the prompt to the labels and produces many meaningless labels; we map the output to the intended label17 when possible and consider meaningless outputs incorrect by default. Results show that irony detection is still an open problem in a zero-shot scenario, particularly in non-mainstream languages.
14huggingface.co/DAMO-NLP-MT/polylm-chat-13b 15Following BLOOM’s recommendations, we use mT0-xxl for English and mT0-xxl-mt for other languages. 16For each instance, we consider the majority vote among all annotators. 17The process was semi-automatic. We checked nonstandard outputs for patterns that refer to the ironic or not ironic label (e.g., “irony”, “ironic”, “Output: ironic”, “[prompt] ironic”, etc.). We were not able to perform this operation for non-Latin scripts, and thus do not report results for these languages.

heading: 5.2 Positionality
text: Santy et al. (2023) have exploited annotators’ sociodemographic metadata to investigate design bias and the intrinsic positionality of datasets and models. Given the rich set of metadata available in MultiPICo, we can explore LLMs’ positionality in the perception of irony for multiple languages.
Following Santy et al. (2023), we grouped annotators according to their demographic traits and computed the mean label for annotators having a certain trait only. We use an LLM to label the instance in a zero-shot approach, using the base prompt in Appendix B.1. For all instances, we take the label produced by the model and compute their Pearson’s correlation coefficient with the groupspecific mean labels.
We find that, for all languages, ChatGPT tends to align with the perspective of young annotators (see Table 6). This is consistent with OpenAI’s report on InstructGPT (Ouyang et al., 2022), where the annotators are described as «quite young (75% less than 35 years old)». Other dimensions — other than student status — show a weaker signal (see Table 11). Characterizing models’ positionality is particularly important with closed-source systems, for which the design choices, training data, and algorithms are unknown and can only be characterized by their observed behaviors (Gallegos et al., 2023; Li et al., 2023; Kotek et al., 2023).

heading: 5.3 Perspective-taking prompting
text: To explicitly model the differences in human views toward given phenomena, recent work has explored ad-hoc prompting, impersonating a specific user (Deshpande et al., 2023; Cheng et al., 2023) or exploiting relevant opinions previously expressed by the user (Hwang et al., 2023). In this context, Beck et al. (2023) have shown that sociodemographic prompting lacks robustness and propose to use it to identify ambiguous instances. However, its effect
is hard to evaluate. MultiPICo provides a novel benchmark to evaluate sociodemographic prompting for irony detection in multiple languages. To showcase this, we construct perspective-based datasets per language. For each sociodemographic trait, we consider labels from annotators with that trait only and aggregate them using majority voting. Then, we divide each dataset into a training (80%) and a test (20%) set. We perform the following experiments:
• No additional information (base): we prompt the model with a Post-Reply pair as discussed in Section 5.1.
• Trait-based perspective-taking (trait): we ask the model to impersonate a person from a specific socio-demographic group by prepending the information, e.g., “You are a self-identified male/female/. . . /Indian/. . . ”.
• Data-driven perspective-taking (data): we extract the most representative examples for each trait. We split the annotators into two groups: one composed of people with the trait and its complement. Among instances with a high agreement in the target group (> .60), we then rank the instances by their P-index computed according to this binary split and select the top 3 ironic and not ironic instances according to the target group annotation. We prompt the model in a few-shot setting, providing the extracted examples and the related labels. Note that this approach is completely data-driven and can be exploited to perform perspective-taking prompting even when the nature of the target group is difficult to describe.
An example of all prompts is in Appendix B.1. We use PolyLM for all experiments. Table 7 shows a widely variable performance across languages. For English, base results show the best performance in many cases, particularly for gender and generation. For Spanish and German, perspective-taking prompting works well, with the few-shot approach consistently superior to the prompt-based one. The data-driven approach also obtains good results for the Dutch language, where the fixed prompts do not improve over the baseline. For French, the fewshot approach is the only one that works; otherwise, the model never predicts irony. Finally, results for Portuguese and Italian are mixed, while the model
does not seem to have a notion of irony for the Arabic language.

heading: 6 Conclusions and Future Work
text: We have presented MultiPICo, a multilingual perspectivist corpus for irony. The corpus consists of Post-Reply pairs from social media platforms, and it is released with disaggregated annotations and annotators’ metadata.
Our analysis of the corpus shows a high variability in the perception of irony across sociodemographic groups, confirming the importance of considering the annotators’ characteristics and background for irony analysis. Some demographic dimensions like age and student status particularly induce a strong polarization of annotations. In some languages, such as German, English, and Spanish, we notice a connection between annotators’ nationality and their sensitivity to irony across their linguistic varieties. It suggests that annotators with a background that is in line with a specific language variety tend to have higher agreement.
Furthermore, we showcase the utility of MultiPICo as a benchmark for perspectivist modeling of irony through experiments of perspective-taking prompting with LLMs. In particular, we compare prompts without sociodemographic information, with trait-based perspectives, and with data-driven perspective examples, showing that it is possible to evaluate models in a perspectivist setting. Moreover, MultiPICo allowed us to perform a positionality analysis of ChatGPT, discovering that tends to be aligned with the youngest annotators.
In the future, we plan to expand the analysis of linguistic patterns that characterize irony across languages and language varieties, looking also at the different types of irony (i.e., situational and verbal irony) involved in the Post-Reply pairs. Furthermore, we plan to mine meaningful clusters of annotators based on their annotation rather than on their demographic traits and to analyze whether a meaningful relationship exists between these groups and those based on sociodemographic information.

heading: 7 Limitations
text: While our proposed resource is the first multilingual corpus of irony distributed with disaggregated annotations, we acknowledge that the choice of languages is still limited and somewhat arbitrary. The sociodemographic information about the annotators is also partial, bound to what was avail-
able from the crowdsourcing platform, and following a discretization of human personal traits that could be perceived as forces (e.g., representing selfidentified gender as a single binary label).
Similarly to Sachdeva et al. (2022); Sap et al. (2022); Forbes et al. (2020), we noticed the ethnicity of annotators was unbalanced. We point out this issue to highlight the limitation of paid crowdsourcing (Santy et al., 2023). Furthermore, as shown by Orlikowski et al. (2023), annotators’ sociodemographics do not always align with the most relevant grouping of annotators according to the language phenomenon under study. However, approaches based on mining perspectives (Lo and Basile, 2023), as opposed to strictly categorizing annotators, may alleviate this issue.
In the vast majority (∼90%) of cases, we downloaded the conversation-starting messages and their direct replies to capture the full conversational context. In a few cases, the downloaded reply was not direct but rather a second-level reply (a reply to a direct reply); thus, some conversational context might be missing.

heading: 7.1 Ethical considerations
text: Our work stresses the necessity to consider and include the subjectivity of the annotators in NLP applications, encouraging reflection on the different perspectives encoded in annotated datasets to minimize the amplification of biases. The proposed corpus, moreover, can be used as a starting point also for investigating and evaluating LLMs across a multilingual spectrum, to make them apt to final users.
For building the proposed resource, we adopted measures to protect the privacy of annotators, and our data handling protocols are designed to safeguard personal information (like anonymization of users’ mentions). Although our attention during the collection of data was focused on ironic content spread online, we acknowledge that some of the material to annotate could contain racist, sexist, stereotypical, violent, or generally disturbing content.
Regarding the annotation process, we aimed to pay annotators fairly, estimating an average rate of 9£ per hour. Moreover, we tried to balance annotators through their nationality and self-identified gender. However, we are aware that considering gender in a binary form is limited. We plan to adopt a more inclusive approach toward non-binary annotators in future work.
Some sociodemographic information had not been managed by us before the annotation process, and, when examining them in this work, we noticed a substantial unbalance for some dimensions, like the self-identified ethnicities. This pattern suggests the need to interact differently with annotators or social communities if we want a diversity of annotators and perspectives in terms of social background.

heading: Acknowledgements
text: This work was funded by the ‘Multilingual Perspective-Aware NLU’ project in partnership with Amazon Alexa. The work of Cristina Bosco is also supported by Compagnia di San Paolo - Bando ex-post 2020 - SteroetypHate. The work of Valerio Basile is also supported by Compagnia di San Paolo - Bando ex-post 2020 - “Toxic Language Understanding in Online Communication - BREAKhateDOWN”. We thank all the people involved in providing and checking the prompt translations.

heading: A Appendix
text: A.1 Annotation process
Instructions provided to the annotators before starting the annotation task:
Is it ironic? In this study, we ask the participants to read a message and a reply, and judge if the reply is ironic.
Irony is a figurative language device that conveys the opposite of literal meaning, profiling intentionally a secondary or extended meaning.
For instance:
message: If ur homeless u probably wouldn’t have a phone.
reply: Yes, and all your belongings would be in a handkerchief tied at the end of a stick. –> irony: yes
message: If ur homeless u probably wouldn’t have a phone.
reply: Yes, you’re right. –> irony: no
This annotation consists of 200 small conversations of tweets and Reddit’s posts and will take more or less 2 hours. There are no requirements for taking part in this study, simply annotate the presence of irony as you perceive it. Thank you for your interest in this research!
Example of task:
Message This man is so completely focused towards engineering riots. Reply Gotta stick to your strengths Is the reply ironic? Ironic or Not ironic
A.2 Sociodemographic information In Table 9, we report the characteristics of each sociodemographic cohort.
In Tables 10 and 8, we report the details of the sets of data and the annotators’ sociodemographic information per language, respectively.

heading: B Appendix
text: B.1 Prompt example In this section, we report the prompts written by native speakers and used for our experiments.
Instruction: You are provided in input (Input) a pair of sentences (Post, Reply) extracted from social media conversations. Your task is to determine if the Reply is ironic in the context of the Post. Please provide in output (Output) a single label "irony" or "not irony".
Input:
- Post: Because the last generation was Gen. Z and there’s no where to go past that, so we gotta start from the beginning.
- Reply: but we should have just named the first generation "Alpha" instead of doing it now
Output:
Base prompt for English
Instruction: You are British. You are provided in input (Input) a pair of sentences (Post, Reply) extracted from social media conversations. Your task is to determine if the Reply is ironic in the context of the Post. Please provide in output (Output) a single label "irony" or "not irony".
Input:
- Post: Because the last generation was Gen. Z and there’s no where to go past that, so we gotta start from the beginning.
- Reply: but we should have just named the first generation "Alpha" instead of doing it now
Output:
Prompt-based perspective-taking prompt for English, for the British perspective.
Instruction: You are British. You are provided in input (Input) a pair of sentences (Post, Reply) extracted from social media conversations. Your task is to determine if the Reply is ironic in the context of the Post. Please provide in output (Output) a single label "irony" or "not irony".
Example 1:
Input:
- Post: I went there about 10 years ago. Costs about £20 to go in and it’s just a greenhouse.
- Reply: You go to look at the plants not the greenhouse. It’s like saying a restaurant is just a bunch of tables.
Output: irony
Example 2:
Input:
- Post: casually joins you. Every time I log out of my bank they’re there. And ad with them in is on repeat in my local branch. How I’m a Zelebrity is still going after all these years is beyond me. And please, God, why do people pretend they can’t tell them apart?
- Reply: I think I’d have to switch banks
Output: irony
Example 3:
Input:
- Post: Can we just stop with this now?
- Reply: I suspect it’s coming to an end....
Output: irony
Example 4:
Input:
- Post: We just outraised Greg Abbott — again. Now we’re going to defeat him. That’s how we overcome his extremism and move Texas forward.
- Reply: Money won’t get you elected
Output: not irony
Example 5:
Input:
- Post: When you’re young, work to learn don’t work to earn. You should prioritise study over work. Go full time uni and part time work.
- Reply: >work to learn don’t work to earn
Output: not irony
Example 6:
Input:
- Post: How none of my reddit posts ever get guilded.
- Reply: I’d guild you if I wasn’t poor, my friend
Output: not irony
Example to label:
- Post: Because the last generation was Gen. Z and there’s no where to go past that, so we gotta start from the beginning.
- Reply: but we should have just named the first generation "Alpha" instead of doing it now
Your output:
Data-driven perspective-taking prompt for English, for the British perspective.
B.2 Computational resources Baseline performance (Sections 5 and B.3) ChatGPT (gpt-3.5-turbo) has a cost of $0.001/1000 tokens. We run PolyLM-13B on 5 v100 GPUs with 16GB VRAM; the experiment took approximately
7 hours. We ran the BLOOM zero-shot experiments on two NVIDIA-A40 GPUs; experiments were completed in around 200 hours. The long processing time for BLOOM is likely because the model tends to generate unnecessarily long outputs (e.g., repeating the prompt) rather than the actual label.
Perspective-taking prompts We run PolyLM13B on 5 v100 GPUs with 16GB VRAM; the experiments took approximately 200 hours in total.
B.3 ChatGPT positionality Table 11 reports the results for ChatGPT’s positionality.

heading: Future Work
text: We plan to adopt a more inclusive approach toward non-binary annotators in future work. Some sociodemographic information had not been managed by us before the annotation process, and, when examining them in this work, we noticed a substantial unbalance for some dimensions, like the self-identified ethnicities. This pattern suggests the need to interact differently with annotators or social communities if we want a diversity of annotators and perspectives in terms of social background