Title: MultiPICo: Multilingual Perspectivist Irony Corpus
Benchmark: MultiPICo
Summary: Perspectivism in NLP models different individual perspectives by leveraging data annotated with subjective opinions. The proposed MultiPICo corpus includes multilingual ironic short conversations from Twitter and Reddit, along with annotator sociodemographic information, allowing for the analysis of demographic influences on irony perception and the benchmarking of large language models' ability to recognize irony across different groups and languages.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Irony detection
Phenomenon defined by authors: Yes
Phenomenon definition: benchmark the ability of large language models to recognize irony, their positionality with respect to sociodemographic groups, and the efficacy of perspective-taking prompting for irony detection in multiple languages

Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Detect irony in text
Task item definition: Text, language, LLM, detection score, positionality of LLM with respect to age
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 18,778
Dataset metadata available: Yes
Dataset metadata detail: language, annotator demographics, sources, human annotation
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail:  positionality with respect to age, demographics of annotators
Metric access: Outputs alone
Authorship: Academia
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: NLP / 
New taxonomy: General Capability / Multilinguality