SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
Authors: Dimosthenis Antypas
Asahi Ushio
Francesco Barbieri
Leonardo Neves
Kiamehr Rezaee
Luis Espinosa-Anke
Jiaxin Pei
Jose Camacho-Collados
Venue: EMNLP 2023 Findings (2023)
Keywords: Social Media
Benchmark
Twitter

Abstract: Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.

SUPERTWEETEVAL: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research
Dimosthenis Antypas' Asahi Ushio Francesco Barbieri² Leonardo Neves²
Kiamehr Rezaee Luis Espinosa-Anke Jiaxin Pei Jose Camacho-Colladoss
Cardiff NLP, Cardiff University, UK* Snap Inc., Santa Monica, CA, USA
3 School of Information, University of Michigan, USA AMPLYFI, UK
[antypasd, ushioa, rezaeek, espieessssiinnsnselll camachocollados] jecardi ff. ac. 2 (fbarbieri, J com 3 pedropei@umich. edu
Abstract
Therefore, unsurprisingly, strand of Twitter- specific NLP research has produced what we would consider now de-facto models and datasets. On one hand, specialized LMs, either pre- pre-trained on mul- tilingual Twitter text alone (Nguyen et al., 2020; DeLucia et al., 2022; Barbieri et al., 2022b), or in- cluding social engagements (Zhang et al., 2022b); and on the other, joint Twitter-specti models and datasets such as TweetEval (Barbieri et al., 2020). However, one area where social media NLP re- search seems be lacking behind ing with appropriate resources the current shift to- wards in-context learning (ICL) that Large LMs (LLMs) enable (Min et al., 2022). Benchmarks such as TweetEval, while helpful, are constrained to tweet classification tasks, crucially neglecting quence tagging. generation and question answering (QA), for instance, which not only are more di- verse, but better test beds for LLMs and comparing fine-tuning vs ICL (Liu et al., 2022).
Despite its relevance, the maturity of NLP for social media pales in comparison with general- purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given task, which is the best performing model and how it compares with others. To alleviate this is- sue, we introduce unified benchmark for NLP evaluation in social media, SUPERTWEETE: VAL, which includes heterogeneous set of tasks and datasets combined, adapted and con- structed from scratch. We benchmarked the performance of a wide range of models on Su- PERT TEETEEALAL and our results suggest that, despite the recent advances in language mod- elling, social media remains challenging.
match-
1 Introduction
There is overwhelming evidence that general- purpose NLP systems suffer from significant drop in performance when exposed to tasks in specialised domains. This has been shown in disparate domains such as the legal, medical and financial, to name a few. Specifically, for these domains, specialised language models (LMs) such as Legal-BERT,, SciBERT, PubMedBERT, or BloombergGPT (Chalkidis et al., 2020; Beltagy al., 2019; Gu et al., 2021; Wu et al., 2023) have shown exhibit lower perplexity and higher down- stream performance across the board. The need for specialised LMs and corresponding evaluation benchmarks is exacerbated in social media, where instead of (or rather, in addition to) a specialised domain, an NLP system has to deal with idiosyn- crasies such as emoji (Miller et al., 2017; Cappallo et al., 2018; Barbieri et al., 2017a), poor capitalisa- tion (Derezynski al., 2015), vulgarisms and col- loquialisms (Camacho-Collades et al., 2020), fast language change (Del Tredici al., 2019; Loureiro et al., 2022a), and dynamic and caafform-speciffe communicative structures (Bastos et al., 2013).
The contributions of this paper can be summa- rized follows. First, we introduce the SUPER- TWEETEVAL benchmark SUPERTWEETTE fills an important gap in the current NLP landscape by unifying diverse social media tasks and datasets beyond tweet classification (e.g., NER, question an- swering or tweet similarity) into one single bench- mark, which we argue will contribute to faster and more replicable experiments on Twitter. Our sec- ond contribution is suite of experiments that serve two purposes. First, to establish baselines using fine-tuned and ICL approaches, and second, for analysing and producing insights from our exper- imental results, namely that overall better perfor- mance is obtained with fine-tuned masked language models when compared to equally sized text gen- eration architectures, and that zero and few-shot approaches generally struggle as they are not cas-
"SUPERTWEETEVAL available at the following link: https: hatsss///ugggngaaae http:////ggggggga...co//aa////// co/dataseet/cardiffnnp super_tweett super_tweeteval

---

ily to adapt for certain social media tasks. In sum, our results show that SUPERTWEETER challenging for general-purpose language models, regardless of their size and domain, and that spe- cialised models are instead more competitive.
section, we describe the datasets used for each task, which we have split into three types: (I) existing datasets that have been included in the benchmark as they are; (2) datasets that have been adapted to suit the needs of the benchmark; and (3) datasets that we have constructed from scratch as part of this paper. which did not exist in previous literature.
is still
2 Related Work
The advent of general-purpose NLP systems (namely LMs) has led to a proliferation of umi- fied benchmarks agglutinating diverse NLP tasks. The General Language Understanding Evaluation benchmark (Wang et al., 2018, GLUE) was one of the first efforts to provide large-scale bench- mark composed of diverse tasks such as natural lan- guage inference or textual similarity. GLUE was composed of relatively simple and homogeneous tasks, and saw automatic systems quickly reach hu- man performance in some cases (Yang et al., 2019). Because of this, SuperGLUE (Wang et al.. 2019) was developed with the same spirit but included a wider range of tasks and settings. Since then, other veneral-purposs benchmarks for language models, especially those of the new generation, have emerged, such MMMU (Hendrycks al., 2021) and BIG-Bench (Srivastava et al., 2022).
3.1 Existing Datasets
Intimacy Analysis (TWEETINTIMACY) Inti- macy is an important social aspect of language communication (Pei and Jurgens, 2020). We use the English subset of the MINT dataset (Pei et al., 2022), which contains 1,983 English tweets an- notated with intimacy scores ranging from I to 5, with 1 meaning "Not intimate all" and 5, "Very intimate".
Meaning-Shift Detection (TEMPOWIC) This task focuses on the understanding of language lution through time which, while popular research topic (Luu et al., 2022: Agarwal and Nenkova, 2022), remains a challenging problem, specifically in Twitter. In SUPERTWEETEVAL we utilise TEM- POWIC (Loureiro et al., 2022b), a dataset com- prised of 3,294 tweets. Here, two tweets from different time periods and a target word are pro- vided, and the goal is to recognise whether the tar- get word's meaning is the same in the two tweets (binary classification).
In terms of social media research, there are many tasks that require modelling textual content. Tweet- Eval (Barbieri et al., 2020) was the first unified benchmark that agglutinated different tasks into the same benchmark. However, TweetEval is limited to tweet classification, including emotion recognition (Mohammad et al., 2018), emoji prediction (Barbi- eri et al., 2018a), irony detection (Van Hee al., 2018), hate speech detection (Basile et al., 201 9a), offensive language identification (Zampieri et al., 2019b), sentiment analysis (Rosenthal et al., 2017). and stance detection (Mohammad et al., 2016). Similar to the evolution of GLUE into SuperGLUE, is
Sentiment Classification (TWEETŠENTIMENT) Sentiment analysis has been extensively studied both in general (Medhat et al., 2014; Wankhade et al., 2022) and social media context (Barbieri et al., 2022a; Marcec and Likic, 2022). In SUPER- TWEETEVAL, we utilise the data presented in the SemEval 2017 Task 4, subtask C (Rosenthal et al., 2017). The data are formatted as a "Topic Based Sentiment Analysis" task where each tweet is given sentiment label on a 5-point scale ('strongly neg- ative', 'ngaative,, 'negative neutral', 'positive', 'strongly positive') regarding specific target. In total 43,011 tweets and 325 different topics are present.
the aim of this paper and construct& benchmark that robust, large, and especially consisting of diverse tasks and settings for social media NLP research, and in particular, Twitter.
3 Datasets
Emotion Classification Similar to sentiment analysis, emotion classifi- cation has been a popular topic of study (Kušen et al., 2017: He al., 2016) and has been used better understand users' behaviours and intentions in social media (Son et al., 2022; Corbett and Savarimuthu, 2022). For our use case, we utilise
SUPERT WELTEVAL includes a variety of Twitter- specific NLP tasks. For each of them, we include a relevant dataset that can be used as a proxy to test the performance of models on that task? In this
Mooee details and justifications for the selection of datasets used in this benchmark are provided in Appendix A.I.

---

the English subset of the 2018 Semßival task 1: Affect in Tweets, subtask: E-c (Mohammad al., 2018). A total of 7,168 tweets are present and are labelled with one or more emotions based on their context. The labels are selected from a taxonomy of 11 emotions (plus neutral indicating the absence of emotion) such as fearr, and joy.²
the system's 's objective is to determine whether the provided dcfimition aligns with the target phrase in the given context (positive) or not (negative).
First, to obtain positive instances with match- ing definitions, we use the definition provided for the gold Wikidata item ID. Then, we associate a negative instance for each target word's positive in- stances. For this, maximum of top 10 candidates were pulled from the Wikidata API by searching for the target phrase of a positive. Then, candi- dates with low page views were eliminated to re- move noise, and negative instances were chosen randomly from the pool of candidate definitions.
Topic Classification (TWEETTOPIC) Topic classification is a method commonly used to per- form targeted analysis on social media data. This is a challenging task, due to the ever increas- ing amount of data produced in social platforms (Weller, 2015; Stieglitz et al., 2018) 8). In SUPER- TWELTEVAL we use the multi-label setting of the TWEETTOPIC dataset (Antypas et al., 2022) for topic classification. The dataset consists of 6,837 tweets that have been assigned one or more topics. The taxonomy of topics used was selected by a team of social media experts from Snap Inc. and consists of 19 broad topics tailored for social media content such as sports or music." 4
Hate Speech Detection (TWEETHATE) The presence of hate speech in social media is an ever increasing problem, with hateful content be- ing spread in various online communities (Udanor and Anyanwu, 2019; Walther and McCoy, 2021). We utilise Measuring Hate Speech (Sachdeva et al., 2022) which consists of 39,565 social me- dia ( (YouTube, Reddit, Twitter) manually annotated comments. The coders were asked to annotate each entry on 10 different attributes such as the pres- ence of sentiment, respect, insults, and others; and also indicate the target of the comment (e.g. age, disability). The authors use Rasch measurement theory (Rasch, 1960) to aggregate each annotator's rating for every label in continuous value which then be mapped to a binary value.
Question Answering (TWEETQA) As genera- tive task, we consider an abstract question answer- ing (QA) task on Twitter. To this end, we rely on TWEETQA (Xiong et al., 2019), which consists of a tweet and an answer as input, with the answer to the question as the output. Note that the an- swer may not be explicitly included in the tweet. The dataset contains 9,489/1,086/ ,,203 tweets for training/validatiionness splits, respectively.
For our needs, only entries extracted from Twit- ter were considered. Each tweet was assigned a label if at least two out of five annotators agreed on it. We opted out of majority rule in order to acquire a dataset that can be used to train models capable of handling real-world, complex data (Mo- hammad et al., 2018; Antypas et al., 2022). A small amount of tweets with more than one label were discarded. The final dataset contains 7,168 tweets and 8 different labels..
NER (TwEETNER7) For Name Entity Recog- nition (NER), we include the TWEETNERT dataset (Ushio et al., 2022b). This dataset contains 6,837 tweets and seven different labels: person, location, corporation, creative work, group, product, while also offering temporal split (train and test splits stemming from different time periods).
3.2 Adapted Datasets
Question Generation (TWEETQG) By lever- aging the TWEETQA dataset, we re-frame it as a question generation (QG) task, where we use the tweet and the answer as the model input, while the question is the output.
Named Entity Disambiguation (TWEETNERD) The original TWEETNERD dataset (Mishra et al., 2022) is a collection of tweets, a target phrase within the tweet and a Wikidata entity ID to which the target phrase refers in the context of tweet. To make the task more accessible for the evaluation of language models, we convert the dataset binary classification task: given the tweet, target phrase and possible definition of the target phrase,
3.3 New Datasets
In addition to the previous datasets that were di- rectly integrated into the benchmark with mini- mal preprocessing or adapted from existing ones,
FFll list emotions can found Appendix: Table 6. Fuul list of topics can be found in Appendix: Table 7.
FFull list of labels can be found in Appendix: Table 8.

---

we also constructed two additional datasets from scratch that complement the initial list of tasks and datasets. These are emoji prediction over 100 la- bels (Section 3.3.1) and tweet similarity (Section 3.3.2).
of Chen et al. (2021) detect trending hashtags for the period between 2020 and 2021, based on the corpora collected for TimeLM (Loureiro et al., 2022a). Afterwards, we randomly selected a di- verse set of hashtags and collected an additional sample of tweets featuring those hashtags (i.c., most common hashtag only appears on 25 pairs), The resulting dataset features 1,000 tweet pairs, with the inclusion of 20% randomly paired tweets for completeness.
3.3.1 Emoji Prediction (TweETEMOJIION
This task aims to expand on previous emoji classification problems with 20 classes (Barbi- eri et al., 2017b, 2018b) by introducing more challenging dataset with 100 different emojis (TWEETEMOJ1100) TWEETEMOJI consists of a more recent corpus, an important feature to consider in the ever evolving setting of social me- dia, and takes into account a wider variety of emo- jis. EWOJi 00 considers tweets with only one emoji present at the end of the text which is re- moved and used as our label to create multi-class classification setting.
A Annotation with by Likert-like scale by three independent an- notators All the annotators were native English speakers and were paid fairly through our insti- tutional student job provider'. The final inter- annotator, as measured by annotator pairwise Spear- man correlation, was 0.70.
All the tweet pairs then rated
For the creation if the dataset, an existing large collection of tweets (37,083,959 (Loureiro al., 2022a) is taken into account to extract the hundred most frequent emoji. For each emoji selected, we collected 500 tweets every day for the time pe- riod between 01-01-2020 to 01-01-2023, In total 7,379,453 new tweets were gathered through the Twitter API utilising the Tware library (Summers, 2013).
SUPERTWEETEVAL. The Unified Benchmark
We convert all datasets presented in the previous section to the same JSON format, unifying them with the same notation, preprocessing and style. Ta- ble I provides an overview of cach task and dataset while also providing example entries.
4.1 Preprocessing
Following tweet collection, we filtered all en- tries that contained more than one emoji and en- tries where the emoji was not present at the end of the tweet. To avoid highly similar tweets that may have different emojis, we also removed near dupli- cated entries. This is accomplished by applying a normalisation step where (1) URLs and mentions are removed, and (2) entries that are considered duplicated based on their lemmatised form ig- nored. Finally, colour variations of the heart, circle, and square emoji were ignored. All emojis present in TWEeTEMOs100 and their distribution can be found in Figure the Appendix.
A common preprocessing pipeline is applied to all the collected datasets aiming to standardise them and provide uniform and easy-to-use format. Firstly, all URL links masked as /URL), both for privacy reasons, and for concentrating the focus of our tasks to the main text context of each tweet. Furthermore, all mentions of non-verified users are masked with @user to maintain their anonymity. Finally, an attempt is made to unify features and label/score naming to increase the datasets' ease- of-use.
4.2 Evaluation Metrics
3.3.2 Tweet Similarity (TWEETSIM)
To unify evaluation metrics for a better understand- ability, we selected and converted all metrics in percentage-basee 0-100 scale, in which higher scores represent better model predictions.
Given the importance of textual similarity dataset in NLP (Cer et al., 2017) and the lack of such datasets in social media, we decided to construct new dataset focused tweet similarity. Given two tweets as input, the tweet similarity task consists of assigning them 0 to score according to their similarity.
TWEETSENTIMENT Macro Averaged Mean Ab- solute Error AM (Baccianella et al., 2009)
'Guidelines are available Appendix
"To avoid breaking anonymity rules, more details about the annotators and compensation will he provided upon accep- tance.
Sampling Similarly to the TEMPOWIC tweet sampling procedure, we followed the approach

---

Task (Dataset)
Example Input
Example Output
NER (TwEETNER7)
Tweet: Winter solstice 2019: short day that 's long on ancient traditions url via _CNN_ Travel
Winter solstice 2019:
@CNN_ @CNN_'rave: product
Emotion Classification
Tweet: Whutever you decide to da make sure makes you #huppy.
Joy, love, optimism
(T ([wEETEnoTION)
Question Generation
Tweet: 5 years seconds, Durren Booth (@darbooth) January 25, 2013 Context: vine
what site does the link lake you
(1 (TwEETQG)
Tweet: hella excited for ios hecause siri reads notificaaions out lond to you [..]]
Name Entity Disambiguation (TwELTNERD)
Target: siri
True
Definition: intelligent personal assistant various Apple devices
Sentiment Classification
Tweet: #ArianaCirande Ari By Ariana Grande 80% Full url #Singer #Actress url
negative neutral
(IWEETSENTMMENT)
Target: #ArianaGrande
Tweet The minute can walk well I' mn going to delta pot
Mcaning Shift Detection
Tweet Then this new delta variant out im vaccinated but stillill likeee" Target: deltu
False
(TEMPOWIC)
Emoji Classification
Tweet: SpiderMAtS back
(TWEETEMOn100)
Intimacy Analysis
Tweet: @user SKY scored less runs just lol
1.20
(T (TWEETINTIMAC)
Question Answering
Tweet: 5 years seconds. Darren Booth (@user) January 25, 2013 Question: which measurreeents time are mentioned?
and seconds
(TWEETQA)
Topic Classification (I (TWEETTOPIC)
Tweet: Sweet, HIOWAvsISU nationally televised night game! Nebraska getting bumped to @FOX Business is just a bonus.
film_iv & video, sports
Hate Speech Detection (TWECTHATE)
Tweet: Support Black Trans youth url
hate
Tweet Similarity
Tweet I wish kuyvee all the hest #hhnaijj
2.33
(TWERTSIM)
Tweet Sammic about to cry to the housemates night #bboaija
Table I: Example input and output for each and dataset included in SUPERTWEETEVAA.
is selected as the evaluation metric for the Senti- ment Classification task. To better integrate it in score and cap the negative values to O. In contrast to F1- scores, MMD A (also used in the original SemEval competition) takes into account the order of the labels and provides a better understanding of the performance of the models.
TWEETSIM & TWEETINTIMACY For both re- gression tasks, Spearman's 's correlation r is used as the main evaluation metric. This metric focuses the relationship between predicted and actual ranks instead of the absolute errors between them (i.e. Mean Absolute Error). Negative values are also capped to
our benchmark, we use 1 - .M MAEM
TWEETNERD & TEMPOWIC For both binary classification tasks we use Accuracy score. The classes in both datasets are relatively balanced (fully balanced in the case of TWEETNERD) and thus Accuracy provides reliable metric.
TWEETEMOTION, TWEETTOPIC and TWEET- NER7 For the multi-label classification tasks of TWEETEMOTION and Topic Classification the stan- dard average macro-F1 score is used. Metrics like Accuracy score, and Jaccard Index were initially considered, however, macro-F1 will encourage the development of more precise and accurate models across classes. Average macro-F1 is also used for the NER task (similar to the TWEETNER7 original paper).
TWEETHATE In this task we utilise combina- tion of micro and macro Fl scores as an evaluation metric. Specifically, we first group all entries clas- sified as hate speech as one class, and together with the not-hate class the micro-FI score is calculated. Then, the macro-F1 for only the hate speech sub- classes is calculated. Finally, we report the average of these scores. This "combined F1" score is selected because: (1) it weights heavily the most important decision (a tweet being hateful or not) and (2) does not penalise unfairly poor performance in low frequency hate speech sub-classes.
TWEETEMOJH109 Considering the large num- ber of labels present in the Emoji Classification task and the fact that some of the emojis can be a close match for more than one entry, Accuracy at top 5 is selected as the official evaluation metric.

---

Task (Datasct)
Train
Valid.
For the regression tasks, the naive model al- ways outputs the average value of the training set (TWEETINTIMACY, TWEETSIM), and for Senti- ment Classification (ordinal classification) the out- 1,203
Test
TWEETNER7 4,616
576
2,807
TWEETEMOTION
6,838
886
3,259
TWEETQG
9,489
1,086
put is always 'negative or neutral'. Finally, for the text generation tasks of QA & QG our naive model always returns the input text, and for the NER task it outputs random tokens assigned with random entities.
TWEETNERD
20,164
4,100
20,075
TWEETSENTIMENT
26,632
4,000
12,379
TEMPOWIC
1,427
395
1,472
TW
50,000
5,000
50,000
TWEETINTIMACY
1,191
396
396
TWEETQA
9,489
1,086
1,203
TWEETTOPIC
4,585
573
1,679
5.2 Fine-tuning
TWEETHATE 5,019
716
1,433
Model Selection For the fine-tuning setting we consider eight different models OPT (Zhang et al., 2022a), FlanT5 (Chung et al., 2022), RoBERTa (Liu et al., 2019), and TimeLM (Loureiro al., 2022a). The selection of the mod- els was done based on: (1) their relatively small size, with the smallest model having 85 million parameters (FlanT5SMALL) and the largest 354 mil- lion (ROBERT&LARGE) (2) The architectures of the models and training process. FlanTS5 is an encoder- decoder model trained with text-to-text format; OPT is decoder only model and its training pus includes large portion of Reddit data; and finally, RoBERTa, a traditional masked language model, and TimeLM which are based on the same architecture, but their training corpus (specialised on social media, and particularly Twitter) makes them an interesting candidate. For the NER task, only the results from the RoBERTa based models reported since adapting OPT and FlanT5 for this task not trivial, and hence this endeavour is left for future work.
TWEETSIM
450
100
450
Table Number tweets the train, validation (Valid.) and test splits for each of the tasks in SUPER-
TWEETEVAL.
TWEETQA & TWEETQG For the evaluation metrics of generative tasks, we employ the answer- span Fl score for TWEETQA following Rajpurkar et al. (2016), and METEOR (Denkowski and Lavie, 2014) for TWEETQG, which has been shown to be a well-correlated metric for QG (Ushio et al., 2022a).
4.3 Statistics
consists of 255,170 tweets across twelve different datasets. For each task, we consider the traininn/validatiionness splits presented in their resource paper (or in the model released by the authors of the papers). Exceptions are the TWEETHATE, TWEETSIM and TWEETE- MOIT 00 tasks where new data splits were created. Table 2 displays the final distribution of tweets in each split for each task.
Overall,
Training The implementations provided by Hug- ginglFace (Wolf et al., 2020) are used to train and evaluate all language models, while we utilise Ray Tune (Liaw et al., 2018) 8) for optimising the number of epochs, learning rate, warmup steps, and batch size hyper-parameters. The typer-parameter opti- misation is done by performing random search over 10 different runs for each model.
5 Experimental Setting
For the evaluation, we rely on the datasets and splits presented in Section 4.3. In particular, we evaluate all models on the test splits. Each dataset uses different evaluation metrie, introduced in Section 4.2.
5.3 Zero & Few Shot
5.1 Naive Baselines
Further to the fine-tume experimental setting, two in-context Icarning settings are established: zero and few shot. Aiming to explore the challenges that arise when testing SUPERTWEETEWA. such settings, we select the larger versions the
To establish lower performance threshold for each task, naive baselines are also included. For the classification tasks (TweLtEmoTiON. TWEET- NERD, TEMPOWIC, TWEETTOPIC, TWEET- HATE, TWEETEMOJ1 a Majority classifier (most frequent class in training) is employed.
Details of the model can be found in the Appendix: Table

---

FlaST SMMALL
127
ROBERTaBASE OPT
ROBERTILARGE
TimeL
TimeLMuage
Naive
TEMPOWIC
63.59
62.84
58,02
67.66
65.08
63,86
68.14
68.41
63.45
TwEErEMon
1.77
0.75
32.18
31.51
31.56
34.09
33.45
35.64
8.55
TWEETEMOTION
37.49
47.73
54,62
$5.36
55.27
57.95
55.61
58.53
4.58
TWEETHATE
71.09
TWERTINTIMACY
45,92
56.90
45.83
41.06
52.44
22.28
68.95
58.53
3.73
TWEETNER?
59.10
60.00
58.20
60.40
2.12
TWEEINERD
76,70
54,79
83.04
84.11
83.19
84,92
83,95
85.30
50.00
TWEETQA
53.64
66.09
19.60
17.43
12.70
TWEETQG
25,36
TWHRTSENTIMENT
6.98
2.38
48.53
49.29
50.75
54.50
51.89
54.65
0.00
TWEETSIM
9.70
5.45
65.99
66.41
74.42
68.15
72.24
74.64
0.00
TOPEITOPIC
23.16
40.53
55.65
58.78
45.34
58.71
36.65
58.84
2.30
Table 3: SUPERTTWETTEVA individual task results of selected models in the fine-tuning setting.
FlanT5 models (FlanT35x. & FlanT5xxx.). OPT- IML, and also text-ada-001 from OpenAl, small version of GPT-3 (Brown et al., 2020), and 1.-pp....-tubbb and are evaluated each task.
of difficulty of each task. Not surprisingly, most models seem to perform relatively well sim- pler datasets such TWEETHATE (best: 0.8254) and TWEETNERD (best: 0.8530). However, the majority of the tasks appear to still offer an ade- quate challenge, with the best performing overall model (LIReLMM,ARGE) achieving an average score of 0,63 across all tasks tested. Specifically, the most difficult tasks appear to be TWFETEMOJ 00 and TWEETQG, where all models perform below 0.5.
In both settings, we prompt the models three times and report the average result of the runs. Specifically, in the few-shot setting we sample dif- ferent examples extracted from the validation set of each task for each run. The number of examples sampled for few-shot was based on the given task. For regression and text generation tasks, we pro- vide five random examples in each prompt, while for classification tasks we include one example per class with maximum of five examples. The prompts used in our experiments are, in their ma- jority, based on the instructions used in the FlanTS (Chung al., 2022), and OPT-IML (Iyer et al., 2023) papers!!.
Finally, regarding the models' architectures tested, our results indicate that the RoBERTa mod- els (and specifically TimeLM1ARGE. display bet- performance across most tasks when compared to FlanTS and OPT counterparts.
Zero & few shot When considering the results of our zero/few shot experiments (Table 4), a clear trend is revealed where most models tested fail to achieve better, even similar, results to those that were fine-tuned. Exception to this is chat-gpt-3.5-turke which in some tasks such as TEMPOWIC and TWEETNERD achieves sim- ilar, but still lower, performance to the fine-tuned models, while it also achieves the best score the TWEETQA. However, its performance must be viewed with caution as due to its closed nature as there is possibility that the GPT models may have already been trained the datasets collected (including test splits) providing them unfair ad- vantage.
As a final note, we forfeit reporting the results of zero/few-shot settings on TWEETNER? and 1WEE1EMOH1100 as our initial experiments were unsuccessful. This is mainly due to: (1) limitations of the models themselves (c.g. FlanT5 models not trained with emojis); (2) evaluation difficulties (TweETEMOJH is evaluated using Accuracy top which leads complications on the few- shot setting as only one emoji is included in the gold standard); and (3) issues that arose with the prompts tested (see Section C in the Appendix).
6 Results
The difference in performance, particularly in the regression and ordinal classification tasks of TWEETINTIMACY and TWEETSENTIMENT sig- nificant. The best performing model, Flan achieves, in a few-shot setting, scores of 29.96 and 25.72 in TWEETINTIMACY and TWEETSENTI-
Fine-tuning The results from the fine-tuning set- ting (Table 3) provide an indication of the level
"The IML version (yyrr al., 2023) is selected as it is trained in similar way FlanT5. 10 ht tps: homp//ai.co//chaaiii
Detailed prompts for each task can be found in Appendix
C.
MENT respectively, which is more than a 50% drop

---

Model
Zero-shot
Few-shot
Model
Zero-shot
Few-shot
TEMPOWIC
Fla
58.22
66.33
TWEETTOPIC TWEETSIM TWEETSENT. TWEETQG TWEETNERD
67.25
67.54
Fyxn'T5xx1.
38.09
65.71
Flan
75.56
73.94
OPT-IMM-1.30
62.14
60.33
OPT-IMI-1.3
54.71
52.80
text-ada-801
33,26
39.29
text-ada-801
35.09
50.02
chat-gpt-3.5-turbo
64,95
68.34
chat-gpt-3.5-tune
69.97
80.28
TWRETEMO.
FlanT5x
30.08
30.19
FlanT5xx
21.76
22.15
FlanT5xxL
32.77
36,63
Flan
21.05
21.87
OPT-IML1,38
19.96
24.66
OPT-IML1,38
20.34
14.51
text-ada-801
0,42
20.06
text-ada-001
19.09
14.49
chat-gpt-3.5-ttrbbe
45.17
51.56
chat-gpt-3.55tuu
23.25
33.3
TWEETHATE
FlanT5xi
51.05
54.87
FlanT5xL
0,50
6,44
FxxnT5xxL.
41.37
43.74
Flan
28.30
25.72
OPT-IML1.30
28.46
23.10
OPT-IMLJ.3
11.09
text-ada-801
35,45
29.94
text-ada-
0,00
0,00
chat-gpt-3.5-tune
63,42
57.9
chat-gpt-3.
42.99
41.22
TWEETQA TWRETINT.
FlanT5xL
28.22
28.98
FlanT5xi
59.46
$4,29
FlanT5xxL
29.96
29.68
FlanT5xxL.
56.76
61.69
OPT-IML1,38
0.00
1.94
OPT-IML1,38
43.91
9,92
text-ada-801
3.17
0,92
text-ada-001
0.00
8.71
chat-gat-3.5-turbo
41.82
53.17
chat-gpt-3.5-turd
68.94
$7.74
FlanT5xi.
53,85
54.33
FlanT5xi.
36.16
34.73
FlanT5xx1.
53,88
64.44
FlanT5xxi.
36.25
37,35
OPT-IML1,s8
50.85
53,50
OPT-IML1,38
13,43
8.59
text-ada-800
17.99
17.99
text-ada-000
0.00
4.16
chat-gpt-3.5-turft
32.90
70.51
chat-gpt-3. 3. 3. 5utbbo"
54.77
48.31
Table 4: SUPERTWEETĒVAL zero few shot results. Best results for each task and setting are bolded. Chat-GPT results (marked with ") are included for completeness. We refrained from highlighting ChatGPT results due to its closed and irreproducible nature, as well as the possibility to have been directly trained on some the test sets.
in performance compared to the scores achieved by the best performing fine-tuned model (68.95 and 54.65). 12
that contain data splits from different time periods (TwEETNER7, TEMPOWIC, TWEETTOPIC, and TWEETNERD).
7 Analysis
Multi-label. In this cluster we include the TweetTopIc and TwEETEMOTION datasets, analysing the models' performance in multi-label classification.
Aiming to acquire a better understanding capabil- ities for the model, and also the challenges that the tasks present, we organise the tasks in smaller sub- benchmarks or clusters that are aimed at testing particular feature, and investigate their perfor- mance. The clusters defined are created based on the nature of each task as well as the features that are present them.
Multi-class. Similar to the previous cluster, we consider the TWEETSENTIMENT and TWEET- HATE datasets to evaluate the models in single- label multi-class tweet classification.
Regression. For this cluster, include the two regression tasks of TWEETSIM and TWEETINTI- MACY, and also consider TWEETSENTIMENT (or- dinal classification).
Temporal For this cluster, all the datasets that feature a temporal aspect are grouped to- gether. In particular, we include those datasets
1he goal of this paper is have strongest models, rather evaluate models oft-of-the-Bee As such, there are not easily solved
We group all datasets that provide information regarding target word or entity that is used in models' predictions TWEEETSNNTMMENT. TEMPo WIC and TWEETNERD).
Target-based.
are tasks such as TWEETEMOn1ON by the selected zero/few models.
13 "Due to the lack of zero/few-shot results for TWEETNERT. we added the subset temporal " that does not include NER.

---

Mode
Model
Big-label
Disamb.
Generation
Multi-class
Multi-label
Regression
Target
Temporal
Temporal*
shoo
56.82
37.46
34.83
34.51
38.34
47.31
49.97
OPT-IMLiN8
58.42
35,59
23.65
16.70
21.38
45.23
43.42
shat-gpt-3.5-turke
67.46
32.47
53.20
49.97
51.25
59.30
63.23
Pew-shot
FlanT5xaL
69.83
43,16
34.73
36.99
39.03
55.12
59.00
OPT-IMI.1.8
56.56
34,01
17.09
16.62
8.09
41.41
40.57
chat-gpt-3. tett-pp--3-sturbe
74.31
51.90
49.56
49.93
50.71
63.28
65.64
E. tuning
SRASE
20.64
58.82
55.06
35,93
44.13
21.38
40.00
47.59
52.72
OPTssM
45.15
75,89
10.61
62,90
$7.07
52.26
67.02
55.62
70.18
TimeL.M
47.24
76.86
68.59
58.68
62.61
69.45
67.78
70.85
Table Aggregated results over each test cluster of zero-shot, few-shot and fine-tuning methods.
Big-label. In this setting we include classification tasks (both multi and single label) that contain high number of labels TTwEETEMOJII with 100 labels and TWEETTOPIC with 19 labels).
erative, sequence prediction, and regression tasks, in addition to challenging classification tasks. This
benehmark is aimed at unifying Twitter-based eval- uation protocols and provides a realistic assessment of models this difficult and important domain. Our evaluation highlighted the challenging nature of the benchmark and the social media domain. particular, the results show how recent LLMs strug- gle with the specialised nature of this domain, with smaller but fine-tuned and more specialised models being more competitive overall. In addition to its evaluation purposes, it can also be used the basis for multi-task learning, as well as for making use of already-traine -rained models.
Generation. TWEETQA and TWEETQG were grouped together to create setting for evaluating the gemeration capabilities of the models.
Disambiguation. As final cluster we consider the tasks TEMPOWIC and TWEETNERD which share the goal of understanding and differen- tiating the meaning of a term between two contexts.
For this analysis, we selected the two best performing models in zero and few shot tings (FlanT5xxL- OPT-IML1.33) along with chat-gpt-3.5-tun
Limitations
and the best model of each architecture from the fine-tuning experiment (TimeL.MLARGE- FlanT5BASE, and 350 Ta- ble 5 displays the results of each cluster. Although the comparison across clusters is not straightfor- ward given the different evaluation metrics, the most challenging settings for all model tested ap- pear be the Big-label and Multi-label clusters where no score greater than 60 is achieved. Finally, the results again highlight that in-context learn- ing models (both zero- and few-shot) generally underperform compared to smaller models fine- tuned on the full training set, with even ChatGPT failing to attain the best score in any of the test clusters. In general, the fine-tuned TimeLMLARGE model achieve the best results across all non- generation clusters, with the fine-tuned FLAN-T5 model achieving the best results on generation tasks.
The main issue of of language variety, as it focuses on English only. By making this first English benchmark publicly available, we hope this can pave the way for fu- ture research to extend the benchmark for other languages.
is its lack
In terms of data, the benchmark only includes one dataset per task. We took this choice to both (1) make the benchmark simpler to understand, and (2) because for most tasks only a single dataset was available. As such, conclusions for individual tasks can be limiting. Also, for some tasks, it is hard know the performance ceiling for models, often reported as human performance. While we could have attempted to provide an upper bound, we be- lieve this is a challenging problem in itself, as also human performance estimates are often unreliable as a performance indicator (Tedeschi et al., 2023). Finally, evaluation is focused setting comparing language models in supervised and zero/few shot settings and only focused on a limited set of L.Mss. We did not intend to pro- vide a new model performing well on all tasks, but rather an assessment of current models in similar
simple
8 Conclusion
In this paper, we have presented a new social me- dia NLP benchmark, SUPERTWEETEVAL. goes beyond simple tweet classification, including gen-

---

conditions. Because of this, we may have not pro- vided the models with their optimal settings. We will release all the evaluation scripts so that other researchers can easily evaluate their model perfor- mance on SUPERTWEETEVAL
classification: A new approach. International Jour- nal of Applied Information Systems, 4(1):48-55
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022a. XIm-t: Multilingual lan- guage models in twitter for sentiment analysis and beyond. In Proceedings of the Thirleenth Language Rexources and Evaluation Conference, pages 258- 266.
Ethics Statement
Our work aims to contribute and extend research in social media and particularly on Twitter. We propose a unified benchmark that can be utilised to train and evaluate new social media models. The under
Francesco Barbieri, Miguel Ballesteros, and Horacio Saggion. 2017a. Are emojis predictable? In Pro- ceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistis: Volume 2, Short Papers, pages 105-111.
datasets collected in SUPERTWEETTE public licences and follow the rules of Twitter API. Moreover, given that the data presented includes user generated content we are committed to respect- ing the privacy of the users. This is achieved by applying the necessary preprossessing to remove user mentions (from non-verified users) and URLs that can be used identify individuals. We also ensured that none of the dataset splits contain more than 50,000 tweets. Finally, regarding the anno- tation of TWEETSIM, fair payment was ensured to be paid for the annotators that took part in its collection.
Francesco Barbieri, Miguel Ballesteros, and Horacio Saggion. 20. Are emojis predictable? In Pro- ceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guisties: Volume 2, Short Papers, pages 105-111, Valencia, Spain. Association for Computational Lin- guistics.
Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEvall: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages (644-1650, Online. Association for Computational Linguistics.
Acknowledgements
Francesco Barbieri, Jose Camacho-Collaae Francesco Ronzano, Luis Espinosa Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018a. Semeval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 24-33 -3.
Jose Camacho- -Colaao- is supported by a UKRT Future Leaders Fellowship. We would like to thank Fangyu Liu and Daniel Loureiro for their involve- ment in specific tasks in the early stages of this project.
Francesco Barbieri, Jose Camacho Collados,, Francesco Ronzano, Luis Aapinosa-Anke Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018b. SemEval 2018 task 2: Multilingual emoji prediction. In Proceedings of the 12th International Workshop Semantic Evaluation, pages 24-33, New Orleans, Louisiana, Association for Computational Linguistics.
References
Oshin Agarwal and Ani Nenkova. 2022. Temporal ef- fects on pre- pre-trained models for language processing tasks. Transactions of the Association for Computa- tional Linguistics, 10:904-921.
Dimosthenis Antypas, Asahi Ushio, Jose Camacho- Collados, Vitor Silva, Leonardo Neves, and Francesco Barbieri. 2022. Twitter topic classifica- tion. Proceedings of the 29th International Con- ference on Computational Linguistics, pages 3386- 3400, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados 2022b. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 358-266, Marseille, France. European Language Re- sources Association.
Stefano Baccianella, Andrea Esuli, and Fabrizio Se- bastiani. 2009. Evaluation measures for ordinal re- gression. 2009 Ninth international conference on intelligent systems design and applications, pages 283-287. IEEE
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Deb- ora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019a. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th international workshop on semantic evaluation, pages 54-63.
Rakesh Balabantaray, Mudasir Mohammad, and Nibha Sharma. 2012. Multi-class twitter emotion

---

Valerio Basile, Cristina Bosco, Elisabetta Fersini. Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019b. SemEval-2019 task 5: Multilingual detec- tion of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Processing for Social Media, pages 163-169, Online. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Erie Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Lc, and Jason Wei. 2022. Scaling instruction finettund language mod- els.
Marco Toledo Bastos. Rafael Luis Galdini Raimundo, and Rodrigo Travitzki. 2013. Gatekeeping twitter: message diffusion in political hashtags. Media, Cul- ture & Society, 35(2):260-277
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language pages 3615-3620
Jacqueline Corbett and Bastin Tony Roy Savarimuthu. 2022. From tweets to insights: A social media analy- sis of the emotion discourse of sustainable energy in the united states. Energy Research & Social Science, 19:102515.
Processing
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings of the international AAAT conference web and social media, volume 11. pages 512-515.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2019. Short-term meaning shift: A distribu- tional exploration. In Proceedings of NAACL-HLT, pages 2069-2075.
Alexandra DeLucia, Shijic Wu, Aaron Mueller, Carlos Aguirre, Philip Resnik, and Mark Dredze. 2022. Ber- nice: A multilingual pre-trained encoder for Twitter. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 6191-6205, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Jose Camacho-Collados, Yerai Doval, Eugenio Martinez-Cámara, Luis Espinosa-Anke Francesco Barbieri, and Steven Schockaert. 2020. Learning cross-lingual word embeddings from twitter via dis- tant supervision. Proceedings of the International AAAI Conference on Web and Social Media, 14(1):22- 82.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Spencer Cappallo. Stacey Svetlichnaya, Pierre Gar- rigues, Thomas Mensink, and Cees GM Snoek. 2018. New modality: Emoji challenges in prediction, antic- ipation, and retrieval. IEEE Transactions on Multi- media, 21(2):402-415. 415.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo. Maricke Van Erp. Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32-49
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pages 1-14, Vancouver, Canada, Association for Computational Linguistics.
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User- seerraeneaated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguis- tics.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arX1v:2010.02559.
Shuguang Chen, Leonardo Neves, and Thamar Solorio. 2021. Mitigating temporal-drift: A simple approach to keep NER models crisp. Proceedings of the Ninth International Workshop on Natural Language
Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gi- anluca Stringhini, Athena Vakali, Michael Sirivianos,

---

and Nicolas Kourtellis, 2018. Large scale crowd- sourcing and characterization of twitter abusive be- havior. In Proceedings of the international AAAI conference on web and social media, volume 12.
is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems. 35: 1950--99996.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Lara Grimminger and Roman Klinger. 2021. Hate to- wards the political opponent: A Twitter corpus study of the 2020 US elections on the basis of offensive speech and stance detection. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity. Sentiment and Social Media Analysis, pages 171-180, Online, Association for Computa- tional Linguistics.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Laaaaho-collados 2022a. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics: Syxtem Demonstrations, pages 251-260, Dublin, Ire- land. Association for Computational Linguistics.
Yu Gu, Robert Tinn. Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3-1)
Daniel Loureiro, Aminette D'Suuza, Areej Nasser Muhajab, Isabella A. White, Gabriel Wong, Luis Espinosa- Leonardo Neves, Francesco Barbi- eri, and Jose Camacho-Colladee 2022b. TempoWiC: An evaluation benchmark for detecting meaning shift. in social media. In Proceedings of the 29th Inter- national Conference on Computational Linguistics. pages 3353-3359) Gyeongju, Republic of Korea. In- ternational Committce on Computational Linguistics.
Saike He, Xiaolong Zheng. Daniel Zeng, Chuan Luo, and Zhu Zhang. 2016. Exploring entrainment pat- terns of human emotion in social media. PloS one, I (1(3)e01 50630.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time waits for one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1444,,558, Seattle, United States. Association for Computational Lin- guistics.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang. Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christo- pher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-iml: Sealing language model instruction meta learning through the lens of generalization.
Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the hasoc track at fire 2019 Hate speech and offensive content identifi- cation in indo-european languages. In Proceedings of the I 11th forum for information retrieval evaluation. pages 14-17
Hang Jiang, Yining Hua, Doug Beeferman, and Deb Roy. 2022. Annotating the Tweebank corpus on named entity recognition and building NI.P models for social media analysis. In Proceedings of the Thir- leenth Language Resources and Evaluation Confer- ence. pages 7199-7208, Marseille, France. European Language Resources Association.
Robert Marcec and Robert Likic. 2022. Using twitter for sentiment analysis towards astrazeneca/oxfood, pfizer/biontech and moderna covid- 19 vaccines. Post- graduate Medical Journal, 98( 11 1611):1444430
Ema Kušen, Giuseppe Cascavilla, Kathrin Figl, Mauro Conti, and Mark Strembeck. 2017. Identifying emo- tions in social media: comparison of word-emotion lexicons. In 2017 5th International Conference on Future Internet of Things and Cloud Workshops (Fi- CloudW), pages 132-137. IEEE
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam. Chris Biemann, Pawan Goyal, and Animesh Mukher- jec. 2021. Hatexplain: benchmark dataset for ex- plainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 35, pages 14475.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:7807.05/18.
Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applica- tions: A survey, Ain Shams engineering journal, 1093-11131
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficient fine-tuning
Hannah Miller, Daniel Kluver, Jacob Tpeeeult-Spicker Loren Terveen, and Brent Hecht. 2017. Understand- ing emoji ambiguity in context: The role of text in

---

emoji-related miscommunication. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, pages 152-161.
Ankita Rane and Anand Kumar. 2018, Sentiment clas- sification system of twitter data for us airline ser- vice analysis. In 2018 IEFE 42nd Annual Computer Software and Applications Conference (COMPSAC), volume 01, pages 769-773.
Sewon Min, Mike Lewis. Luke Zettlemoyer, and Han- nanch Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791-2809.
Georg Rasch. 1960. Studies in mathematical psychol- ogy: I. probabilistic models for some intelligence and attainment tests.
Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020. Temporally-inforne inforleed analysis of named entity recog- nition. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605 7017-7667, Online. Association for Computational Linguistics.
Shubhanshu Mishra, Aman Saini, Raheleh Makki, Sneha Mehta, Aria Haghighi, and Ali Mollahosseini. 2022. Tweetnerd-end to end entity linking bench- mark for tweets. arXiv preprint arXiv:2210.08122
Saif Mohammad, Felipe Bravo-Marquez. Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval- 2018 task I: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1-17, New Orleans, Louisiana. Association for Computational Linguistics.
Sara Rosenthal. Noura Farra, and Preslav Nakov. 2017. SemEval-2001 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-200 2017), pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguisties.
Shihab Elbagir Saad and Jing Yang. 2019. Twitter sen- timent analysis based on ordinal regression. IEEE Access, 7: 16666777-633685
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-22 2016). pages 31- 41, San Diego California. Association for Computa- tional Linguisties.
Pratik Sachdeva, Renata Barreto, Geoff B Alexan- der Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leverag- ing rasch measurement theory for data perspectivism. In Proceedings of the 1tt Workshop on Perspectivist Approuches to NLP @ @REEC2022, pages 83-94, Mar- seille, France. European Language Resources Asso- ciation.
Dat Quac Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language System Demonstrations, pages 9-14.
Mattia Samory, Indira Sen, Julian Kohne, Fabian Flöck, and Claudia Wagner. 2021. "call me sexist, but...: Revisiting sexism detection using psychologi- cal scales and adversarial samples. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 573-584.
Processing:
Nedima Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit- Yan Yeung. 2019. Multi- lingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- pages 4675-
Jihyeong Son, Changhyun Nam, and Sonali Diddi. 2022. Emotion or information: What makes consumers communicate about sustainable apparel products on social media? Sustainability, 14(5):2849.
guage Processing (EMNLP-IJCC)) 4684, Hong Kong, China. Association for Computa- tional Linguisties.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.06611
Jiaxin Pei and David Jurgens. 2020. Quantifying inti- macy in language. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 3307-5326.
Jiaxin Pei. Vitor Silva, Maarten Bos, Yozon Liu, Leonardo Neves, David Jurgens, and Francesco Bar- bieri. 2022. Semeval 2023 task 9: Multilingual tweet intimacy analysis. arXiv preprint arXiv:2210.01
Stefan Stieglitz, Milad Mirbabaie, Bjöm Ross, and Christoph Neuberger. 2018. Social media analytics- challenges in topic discovery, data collection, and data preparation. International journal of informa- tion management, 39:156-163
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Carlo Strapparava and Rada Mihalcea. 2007. Semeval- 2007 task 14: Affective text. In Proceedings of the fourth international workshop on semantic evalua- lions (SemEval- 2007), pages 70-74.

---

Ed Summers. 2013. Archive tweets from the command line. Online] https://pypi. org/projectlcitepyy
and Interpreting Neural Networks for NLP. pages 353-355, Brussels, Belgium. Association for Com- putational Linguisties.
Simone Tedeschi. Johan Bos, Thierry Declerek. Jan Hajic, Damiel Hershcovich, Feduard H Hovy, Alexan- der Koller, Simon Krek, Steven Schockaert, Rico Sennrich, et al. 2023. What's the meaning of super- human performance in today's nlu? arXiv preprint arXiv:2305.08414.
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on senti- ment analysis methods. applications, and challenges. Artificial Intelligence Review, 55(7):5731
Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection T Twitter. In Proceedings of the First Workshop NIP and Computational Social Science, pages 138- 142, Austin, Texas. Association for Computational Linguisties.
Collins Udanor and Chinatu C' Anyanwu. 2019. Com- bating the challenges of social media hate speech in polarized society: A twitter ego lexalytics approach. Data Technologies and Applications.
Asahi Ushio, Fernando Alva-Manchego, and Jose Gamacho-Colladoss 2022a. Generative language models for paragraph-level question generation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 670- 688, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACI Student Research Workshop, pages 88-93, San Diego. California. Association for Computational Linguis- tics.
Asahi Ushio, Francesco Barbieri, Vitor Sousa, Leonardo Neves, and Jose Camacho-Collados. 2022b, Named entity recognition in Twitter: A dataset and analysis on short-term temporal shifts. In Proceedings of the 2nd Conference of the Asia-Puciffe Chapter of the As- sociation for Compitational Linguistics and the 12th International Joint Conference on Natural Language Processing Volume I: Long Papers), ,. pages 309-319, Online only, Association for Computational Linguis- tics.
Katrin Weller. 2015. Accepting the challenges of social media research. Online Information Review.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 48, Online. Association for Computational Linguistics.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 8 task 3: Trony detection in en- glish tweets. In Proceedings of The 12th Interna- tional Workshop on Semantic Evaluation, pages 39 50.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023, Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564.
Bertie Vidgen, Scott Hale, Ella Guest, Helen Mar- getts, David Broniatowski, Zeerak Waseem, Austin Botelho, Mathhe Hall, and Rebekah Tromble. 2020. Detecting Fast Asian prejudice on social media. In Proceedings of the Fourth Workshop on Online Abuse and Hurms, pages 162-172, Online. Association for Computational Linguistics.
Wenhan Xiong, Jiawei Wu, Hong Wang, Vivek Kulka- mi, Mo Yu, Shiyu Chang. Xiaoxiao Guo, and William Yang Wang. 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 5020- 5031, Florence, Italy. Association for Computational Linguistics.
Samantha Walther and Andrew McCoy. 2021. Us ex- tremism on telegram. Perspectives on Terrorism, 100-121
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: a stickier benchmark for general-purpose language understand- ing systems. In Proceedings of the 33rd International Conference on Neural Information Processing Sys- tems, pages
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNL.P. Analyzing
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the type and target of offensive posts social media. In Proceedings of the 2019 Conference of the North American Chapter of the

---

Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 1415-1420, Minneapolis, Minnesota. Association for Computational Linguisties.
more challenging setting for more recent models and architectures. Finally, as it was part of Se- mÉval competition TWEETSENTIMENTI already well-known and tested dataset.
Marcos Zampieri, Shervin Malmasi, Preslav Nakow, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and cat- egorizing offensive language in social media (Oi- fensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75 75-86, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguisties.
Similar for the Sentiment Analysis task, the dataset pool for the TWEETEMO- TION task was rather limited when considering our needs (multi-label classification & Twitter based) (Balabantaray et al., 2012; Strapparava and Mihal- cea, 2007). Again the data selected is well tested SemLäval dataset that fits the needs of our task.
TWEETEMOTION
Susan Zhang. Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022a. Opt: Open pre-trained transformer language models.
TWEETNER7 Finally. the TWEETNER7 dataset was selected instead of similar ones (Rijhwani and Preotiuc-Pietro, 2020; Jiang et al., 2022; Derezynski et al., 2017: ?) it provides larger taxonomy of entities, larger dataset with uniformed distribution, and temporal characteristics (a more recent corpus) which are appreciated for building the sub-clusters used.
Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park. Brian McWilliams, Jiawei Han. and Ahmed El-Kishky. 2022b. Twhin-bert: socially-enriched pre-trained language model for multilingual tweet representations, arXiv preprint 220.:22009,0766..
A.2 Detailed Classes
A Datasets
1. anger
8. pessimism
2. anticipation
9. sadness
A.1
Dataset Selection
3. disgust
10. surprise
All the datasets used in this benchmark were care-
4. fear
11. trust
fully selected based on their difficulty level, cover-
5. joy
12. neutral (no emotion)
age they offer, and availability (licence). Below we discuss the selection process for tasks that large selection of pre-existing datasets exists.
love
optimism
Table 6: Emotions present TWEETEMOTION.
TWEETHATE For this task we considered a list of 13 different hate speech detection datasets Twitter (Sachdeva al., 2022; Samory al., 2021;
1. arts & culture
12. music
Grimminger and Klinger, 2021: Mathew et al.. 2021; Zampieri et al., 2019a: Davidson al., 2017;
2. business & entrepreneurs
13. news & social concern
3. celebrity pop culture
14, other hobbies
4. diaries & daily life
15. relationships
Waseem and Hovy. 2016; Waseem, 2016; Ousid-
5. famliy
16. science & technology
houm et al., 2019; Basile et al., 2019b; Mandl et al., 2019; Vidgen et al., 2020; Founta et al., 2018). The final dataset contains five distinct hate-speech classes, in contrast other available options, e.g. two hate classes present in SemEval's 2019 task 5 (Basile et al., 2019b), and also utilises more clear taxonomy than other available datasets (e.g. HateXplain, Mathew al. AAAI, 2021 (Mathew et al., 2021)).
6. fashion & style
17. sports
7. film & video
18. travel & adventure
8. fitness & dining
19. youth & student life
9. food & dining
10. gaming
Table 7: Topies present in TWEETTOPIC.
1. hate gendeg
hate_origin
hate_ _ace
hate didabbiiyy
TWEETSENTIMEN We opted of using datasets that consider sentiment analysis as a clas- sification (Rane and Kumar, 2018) or regression (Saud and Yang, 2019) task and instead use TWEET- (astt based on a five point scale) as
3. hate_sexuality
7. hate_age
4. hate_ religion
8. not hate.
Table 8: Classes present in TWEETHATE.

---

gold label
validation
gold label
validation
test
900
80
800
518
52
519
2
627
83
827
517
52
518
827
514
4
813
81
812
62
504
50
503
5
812
61
812
497
90
497
796
493
482
7
8
791
79
791
403
48
403
790
461
48
787
460
10
8
772
77
772
456
46
457
757
751
444
443
15
746
75
747
436
44
457
729
727
73
728
418
418
16
726
73
727
414
41
414
18
718
72
719
382
38
382
19
725
71
715
370
37
370
705
21
097
70
687
355
36
355
692
692
347
35
348
692
24
087
69
688
337
34
337
680
381
679
27
668
67
669
324
32
323
667
308
645
207
30
636
63
636
24
285
29
266
615
278
610
61
609
271
27
270
38
609
61
609
257
26
256
35
589
60
588
245
24
245
36
597
80
598
225
22
225
38
593
59
583
220
22
219
39
592
59
581
213
21
213
589
195
41
584
59
584
181
18
181
562
56
3
165
16
555
152
44
545
55
545
4
150
15
150
545
534
53
534
47
525
53
526
50
50
521
52
521
50
Figure Emojis distribution each split.
A.3 Annotator Guidelines of TWEETSIM
therefore, consists of scoring how related or similar two tweets according to the following scale:
The dataset will be composed of pairs of tweets and relatedness score. The annotation task will,
(5) Tweets are equivalent, even some minor de-

---

tails may differ (e.g., commenting about the same situation in different ways, one being more com- plete than the other, etc.)
Shawnee #njsoccer @user Target: Shawnee
Definition: city in Pottawatomie County, Okla- homa, United States
(4) Almost equivalent, refers to the same situa- tion/event/person but with possibly relevant differ- ences, such as missing significant details.
Options: yes, no
TWEETQA
(3) Not equivalent, but shares details about simi- lar situationnevent/person Could be tweets around similar event but with different emotion sen- timent towards it.
Answer based on context:
Context: years in 5 seconds. Darren Booth ( (@dar-
booth) January 25, 2013
(2) Categorically related, tweets are on the same topic or category (c.g. sports, politics).
Question: what site does the link take you to? Answer:
(1) Loosely related, there is something minor in common (e.g. same energy/sentiment same type, etc.).
TWEETQG
Write question based on this tweet and context. Tweet: 5 years in 5 seconds. @ (darbooth) January 25, 2013 Context: vine Question:
(0) No relation, the tweets do not have anything in common.
Darren Booth
Please note that only exact numbers should be used (e.g. or 3) without any decimal.
B Models Details
TWEETNER7
C Zero-shot Prompts
Entity Definition:
We list the prompts used for each task in the zero- shot setting. For the few-shot setting we follow similar approach while adding 2-5 (depending on the task) examples at the beginning of each prompt.
1. corporation:
Names corporations (c.g.
Google).
2. creative_ work: Names of creative works (e.g.
Bohemian Rhapsody).
3. event: Names of events (e.g. Christmas, Super Bowl).
TEMPOWIC
Tweet I: "In this bullpen, you should be able ask why and understand why we do the things we do." @user #pitchstock2020 user
4. group: Names of groups (e.g. Nirvana, San
Diego Padres).
location: Names that are locations (e.g France). 6. person: Names people (e.g. Virginia Wade), product:
Tweet 2: Castro needs to be the last bullpen guy to pitch.
Name of products (e.g. iPhone).
Does the word "bullpen" mean the same thing in these two sentences?
Identify and categorize named entities in the
Options: [ yes, no
following tweet:
Tweet: New music coming soon via # columbia
TWEETSIM
@Columbia Records
How similar are the following two tweets?
newmusic # photooftheday # listentothis Columbia Records UK URL
Tweet India is With @republic #Immortal-
Sushant #FreeAnujNow #CantBlockRepublic #Na- tion_ With R Bharat
TWEETTOPIC
Tweet 2: Trending On 5 Number Retweet And Comment For 1st #Nation_ With_R Lith__R_oo R_haratt Give the answer scale from 0 5, where 0 is "not similar all" and 5 is "means the same thing".
Which topics from the options below are present in the following tweet?
Tweet: Philadelphia clearly didn't take a page out of the @user Game playbook of firing everything net, make the opposing goalie beat you. There's minutes left and the Flyers have 16 shots Options:
TWEETNERD
Based on the tweet the definition of the target
[ arts _rts_& trr____uluttre ness & __entreprrenuur, celebrity_& Lettt__ppp___utur_e_ diaries & daily family, fashion & style,
busi-
correct?
Tweet: No. Eastern leads 3-0 at halftime against

---

Model
Parameters
Link
Citation
RoBERTaBASE
123M
Chung al. (2022)
RoBERTaLMA
354M
http:://laagppnaaeeeecrrrrrrrrrr
TRASE
123M
https///ieiiiiiieocccccerreeiirr Loureiro et al. (2022a)
TimeLMaARC ARGE
354M
OPT
125M
Zhang al. (2022a)
3PT
350M
OPT-IML138
1.3B
Iyer et (2023)
FIaATS
80M
FlaSTSBASE
250M
Chung (2022)
F5x
11B
http:///waghhiaaaeeccc//////////
text-ada-801
350M
Brown et al. (2020)
5pt-3.5-turre
175B
(2020)
Table Number parameters, link to implementation, and reference for each model used in our experiments.
food_& __dining, gaming, fitness_&_haet &_health, _educctioona,, other_hobbie
film IV tv_&v__rv_ tV_&_videoo video,
fitness
hate_ religion, hate_origin, hate_age age, not bate|
hate_ tii__diiabllity
music, news_&_social_conccee relationships, science_6 & setthhhnology. sports, travel adventure, youth d__student__i life
TWEETINTIMACY
How intimate is the following tweet?
TWEETSENTIMENT
Tweet: thank u, nxt
What is the sentiment of this tweet regarding the target?
Give the answer on scale from 0 5, where 0 is "not intimate at all" and "very intimate".
Tweet: #ArianaGrande Ari By Ariana Grande 80% Full (URL) #Singer #Actress URL
TWEETEMOJ110
Target: #ArianaGrande
Select the top 5 emojis that best describe the following tweet:
Options: [ "strongly negative', 'negative", 'negative neutral", "positive", "strongly positive"]
Tweet: Besties how tf do i get skinny in 12 days
EMOTION
Which emotions from the options below are ex- pressed in the following tweet?
Tweet: @user @ You Dont worry In- dian army is on its ways to dispatch all Terrorists
to Hell
[ l'anger', 'anticipationn, 'disgust', 'fear', 'joy", 'love', 'optimism', 'pessimism', 'sadness', 'surprise", 'trust"
Options:
TWEETHATE
Classify the following tweet as hate speech based on the options.
Tweet: If you're re angry and hate someone because of the color of their skin and you need shoot someone then you're re a s* bitch. You're NOT SUPERIOR. YOU ARE WEAK P*NK*SS B*TCH. F*CK YOU AND YOUR GUNS.
Options: hateer, hate_race hate sexuality,


Reviews:
[{"review_id": "jvwBunCxqk", "reviewer": "Reviewer_5755", "summary": "", "questions": "A) how did you choose the additional tasks/datasets to include in SuperTwitterEval?  How did you choose those you retained from TwitterEval?  How do you know the new tasks are relevant to social media NLP researchers in general and are sufficiently difficult?\n\nB) Why didn't you include stronger baselines to demonstrate the difficulty of SuperTwitterEval in the face of the latest LLMs?  For example chatgpt/gpt-4 instead of ada-001, or PaLM instead of Flan-T5.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort dedicated to review our manuscript. Below we identify the main points raised and attempt to address them.\n\n***“A) how did you choose the additional tasks/datasets to include in SuperTwitterEval? How did you choose those you retained from TwitterEval? How do you know the new tasks are relevant to social media NLP researchers in general and are sufficiently difficult?”***\n\nThank you for the details about the selection of SuperGLUE tasks. Indeed, our selection of tasks was somewhat different from how it was done in SuperGLUE. In particular, we aimed at having a diversity in terms of real-world tasks, which were carefully curated before being included in the benchmark. Then, once the task was decided, we gathered several datasets whenever available before deciding on the dataset to be introduced in the benchmark. Our goal was to provide challenging datasets, which were distinctive from the TweetEval benchmark focused on tweet multiclass classification. For example, for the hate speech task we went through a list of 13 different hate speech detection datasets on Twitter, finally deciding on one dataset based on the difficulty and coverage of the dataset, as well as the licence. In this case, the final dataset contains five distinct hate-speech classes, in contrast to other available options, e.g. two hate classes present in SemEval's 2019 task 5, and also utilises a more clear taxonomy than other available datasets (e.g. HateXplain, Mathew et al. AAAI, 2021).  A similar process was followed for all the tasks included. \n\nIt is also worth noting that a few tasks present consist of more challenging/advanced versions of the original TweetEval benchmark (e.g. multi-label instead of multi-class emotion classification, or 100 emoji labels instead of 20 in emoji prediction). \n\nWe will provide more details about the criteria we followed for selecting each individual dataset in the camera ready version, alongside size / usage comparisons with similar resources where available.\n\n***“B) Why didn't you include stronger baselines to demonstrate the difficulty of SuperTwitterEval in the face of the latest LLMs? For example chatgpt/gpt-4 instead of ada-001, or PaLM instead of Flan-T5.”***\n\nWe have computed ChatGPT results, available below. We initially refrained from using OpenAI’s larger/newer models due to: (1) the closed nature of these proprietary models limits the research we can perform and there is high probability that the GPT models may have already been trained on the datasets (including test splits) collected and the unfair advantage that this would give them; and (2) budget, i.e., we have access to a small number of medium-spec GPUs and some funding for using LLMs APIs, but properly running experiments with, e.g. GPT-4 would incur 10x the costs of our current experiments.  \n\nWhat follows are the zero-shot results of ChatGPT compared to the results reported in the paper:\n|                 | ChatGPT | Best zero-shot | Best fine-tuned |\n| --------------- | ------- | -------------- | --------------- |\n| TempoWic        | 0.6495  | 0.6214         | **0.6841**          |\n| SemEval-Emotion | 0.4517  | 0.3277         | **0.5853**          |\n| Tweet-Hate      | 0.6342  | 0.5105         | **0.8254**          |\n| Tweet-Intimacy  | 0.4182  | 0.2822         | **0.6895**          |\n| TweetQA         | 0.3290  | 0.5388         | **0.6609**          |\n| Tweet-NERD      | 0.6997  | 0.7556         | **0.853**           |\n| TweetQG         | 0.2325  | 0.2176         | **0.4404**          |\n| Tweet-Sentiment | 0.4299  | 0.283          | **0.5464**          |\n| TweetSim        | 0.6894  | 0.5946         | **0.7464**          |\n| TweetTopic      | 0.5477  | 0.3625         | **0.5884**          |\n| TweetEmoji      | 0.1650  | \\-             | **0.3564**          |\n| Tweet-NER7      | 0.1804  | \\-             |  **0.604**           |\n\nAs can be observed, ChatGPT performs better than the other models tested in a similar zero-shot setting, with TweetQA and Tweet-NERD being the only tasks where it fails to outperform FLAN-T5-XXL (scores: 0.5388 and 0.7556 respectively). However, when compared to the best performing fine-tuned models, ChatGPT fails to achieve greater scores to the fine-tuned models with differences ranging between 0.03 (TempoWic) to 0.4236 (Tweet-NER7) in favour of fine-tuning.\n\nThe results reinforce our conclusion that there is room for improvement and that SuperTweetEval remains challenging also for recent closed-source LLMs. \n\nRegarding open-source models, FLAN-T5 was selected as it has been shown to be the strongest instruction-fine tuned available for this type of tasks at the time of the writing (while we acknowledge that it is hard to compare across models and reviewers may believe there are stronger models). Thank you for the suggestion of PaLM, we will also add these results in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper identifies a need in the publicly available social media datasets and is contributing a new dataset meant to fill that need.  This is timely for evaluating the gap in performance of the latest LLMs and related models. The overall dataset contributes a total of 5 new datasets if you consider the re-purposed datasets as new datasets.  The analysis includes a reasonable set of baselines, including a variety of model types and training types, but leaves plenty of room for contribution from future papers in terms of improving on the tasks.", "reasons_to_reject": "The biggest concern I have is that it is hard to tell if the new dataset is really introducing a challenge to modern LLMs as claimed, because none of the latest models were evaluated on it.  The only OpenAI model evaluated was text-ada-001, which is a very old and the smallest size model available.  For example, evaluating at least ChatGPT on the tasks would have given a better idea on whether the new tasks are really going to be difficult for the latest models.  Even better would have been to see how GPT-4 does on the dataset.  Google's flan-T5-xxl is a good baseline, but what about Palm-Bison or something like that?\n\nAnother concern is that the paper never describes how the new tasks were selected.  In contrast, SuperGLUE describes a very practical way of determining their new set of tasks, \"The remaining tasks were identified from those submitted to an open call for task proposals and were selected based on the difficulty for current NLP approaches.\" (SuperGLUE paper, see [1] below)  Such an open call or survey of those working in the field of social media NLP doesn't appear to have been done in this case (unless I missed this in the paper).  Are these new tasks really relevant to most social media NLP researchers or only to the authors?  How do they know?  \nIt's also hard to know if the new datasets in SuperTweetEval were selected based on difficulty because we don't know what alternative datasets they considered or how they chose those that made it into the dataset.  Above concerns about using stronger baselines in the report compound the concerns, as we are not sure if these tasks were chosen due to actual difficulty or maybe just perceived difficulty by the authors.\n\n[1] Quotes taken from Super Glue paper here: https://paperswithcode.com/dataset/superglue#:~:text=SuperGLUE%20is%20a%20benchmark%20dataset,language%20understanding%20technologies%20for%20English.", "questions_to_authors": "A) how did you choose the additional tasks/datasets to include in SuperTwitterEval?  How did you choose those you retained from TwitterEval?  How do you know the new tasks are relevant to social media NLP researchers in general and are sufficiently difficult?\n\nB) Why didn't you include stronger baselines to demonstrate the difficulty of SuperTwitterEval in the face of the latest LLMs?  For example chatgpt/gpt-4 instead of ada-001, or PaLM instead of Flan-T5.", "paper_topic_and_main_contributions": "NLP for social media is much more immature than in general.  In part, this could be caused by a lack of competitive and public social media datasets.  To address this shortcoming the authors developed the SuperTweetEval dataset, which (like SuperGLUE vs GLUE) broadens the tasks and datasets available in TweetEval.  The new dataset collection includes existing datasets that are unchanged (7), datasets that have been adapted for the new benchmark (3), and entirely new datasets (2).  In all the dataset covers a total of 255,170 tweets across 12 task-specific datasets.  They also provide a number of baseline performance from multiple types of models including decoder only, encoder/decoder, and encoder only architectures.  The best fine tuned model is a BERT model that has been pretrained on twitter data, while the best zero/few-shot model was a FLAN-T5 model.  Limitations listed include a lack of non-English data, single dataset per task, and a limitation on the number of baselines provided.  In addition the work noted that in terms of ethical concers they attempted to follow all rules in respecting privacy of twitter users and in appropriately compensating any annotators.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a", "Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {}}, {"review_id": "3lidWEfLmq", "reviewer": "Reviewer_Anux", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and the two issues raised. \n\n***“As the task is to test the ability of NLP models, there is no reason to skip large GPT-3 models, and I cannot give my judgement if I did not see the performance on large GPT-3. As we know, lots of challenging tasks now can be achieved with good performance.”***\n\nWe have added the results of the requested GPT baseline (please see the B. response to Reviewer 1). We can confirm that the tasks are challenging also for ChatGPT. Initially we refrained from using OpenAI’s larger/newer models due to: (1) the closed nature of proprietary models limits the research we can perform and it is questionable any insights may be actionable by the community; (2) the high probability that the GPT models may have already been trained on the datasets (including test splits) collected and the unfair advantage that this would give them; and (3) academic budget, i.e., we have access to a small number of medium-spec GPUs and some funding for using LLMs APIs, but properly running experiments with, e.g. GPT-4 would incur 10x the costs of our current experiment. \n\n***“As a benchmark dataset, I would like to see a much large corpus.”***\n\nThe SuperTweetEval benchmark consists of over 255,000 instances in total (the number of tweets being even larger), with the smallest dataset containing 1,000 entries. Therefore, despite the specialised nature of this dataset, SuperTweetEval is even comparable in size or larger than similar unified benchmarks such as  SuperGLUE - see Table 1 of the SuperGLUE paper for example for a comparison (Wang et al. 2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors bring an important task as the social media channel is important task for NLP area.\n2. This paper provide a dataset which can be used to benchmark social media performance.", "reasons_to_reject": "1. As the task is to test the ability of NLP models, there is no reason to skip large GPT-3 models, and I cannot give my judgement if I did not see the performance on large GPT-3. As we know, lots of challenging tasks now can be achieved with good performance.\n2. As a benchmark dataset, I would like to see a much large corpus. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author proposes a challenging, unified and  heterogeneous benchmark for social media NLP research. As the authors, although the NLP techniques advance significantly, however, social media is still a challenging task. Authors test some models to prove their concept. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I2WI25i5rj", "reviewer": "Reviewer_B3Tx", "summary": "", "questions": "A. It is claimed that the paper is built upon TweetEval, however it seems to be excluding the datasets in that benchmark. Did you include all the datasets from TweetEval in your benchmark? If not what is your reason for not including them?\n\nB. For Tweet-QG, what is the reason behind reframing the task and not using it as is?\n\nC. For similarity and intimacy evaluation, what was the reason of not using Mean Absolute Error? Overall, I guess the benchmark could include more evaluation metrics for evaluating different aspects of the existing datasets.\n\nD. For the Tweet-Emoji zero-shot or few-shot experiments, was it not possible to prompt the system to produce the top 5 related emojis? (I guess that should be doable at least for the GPT model)\n\nE. You mentioned there were the issues that arose with the prompts tested, can you explain more?\n\nF. Why are some missing numbers for some of the models for TweetQA and Tweet-QG in Table 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort dedicated to review our manuscript. Below we identify the main points raised and attempt to address them.\n\n***“The main concern about the paper in my opinion is not using stronger existing models (the models are mainly the smaller size chosen from each family of models) …” “... If the best-performing LLM can do pretty well on these tasks, then there is not much room for improvements and therefore the benchmark cannot be seen as a strong and challenging test bed for future models. …”***\n\nWe added the baseline requested as reference baseline (ChatGPT), but please see our answer to reviewer 1 in relation to the justification of the models chosen, which is also included in Section 5.2 of the paper, and in particular why we refrained from evaluating closed models. And in particular to this concern, we would like to stress the utility of the benchmark irrespective of the performance of newer LLMs. In addition to its challenging nature, this dataset will serve as a test bed and for fine-tuning smaller models that may be developed in order to be used at a large scale, or simply as an analysis proxy. So far there are no social benchmarks such as the one we propose, which can serve to measure the progress of newer models and perhaps in the future, to prove that newer models can indeed be successfully used in all social media tasks (for which so far there is no evidence but our benchmark can help test these newer models).\n\n***“Also, it would be really helpful to have an oracle (or even a proxy) for these tasks (either human oracle or any other relevant oracles) to provide the upper bound on the tasks as well. This can at least provide some intuition on how easy/difficult each task is.”***\n\nWe agree that a human oracle could provide a useful estimate, although with some caveats as also acknowledged in the limitations section and explained in Tedeschi et al. (ACL 2023). Nevertheless, for most tasks this estimate can be computed from the inter-annotator agreement or similar metrics. We will add this information in the Appendix in order to provide a general idea on the tasks’ difficulty.\n\n***“A. It is claimed that the paper is built upon TweetEval, however it seems to be excluding the datasets in that benchmark. Did you include all the datasets from TweetEval in your benchmark? If not what is your reason for not including them?”***\n\nWith the exception of the SemEval-Emotion task, which utilises the same dataset but in a more realistic multi-label setting, none of the datasets present in TweetEval were used. The aim of SuperTweetEval is to expand on the original benchmark by creating new and challenging tasks while adjusting the difficulty of existing ones by better linking them to real-world tasks.  For example, the emoji prediction task of TweetEval consists of only 20 emojis while in SuperTweetEval we consider 100 emojis. Similarly, the sentiment analysis  task in SuperTweetEval is structured as topic based sentiment analysis in a 5 point scale, while TweetEval presents it as a classification approach. To achieve our goal we had to introduce new datasets that will be sufficiently diverse in terms of scope, while challenging for the current models available.\n\n***“B. For Tweet-QG, what is the reason behind reframing the task and not using it as is?”***\n\nWe are not sure that we fully understood the question. Tweet-QG is a modified version of the original Tweet-QA dataset structured as a question generation task, instead of the original question answering framing of Tweet-QA. We will make this clear in the paper to avoid confusion.\n\n***“C. For similarity and intimacy evaluation, what was the reason of not using Mean Absolute Error? Overall, I guess the benchmark could include more evaluation metrics for evaluating different aspects of the existing datasets.”***\n\nFor the similarity and intimacy evaluation we utilise the same metrics used in the original SemEval tasks (SemEval-2017 task 1 & Semeval 2023 task 9) . We agree that including more evaluation metrics is beneficial when studying an individual task. In the benchmark we only include a single metric for each task for simplicity, but we will include the option to compute alternative metrics for each task. \n\n***“E. You mentioned there were the issues that arose with the prompts tested, can you explain more?”***\n\nThe prompts used in our experiments are mostly modified versions of the prompts presented in the FLAN-T5 paper. However, for several tasks (e.g. Tweet-NER7, Tweet-Emoji) existing literature related to zero/few shot learning is limited (i.e., the performance may improve with a more optimised prompting methodology).\n\n***“D. For the Tweet-Emoji zero-shot or few-shot experiments, was it not possible to prompt the system to produce the top 5 related emojis? (I guess that should be doable at least for the GPT model)”***\n\nWe evaluated the selected models on the Tweet-Emoji task in zero/few -shot experiments, however their performance was very poor (not better than a random baseline).These prompts are available in Appendix C. Therefore, based on our prompts and models used, the results suggest that this task cannot be accomplished just by prompting, which may be due to several factors such as the large number of labels that may confuse the models, or the specificity of the task (the original T5 training corpus is not focused on social media and may not contain emoji). There may be more subtle ways to solve the task, but proposing new models was out of the scope of this paper, and we hope that our benchmark could encourage further research in this area. \n\n***“F. Why are some missing numbers for some of the models for TweetQA and Tweet-QG in Table 3?”***\n\nFor the Tweet-QA and Tweet-QG tasks we do not report results from the RoBERTa based models as only generative models were tested (i.e., only generative models were used on this task). We will add a footnote to explain this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper builds upon the exiting benchmark (TweetEval) and extends it by modifying and creating new datasets to include the generation tasks as well.\n\n2. The paper unifies 12 datasets (a mixture of classification and generation tasks) as a benchmark and conducts analysis on them and establish baselines and insights for further research in the domain of NLP for social media and tries to fill in the gap that currently exists in the field of social NLP.\n\n3. The taxonomy of the task clusters presented in section 7 is interesting as it organizes the datasets into similar categories and that can be useful for research questions on specific types of tasks.", "reasons_to_reject": "The main concern about the paper in my opinion is not using stronger existing models (the models are mainly the smaller size chosen from each family of models) on the tasks to better support the claim of the authors as the proposed benchmark can provide insights on how current models perform. Although the experiments conducted in the paper show the differences between different modeling strategies to some extent, it would be difficult to know how much room for improvement exists for different tasks and which aspects need to be worked on to further improve models. Therefore, it would be difficult to conclude that the benchmark is actually providing a challenging test bed for models.\nIf the best-performing LLM can do pretty well on these tasks, then there is not much room for improvements and therefore the benchmark cannot be seen as a strong and challenging test bed for future models. With the reported numbers one can not easily figure out if the benchmark poses challenging questions for researchers to answer.\n\nAlso, it would be really helpful to have an oracle (or even a proxy) for these tasks (either human oracle or any other relevant oracles) to provide the upper bound on the tasks as well. This can at least provide some intuition on how easy/difficult each task is.", "questions_to_authors": "A. It is claimed that the paper is built upon TweetEval, however it seems to be excluding the datasets in that benchmark. Did you include all the datasets from TweetEval in your benchmark? If not what is your reason for not including them?\n\nB. For Tweet-QG, what is the reason behind reframing the task and not using it as is?\n\nC. For similarity and intimacy evaluation, what was the reason of not using Mean Absolute Error? Overall, I guess the benchmark could include more evaluation metrics for evaluating different aspects of the existing datasets.\n\nD. For the Tweet-Emoji zero-shot or few-shot experiments, was it not possible to prompt the system to produce the top 5 related emojis? (I guess that should be doable at least for the GPT model)\n\nE. You mentioned there were the issues that arose with the prompts tested, can you explain more?\n\nF. Why are some missing numbers for some of the models for TweetQA and Tweet-QG in Table 3?", "paper_topic_and_main_contributions": "The paper introduces a benchmark consisting of multiple social media datasets (Twitter mainly) and tasks (10 classification and 2 generation) with the aim of providing a test bed for NLP models for social media. The paper also includes the results of the experiments of different models on the provided benchmark. The contributions of the paper are twofold: first, introducing a dataset “SuperTweetEval” which unifies 12 different datasets and tasks and second, a set of experiments to show how different models perform on the proposed tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work presents a benchmark for NLP evaluation in social media, SuperTweetEval. 

The reviewers scores vary considerable from borderline/mediocre to good/strong with respect to both soundness and excitement. One major point of agreement among reviewers has to do with the lack of evaluation of latest/best-performing models to date to provide a wider picture of the challenges & limites presented by the benchmark. The authors add models such as Chat-GPT to the papter as requested in order to provide a wider picture. The authors also provide further justifications during rebuttal for the models selected. This aspect should be reinforced in the camera-ready version of the paper. The authors should clearly describe how the datasets in SuperTweetEval were selected, how are they relevant to social media NLP research, and how they compare to previously existing similar resources.


The points of divergence are: the size of the dataset and the lack of human baselines. The authors address this issues during rebuttal and in my opinion should not preclude publication.
Decision: Accept-Findings