Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
Authors: Minje Choi
Jiaxin Pei
Sagar Kumar
Chang Shu
David Jurgens
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
social information
benchmark
TL;DR: We curate a benchmark of NLP tasks for measuring the social information capabilities of LLMs, and conduct various experiments showing how good existing LLMs are.

Abstract: Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand social language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor \& sarcasm, offensiveness, sentiment \& emotion, and trustworthiness.  In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The resources are released at https://github.com/minjechoi/SOCKET.

Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with the SOCKET Benchmark
Minje Choi Jiaxin Peit* Sagar Kumar Chang Shu David Jurgens
Uniiersityy of Michigan, Ann Arbor, MI, USA NNrttheastern University, Boston, MA, USA University of Cambridge, Cambridge, UK
[minje, pedropei, jurgens @umi ch. edu sagmar..sggnoorthhstteee edu
.ukuk
Abstract
2021), the NLP field has limited abilities test it. Here, we introduce SocKET, new benchmark for evaluating social knowledge.
Large language models (LLMs) have been shown to perform well at variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms in- cluding conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMSs understand social language. Here, introduce a new theory- driven benchmark, SOCKET, that contains 58 NL.P tasks testing social knowledge which group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, trustwor- thiness, and other social factors. In tests the benchmark, we demonstrate that current mod- els attain only moderate performance reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero zero-shot evaluations, we show that pretrained models already possess some innate but limited capa- bilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides systematic way analyze model performance on an important dimension of lan- guage and points to clear room for improve- ment to build more socially-aware LLMs. The resources are released at https:/ //github. com/min htt.//mijjjechccccm SOCKET.
Evaluating NLP systems has remained key component for benchmarking the field's progress. Indeed, the rapid replacement of traditional mod- els by LLM- based approaches was strongly moti- vated by substantial gains by LLMs on a varicty comprehensive Natural Language Understanding (NLU) benchmarks like SuperGLUE (Wang et al.. 2019) and Natural Questions (Kwiatkowski et al., 2019). However, despite the fundamental social aspect of language, comprehensive benchmarks of social language remain absent. Instead, existing computational studies of social language have built individual datasets and models for specific types of information like empathy (Sharma et al., 2020), politeness (Danesu--Niculssuu-iie et al., 2013), and humor (Van Hee et al., 2018). While benefi- cial, these semantic-level tasks omit broader social and narrative-level information (Li et al., 2021) and present only narrow view of model performance.
We introduce SOCKET (Social Knowledge Evaluation Tests), a theory-grounded, systsgrrrondddddssttttttti collection of 58 social language tasks.! SOCKET covers five categories of social information: timent & emotion, trustworthinesss humor & sar- casm, offensiveness, and social factors, each mo- tivated by specific theories. To examine models' generalizability, SOCKET includes four task for- mats: classification, regression, pairwise compari- son, and span identification. This construction aims at assessing not only NLP models' performances on individual tasks but their ability to perform mul- tiple task types and productively benefit from related tasks and task categories during learning. Our study offers the following three contribu-
1 Introduction
Interpersonal communication is more than just what is said. Understanding communication re- quires reasoning not only about the content of message but also the social implications drawn from that message (Halliday, 1995). As NLP sys- tems, partieularly Large Language Models (LLMs), are increasingly used in interpersonal settings, these models' abilities to understand social knowl- edge become critical. However, despite the recog- nized need for social knowledge (Hovy and Yang,
'The choice of the term "social knowledge" in framing stems from its use for broad category in psychology (e.g.. Turiel, 1983; Adolphs, 2009) that matched the capahilities we are interested in.
equal contribution

---

tions the research community. (1) We moti- vate theoretically-groundee organization of so- cial tasks ($2) and subsequently introduce a new easy-to-use benchmark, SOCKET, that system- atically organizes 58 tasks ($3). (2) We bench- mark multiple current LLM approaches to mul- titask NLU via standard supervised training and zero-shot LLMs ($4). Across all tests, our results show that baseline LLMs perform moderately, at best, but offer promising signs of being able to leverage task correlations. (3) We test the abili- ties of models to make use of cross-task transfer ($5) showing multi-task training on strongly cor- related tasks can maintain or even improve perfor- mance in specific tasks, but doing so on weakly correlated tasks can hurt the overall performance of LLMs ($6). We release our framework code and prepackaged datasets at https://githost com/ minjechoi/SOCkar and https: //wgggingaacc.. co boo/atttccts/Blbaaal....
the-art results close human performance in their respective settings. Studies such as Park al. (2021) have also leveraged explicitly-given norms to train models to be more accurate in context- specific situations.
However, these plausible results may be achiev- able solely by focusing the statistical and syn- tactical instead of the social aspects of language. Whether to make advances language understand- ing in research or to ensure reliability and safety in deployment, it is of vital importance to study whether models are truly capable of gaining a gen- eralizable understanding of social factors before employing them for tasks that require such knowl- edge (Hovy and Yang. 2021). The necessity for such understanding is exemplified by studies show- ing that, when measuring the same concept, the performance of a model can vary greatly when tested on a different dataset due to factors such as changes in dialect, speaker demographics, and dataset domain (Miller et al., 2020; Blodgett et al., 2016; Wang et al., 2022a).
2 Social Information in Natural Language Processing
Despite this importance, efforts towards ag- gregating and synthesizing various datasets into themes have been less practiced. One notable ex- ception is the work of Kang and Hovy (2021). where the authors combine existing datasets on different linguistic styles to introduce a benchmark that enables them to study cross-style language un- derstanding. Similarly, we present benchmark curated from over fifty different tasks on different aspects of social information, which we group into five distinctive categories.
Language is inherently social, as meaning is con- structed through social interactions (Wittgenstein, 1953). A substantial body of research in linguis- tie theory and communication studies have ex- amined how social knowledge is communicated via language understanding. Theories of lan- guage grounded in interaction and communication systems such as Systemic Functional Linguistics (SFL) by Halliday et al. (1989) assert that the func- tion and appropriacy of language in given context is the key to our understanding of language and its use (Eggins, 2004; Allan, 2007; Halliday et al., 1989; Halliday, 2004). We use these insights to probe linguistic models for their ability to capture social information, which we define as information conveyed through text about broader metatextual function and contextual appropriacy of the utter- ances in conversation.
Examining the social knowledge of LLMs LLMs are ubiquitous in NLP and their success is at- tributed to the ability to capture language charac- teristics from the immense amount of text seen in pre-training and to effectively apply this in- formation on downstream tasks, achieving state- of-the-art performances in many language under- standing tasks (Chung et al., 2022a). LLMs have demonstrated less success when solving tasks di- rectly related to social knowledge. For tasks that require social information such as detecting sarcasm (Farha et al., 2022) or patronizing lan- guage (Perez-Almending et al., 2022), recent mod- els exhibit only moderate performance. One major challenge is that compared to humans, LLMs have less capability to make predictions outside of the provided input and must perform reasoning only based on their innate social information (Sap et al.,
NLP Studies on Social Information Numerous studies have contributed to the development of datasets and models aimed toward identifying nu- anced social information in language across diverse contexts. Computational linguists have modeled multiple forms of social information in language like sentiment (Buechel and Hahn, 2017), polite- ness (Fu et al., 2020), humor (Meaney et al., 2021), offensiveness (EISherief et al., 2021), and intimacy (Pei and Jurgens, 2020), often achieving of-

---

2019b; Zhou et al., 2020), Yet, it is this very social knowledge that is crucial for human interactions and conversations and a milestone that should be reached for LLMs to engage in meaningful commu- nications with humans (Mahowald et al., 2023).
3.1 Task Selection Process
The task curation process began with a system- atic review of literature on social from linguistics, communications, and psychology identify likely categories of social knowledge. Then, possible datasets and tasks were identified through a system- atic review of datasets published at ACL, EMNLP, NAACL, EACL, LREC, and SemEval since 2015, In this first pass, we selected more than 100 datasets and tasks to detect different types of social infor- mation in language (ef. Table I in Appendix B.9 for all candidate datasets and tasks). Tasks were selected based on membership five categories of social language (described next) that are motivated as core aspects social language understanding. For each category, we include tasks of sev- eral distinct objectives: binary and multi-class classification, regression, pairwise similarity de- tection, and span identification. Where possi- ble, we aim for diversity within categories and ensure one task for each objective. Candidate tasks were removed if it was found that train- ing bert peart-bbss-unccase model on the task achieved test performance over 0.95, which would provide little insight into progress at recognizing social information.
More recently, general-purpose LLMs trained with instruction-based prompts have been known achieve strong performances, putting them to use several practical domains such as summarization, question answering, and classification (Sanh et al., 2022). A newly emerging trend is to use curated prompts to identify the psychological capabilities of instruction-guide LLMs. Ruis et al. (2022) and Hu et al. (2022a) examine pragmatic under- standing capabilities using prompts. Coupled with additional steps such as reasoning, this prompt-based approach has large potential for understanding whether LLMs can pro- vide reasoning capabilities like humans.
(CoT)
The Inter-relatedness of Social Information So- cial language understanding requires accurately perceiving different dimensions and facets of com- munication that relate to one another. Interper- sonal communication makes frequent use of humor (Schnurr, 2010), mitigation, also known as hedg- ing. (Schneider, 2010), and swearing as norm violation (Stapleton, 2003) in defining the con- tours of the social context for the speakers. Of- ten, the pragmatics of these different dimensions of social language use are intertwined: commu- nication with one dimension influences the inter- pretation of another, c.g., politeness and offensive speech (Culpeper, 2021), humor and politeness (At- tardo, 2008), humor and offensiveness (Alberts, 1992), and mitigation and empathy (LI Hai-hui, 2019). Understanding one of these dimensions requires models to have the ability to recognize the related dimensions. While past computational work has largely focused on single dimensions, SOCKET fills a key gap by testing whether models can accurately recognize multiple, interrelated so- cial dimensions-ane whether models can benefit in their understanding from cross-task transfer.
While this process identified many candidate tasks in multiple categories, the benchmark still defines only partial progress in social knowledge capabilities. Some abilities recognized by social sciences such deceit have only proposed (Ott et al., 201 1), providing limited data to measure progress. However, recognizing these as limitations (discussed in more detail in $8), SOCKET provides a diverse set of tasks and ca- pabilities, described next, for the field to begin to measure progress.
two tasks
3.2 Task categories
Inspired by theories in interpersonal communica- tion and interpersonal pragmatics, we provide a thematic organization of the tasks in SOCKET into five related categories of social knowledge: Humor & Sarcasm, Offensiveness, Sentiment & Emotion, Social Factors, and Trustworthiness.
3 The SOCKET Benchmark
Humor & Sarcasm The practice of humor conversations and interactions plays a key role
Here, we describe the steps taken to curate SoCKET as robust benchmark for identifying so- cial information embedded language in interper- sonal communication contexts.
Other task types were initially considered (e.g., genera- tion, paraphrasing) but such tasks were not feasible for all models and often were less standardized in their evaluation, complicating cross-task comparison if included.

---

1dbb
Hiwee save
brmagen (hkee a. 4030
13.00
1391
Screer Daemm
Eatubiig flie 2011)
013
TIF1
(Minsey 201)
034
121
Setemit
chas and new
2013
2084
eaicoa--uaa (Wkge c-d-
2.400
13.
280
Spaee1 Baadin
caderiiaa tchan 2013)
2/733
4_510)
2018)
1089
200
tramm
2011
1227
hastoend en dec
13-3
(.91
2011
2001
seake Faat the ood da 2013
Jate
21.77
03.
2.91
Sediect& Rradin
treet peee Buriesora 2019
19080
011
30.21 36
01.6%
034
1911
Setemi
lest and
1981)
7001
115
bare
01.03
134
7991
Racilledd 2015
1304
0911
Fiar
seplccaaeert55 www.erttttSeeeee Deerd 3011
13.5
2.00
Scaid Pack
kouu pellicos ditoni 2011)
015
4439
Scdli
(Pd
2367
(1ioar, 2012
38:00
034
2.9)
bugiin a-. a01y
(601
F15
iern (lgaat 2017)
<boura
36300
035
1.0
Tranation
buggke (S di 3005
4443
015
201130
14
134
791)
kepal (/hiny on Kie
1108
116
315-1
2000
ataal 3320
pike
unge ill
04381
034
2.811
preiprack que (Marea 2000
udlloweppii (Wang Fata, 2017:
1417
CL5
2070
2013
much
11:28
Offeatrnio
tecc eitoalic Stangiot
14:00
0355
120
Trowstiioo
(Podoo 2.03
015
107-40
Table I: A list of the datasets covered in the SOCKET benchmark, A total of 58 tasks in 5 categories of social information. Included are each task's sample size, task type and evaluation metrie used in the original paper. SOCKET covers four types of tasks: classification (CI.S), regression (RFG), pair-wise comparison (PAIR), and span identification (SPAN). F1, F1-M and F1-m indicate binary FI, macro El and micro Fl scores,
in maintaining and forming positive social rela- tions (Holmes, 2006; Brown et al., 1987; Ziv, 2010). We differ Humor & Sarcasm from Trustworthi- ness social information category because while both categories consider non-cooperative behaviors (Grice, 1975), humor is considered to be prosocial (Attardo, 2008). In instances where the humor is not considered to be prosocial and is instead of a derogatory nature, we consider it to be in the Offensiveness category. By nature, humor is a sub- jective concept that can differ depending on both demographic and contextual factors (Ruch, 2010), making humor detection a difficult task for LLMSs SOCKET includes a number of tasks on humor that can occur in various contexts such as in social media (Meaney et al., 2021), short jokes (Meancy et al., 2021), and news headlines (Hossain et al., 2020). We also include tasks that require detecting relevant concepts of humor such as sarcasm (Kho- dak et al., 3) and irony (Van Hee et al., 2018). Offensiveness Detecting offensiveness using com- putational methods has gained large attraction in recent years due the ubiquity of online com- munication and the necessity to implement auto- mated content moderation to combat abusive be- haviors (Spertus, 1997). However, most existing studies only focus on limited types of offensive languages (Jurgens et al., 2019). In this study, we consider offensiveness to be any explicit or implicit language directed towards individuals, entities, or groups (Waseem et al., 2017), and the tasks chosen are representative of this understanding. SOCKET includes a list of offensiveness detection tasks cov- cring different levels of harmful content and abu- sive language including both explicit and implicit
hate (EISherief et al., 2021), abuse (Vidgen et al., 2021), and humor-related offensiveness (Meaney et al., 2021). We also include forms of bias directed towards people and groups, as social bias enforces harmful stereotypes (Sap et al., 2020).
Sentiment & Emotion Emotion is core element of interpersonal communication that can be com- municated through human language in several as- pects (Majid, 2012; Barrett et al., 2007). Social information is crucial in the ability to not only communicate, but also feel emotion. Theories of discretized emotion (Ekman, 1992) have been supported by empirical findings that humans use discrete labels learned through language to direct their emotional responses to stimuli (Lindquist and Barrett, 2008). Moreover, emotional responses have been shown to direct communication with peers (Lee et al., 2020), and expressing certain emotional responses- succh as anger- have been shown to have social ramifications (Keltnr et al., 1993). Interpreting emotions from text using com- putational tools has been popular research topic across numerous areas in social sciences, enabling new discoveries at unprecedented scale (Jackson et al., 2022). In SoCKET, we include wide range of tasks from various domains such as daily dialogue (Li et al., 2017), written responses news stories (Buechel and Hahn, 2017), and tweets using textual syntax (Mohammad et al., 2018), and also emojis (Barbieri et al., 2018).
Trustworthiness People can detect cues in lan- guage that determine the trustworthiness of mes- sage (Newman et al., 2003), leading to studies that aim to quantify the level of trust in text using com- putational methods (Choi et al., 2020). In particu-

---

lar, this direction has gained attention from NLP communities following increased needs combat and mitigate potential harms coming from the gen- eration and dissemination of false information in online spaces (Wu et al., 2019). In SOCKET we in- clude tasks that require identifying perceived trust from several dimensions: impartiality (Pryzant et al., 2020), deception (Ott et al., 2011), propa- ganda (Martino et al., 2020), rumor (Ma et al., 2017) and bragging, as it is considered to be "un- plain speaking" (Haiman, 1998; Jin et al., 2022). Other Social Factors Finally, we include tasks of more discursive and rhetorical type, that understood be more reliant on the contextual elements of social distance, power, and solidar- ity. In SOcKET, the tasks included are empa- thy (Buechel et al., 2018), politeness (Hayati et al., 2021; Fu et al., 2020), intimacy (Pei and Jurgens, 2020) and complaints (Preofiuc-Pietro et al., 2019). Politeness, like humor, is understood to be a non- cooperative prosocial behavior but unlike humor, is concerned with the act of "saving face" (Brown and Levinson, 1987). Empathy, shown be closely related to politeness (Fukushima and Haugh, 2014), is heavily reliant on social positions in the context of the conversation (Macagno et al., 2022). Inti- macy, however, has been shown to be more depen- dent on notions of time and space between people in dialogue (Márquez Reiter and Frohlich, 2020).
all tasks, including 269,246 samples in the test set. However, experimenting with an evaluation of size can be prohibitive due to model size, available resources, and considerations of the environment. Therefore, we also release a subset of our data as SOCKETTE (SoCKET but Tinier) that contains at most 1000 items per task in the test set, reducing the test set to 43,731 samples. In Appendix B.3, we show that performance on SoCKETTE: is highly correlated and we hope that this smaller subset enables more rapid progress.
Benchmarks on the Social Knowledge Capabilities of LLMs
We first train and evaluate several commonly used multitask LLMs on our datasets to obtain bench- mark results, which provide a first glimpse of how good LLMs are at learning social knowledge tasks. Experiment details are described in Appendix $B.
4.1 Training Methods
BERT-based Finetuning We first apply the stan- dard process of fine-tuning pretrained LLMs. We select two of the most popular LLMs BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as well as two lightweight mod- els known to achieve high performance on fine- tuning tasks - DeBERTa- V3 (He et al., 2021) and MiniLM (Wang et al., 2020).
Prompt-base based finetuning Prompt-based finetun- ing has emerged as flexible and effective means of adapting models downstream tasks (Wei al., 2021). As a benchmark, we include the perfor- mances of a T5 model (Raffel et al., 2020) trained on each task via finetuning. We manually design prompts for each task. For classification tasks, we use verbalizers to map the class to word labels and for regression tasks, we adopt method similar to Gao et al. (2021) in that we use two anchor words "Yes" and "No" and consider the probability of pre- dicting "Yes" the final score. For span-based tasks, we train the model to directly generate the sequence outputs. A list of prompts can be found in Table 8 and Table 9 in the Appendix.
3.3 Dataset Summary
The final SOCKET benchmark contains 58 tasks from 35 datasets, grouped into the five categories shown in Figure 1. We denote multiple tasks from the same dataset by adding the task name as a suffix following the dataset name and # symbol.
The collection of tasks chosen for SoCKET makes a comprehensive benchmark to measure language models' abilities to capture underlying so- cial information. Motivated by theories of systemic functional linguisties and interpersonal pragmatics, SocKET cuts across number of dimensions of interpersonal communication, allowing also be a tool to better understand and interpret co-learning abilities and dependencies in sociolinguistic tasks. Having this ability allows researchers and users to more efficiently and effectively deploy NLP meth- ods by providing empirical results on the limits and affordances of variety of out-of-domain social language tasks.
Zero-shot predictions We further apply our de- signed prompts to test the performances of LLMs in zero-shot setting where no further finetun- ing is performed. Using the same prompts pro- posed in Table 8, we test SOCKET on several widely used LLMs: GPT (Radford et al., 2018). GPT-J-6B (Wang and Komatsuzaki, 2021), OPT
In total, SOCKET spans 2,616,342 items across

---

Calegery
Model
well dimension No. panma (R)
Himor Strcaom
Offirn.
Fio.
Factors
Trust
REG
SPAN
Ang
haseline
majority
rankkm
0.27
0:40
0.35
0.42
0.17
0.12
0.25
0.36
0.41
0.35
0.58
0.39
0.34
0.51
0.50
0.50
0.00
0.00
0.32
0.32
ElouherAl-gpij--
0.39
0.35
0.29
0.35
0:28
0.32
0.26
0.50
0.08
0.32
alpaciemalive
0.33
0.44
0.45
0.55
0.31
0.43
0.48
0.57
0.17
0.45
bigscience blooma--
0.43
0:45
0.19
0.56
0.09
0.18
Ceebbrs-Ceeebraat GPT-4-78
6.7
0.34
0.52
0.36
decepoda reseurch llatio
13
0.31
0.53
0.40
2ero-drot
fueehook-npe-
13
0.28
0.25
0.49
0.31
gongle fla--5-add
0.63
0.55
(5-3b
0.49
0.36
0.56
GPT:3
0.67
0.56
bon base uncased
0.11
0.78
0.16
0.65
0.70
0.62
0.70
0.79
0:77
0.55
0.71
mibertt- base
0,086
0.79
0.77
0.68
0.72
0.63
0.70
0.83
0.79
0.64
0.72
finouniig
deberta-3
0,058
0.83
0.77
0.70
0.72
0.66
0.73
0.57
0.79
0.63
0.75
Minil M
0.086
0.77
0.72
0.61
0.67
0.58
0.66
0.78
0.69
0.57
0.67
15
0.25
0.68
0.72
0.55
0.39
0.47
0.65
0.66
0:45
0.54
0.62
Table 2: A comparison of the benchmark performances of different models and training schemes. Best-performing instances are shown in bold. The best performing parameter size for each zero-shot model shown (cf. Figure I). A full comparison of all models across all settings can be found in Table 4 in the Appendix. The performances each individual task using a DeBERTa-V3 model can be found in Table 10 in the Appendix.
(Zhang et al., 2022), T5 (Raffel et al., 2020),
two models, google- saalee-fan------- xxl and GPT3.5, are much closer in performance to supervised models.
LLaMA (Touvron al.. 2023a), LLaMA-2 (Tou-
vron et al., 2023b), BLOOM (Workshop et al.,
Social knowledge can be hard to infer Our bench- mark results reveal that even our best-performing model leaves significant room for improvement, scoring just above 0.7 overall- compared with the models' analogous performance syntactic and discourse NLU tasks (He al., 2021) which are often much higher. A comparison among cate- gories of social knowledge reveals that humor & sarcasm is generally the easiest to detect, while trustworthiness is the hardest. This performance gap can be attributed to the level of understand- ing required for each dimension while detecting humor or other social emotions can often be cor- related with cues such sentiment, detecting the level of trust within sentences requires more under- standing of the context and may be harder to detect using computational models (Choi et al., 2020). At a task level, we observe that models struggle most in span detection tasks. This is complex task due to its open-ended nature, and thus SERT-based fine- tuning does not perform well as in other types of tasks. We highlight that learning the various aspects of social knowledge is indeed challenge for current LLMs, and thus call for the need for future models with improved social capabilities.
2023), BLOOMZ (Muennighoff et al., 2022),
FLAN-TS (Chung et al., 2022b), RedPajama (Com- puter, 2023), and Alpaca (Taori et al., 2023; Wang et al., 2022b). We also evaluate the performance of GPT-3.5 using OpenAI's API. Samples for which a model does not provide appropriate label are automatically marked as incorrect. For each LLM variant, test zero-shot results for different model sizes ranging between 110M and 13B parameters, which we report in Table 4 in the Appendix.
4.2 Results
We compare model performances across category type and task type as shown in Table 2. Each re- ported value the average of the scores on ev- ery task within the specified group. The ratio- nale behind using a unified average score is to provide a high-level comparison of the perfor- mances of zero-shot and fine-tuned models un- der various settings, including task type (regres- sion/classification/aaar/ of social knowledge.
DeBERTa-Ve V3 achieves the best overall perfor- mance after full training on each of the SOCKET datasets, followed by other BERT-based models.
Supervised models significantly outperform zero-shot models Table 2 reveals that despite be- ing much smaller in the number of parameters, fine- tuning supervised models such as MiniLM leads to much better performance than zero-shot mod- els using trate-of-the-an L.LMs. All the zero-shot LLMs performed poorly, many on par with random baselines, apart from FLAN-T5. Figure shows
The pased finetuning of T5 performs worse than standard finetuning, especially on the pairwise classification and regression tasks. Meanwhile, most zero-shot models perform only slightly better than the baseline, indicating that prompting alone does not elicit correct social knowledge- thouledgetthough though

---

ALPACA
Average score across all tasks
0.5
BLOOM
BLOOMZ
13
1bn
FLAN-TS
GPT D.PEo.oooss-. fu
GPT
bloom
Iomn
GPT-J
020-136
0.3
ham-lhh
LLAMA
LLAMAZ
MPT
0.2
blaam 5t0m
GPT
lomb 130
OPT
Calange
15-3c
RedPajama
GIT-5DOM
8
10
No. params (billions)
Figure I: A comparison of LLMs on the aggregated scores tested on SocKET under zero-shot settings. The overall performances vary greatly by model architecture, while larger models do not always guarantee better performance.
a detailed picture of how different LLM parame- ter sizes influence the ability to comprehend social knowledge tasks in zero-shot setting. Surpris- ingly, find that of the various training schemes FLAN-TS is by far the most effective for inferring social knowledge, even with relatively small mod- els. We speculate this performance is due to its initial pretraining on more than 1,000 tasks.
When models are able to answer the question, are they right? Restricting only to instances in which model outputs valid answer reveals het- erogeneity among different model groups (Fig- ure 3), showing an interplay between model size, coverage, and performance. For urchitectures such as FLAN-TS or BLOOMZ we observe posi- tive correlation between parameter size and perfor- mance, both in its ability to understand instructions and to make correct predictions. On the other hand, for certain architectures having larger parameters can actually make it worse at understanding in- structions (e-g. LlaMA) predicting correctly (e.g. ОPT), Recognizing that measuring of instruction understanding and the accuracy of an LLM both depend on how strictly one chooses to map the pre- dictions to an answer, overall, our results suggest that while LLMs do contain the potential for under- standing social knowledge, additional steps such finetuning instruction tuning are likely needed for better social understanding.
More parameters do not guarantee more social knowledge Another general trend we observe is weak correlation between the number of parame- ters and overall performance within the same model architccture (p- 0.266, p= -08). This is to some extent determined by the model's ' ability to under- stand the task itself given an instruction prompt as well as a sample input, as larger models are capa- ble of understanding wider variety of tasks (ef. Appendix Table 6). Of course, it is also possible that larger LLMs could encode a greater amount social knowledge through their greater parame- ter sizes. Interestingly, we observe that for some models, larger size does not always guarantee bet- ter performance. This is the case especially for BLOOM, T5 and GPT, where the largest model is not always the best performer within the group.
5
Do we see Cross-task Transfer of Social Knowledge?
Models varied the ability to follow instruc- tions (Appendix Table 6). As expected, instruction- tuned models like FLAN-T5 and Alpaca are gener- ally able to follow the prompt instructions, while other models may generate answers that are not provided in the options. For our social tasks, instruction-follloiing was not significantly corre- lated with model size (p=0.08, p=0.60). Thus, lower model performance in Figure/ is, in part, due to models being unable to answer questions relating to social knowledge.
In this section, we examine the relations and de- pendencies between tasks using the predictions of LLMs trained on different tasks and test for depen- dencies between tasks that are predicted by theory. Quantifying Task Dependency We quantify the dependency between two tasks as follows. We fine- tune pretrained LLM on task L obtain model mA, which is used make predictions on the test set of another task fg. The correlation between the predicted values from model ma and the truc labels the test set of t is considered as the task

---

Humor Sarcasm
1.0
results show that hate speech datasets are often overfit and do not generalize well to other similar datasets Figures 2 & 6, however, show that of the seven datasets included in SOCKET, five of them included at least one task which showed compara- ble correlations when tested both within and out of domain. Indeed, PersonDirreteelAbuss. task labeled for offensive language specifically directed towards individual, is actually predieted better by models fine-tuned on jigsaw# tasks than it was on its own.
0.8
Offensiveness
0.6
Sentiment & Emotion
0.4
Social Factors
Trustworthiness
0.0
H 5
Offens.
G Soc. Trust.
Figure 2: Heatmap task dependency among all task pairs, annotated at category level. Each value represents the absolute strength of correlation between the true lahels of the test set of specific task (columns) and the predictions made on that task using model trained on a different task (rows). We observe strong correlations, especially within the Offensiveness, Sentiment & Emo- tion, and Social Factors categories. A larger version labeled the task level is shown in Appendix Figure 6.
Interestingly, correlations are scarce within the Humor & Sarcasm, and Trustworthiness categories. This is consistent with findings from (Hu et al., 2022b) which show that models without exposure to linguistic forms lack the requisite social infor- mation to perform well on non-literal pragmatic phenomena such as humor and deceit.
Another notable individual task is humor_ rating from the Humor & Sarcasm dataset, which per- forms well as both the fine-tuning and predicted task alongside a number of tasks from the Emotion & Sentiment ceateorry-aartcuune emotion tasks, as well as hateoffensive in the Of- fensiveness selegory-whice labels comments as either "hateful," "offensive," or neither. While re- lationships between offensiveness and humor have been theorized as early Freud (1960) and sen- timent recognition has been shown to bolster of- fensive language detection (Liu, 2012), relatively little has been said regarding connections between the three categories and thus, this result presents an opportunity for further research.
dependency that la has on 4B. We report the ab- solute correlation value, as negatively correlated tasks are still informative. We describe how the correlations are obtained across different task types in the Appendix ($B.6). Span identification tasks are omitted from this analysis, resulting in 55 x 55 scores. We also measure the pairwise correlation between models mA and mB as well as task de- pendency to gain an additional perspective of task similarity. Details for the model correlation can be found in Appendix $B.6 and Figure 7.
discretized
The task dependencies for all task pairs, shown in Figure 2, reveal salient block structures within the category,* 4 especially for the Offensiveness, Sentiment & Emotion, and Social Factors cate- gories, suggesting the existence of shared knowl- edge within our thematically grouped tasks. These correlations align with existing findings from inter- personal pragmatics the relationships between social knowledge. For instance, increased self- disclosure or pain-related interactions are known to promote both intimacy (questionintimacc) and em- pathy (empathy) (Parks, 1981; Cano and Williams, 2010), two elements within the Social Factors cat- egory, while the usage of emojis (tweet_emoj) as effective symbols are indicative of emotional states such valence (enoeunk*_valennc) and arousal (emobank## arousal) (Fischer and Herbert, 2021), which belong to the Sentiment Emotion category.
We observe that politeness shows strong transfer with many of the offensive and hate speech detec- tion tasks in the SOCKET benchmark. In particular, those tasks with high correlation within the offen- sive category are highly correlated in predicting the politeness classification task. This finding is supported by literature showing that impoliteness can fall under the umbrella of offensive language (Bączkowska, 2021) and, although key differences exist in the pragmatics of the two, the constructs are closely related (Parvaresh, 2023; Culpeper, 2021).
Interestingly, regression tasks (from the ha- hackathon, emobank, and empathy datasets) in general have strong correlations with several other tasks. This trend suggests that tasks labeled with continuous variables may have more expressive power compared to ordinal or nominal categoriza- tion, and thus have a higher potential for stronger
The Offensiveness category shows mixed results in comparison with Arango et al. (2019), whose
"See Figure 6 for fully labeled version.

---

task dependencies. However, the magnitude of the correlation may be influenced by the relative value distributions of different correlation methods. This finding calls for a need for more datasets with continuous labels, which requires more effort but allows models to capture more fine-grained con- cepts of social knowledge.
Model type Category-wise
Category
Single task
All tasks
Humor & Sarcasm
0.76
0.76
0.74
Offensiveness
0.76
0.76
0.76
Sentiment & Emotion
0.64
0.64
0.62
Social Factors
0.67
0.67
0.66
Trustwortbiness 0.66
0.64
Table 3:
The performances of different multi-task set- tings aggregated category level. Numbers with indicate cases where the prediction results significantly differ from the single task setting (paired t-tests).
6 Can Multi-task Training improve Social Knowledge?
Our findings reveal significant task transfer, both within and across task categories, which hints at shared knowledge among tasks. Linguistics studies of social language also note the interrelated percep- tions of different dimensions such as humor and offensiveness (Culpeper, 2021; Attardo, 2008; Al- berts, 1992: LI Hai-hui, 2019). We now examine whether LLMs can learn a more robust sense of social knowledge by training on multiple tasks. Experimental Setup Recent studies have explored the possibility of multi-task training on LLMs, which is training a single model on several different tasks simultaneously, with effects of improving its performance on both seen and unseen tasks (Agha- janyan et al., 2021; Padmakumar et al., 2022). We apply multi-task training on SOCKET, but make one clear distinction from prior work. Whereas pre- vious studies have shown that multi-task training is especially effective when the grouped tasks are of similar types (Padmakumar et al., 2022), we intro- duce a new setting by grouping tasks instead by our defined categories of social knowledge, We expect that same-category tasks contain social knowledge that can be shared across tasks, resulting in LLMSs that learn a more robust concept of the specific dimension than when trained on single tasks.
feet on task performance (Table 3). Although some tasks did benefit from being co-trained within cat- egory (Appendix Table 10) particularly the Of- fensiveness category-wher aggregated the cat- egory level, the average performance is worse. In particular, the Humor & Sarcasm and Trustworthi- ness categories have the lowest levels of within- task and cross-task dependencies ($5). The per- formance drop is less strong in categories with high dependency, indicating that while multi-task training similar tasks may not always improve performance, task-relatednes can help preserve performance when also learning task-specifie new concepts. Together, our results suggest multi-task training on unrelated social tasks hurts overall performance a result contrary to social science ex- pectations of how social information is processed- and points to need to further investigate cases when applying multi-task training as practice to improve the social knowledge of LLMs.
Conclusion
People increasingly interact with LLMs natural conversation. To what degree are these models able to pick up on the social cues? To help answer this question, we introduce SOCKET, an NLP bench- mark to evaluate how well models perform at learn- ing and recognizing concepts of social knowledge. We provide benchmark results using several popu- lar models and provide case studies of studying the inherent social capabilities of LLMs in a zero-shot setting. Surprisingly. LLMs perform moderately at best, with even large LLMs (> 10b parameters) varying widely in their abilities. Additionally, we show that there exist significant task dependencies both within and across task categories, and that multi-task training on task categories can affect model performance. Our work contributes the broader NLP community by fostering future ef- forts toward building and evaluating more socially responsible and coherent LLMs.
A popular method for multi-task training is pre- finetuning (Aghajanyan et al., 2021; Shi et al., 2022), which involves first stage of finetuning on shared
multiple tasks using task-specific heads encoder, then re-using the encoder for downstream tasks, We apply preftinetnning in two different set- tings: (1) category-wise tasks, where we perform pre- pre-finetuning on tasks grouped to the same cate- gory, and (2) all tasks, where all tasks of SOCKET are included in the pre-finetuning stage. Consistent with prior work, we perform the second finetuning stage on individual tasks using the pre- pre-finctuned model initial weights (Aghajanyan al., 2021). Other training details are identical $4.
Results Multitask training had little to negative ef-

---

8 Limitations
get a more robust assessment of model capabilities. c.g., multiple tests of model's ability to recognize humor, order to avoid the pitfalls of ascribing progress on the basis of a single task alone (Sub- ramonian et al., 2023). Nevertheless, SOCKET omits several notable dimensions or forms of so- cial knowledge. Some social aspects of language such pragmatic polysemy (Carston, 2021; Apres- jan, 1974) and idioms (Strässler, 1982) either had too few similar datasets to form a theory-backer category, or there were no existing NLP datasets to test the construct. The latter is the case, espe- cially in the case of linguistic techniques unique to recognize when a speaker is adopting community- specific dialects such as African-Americal En- glish (Hyter et al., 2015; Rivers et al., 2012; Allan, 2007) and Queer Language (Barrett, 2006; Hueb- ner, 2021; Harvey, 2000).
Cross-cultural and multilingual expansions Culture is an important aspects of understanding language, especially within the broader setting of multilingual NLP. In this study, however, make clear distinction between cultural knowledge and social knowledge, the latter of which is our focus for this study. Our work is grounded in social- psychological theory and the sociolinguistics of interpersonal communication, especially dyadic communication. Such studies are often aimed at phenomena that are widely shared across cul- tures while recognizing that cultural variation exists within how those phenomena are perceived. In con- trast, work in anthropology or cultural studies pro- vides a different perspective and grounding. Such work frequently focuses on cross-cultural perspec- tives and what is or is-not shared across cultures. For example, in language, the interpretation of whether something is polite can depend on gender norms (Mills, 2004) and cultural (Lorenzo-Dae Dus and Bou-Franch, 2003), highlighting the potential con- text sensitivity. Similarly, the perception of toxicity can depend on the cultural identities of the reader (Sap et al., 2019 9a; Ghosh et al., 2021). While highly valuable to study, cultural knowledge is a separate construct from social knowledge (though interre- lated) and not the focus of this benchmark, though we hope that our work inspires other benchmarks to help assess such differences.
Social language understanding happens within a static, unspecified context for the current tasks in SoCKET. However, the social context in which message is said can dramatically alter its mean- ing, NLP is just beginning to incorporate the social context into language understanding (Hovy and Yang, 2021). While handful of datasets have be- gun to explore modeling context explicitly, such as through the preceding conversation (Pavlopoulos et al., 2020; Menini et al., 2021), the identity of the speaker (Almagro et al., 2022), the social re- lationship between speakers (Jurgens et al., 2023), or explicit social norms (Park et al., 2021), there currently too few of such tasks to compose comprehensive benchmark with which to measure progress. Future datasets and benchmarks will be needed to study understanding social knowledge when controlling for context.
Regarding multilingual data, SOCKET currently contains tasks based in English due to the limited availability of tasks in non-Enge non-English. While there are few datasets such as HAHA (Chiruzzo et al., 2020) in Spanish and DeTox (Demus et al., 2022) in German, bers yet to provide meaningful grouping This highlights the importance of constructing datasets and frameworks capable of capturing social knowl- edge for a wide variety of languages, which we consider an important future step.
not able find sufficient num-
Thus, SocKET represents a starting point for modeling models' abilities and provides room for improvement via the addition of new categories constructs, as additional data becomes available. Further inclusion of other dimensions and corre- sponding tasks should be an ongoing goal.
Additional dimensions and forms of social knowledge Interpersonal communication con- veys richness of different social information and despite extensive literature review and data cu- ration process, we fully acknowledge that other dimensions of social knowledge are not included in our current benchmark. creating SocK ET, our aim was to focus on diverse categories so- cial knowledge that have multiple tasks in order to
Benchmarks as markers of progress SOCKET fills a current gap for assessing the capabilities of LLMs on understanding social language. How- ever, benchmarks constructs have been rightly critiqued as markers of progress in NLP (e.g., Bow- man and Dahl, 2021; Schlangen, 2021: Subramo- nian et al., 2023), due to aspects such as changing or narrowing the field's definition of task, overem- phasizing or overselling progress in a particular

---

area, or encouraging leaderboard chasing. In de- signing SOcKET, we aimed directly address the pitfalls of benchmark design by selecting diverse set of social language understanding tasks that mir- rored human capabilities recognized in social sci- ence studies; this selection helps ensure a broad measure of performance and that "progress" is not due to improved performance one type of task. However, the benchmark itself does not capture all of social knowledge (nor do we claim as such) and we view it only as a starting point-a yardstick by which to mcasure current systems- with a need for new tasks and benchmarks as models advance in their social reasoning capabilities.
print, which we avoid for both financial and envi- ronmental reasons (Hu et al., 2021; Liu et al., 2022; Lester et al., 2021).
Ethical Considerations
The interpretation of social information in commu- nication is highly subjective in that it can largely vary depending on demographic and contextual factors. Nevertheless, several NLP datasets cre- ated via crowdsoureing, which raises concerns on whether the dataset's labels are truly representative of our society (Talat et al., 2022). Even within our benchmark, there the possibility that for tasks such as offensiveness or humor the crowdsourced labels may undermine phrases that might disregard specific demographic group, which may be in- evitably picked up by LLMs that are trained and evaluated on these datasets. Improved versions of our benchmark should include datasets that are more inclusive in such contexts, which we call for future work.
The use of single metric to measure progress in an area or task can mask meaningful insight and fail contextualize performance. While we follow common practice NLP (e.g., Wang et al., 2018, 2019; Muennighoff et al., 2023) and report a single mean score in Table 2, the design of SOCKET in- cludes specific task categories and types designed to easily and meaningfully inspect what is ulti- mately contributing to the single score c.g., models performing well in classification but poorly in span recognition? Nevertheless, this design is a trade-off: A single score can and likely does pro- mote leaderboard chasing by setting a clear goal to pursue, while completely disaggregated scores like those in Table 4 become unwieldy and make it hard to assess whether meaningful progress is being made when comparing two models. Here, we have opted to report both the overall average and averages for each category and type (10 scores total) in an attempt to balance these two tensions.
There has been increasing concern over the amount of computing resources required for con- ducting deep learning research scale, especially regarding LLMs where task performance is im- proved through larger datasets, increased model parameters, and longer training hours. The time and amount of computing resources required for training LLMs has become nontrivial (Bender et al., 2021), and it has been increasingly aware among machine learning practitioners to consider the car- bon footprint of models and computing methods to minimize risks of global warming. This, com- bined with limited transparency of experiment re- sults, may harm the very concept of open science. Keeping this in mind, we focused on conducting casily reproducible experiments that can be run on single GPU within the time frame of hours or a couple of days at the longest. Some of our findings contribute towards this rightful direction, can be in our investigation multi-task training. More importantly, we highlight the fact that the main contribution of our study is a thoroughly de- signed public framework of tasks for examining the social knowledge of LLMSs. While it is indeed important to develop and improve LLMs that perform better on several tasks, we believe that correctly evaluating the level of social knowledge engraved in these models is an equally important task. Without such scrutiny, the users of LLMs deployed practical settings may be vulnerable to
Technical limitations One major limitation of the current benchmark is we only tested LLMs that have up to 13B parameters. Recent studies show that the LLMs may start to show emergent abilities when they are scaled up above certain threshold (Wei et al., 2022). Due to limited computational and financial resources, we are not able test all very large language models, though we welcome future researchers to work on our benchmark and evaluate the sociability of new and larger LLMss.
Finally, our zero-shot model performance used curated prompts on pretrained models without any further finetuning. While it is widely known that instruction-basse based finetuning specific downstream tasks can greatly improve performance, we delib- crately chose not to do so. Finetuning LLMs with billions of parameters can leave large carbon foot-

---

socially undesirable or unethical content. We sin- cerely hope that our efforts in producing SoCKET can ease difficulties of conducting future studies that aim to examine and improve the social under- standing of LLMs.
Salvatore Attardo. 2008 Semantics and Pragmatics of Humor. Language and Linguisties Compass. 2(6):1203-122
Francesco Barbieri, Jose Camacho-Colladee Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemEval 2018 Task 2: Multilingual Emoji Prediction. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 24-33, New Orleans, Louisiana, Association for Computational Linguistics.
Acknowledgments
The authors thank reviewers for their timely and valuable feedback on the paper, with a special shout-out to RI for their very detailed feedback which certainly made this paper better. We also thank the members of the Center for Social Media Responsibility, especially Paul Resnick and James Park for their support which enabled the initiation of this project. This work was supported by the National Science Foundation under Grant Nos. IIS- 2007251, IIS- TIS-2143522 214522,, and 2137469. The third au- thor was partially supported by grant SES-2200228 from the National Science Foundation.
Lisa Feldman Barrett, Kristen A Lindquist, and Maria Gendron. 2007. Language as context for the per- ception of emotion. Trends in cognitive sciences, 11(8):327-332.
Rusty Barrett. 2006. Queer talk. Encyclopedia of Lan- guage & Linguistics, 10:316-323.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Dehora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Workshop Semantic Evaluation, pages 54-63 Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
References
Ralph Adolphs. 2009. The social brain: neural basis of social knowledge. Annual review of psychology, 60:693-716.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, FAceT '21, page 610-623.
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with rre-iintuning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Com- putational Linguistics.
Julia Birke and Anoop Sarkar. 2006. A Clustering Ap- proach for Nearly Unsupervised Recognition of Non- literal Language. In 11th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 329-336, Trento, Italy. Association for Computational Linguistics.
JK Alberts. 1992. Teasing and sexual harassment: Double-bind communication. Constructing and re- constructing gender: The links among communica- tion, language, and gender, 10:185.
Su Lin Blodgett, Lisa Green, and Brendan O' O'Connor. 2016. Demographic dialectal variation in social media: case study of African-American English. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1119 19110,,,,1111111111 Austin, Texas. Association for Computa- tional Linguistics.
Keith Allan. 2007. The pragmatics of connotation. Jour- nal of Pragmatics, 1946)1004711077.
Manuel Almagro, Ivar R Hannikainen, and Neftalí Vil- lanueva. 2022. Whose words hurt? contextual deter-
Samuel R Bowman and George E Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In 2027 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 4843 4855. Association for Computational Linguistics (ACL).
minants of offensive speech. Personality and Social Psychology Bulletin, 48(6):937-953.
Ju D Apresjan. 1974. Regular polysemy.
Aymé Arango, Jorge Pérez, and Barbara Poblete. 2019, Hate speech detection is not as easy as you may think: closer look at model validation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Re- trieval, SIGIR' 19, page 45 54, New York, NY, USA. Association for Computing Machinery.
Penelope Brown and Stephen C. Levinson. 1987. Politeness: some universals in language usage. Number 4 in Studies in interactional sociolinguis- tics. Cambridge University Press, Cambridge [Cam- bridgeshire]: New York.

---

Penelope Brown, Stephen C Levinson, and Stephen Levinson. 1987. Politeness:
Together Computer. 2023. Redpajama: An open souree recipe to reproduce llama training dataset.
Some universals in lan- guage usage, volume 4. Cambridge university press.
Harald Cramér. 1999. Mathematical methods of statis-
Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Un- gar, and João Sedoc. 2018. Modeling empathy and distress in reaction to news stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural I Language Processing (EMNLP 2018).
tics, volume 43. Princeton university press.
CrowdFlower. 2016. The emotion in text. published by crowdflower.
https:
//data. wwwrrdddrrooollofeeeeeee sent imenttieetteemm seaattt-nallssiiiini innaiss--i-tex- Accessed: 2023-01-14.
Sven Buechel and Udo Hahn. 2017. EmoBank: Study- ing the Impact of Annotation Perspective and Repre- sentation Format on Dimensional Emotion Analysis. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 778, Valencia, Spain. Association for Computational Lin- guistics.
CrowdTruth. 2016. Short text corpus with focus on humor detection. Original-dates 2016-05- 10T12:48:54Z.
Jonathan Culpeper. 2021. Impoliteness and hate speech: Compare and contrast. Journal of Pragmatics, 179:4- 11.
Anna Bączkowska. 2021. "You're re too thick to change the station" - Impoliteness, insults and responses to insults on Twitter. Topies in Linguistics, 22(2):62- 84.
Giovanni Da San Martino, Alberto Barrón-Cedent
Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In Pro- ceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377-1414, Barcelona (online). International Committee for Computational Linguis- tics.
Annmarie Cano and Amanda de C Williams. 2010. Social interaction in pain: Reinforeing pain behaviors or building intimacy? PAIN@, 149(1):9-11 11.
Robyn Carston. 2021. Polysemy:
Pragmatics and sense conventions. Mind & Language, 36(1):108-133.
Cristian Danescu-Niculescu-Mizll Moritz Sudhof, Daniel Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factorr. In Annual Meeting of the Association for Computational Linguistics.
Luis Chiruzzo, Santiago Castro, and Aiala Rosá. 2020. HAHA 2019 dataset: A corpus for humor analy- sis in Spanish. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 5106-5112, Marseille, France. European Language Resources Association.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech De- tection and the Problem of Offensive Language. Pro- ceedings of the International AAAT Conference on Web and Social Media, 11(1):512-51 515- Number: 1.
Minje Choi, Luca Maria Aiello, Krisztián 7solt Varga, and Daniele Quercia. 2020. Ten social dimensions of conversations and relationships. In Proceedings of The Web Conference 2020, pages 1514-
Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. Detox: A comprehensive dataset for German offen- sive language and conversation analysis. In Proceed- ings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 143-153, Seattle, Washington (Hybrid), Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022a. Scaling instruction-finetueed language mod- els. arXiv preprint arXiv:2210.1 11111..0......6...
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022b. Scaling cestruction-finetunee language models.
Suzanne Eggins. 2004. Introduction to systemic tional linguistics. A&c Black.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence model- ing. CoRR, abs/ 411.3555..
Paul Ekman. 1992. An argument for basic emotions.
Cognition and Emotion, 6(3- Publisher: Routledge _eprint: https://doi.org/10.. 19880 1068.
1699-000.

---

Mai ElSherief, Caleb Ziems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent Hatred: A Bench- mark for Understanding Implicit Hate Speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345- 363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41-58, Brill.
John Haiman. 1998. Talk is cheap: sarcasm, alienation, and the evolution of language. Oxford University Press, Oxford. OCLC: 252598275.
Michael AK Halliday, 2004. Introduction: How big is a language? On the power of language. The language of science, -2.
Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson and Walid Magdy, 2022. Semeval-2022 task 6: is- arcasmeval, intended sareasm detection in english and arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802-814.
Michael Alexander Kirkwood Halliday. 1995. Dis- course in society: Systemic functional perspectives. 50. Greenwood Publishing Group.
Michael Alexander Kirkwood Halliday, Ruqaiya Hasan, et al. 1989. Language, context, and text: Aspects of language in a social- secial-semiotic perspective. Oxford University Press Oxford.
Brigitte Fischer and Cornelia Herbert. 2021. Emoji as affective symbols: affective judgments of emoji. emoticons, and human faces varying in emotional content. Frontiers in psychology, 12:645173
Keith Harvey. 2000.
Describing camp talk: lan- gunge/prugmmatie/poliiicc Language and Literature: International Journal of Stylistics, 9(3):240-2)) 2600.
Sigmund Freud. 1960. Jokes and their relation the unconscious. WW Norton& & Company.
Shirley Anugrah Hayati, Dongyeop Kang, and Lyle Ungar. 2021. Does BERT Learn Humans Per- ceive? Understanding Linguistic Styles through Lex- ica. In Proceedings of the 2027 Conference on Empir- ical Methods in Natural Language Processing, pages 6323 13311, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Liye Fu, Susan Fussell, and Cristian Danescu- Niculescu- Mizil. 2020. Facilitating the Communication of Politeness through Fine-Grained Paraphrasing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 5127-5140, Online. Association for Computa- tional Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with diieiennttdsetttgeeed embedding shar- ing. CoRR, abs/21 11 1.95433.
Saeko Fukushima and Michael Haugh. 2014. The role of emic understandings in theorizing im/politeness: The metapragmaties of attentiveness, empathy and anticipatory inference in Japanese and Chinese. Jour- nal of Pragmatics, 74: 165-179.
Janet Holmes, 2006. Sharing a laugh: Pragmatic aspects of humor and gender in the workplace. Journal of Pragmatics, 38(1):26-50. Special Issue: Gender and Humor.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2816-3830, Online. Association for Computa- tional Linguistics.
Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEval-20200 Task 7: Assess- ing Humor in Edited News Hcadlines. In Proceed- ings of the Fourteenth Workshop on Semantic Eval- uation, pages 746-758, Barcelona (online). Interna- tional Committee for Computational Linguistics.
Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In International Conference on Intelli- gent Text Processing and Computational Linguistics, pages 152-165. Springer.
Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In The 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. Associ- ation for Computational Linguistics.
Deepanway Ghosal, Navonil Majumder, Rada Mi- halcea, and Soujanya Poria. 2020. Utterance- level Dialogue Understanding: An Empirical Study. ArXiv:2009.11 13902 [cs].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Sayan Ghosh, Dylan Baker, David Jurgens, and Vin- odkumar Prabhakaran. 2021. Detecting cross- geographic biases in toxicity modeling on social me- dia. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), ), pages 313- 328.
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022a. A fine- grained comparison of pragmatic language under- standing in humans and language models. arXiv preprint arXiv:2212.06M0)

---

Jennifer Hu, Sammy Floyd. Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022b. A fine-grained comparison of pragmatic language understanding in humans and language models. ArXiv:2212.0680 [cs].
in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNI.-IJCNLP)) pages 1696-1706, Hong Kong. China, Association for Computational Linguis- tics.
Daniel R Huebner. 2021. Anachronism: The queer pragmatics of understanding the past in the present. The American Sociologist, 52(4):740-76.
Dongyeop Kang and Eduard Hovy. 2021. Style is NOT single variable: Case studies for cross-stylistic lan- guage understanding, In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 2376-2387, Online. Association for Computational Linguistics.
Yvette D Hyter, Kenyatta O Rivers, and Glenda DeJar- nette. 2015, Pragmatic language of african american children and adolescents. Topics in Language Disor- ders, 35(
Joshua Conrad Jackson, Joseph Watts, Johann- List, Curtis Puryear, Ryan Drabble, and Kristen A. Lindquist. 2022. From text to thought: How ana- lyzing language can advance psychological science. Perspectives on Psychological Science, 17(3)805- 826. PMID: 34606730.
Dacher Keltner, Phoehe C Ellsworth, and Kari Edwards. 1993. Beyond simple pessimism: effects of sadness and anger on social perception. Journal of personal- ity and social psychology, 64(5)740.
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra- halli. 2018. A Large Self-Annotated Corpus for Sar- casm. In Proceedings of the Fleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ro- nan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saa- dia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2021. Can Ma- chines Learn Morality? The Delphi Experiment. Publication Title: arXiv e-prints ADS Bibcode: 2021arXiv2110772
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452 466.
Jigsaw. 2017. Toxic Comment Classification Challenge.
Jigsaw. 2019. Unintended Bias in Toxicity Classifica- tion.
Mali Jin, Daniel Preotiuc-Pietro, A. Seza Doğruöz, and Nikolaos Aletras. 2022. Automatic Identification and Classification of Bragging in Social Media. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1:
Erik Körner, Gregor Wiedemann, Ahmad Dawar Hakimi, Gerhard Heyer, and Martin Potthast. 2021. On Classifying whether Two Texts are on the Same Side of an Argument. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10130-10138, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Long Papers), pages 3945-3959, Dublin, Ireland. As- sociation for Computational Linguistics.
David Jurgens, Libby Hemphill, and Eshwar Chan- drasekharan, 2019. A just and comprehensive strat- egy for using NLP to address online abuse. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 3658- 3666, Florence, Italy. Association for Computational Linguistics.
Kent M. Lee, Kristen A. Lindquist, Nathan L. Arbuckle, Samantha M. Mowrer, and B. Keith Payne. 2020. An indirect measure of discrete emotions. Emotion, 20(4):659-676
David Jurgens, Agrima Seth, Jackson Sargent, Athena Aghighi, and Michael Geraci. 2023. Your spouse needs professional help: Determining the contex- tual appropriateness of messages through modeling social relationships. nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 10994- 11013.
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Proceedings of the sl An-
Dongyeop Kang, Varun Gangal, and Eduard Hovy. 2019, (Male, Bachelor) and (Female, Ph.D) have differ- ent connotations: Parallelly Annotated Stylistic Lan- guage Dataset with Multiple Personas. In Proceed- ings of the 2019 Conference on Empirical Methods
Yan Li, Manoj a Thomas, and Dapeng Liu. 2021. From semantics to pragmatics: where IS can lead in Natu- ral Language Processing (NLP) research. European Journal of Information Systems, 30(5):669-590.

---

Yanran Li, Hui Su. Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog. A Manually Labelled Multi-turn Dialogue Dataset. In Proceed- ings of the Eighth International Joint Conference on Natural Language Processing Papers), pages 986-995, Taipei, Taiwan. Asian Fed- eration of Natural Language Processing.
B.W. Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) Protein Struc- ture, 405(2):442-451
( (Volume I: Long
J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense. In Proceedings of the 15th International Workshop on Semantic Evaluation SemEvvll-2211 pages 105-119, Online. Association for Computa- tional Linguistics.
LI Hai-hui. 2019. Mitigation and Pragmatic Empathy. Journal of Literature and Ari Studies, 9(2).
Kristen A. Lindquist and Lisa Feldman Barrett. 2008. Constructing Emotion: The Experience of Fear as a Conceptual Act. Psychological Science, 19(9):898-
Stefano Menini, Alessio Palmero Aprosio, and Sara Tonelli. 2021. Abuse contextual, what about nip? the role of context in abusive language annotation and detection. arXiv preprint 203.:::213..1999
903.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020. The effect of natural distribution shift on question answering models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6905-697 69.6. PMLR.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61 61-68, Dublin, Ireland. Association for Computational Lin- guistics.
Sara Mills. 2004. Class, gender and politeness. Multi- lingua, 23.
Anirudh Mittal. Pranav Jeevan P. Prerak Gandhi, Diptesh Kanojia, and Pushpak Bhattacharyya. 2021. "So You Think You'l re Funny?": Rating the Humour Quotient in Standup Comedy. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10073-10079, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.
Nuria Lorenzo-Dur and Patricia Bou-Franch, 2003. Gender and politeness: Spanish and british un- dergraduates' perceptions of appropriate requests. Género, lenguaje traducción, pages 187-199.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018, Semeval- 2018 task I: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1-17.
Jing Ma. Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation struc- ture via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 708-717, Vancouver, Canada. Association for Computational Linguistics.
Abhinav Moudgil. Short Jokes.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2006-2029.
Fabrizio Macagno, Chrysi Rapanta, Elisabeth Mayweg- Paus, and Mercè Garcia-Milà. 2022. Coding empathy in dialogue. Journal of Pragmatics, 192:116-132.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Scao, M Saiful Bari. Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf. et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint 2rXiv:221..07780
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought large language models: cognitive perspective. ArXiv:2301.066 [cs].
Asifa Majid. 2012. Current emotion research in the language sciences. Emotion Review, 4(4):432-44) 443.
Rosina Márquez Reiter and David M. Frohlich. 2020. A pragmaties of intimacy. Internet Pragmatics, 3(1):1 33.
G. Da San Martino, A. Barrón-Cedeñoo H. Wachsmuth, R. Petrov, and P. Nakov. 2020. SemEval-2020 Task II: Detection of Propaganda Techniques in News Articles. NxXiv22009.02696 [cs].
Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeñño, Rubén Miguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram

---

Hasanain, Nikolay Babulkov, Alex Nikolov, Gau- tam Kishore Shahi. Julia Maria Struß, and Thomas Mandl, 2021. The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact- Checked Claims, and Fake News. In Advances in Information Retrieval, Lecture Notes in Computer Science, pages 639-649, Cham. Springer Interna- tional Publishing.
15th International Workshop on Semantic Evaluation (SemEval-200) 2022),, pages 59-69 Online. Association for Computational Linguistics.
Jiaxin Pei and David Jurgens. 2020. Quantifying inti- macy in language. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307-5328 Online. As- sociation for Computational Linguisties.
Matthew L. Newman, James W. Pennebaker, Diane S. Berry, and Jane M. Richards. 2003. Lying words: Predicting deception from linguistic styles. Person- ality and Social Psychology Bulletin, 29(5):665-675.
Jiaxin Pei and David Jurgens. 2021. Measuring
Sentence-Level and Aspect-Leve (Un)certainty in Science Communications. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9959-10011, I, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
PMID: 15272998.
Ulf Olsson, Fritz Drasgow, and Neil Dorans. 1982. The polyserial correlation coefficient. Psychome- trika, 47(3):337-347.
Carla Perez-Almendee
Luis Espinosa-Anke and Steven Schockaert. 2022. SemEval-2222 task 4: Patronizing and condescending language detection. In Proceedings of the 16th International Workshop on Semuntic Evaluation 307, Seattle, United States. Association for Compu- tational Linguistics.
Myle Ott, Yejin Choi. Claire Cardie, and Jeffrey T. Han- cock. 2011. Finding Deceptive Opinion Spam by Any Stretch of the Imagination. In Proceedings of the 49th Annual Meeting of the Association for Compu- lational Linguistics: Human Language Technologies, pages 309-319, Portland, Oregon, USA. Association for Computational Linguistics.
pages 298-
Denis Peskov, Benny Cheng, Ahmed Elgohary. Joe Bar- row, Cristian Danescu-Nire Biwecu--iuueeuu--Miiii and Jordan Boyd-Graber. 2020. It Takes Two to Lie: One to Lie, and One to Listen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 381 1-31-,,,4 Online. Association for Computational Linguistics.
Vishakh Padmakumar, Leonard Lausen, Miguel Balles- teros, Sheng Zha, He He, and George Karypis. 2022. large-
Exploring the role of task transferability scale multi-task learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2542-2550, Seattle, United States. Association for Computational Lin- guistics.
Martin Potthast, Tim Gollub, Matti Wiegmann, Benno Stein, Matthias Hagen, Kristof Komlossy, Sebstian Schuster, and Frika P. Garces Fernandez. 2018. We- bis Clickbait Corpus 2017 (Webis-Clickhait-17)
Chan Young Park, Julia Mendelsohn, Karthik Radhakr- ishnan, Kinjal Jain, Tushar Kanakagiri, David Jur- gens, and Yulia Tsvetkov. 2021. Detecting Commu- nity Sensitive Norm Violations Online Conversa- tions. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 3386-3397, Punta Cana, Dominican Republic. Association for Computational Linguistics.
John Pougué Biyonn. Valentina Semenova, Alexandre Matton, Rachel Han, Acrin Kim. Renaud Lambiotte, and Doyne Farmer. 2021. DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates.
Daniel Preotiuc-Piete Mihaela Gäman, and Nikolaos Aletras. 2019, Automatically Identifying Complaints in Social Media. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, ACL..
Malcolm R. Parks. 1981. Ideology in interpersonal communication: Off the couch and into the world. Annals of the International Communication Associa- rion, 5(1)
Reid Pryzant, Richard Dichl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically Neutralizing Subjective Bias in Text. Proceedings of the AAAI Conference on Artificial Intelligence, 38(01):480-44) 489. Number: 01.
Vahid Parvaresh. 2023. Covertly communicated hate speech: cospus-assisted pragmatic study. Journal of Pragmatics, 205:63-77.
John Pavlopoulos. Jeffrey Sorensen. Lucas Dixon, Nithum Thain, and lon Androutsopoulos. 2020. Tox- icity detection: Does context really matter? In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4296- 4305.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. 2018. Improving language under- standing by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res,, 11:140:1-1400
John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. SemEval-202. 2021 Task 5: Toxic Spans Detection. In Proceedings of the

---

Sudha Rao and Joel Tetreault. 2018. Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Bench- marks and Metries for Formality Style Transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- fional Linguislic:: Humun Language Technologies, Volume 1 (Long Papers), pages 129-140, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Linguistics, pages 5477-5490, Online, Association for Computational Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLPP, pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguisties.
Ann- Ann-Katrin Reuel, Sebastian Peralta, João Sedoc, Gar- rick Sherman, and Lyle Ungar. 2022. Measuring the Language of Self- Disclosure across Corpora, In Find- ings of the Association for Computational Linguis- tics: ACI. 2022, pages 1035-1047, Dublin, Ireland. Association for Computational Linguistics.
Klaus R. Scherer and Harald G. Wallbott, 1994, "Ev- idence for universality and cultural variation of differential emotion response patterning": Corree- tion. Journal of Personality and Social Psychology, 67(1):55- 555 Place: US Publisher: American Psy- chological Association.
Kenyatta O Rivers, Yvette D Hyter, and Glenda DeJar- nette. 2012. Parsing pragmatics. The ASHA Leader, 17413):14-17.
David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference Natu- ral Language Processing (Volume Short Papers), pages 670-674.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEvall22117 task 4: Sentiment analysis in Twitter. Proceedings of the 11th International Workshop on Semantie Evaluation (omEvall-201)) pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguistics.
Willibald Ruch. 2010. The sense of humor: Explo- rations of a personality characteristic. volume 3. Walter de Gruyter.
Stefan Schneider. 2010. Mitigation. pragmatics, pages 253-269.
Handbooks of
Stephanie Schnurr. 2010. 13. humour. Interpersonal pragmatics, 6:307.
Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. Large language models are not zero-shot communi- cators.
Ashish Sharma, Adam Miner, David Atkins, and Tim Al- thoff. 2020. A computational approach to understand- ing empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263-5276, Online. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero- zero-shot task gener- alization. In International Conference on Learning Representations.
Yiwen Shi. Taha ValizadehAslani. Jing Wang, Ping Ren, Yi Zhang, Meng Hu, Liang Zhao, and Hualou Liang. 2022. Improving imbalanced learning by pre- finetuning with data augmentation. In Fourth In- ternational Workshop on Learning with Imbalanced Domains: Theory and Applications, pages 68-82. PMLR.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pages 163 1-1642, Seattle, Washington, USA. Asso- ciation for Computational Linguistics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019a. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668-1678.
Maarten Sap. Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Im- plications Language. In Proceedings of the 58th Annual Meeting of the Association for Computational
Ellen Spertus. 1997. Smokey:
Automatic recognition of hostile messages. In Anaifiaii, pages 1058-1065.
Karyn Stapleton. 2003. Gender and swearing: A com- munity practice. Women and Language, 26(2):22.

---

Gerard Steen, Aletta G. Dorst, and J. Berenike Her- mann, editors. 2011. A method for linguistic metaphor identification: from MIP to MIPVU. Num- ber 14 in Converging evidence in language and com- munication research. Benjamins, Amsterdam.
Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan. Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Jürg Strässler. 1982. Idioms in English: A pragmatic analysis, volume 183, Gunter Narr Verlag.
Elliot Turiel. 1983, The development of social knowl- edge: Morality and convention. Cambridge Univer- sity Press.
Arjun Subramonian, Xingdi Yuan, Hal Daumé III, and Su Lin Blodgett. 2023. It takes two to tango: Nav- igating conceptualizations of NLP tasks and mea- surements of performance. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023, pages 4234-3279, Toronto, Canada. Association for Computational Linguistics.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th Interna- rional Workshop on Semantic Evaluation, pages 39- 50, New Orleans, Louisiana. Association for Compu- tational Linguistics.
Thakur Ashutosh Suman and Abhinav Jain. 2021. AS- tarTwice SemEval-2021 task 5: Toxic span de- tection using ROBERTa-CUL CRF, domain specific pre- training and seff-training. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval- 2eme-vl--022), pages 875-880, Online. Association for Computational Linguistics.
Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Introducing CAD: the Contextual Abuse Dataset. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2289-2303, Online. Association for Computational Linguistics.
Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general- purpose language under- standing systems. Advances in neural information processing systems, 32.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Associa- tion for Computational Linguistics.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instruction-folloiingg Ilaaa model. ht https:/ //gttbb/ comm/attu-laa stanford stanford_alpaca.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
Ben Wang and Aran Komatsuzaki. 2021. J-6B: A 6 Billion Parameter Autoregressive Language Model. https://githut com/ k kingoflolz,
GPT-
transformer---eeer jaane
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA. Curran Associates Inc.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer. Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinct, Todor Mihaylov, Pushkar Mishra. Igor Moly- bog. Yixin Nie. Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu,
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022a. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages
Seattle, United States. Association for Computational Linguistics.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions.

---

Zijian Wang and Christopher Potts. 2019. TalkDown:
Leon Weber, Long Phan, Loubna Ben allal, Lu- dovic Tanguy, Manan Dey, Manuel Romero Muñoz. Maraim Masoud, María Grandury. Mario Saško, Max Huang. Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Moham- mad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Pe- ter Henderson, Pierre Colombo, Priscilla Amuok. Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silber- berg. Suhas Pai, Sydney Zink, Tiago Timponi Tor- rent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina. Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Ta- lat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajy- oti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang. Harshit Pandey, Hendrík Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Sai- ful Bari, Maged S. Al-shaibani, Matteo Manica, Ni- hal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim. Tali Bers, Thibault Fevry, Trishala Neeraj. Ur- mish Thakker, Vikas Raunak, Xiangru Tang, Zheng- Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh. Adam Roberts, Hyung Won Chung. Jaesung Tae, Jason Phang. Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper. Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sansevicro, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, San- chit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil. Tim Dettmers. Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lover- ing, Dan Garrette, Deepak Tunuguntla, Ehud Reiter. Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bog- danov, Genta Indra Winata, Hailey Schoelkopf, Jan- Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Minuna Clinciu, Na- joung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais. Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz. Verena Rieser, Vitaly Protasov. Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdennč Kasner, Al- ice Rueda, Amanda Pestana, Amir Feizpour, Am- mar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj. Arash Aghagol. Are- zoo Abdollahi, Aycha Tammour, Azadeh HajiHos- seini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contrac- tor, David Lansky, Davis David, Douwe Kiela,
A Corpus for Condescension Detection in Context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNTLP) pages 3711- 3719, Hong Kong, China. Association for Computa- tional Linguistics.
Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Lan- guage Online, pages 78-84, Vancouver, BC, Canada. Association for Computational Linguisties.
Zeerak Waseem and Dirk Hovy. 2016, Hateful Sym- bols or Hateful People? Predictive Features for Hate Speech Detection Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88- 93, San Diego, California. Association for Computa- tional Linguisties.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.0165..
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint 220..
Ludwig Wittgesstein, 1953. Philosophical Investiga- tions. Basil Blackwell, Oxford.
BigScience Workshop, i, Teven Le Seao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné. Alexandra Sasha Luc- cioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush. Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Alhert Vil- lanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major Iz Belt- agy. Huu Nguyen, Lucile Saulnier, Samson Tan, Pe- dro Ortiz Suarez, Victor Sanh, Hugo Laurençon. Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Lev- kovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina. Hieu Tran, Ian Yu. Idris Abdulmumin, Isaac Johnson, Itziar Gonvalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Al- mubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra,

---

Duong Nguyen, Edward Tan, Emi Baylor, Ez- inwanne Ozoani, Fatima Mirza, Frankline Onon- iwu, Habib Rezanejad, Hessie Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim El- badri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nañis Abrar, Nazneen Ra- jani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An. Rasmus Kromann, Ryan Hao, Samira Al- izadeh, Sarmad Shubher, Silas Wang, Sourav Roy. Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le. Yoyo Yang. Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-FEscalada, Ayush Singh, Benjamin Beilharz, Bo Wang. Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Ranga- sai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato. Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pamics, Maria Castillo, Mari- anna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg. Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok Deshmukh, Shubhanshu Mishra, Sid Ki- blawi, Simon Ott, Sinee
Workshop on Semantic Evaluation, pages 75-86, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- Mona Diab, Xian Li, Xi Victoria Lin, al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXi::200500066
Yunxiang Zhang and Xiaojun Wan. 2022. MOVER: Mask, Over-generate and Rank Hyperbole Gener- ation. ArXiv:2109.07726 [es].
Srishti Ku- mar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Ya- nis Labrak, Yash Shailesh Bajaj. Yash Venkatraman, Yifan Xu. Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b- roob-parameter open-access multilingual language model.
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense pre- trained language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34. pages 9733-9740.
Liang Wu, Fred Morstatter, Kathleen M Carley, and Huan Liu. 2019. Misinformation in social dia: definition, manipulation, and detection. ACM SIGKDD Explorations Newsletter, 21(2):80-90.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.
2019a. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 1415-1420, Minneapolis, Minnesota. Association for Computational Linguistics.
Avner Ziv. 2010. The social function of humor in inter- personal relationships. Society, 47(1): 1-18. 1-18..
Marcos Zampieri, Shervin Malmasi, Preslav Nakov. Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and cat- egorizing offensive language in social media (Of- fensEval). In Proceedings of the 13th International

---

A Details on dataset processing
the evaluations of both versions, demonstrating that SOCKETTE is indeed representative sample of SOCKET.
A.1 Benchmark construction ($3)
The SoCKET dataset consists of 58 tasks from 35 unique, public datasets. The datasets that make up the benchmark dataset are processed in way that is meant to balance uniformity across datasets and tasks while minimizing deviations from the original dataset.
B.4 Details on language model finetuning ($4,
$5, $6)
B.4.1 Task-specific heads ($4, $5, $6)
As our benchmark consists of four different task types: classification, regression, sentence pair de- tection, and span identification we maintain a unified structure for each task where each sample is fed into the encoder of LLM, and the out- put states are then fed into task-specific head layer. For span detection tasks, we feed the last hid- den layer into a bidirectional GRU (Chung et al., 2014), and then the output vectors of the GRU into a linear layer that transforms each vector into a dimension of 3, corresponding to the [B,I,O] la- bels for each token, following earlier work in span identification (Suman and Jain, 2021). For all other tasks, we feed the last hidden state of the encoder corresponding to the [CLS] token into separate classifierrregression head consisting of two linear layers of hidden size 768 and a dropout probabil- ity of 0.1. We use the mean squared error loss for regression tasks and the cross-entropy loss for all other tasks.
For all datasets, key changes from the original dataset are twofold:
Duplicates and unlabeled items are removed from all datasets. If duplicates occur across data splits, the splits are recombined, reshuf- fled, and split.
All datasets are split 80%/10%/10% be- tween train/test/dev splits, respectively. Any datasets not split 80%//0%//10 are recom- bined, reshuffled, and split 80%/10%%/10%
All datasets made compatible with the Hug- ging Face Datasets package.
Experimental Details
B.1
Computational resources ($4, $5, $6)
All of our experiments were conducted on an Ubuntu 22.04.1 machine installed with NVIDIA RTX A5000 and A6000 GPUs. The Python pack- ages used in our experiments include Pytorch 1.13. Transformers 4.21.3, and Pytorch Lightning 1.6.4.
B.4.2 Training strategies for language model finetuning ($4, $6)
When training models for the benchmark ($4) and the multi-task ($6) experiments. the learning rate was linearly increased for 6% of the training steps up to 1c-5 and linearly decreased afterward. All models were trained for a maximum of 10 epochs using three different seeds, with early stopping after validation performance did not increase for three consecutive epochs.
B.2 Comparison of all models
Table contains detailed version of Table 2, where the scores of every single task are presented.
B.3 Details on the comparison between SOCKET and SOCKETTE
Our multi-task training in $6 requires two stages of training: (1) a pre-finetuning stage that simulta- neously trains a model on multiple different tasks, and (2) a finetuning stage that loads the model trained from (1) and finetunes it to a single task. the first stage, a single batch can include several different tasks and produce different types of losses. To obtain a unified loss that is differentiable, we aggregated the loss for each sample and sum them up. which we use for backpropagation. For both stages, we use the same aforementioned training steps and learning rate strategy.
32 of 58 tasks contained more than 1,000 test samples, resulting in a disparity between the sizes the original SOCKET and SOCKETTE variants. To test that both datasets still offer comparable evaluations for testing models, we compare their scores for supervised model and compare test set performances. For each task, we train a deberta- v3- model. evaluate using the test sets of both versions, and compute the correlation between each setting using Pearson's r score. We provide eval- uation results of SOCKETTE for our models in Table 5. Also, we show through Table 7 and Fig- ure 5 that there exists a strong correlation between
For all settings, the training batch size was set to 32 with 16-bit precision enabled. Validation was

---

Fadie
rugett
023
1/4
M12
0.23
B61
0.30
0.34
14:0
1922
Ekvleshl pr jái
0.35
0.39
0.31
0.26
0.30
0.36
0.90
E/6
10:33
0.45
833
0142
0.56
0.35
hyponnecteaee Iol
26
0.0
0.3
0850
0223
1145
0.11
0.31
ligh lasen
83
0.30
0.3
0.35
8.58
0.51
6.11
0.75
BA2
01.8
BA1
11.0
510
0.12
Apparacideesp//
063
1359
0143
0.53
1336
0.43
woddead Carlon (PP Dold
0.99
31
D11
0.18
5.00
0.5
0.33
6/6
0.14
costrax Cortin LPP2278
22
0.32
B10
6118
D166
4.77
538
0.19
coopal-rneee Iaze-7)
65
0..6
122
0.38
0.34
0032
6.1
0.55
tacchal-cpp-15
22
0120
124
suchool oge 150
060
1000
0.35
1129
613
0.11
prolotin Shbe
0.25
140
0645
BA1
0.43
prople tome
DA1
0.39
posfo-tacd-ed
196
0652
0.59
03.4
0883
6.17
0.5
125
0.300
6.55
0.23
635
0.35
100
1021
6.15
8.55
0.56
477
23
111
1636
0.23
540
0.30
0.30
0:13
(5-10
3358
035
0.56
0.31
6.5
0.35
0.33
0.33
0.37
0148
0.53
Thaas Th-chr
6.54
UP133
0.64
0.5
0457
0.53
0.57
0349
088
021
0.6
rshons-tard
0.89
0.18
877
R45
0.33
0.30
0383
1179
664
012
Miilld
072
0411
0.57
0.58
0.51
0.8
0.53
647
0.55
Table comparison of the benchmark performances of different models and training schemes. Best-performing instances are shown in bold.
Cinggry
Midli
Ko poomciilirss
Brear & facein
Difesheees Sreisvet& Trasia
01.5
RATR
650
SNW
Iadix
+ejely
1.17
0.35
020
[kenterllammj An
0.55
0.55
6.31
0.35
3.30
0.6
111
0.25
516
0121
0.25
0.43
tun
0.22
"rg in leeen
33
066
0.30
0.1
8.5
0.37
33
0.51
0.17
hiackenee Now hipolinns Naveze
1.27
1130
0200
0.56
0.38
0.10
Iepoccweddeeno fol
44
044
1.30
0.43
BA1
1142
13.4
1032
0.0
0.43
011
31
D00
816
0.54
E00
0.5
1.19
0.89
6.00
3.10
Cantrn
0,39
0.3
611
015
D00
13.3
3.10
undten Cordina CPTE77
6.2
1.0
1030
0.5
0.33
0.3
0.13
3.16
D65
0.30
8.1
033
8.53
0.5
D.17
0.40
bachhol-pp-122
140
0120
4.56
1324
635
6.7
031
1630
6.17
021
0.13
0.55
6.6
0.24
pnegle Ean---an andd
0.58
0.30
BA1
0.39
D16
0.30
prole
120
0550
DA1
0449
peopl fa
055
1.5
BAT
055
0.90
6114
0.41
142
0.39
6.5
0.37
025
0.30
122
0.22
0.6
U22
0.43
115
1114
0.21
037
0.33
D10
0.11
(5.30
13
621
015
633
0.19
0.15
536
3.19
6:1b
015
verhesoeepaan Rothi wegggeee-e INCIT--1
0.39
127
1055
0.34
526
0.25
0.39
1.30
5115
0.0
Non basc excared
011
0.58
026
RA5
0.33
562
0.70
0.78
4777
555
011
6.0
0.32
6/6
MaliM
0166
0.1
675
041
0187
696
0.66
Table 5: A comparison of the benchmark performances of different models and training schemes on the SOCKETTE test set (a subset of SoCKET). Best-perforning instances are shown in bold.

---

Moldd
prrnscilied Herer
Dffeciiton
BEC
chiddai apwa
1500
110
0.09
0.76
D.61
1656
033
1633
266
034
035
326
142
0.31
5776
1096
3.0
0766
0.59
8.55
3.5
0.53
0.56
(191
1901
150
0.60
8.55
0.9
0166
1.0
110
100
D052
1001
256
0.00
110
0.00
0.59
3.15
1988
0.79
paele
1281
150
1052
0.50
0355
peplo
1381
1222
8:30
unchaa Cantraa GrESic
1030
1090
8310
1631
2.66
6.30
9.6
0.30
0.34
D66
6700
1981
8.00
0.50
230
0.60
8.85
581
0.36
0.66
633
decpeta 2017
1075
3.50
1851
0.00
0.87
230
dac couurt
104
2168
1650
6.5
1036
1862
1872
2.80
6.80
333
033
164
faclandin opt-3
3700
8.54
6.5
195
180
3.98
100
005
0.8
066
0.73
8.50
031
561
nyy descceuuae BuPajana INCITE Jarra
1001
190
3.99
100
093
0.59
1022
1980
238
0823
246
0384
0.8
379
013
370
13:3
0640
0.30
0.49
0.54
022
0.55
044
p-1bb
1991
0314
1123
0.3
238
0.35
Tlaach ge-35
8.05
540
100
110
0.99
100
840
180
580
009
1.00
156
Table 6: The fraction of samples that each LLM can make inferences given the instruction prompts, when tested in a zero-shot setting.
0.55
flan-t5-xxl
Group
flan-15-xl X
GPT-J
0.50
flan-t5-large
BLOOM
Overail score across all tasks
bloomz-7bp
BLOOMZ
0.45
gpt-j-6b
flan-t5-base
redpajama
GPT
0.40
opt-6.7b
15-11b
ppt-13b
opt-1.5
mut Dww. lbl
b1oomz-560n
ALPACA
LLAMA
t5-3b
llama
flan-t5-sma
OPT
0.35
m
GPT-6. 78 bloom-3b
llama-13b
bloom-1b1
opt-2.7b
FLAN-TS
LLAMA2
0.30
lla mama2-7b
MPT
bloom-560nt
bloom-7b1
0.25
GPT-111M
base
GPT-2.78
0.20
GPT-590M
S-lagg
t5 t5-sthall
0.15
0.2
0.3
0.4
0.5
0.6
0.7
Ratio valid samples
Figure 3: A comparison the ratio of valid samples which the LLM was able to make an inference given the correct instruction prompt (x-axis) versus the overall scores when limited to the samples that the model was capable of making an inference (y-axis).
made after each training epoch on the validation set using Pearson's correlation added by and di- vided by 2 for regression tasks and macro Fl score for all other tasks. If there were multiple tasks con- sidered due to multi-task training, the average of all task performances was used as the final validation score.
B.5 Details on prompt-based finetuning ($4, $5)
We use fix prompts fine-tuning for all the prompt- based models. The batch size was set as 32 for training. For every single task, we set 10 as the max epoch and do early stopping based on the validation loss. The learning rate is set as 5e-5.
For classification tasks, the model is fine-tuned to generate the target label. For regression tasks, we first normalized the scores into (0,1) and then split the labels into two groups. The model fine-

---

x
Group
flan-t5-xl
GPT-J
0.5
bloomz-7b1
flan-tS-large
BLOOM
0.997, indicating SOCKETTE can be reliably deployed Overall score across tasks
alpaca-native
BLOOMZ
redpajama
flan-t5-base base
Bloomz-11
GPT
0.4
opt-1 0
50o0m--560m
ALPACA
nnA5-mmal
LLAMA
gpt-i-b
GPT
7B
2.7.
bloom-3b
OPT
t5-11b
0.3
opt-13b
om-
llama-7b
llama2-7
FLAN-TS
LLAMA2
bloom-7b1
opt-6.7b
mpt-7
instrucc
MPT
13b
0.2
bloom- 560m
15-3
GPT-2.7
base ts-large GPT-590M shall
0.1
GPT-111M
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Ratio valid samples
Figure 4: comparison ratio of valid samples which LLM was able make an inference given the correct instruction prompt (x-axis) versus the overall scores across every sample in the test dataset where failed predictions are considered incorrect (y-axis).
Pearson's r: 0.997
1.0
task
SoCKET
SoCKETTE
contextual ahaseffldennttyyirrcccedddd
0.62
0.58
5OCKETTe score
contextuul-auseet hrr.aheeeeeerrroriitttttttuu
0.8
croddflower
0.24
0.22
dailydialog
hasbiseedinplicaiionn hateoffensive
0.86
0.87
0.6
0.93
bumor-pairs
0.98
0.97
implicit-bon hetiiliitteeppiiiiccc....
0.72
0.4
hatpiiithaateimmliiii__iie hate
0.71
0.72
implicit-hatollaiiii hatefincitemeot hate
0.68
umplicit- bateHlafctioritty hate
0.60
0.69
0.2
hatetstereotypicct
0.68
implicit- implicit-hneethheuaee hnte@threateningg... hate
0.63
0.67
0.5
1.0
implicit- beiiitt-attehhire grievance hate
0.70
intentyn
0.75
0.73
SOCKET score
iissaeiidentiyy.o
0.80
igsaweinsultt
0.88
0.87
lgsaweobseeee
0.91
Jisaaeseverre
0.73
0.74
Figure 5: For each of the 32 tasks SOCKET con- taining more than 1,000 test samples, evaluate the performance of a deberta-v3 model trained on a single SOCKET task on both the original test set as well as the
jigsawethreat
0.85
1.00
jigsawtoxic
0.90
0.90
neutralizing bias pairs
0.98
0.98
offensiveyn
0.82
0.82
sare
0.74
0.77
smaller SOCKETTE variant. The correlation between
sentireebunk
0.97
0.97
sexyn
0.79
0.77
the two scores results in a high Pearson's 's r score of
loxic-span
0.68
tweet_emoji
0.34
0.32
ceet_cnttionn tweet tweet_ sentiment
0.81
for more rapid model testing.
0.71
0.69
two-to-lisfrcceiver, truthh
0.59
0.60
pro-to-listsender_trutt
0.58
0.58
Table 7: comparison of the evaluation scores between the test for SocKET versus SoCKETTE when evaluated on DeBERTa-v3 model trained on single- task setting.
tuned to predict "yes" or "no" regarding the prompt question. During inference, the probability of the "yes" token is used as the prediction score. For span tasks, we directly train the model to generate the full answer.

---

B.6
Details zero-shot predictions ($4, $5)
B.9 List of all potential tasks and datasets for SoCKET
We use manually designed prompts for all the zero- shot prediction tasks and the prompts are shown in Table 8.
B.7
Computing correlation scores of task dependencies ($5)
Because our framework consists of several task types, it is challenging to obtain unified metric of correlation actoss different task comparisons. We use the following rules obtain correlation values:
Regression task & regression task: We com-
pute the Pearson's ' correlation coefficient of the two arrays.
Regression task & binary classification task: We compute the point biscrial correlation co- efficient of continuous array and a binary array.
Regression task & multi-class classification task: We set up linear regression task using the one-hot coded values of the multi-class array as independent variables and the con- tinuous array as the dependent variable. We report the root of the R-squared value of the regression as correlation (Olsson et al., 1982).
Binary classification task & binary classifica- tion task: We compute the Matthews' corre- lation coefficient (Matthews, 1975) from the two binary arrays.
Binary or multi-task classification task & multi-class classification task: We compute the Cramer's V score (Cramér, 1999) from the two arrays of categorical variables.
B.8 Computing pairwise model sinilarities ($5)
We quantify the model similarity between two tasks follows. We finetune a pretrained LLM on task L to obtain a model m, and another LLM on task t to obtain mg. We obtain pairwise model simi- larities by inferring both models on a sufficiently large dataset- in this case the entire test set of all tasks-and computing the correlation of the two inferred arrays. We construct undirected graph (Figure 7) where the thickness and color represent absolute correlation strength and polarity between the two models. The addition of polarity enables us to further discover strong negative correlations with task pairs such as politeness and offensiveness.

---

Type
Task
Question(Option
PAIR balkdown pains
quote "test_a" and its context "test_ b quote condowendingg
['No', 'Yos
REG
habackaboonMumor..ooo rating
Detemnine the degree of humoe of the given senience: "lext", The score should be ranging from
and be decimal. is humoceus and vory bumorous. For the sentence: "text". is it humerous?
CLS
mabackatoontis_bo humor bumoor
['No', 'Yes Yee',, NNo
PAIR
humor pairs
first sentence is "textaa". second sentence "text_b". fins sensence funnier
thn the second sentenee?
CLS
sure
For the sentence: "lex",, surcastic?
Yeer', NNo"
CLS
Iw sweet, Jonny
For the sentence: "text" 18 tronic
Noo; "Ye
CLS
contextual- ahveelidennty DivectoddMuuse
For the sentenee: "lex"..s RI illrrity directed abuse!
Noo. Yee'
CLS
contecttall aluse Parsen Diecttalllleeee
For the sentiene:: "text", il persow directed abwse!
No', Yes
RPG
libaac watluccaatoooottooo rating
Detemmin the degree of offemee the given sentence: "lout". The sonee should ranging
from (1.0 to 5.0, and can be decinul.
CLS
Inshisedimpliatiin
For the ventones: "tes", does il imply some biases?
N'oo, YYos
CLS
Intexdeensiv
For the senteeee: "texi" is it hite offemmive?
(Hate', "Offensive', NNi-
ther']
CLS
implicit Saaeiit-aateeee Eateeepplii hate
For the sentenee: 'text" is explicit hate?
[Noo' "Ye
CLS
implicit batctimplici hate
For the sentence: 'text, is implicily hateful?
[NNo "Yos"
CLS
implicti batevinciommnt hale
For the senience: 'lxt', 18 bateful inciemeent act!
[No "ee
CLS
Saapliss---tttileeeiiie hate
For the sentinee: "rex". 18 8 inferiovity lite!
Noo YYer
S
implicii- Holckitroosssss latt
For the
"lex!"
Bateful messoge invelving sterentype??
No' YYex
CLS
implics hattsslkeskhheeenng hote
For the soriinee: "rex"" is it hateful in thresteiing way?
No' YYo
CLS
Cmppicii--atoohhie prievance Jule
For the sontenee: "tex", while prievance lnte?
(NNo. YYo
CLS
intentyn
For the senicnes: "texi" is il inteniiona??
['No', 'Yo'
CLS
Hggawiieesrity hase
For the sentence:: "text,, idencity
NoNo
CLS
Hgaatinnult
For the senience: 'text" 18 it an insuli?
Noo YYe
CLS
For the sentenee: 'tex", obscene?
No. Yes
CLS
Resaweeeeeree Joxic
For the seniinne: 'text" is severely tosic?
Noo. Yex
C1S
For the sentinee: "tex". it threar?
['No', YYa
CLS
For the seniencs: 'text" it toxic?
['No', 'Yes
CLS
uffensiveyn
For the sentence: 'text' is it offensive?
['No', YYes
CLS
For the sentence: 'text" sexist?
I'No', 'Yes 1
SPAN
toxic span
toxic-span
the sentence: 'tex:" which purt of it can be idemirsed as toxic?
CLS
offensive
For the sentence: "text, offensive?
NoNo YYer
CLS
crowdfoower
For the senienee "tex"", what its emetion?
('empty'.
"sadness"
'enthosiasm', 'neurral
'warry'.
'love'. 'Inn'
'hute'
'happiness'.
rellet"".
'boredom'.
'urprise' 'anger"
CLS
dailydiaogg
For the given comersation, "lest", what is its emotion?
I'no emotion
'anger"
disgust'.
Yeur 'hap-
pincss'.
'sadness"
Surpeise]]
RFG
cinohankRamusal
Given the VAD model emoticn, détermine the degree of arousal the given sentence:
"lest". The score should be ranging from 0.0 5.0, and can be decimal.
REG
cmooonnk-oomisanee
Given the VAD model cmotion. determine the degrce of dominaes of the given sentenes:
"lest". The score should be ranging from 0.0 5.0, and can be decimal.
REG emobankdrvaeenee
Given the VAD model of emosion. determine the degree of valence of the given sentence:
"tout" The score should be ranging from 0.0 S.O, and can he decimal.
SPAN
cmottion-quan empatbytdisteess
the sentence: "text" which part it expresses sh shong cnocion??
REG
Determine the degree of distress of the given sentence: "text", The score should ranging
from to 7.0, and can be a decimnll
CLS
kmppthg-kisttess. bin
For the senienes: "vexi" is it showing distroes?
(Noo, YYe
PAIR
same side -uirs
For the sentences: "eex_a" and "sext_"" are they on the same side?
NoNo,, YYes
CLS
semitreehank
For the
"tex", positiv?
['Yes', 'No"
CLS
tweet emaji
For the sentenee: "tex", what is the emoyi that can he added to?
20 emmjji
CLS
For the
"tex", what emotion?
('anger', "joy". Topti
mism', Saddevel] ["negative, "ncutrall ppo-
For the
"tes!"
sentimet?
Itive')
compliints
For sontenee: "texi" it complaint?
['No', YYos]
RBG
emputhy Rempothy
Desermine the dogree of empulity of the given sentenco: "text". The score should be ranging
from 11.0 and be decimal.
CLS
emputhyeempatly bin
For the sentence: "text" is it expressing emputhy?
[NNo', 'Yes ]
CLS
hayai. politencss questieninimayy
For sentence: "text', it polite? For the sentence: 'text', how intimate do you think is?
['Ne', 'Yes intimate'. 'Inti
CLS
('Very
mate'.
'Somewhat inti
mate", "Not very intimatc'
NNot intimate", "Not inti-
mate at all"
CTSS
sanfoooloolrenes
For the "lexi" is 8 polite?
["Yes", 'No"
CLS braggingttrrgg_uciiee achievement
For the sentence: 'text", is it bragging about an achievement?
['No', 'Yes'
CLS
braggingtbragg... action
For the sentence: 'text', At bragging about an action?
['No', 'Yes"
CLS bragginggtrag__power possession
For the sentence: 'text', is it bragging about possession?
['No', 'Yes 1
CLS
bragginnghbrag..cmm pnit
For the sentence: 'tex',, Rt bragging abous trui??
[NNo,,
CLS
hypo-1
For the senience: "text" 18 a hyperbole?
[Noo 'Yes
PAIR
newttaliiiggbia-paan
For the sentenecs: "wxt_a" and "lext_"", which one is biasel??
['th first sentence is bi-
ased'. "the second sen-
lence blavad")
SPAN
propugencs-ppun
In the sontence: "iest" which pait of it can be ideetiiied the propigunda?
C1S
tumorémmnor
For the vertires: "tex", is it momor?
[No
61S
Swo-o--ichovviegg Iruth
For the sertione: "lest" will it perotivol as lie by the receiver?
Nee, Noo
CLS
Iwo-so-lishendar Iruth
For the sentines "lest", is the sender intending tell lie?
Yoy. 'No']
Table 8: The manually designed prompt questions and options used for each task.

---

Type
Task
Question
REG
Hubackalbonthumeer.r.. rating
For the sentence: "text", .. is it humnrouus?
REG
hahackathonfoffense. rating
For the sentence:"
offensive?
REG
cmobankFarousul
For the sentence:"lex" , is the presented emotion highly arousal?
REG
emobank#dominance
For the sentence:"texi"" is the presented emotion highly dominant?
REG
emobannkWvalence
For the sentence:"tex"" is the presented emotion positive?
REG
cupaahykdistness lidistress
For the sentence: "text"" does show distress?
REG
emparhytkemptthy
For the sentence:" does show empathy?
Table 9: The manually designed prompt questions for fine-tuning regression tasks over the model.
hntociathonaiicc
Humor Sarcasm
beDCL rony
"nhhekothenehumee rating
helaccatthon doflerse rating
Insewetock
Prou..
Offensiveness
twoot, Incentin
offensiveon
0.8
(ochwelsenttn eeceeiiy hitts
conteotuelrabueee Midentin Cereiniyylleetee Birehiyeeeeeeee..oo
impliciii wteeemppic C hala
Servh
contextual Pe binccit BFrcceddooo dibuso
imslrtt-hatte hastttthtttwtiie eioeedidtty
hato
merenn hete
impliiti irppich-ettwrtee hetwirheeeten
croe
emolion
beset Unnert
emoi
emrbann ja dentrianee
Sentiment & Emotion
omd banbanccaann auuussll
dallylision
same-ide-po pairs
Social Factors
comllonntmeey
comp complnnis uinis
ompatty cringaahy pathy
slonng
rumora nimor teol
bragginge
Trustworthiness
saggingabra
hypost
breggtnpetre sebreg posseaiin
buth
Precaiver both
Figure 6: A detailed heatmap of Figure 2 showing task dependency among all task pairs well task labels. Each value represents the absolute strength of correlation between the true labels of the test set of specific task (columns) and the predictions made on that task using a model trained on different task (rows).

---

Edge Coloring
Highly Negatively Correlated
Highly Positively Correlated
Node Legend
Sentence Pairs
Social
Humor & Sarcasm
Regression
Offensiveness
Trustworthiness
Classification
Emotion & Sentiment
distr@ss_ib _iin
receiver truth
sarc
sendem truth
irony
distress
empathy
Questionlntimacy
dom
humor
complaints
humor Trating
Has Basieddllllii
arousal
tweet emoji
stereofypícal hate
crowdflowwr
offensiveyn
rumon bool
impligh_han hate
tweet @emotoon
Hate
tweet sentiment
offense rating
identity_haae
empathy bin
same-side-pairs
valence
dailydialog
intentyn
tweet offensive
StanfordPoliteness
toxic
sexyn
hayati politeness
SentiTreeBan
insult
explicit hate
obscene
hypo- PersonDirectedAbuse
sever toxic
Figure 7: Weighted, undirected graph of model correlations. Each edge between nodes i and j weighted by the correlation between predictions from model fine-tuned task i and predictions from model fine-tuned on task j. evaluated on the entire SoCKET dataset. Nodes are sized proportionally to their weighted degree and Yifan Hu algorithm applied for layout, with minor adjustments for readability. Refer to $B.8 for details how the pairwise score for each edge was computed. We observe strong positive correlations similar to Figure 2, especially within the Sentiment & Emotion category and the Offensiveness category. We also see that across-categon transfers may happen in negative direction such suyyati_politenes and several Offensiveness tasks.

---

Task name
Tisk type
Maj.
Single
Categorywise
Humor & Sarcam
hahnckkahon
rating
RBG
0.5
0.68
0.67
0.66
hahockathon
is Jumoor
CLS
0.49
0.38
0.93 0.00
0.93 0:00
0.91 0.01
humor-pairs
humor pains
PAIR
0.50
0.34
0.9% 0.0
0,98 000
0.97 0.00
(weeL_rooy
tweeLirony
CLS
0.49
0.28
0.80 0.01
0.80 0.00
0.75 0.03
Offensieeness
contextual-
DemiityDireeieeAAve PereeeDireessAhures
CLS
0.35
0.50
0.61 0.02
0.62 0:00
0.63 0:01
contextual l-ahese
C1.
0.34
0.54
0.58 0.02
0.57
hahackathon
offense rating hushiasedimpicatioon
REG
0.5
0.5
0.91 0.01
0.91 0:00
0.92 0.01
hnsbissciippication
CLS
0,50
0.37
0.86
0.86 0.00
0.870.00
bateofTensive
bateoffensive
CLS
0.27
0.93 0.00
0.95 0:01
0.95 0.01
implicit. hate
explicit hate implicii incitement hate inferiority. hate
C1.S
0.37
0.49
0.72 0.00
0.74 0.01
0.73
implicit
CLS
implicit-hate
CLS
0.38
0.49
0.67 0.01
0.68 0.01
0.67 0.00
implicit hate
CL.S
0.34
0.59
0.59
0.58
implici-- hule
steroulypical hale threatening, hate grievance hate
CLS
0.37
0.49
0.68 0.01
0.66 0:01
0.68 0.01
implicit-hate
CL.S
0.33
0.50
0.60 0.02
0.64 001
0.65 0:04
implicit-hate
CLS
0.01
0.71 0.07
intentyn
intentyn
CLS
0.47
0.42
0.75 0.00
0.74 0.01
0.74 0.01
jigenw
identity hule
jigsow
insalt
CLS
0.38
0.49
0.88 0/00
0.87 0.01
0.88 0.01
jigsaw
obscene
CLS
0.38
0.49
0,90
0,91 0:00
0,91 0.01
jigsow
severe toxic
CLS
0.34
0.50
0.74
0.73 0022
0.71 0.02
jigsow
threat
CLS
0.34
0.50
0.85 0.62
0.81 0.04
0.80
jissow
texic
CLS
0.01
ollensiveyn
oflemsiveyn
CLS
0,50
0.37
0.82 0.00
0.83 0.01
0.83
sexyn
sexyn talkdown pairs
CLS
0.79
talkdown pairs
PAIR
0.50
0.33
0.89 0.01
0.88 0.01
0.88 0.00
toxic-spun
soxic-spum
SPAN
0.68
0.65 0.25
0.67 0.05
tweet offensive
rwest offensive
CLS
0.48
0.01
0.80
Emottion
cawddllower
crowdllower
CLS
0.06
0.03
0.24 0.00
0.23 0.01
0.22 0.00
dailydialng
dailydiolng
C1.S
0.07
0.43
0.47 0.00
0.47 0.02
emobunk
urousal
REG
0.5
0.80 0.02
0.81 0,02
0.79 0.02
emobank
dominance
REG
0.5
0.5
0.75
0.75 001
0.73
emobank
0.5
0.92
emotioonsppn
emnttion-ppn
SPAN
0
0
0.96 0.01
0.89 0.00
0.86 0:03
empathy
distress
RBG
0.5
0.75
0.72
empathy
distresa bin side pains
CLS
0.50
0.31
0.68 0.01
0.69 0.03
0.65 0.02
side
PAIR
0.48
0.35
0.66
0.70 0.09
0.76
sentitecbunk
sentineebunk
CLS
0.50
0.31
0.97 0.00
0.96 0.00
0.96 0.01
twest emajs
rweet emoji tweet emotion tweet sentinent
CLS
0.04
0.02
0.34 0.00
0.33 0.00
0.33
tweeL emmtion
CLS
0.01
weet_sestimen
CLS
0.32
0.22
0.71 0.00
0.71 0.01
0.69 0.01
Social Fartors
complaints
complaints empathy empathy. hin bayali poliseness guestioninti stanfordpoliieness
CLS
0.50
0.36
0.92
0.92 0.00
0.91 0.01
empathy
RBG
0.5
0.70
0.71 008
0.70
empathy
CLS
0.50
0.33
0.63 0.02
0.62 0.1
0.599..0
hayaii politemess
0.05
guestionintimaey
CLS
0.16
0.06
0.49 0.08
0.48 0.02
0.42
stanfonipoliteness
CLS
0.50
0.36
0.70 0.02
0.71 0.0
0.72 0.02
Tnssworthinees
hragging
brng achievement
CLS
0.36
0.74
0.69 0.01
0.76 0.01
bragging
brag_activn action
CLS
0.35
0.50
0.59 0.02
0.57 0.06
0.59 0.94
brageing
brag possessooiinn
0.70
0.66
bragging
brgg trait hypo-1
CLSS
0.34
0.50
0.67 n/9
0.59 0.07
0.61 0.04
hypo-1
CLS
0.48
0.41
0.74
0.71 001
0.69
neutralizing hias pairs
neutralizing biss pains
PAIR
0.50
0.33
0.96 0.01
0.96 0.01
0.96 0.00
propaganda-span
propaganda-span
SPAN
0.22
0.23 0:03
0.24 0.00
namor
namor bool
C1.S
0.49
0.02
twe-to lie
receiver_ iruth sender
CLS
0.38
0.49
0.57 0.02
0.57 0,02
0.53 0.01
CLS
0.59
Table 10: Detailed table of performance scores from comparing single-task vs multi-task trained models Section 6 (refer to Table 3 in Section 6). There are significant gains from the two multi-task settings in the Humor & Sarcasm category, where the tasks in general have low task dependency (ref. Section 5). However, for other categories we see several instances of tasks where multi-task trained model have greater performance.

---

Paper/Dataset Title
Tasks
Reference
Automatic Identification and Classification of Bragging in Social Media
Bragging (Achievement)
Jin et al. (2022)
Automatic Identification and Classification of Bragging Social Media (Jin et al., 2022)
Bragging tion)
(Ac-
Jin et al. (2022)
Automatic Identification and Classification Bragging in Social Media
Bragging (Pos- session)
Jin et al. (2022)
Automatic Identification and Classification of
Bragging (Trait)
Jin et al. (2022)
Bragging in Social Media
Automatically Identifying Complaints in Social
Complaints
Preofiuc-Pietro et al. (2019)
Media
Introducing CAD: the Contextual Abuse Dataset
Identity Based
Vidgen al. (2021)
Hate
Introducing CAD: the Contextual Abuse Dataset Introducing CAD: the Contextual Abuse Dataset
Individual Hate
Vidgen et al. (2021)
Kroup-Baseh
Vidgen al. (2021)
Hate
Introducing CAD: the Contextual Abuse Dataset Sentiment Analysis in Text
Counter Speech
Vidgen et al. (2021)
Emotion
CrowdFlower (2016)
DailyDialog: A Manually Labelled Multi-turn Di- alogue Dataset
Emotion
Li et al. (2017)
EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimen- sional Emotion Analysis
Emotion lence)
(Va-
Buechel (2017)
and
Hahn
EmoBank: Studying the Impact of Annotation Emotion
Buechel and (2017)
Hahn
Perspective and Representation Format on Dimen- sional Emotion Analysis
(Arousal)
EmoBank: Studying the Impact of Annotation Perspective and Representation Format Dimen- sional Emotion Analysis
Emotion (Domi- nance)
Buechel and (2017)
Hahn
Detecting Emotion Stimuli in Aanotion-Bearing
Emotion
Ghazi et al. (2015)
Sentences
Measuring the Language of Self-Disclosure across
Disturbance
Reuel et al. (2022)
Corpora
Measuring the Language of Self-Disclosure across Corpora
Empathy
Reuel et al. (2022)
SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense
Humor Rating
Meaney et al. (2021)

---

Semīīval 2021 Task HaHackathon, Detecting and Rating Humor and Offense
Funny (boolean)
Meaney al. (2021)
SemEval 2021 Task HaHackathon, Detecting
Offensiveness
Meaney al. (2021)
and Rating Humor and Offense
Social Bias Frames: Reasoning about Social and Power Implications of Language
Biased Implica-
Sap et al. (2020)
tion
Social Bias Frames: Reasoning about Social and Power Implications of Language
Intent
Sap et al. (2020)
Social Bias Frames: Reasoning about Social and Power Implications of Language
Offensiveness
Sap et al. (2020)
Social Bias Frames: Reasoning about Social and Power Implications of Language
Sexism
Sap et (2020)
Automated Hate Speech Detection and the Prob-
Offensive
Davidson et al. (2017)
lem of Offensive Language
Does BERT Learn as Humans Perceive? Under- Politeness
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn Humans Perceive? Under- Positivity
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn as Humans Perceive? Under- Anger
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn as Humans Perceive? Under- Disgust
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn as Humans Perceive? Under- Fear
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn as Humans Perceive? Under- Joy
Hayati et al. (2021)
standing Linguistic Styles through Lexica
Does BERT Learn Humans Perceive? Under-
Sadness
Hayati et al. (2021)
standing Linguistic Styles through Lexica
ScmEval-2020 Task 7: Assessing Humor in Edited News Headlines
Funnier
Sc-
Hossain et al. (2020)
quence
MOVER: Mask, Over-generate and Rank for Hy- perbole Generation
Hyperbole
Zhang and Wan (2022)
Latent Hatred: A Benchmark for Understanding
Explicit Hate
ElSherief et al. (2021)
Implicit Hate Speech
Latent Hatred: A Benchmark for Understanding
Implicit Hate
ElSherief al. (2021)
Implicit Hate Speech
Latent Hatred: Benchmark for Understanding Incitement
EISherief et al. (2021)
Implicit Hate Speech
Latent Hatred: Benchmark for Understanding Inferiority
ElSherief et al. (2021)
Implicit Hate Speech
Latent Hatred: A Benchmark for Understanding
Stereotyping
ElSherief et al. (2021)
Implicit Hate Speech
Latent Hatred: A Benchmark for Understanding Threat
ElSherief al. (2021)
Implicit Hate Speech
Latent Hatred: Benchmark for Understanding Offensive
ElSherief al. (2021)
Implicit Hate Speech
Latent Hatred: Benchmark for Understanding Irony
ElSherief et al. (2021)
Implicit Hate Speech
Latent Hatred: A Benchmark for Understanding
Other Hate
ElSherief et al. (2021)
Implicit Hate Speech

---

Toxic Comment Classification Challenge
Identity-Bast
Jigsaw (2017)
Hate
Toxic Comment Classification Challenge
Insult
Jigsaw (2017)
Toxic Comment Classification Challenge
Obscenity
Jigsaw (2017)
Toxic Comment Classification Challenge
Severe Toxicity
Jigsaw (2017)
Toxic Comment Classification Challenge
Threat
Jigsaw (2017)
Toxic Comment Classification Challenge
Toxicity
Jigsaw (2017)
Automatically Neutralizing Subjective Bias in Text
Bias
Pryzant et al. (2020)
SemEval-2020 Task 11: Detection of Propaganda
Propaganda
Da San Martino et al.
Techniques in News Articles
Technique
(2020)
Quantifying Intimacy in Language
Intimacy
Pei and Jurgens (2020)
Detect Rumors in Microblog Posts Using Propaga- tion Structure via Kemel Learning
Rumor
Detec-
Ma et al. (2017)
tion
On Classifying whether Two Texts are on the Same Side of an Argument
Stance
Körner et al. (2021)
A Large Self-Annotated Corpus for Sarcasm
Sarcasm
Khodak et al. (2018)
Recursive Deep Models for Semantic Composi- tionality Over Sentiment Treebank Facilitating the Communication of Politeness
Sentiment
Socher et al. (2013)
Politeness
Fu al. (2020)
through Fine-Grained Paraphrasing
TalkDown: A Corpus for Condescension Detec- tion Context
Condescension
Wang and Potts (2019)
SemEval-222 2022 Task 5:
Toxic Spans Detection
Toxicity
Pavlopoulos et al. (2021)
SemEval 2018 Task 2: Multilingual Emoji Predic-
Emoji
Barbieri et al. (2018)
tion
SemEval-2018 Task 1:
Affect in Tweets SemBval-2218 Task 3: Irony Detection in English
Emotion Irony
Mohammad et al. (2018)
Van Hee al. (2018)
Tweets
Predicting the Type and Target of Offensive Posts in Social Media
Offensiveness
Zampieri et al. (2019a)
SemEval-2011 Task 4: Sentiment Analysis in Twit- ter
Sentiment
Rosenthal et al. (2017)
It Takes Two to Lie: One to Lie, and One to Listen It Takes Two to Lie: One to Lie, and One to Listen "So You Think You'te re Funny?": Rating the Hu- mour Quotient in Standup Comedy
Sender Truth Receiver Truth
Peskov al. (2020)
Peskov al. (2020)
Humor Rating
Mittal et al. (2021)
DEBAGREEMENT: comment-reply dataset for (dis)aggetment detection in online debates
Stance
Pougué-Biyong et (2021) Nakov et al. (2021)
The CLEF-2021 CheckThat! Lab on Detecting Check Chethy Claims, Previously Fact-Checked Claims, and Fake News
Trustworthiness
Finding Deceptive Opinion Spam by Any Stretch
Deceipt
Ott et al. (2011)
of the Imagination
Finding Deceptive Opinion Spam by Any Stretch
Fact
Ott al. (2011)
of the Imagination
Clustering Approach for Nearly Unsupervised
Nonliteral
Lan-
Birke and Sarkar (2006)
Recognition of Nonliteral Language
gauge
Detecting Community Sensitive Norm Violations
Community
Park et al. (2021)
in Online Conversations
Norms
Can Machines Learn Morality? The Delphi Exper-
Moral
Judge-
Jiang et al. (2021)
iment
ment

---

SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter
Hate Speech
Basile al. (2019)
SemEval-2019 Task 6: Identifying and Categoriz- ing Offensive Language Social Media (OffensE- val)
Offensiveness
Zampieri et al. (2019b)
CivilComments
Toxicity
(Jigsaw, 2019)
CivilComments
Very Toxic
(Jigsaw, 2019)
(Male, Bachelor) and (Female, Ph.D) have differ-
Gender
Kang et al. (2019)
ent connotations: Parallelly Annotated Stylistic
Language Dataset with Multiple Personas
(Male, Bachelor) and (Female, Ph.D) have differ-
Age
Kang et al. (2019)
ent connotations: Parallelly Annotated Stylistic
Language Dataset with Multiple Personas
(Male, Bachelor) and (Female, Ph.D) have differ-
Country
Kang et al. (2019)
ent connotations: Parallelly Annotated Stylistic
Language Dataset with Multiple Personas
(Male, Bachelor) and (Female, Ph.D) have differ- ent connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas
Political view
Kang et al. (2019)
(Male, Bachelor) and (Female, Ph.D) have differ- ent connotations: Parallelly Annotated Stylistic
Education
Kang et al. (2019)
Language Dataset with Multiple Personas
(Male, Bachelor) and (Female, Ph.D) have differ-
Ethnicity
Kang et al. (2019)
ent connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas
Webis Clickbait Corbus 2017
Clickbait
Potthast al. (2018)
VU Amsterdam Metaphor Corpus Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications Dear Sir or Madam, May Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for For- mality Style Transfer
Metaphor
Steen al. (2011)
Uncertainty
Pei and Jurgens (2021)
Formality
Rao and Tetreault (2018)
International Survey Emotion Antecedents and Reactions
Sentiment
Scherer and Wallbott (1994)
Short Jokes
Joke
Moudgil
Short Text Corpus with Focus on Humor Detection
Joke
CrowdTruth (2016)
Hateful Symbols or Hateful People? Predictive
Sexism
Waseem and Hovy (2016)
Features for Hate Speech Detection on Twitter
Hateful Symbols or Hateful People? Predictive
Racism
Waseem and Hovy (2016)
Features for Hate Speech Detection on Twitter Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad Personality through Twitter Behavior Utterance-level Dialogue Understanding: An Em- pirical Study
narcissism
Preofiuc-Pictro et al. (2019)
psychopathy
al. Preofiuc-Pietro (2019)
Machiavellianism
Preofiuc-Pietro et (2019)
Emotion
Ghosal et al. (2020)
Table 11: Table of all the datasets considered when curating the SoCKET Benchmark.


Reviews:
[{"review_id": "qqLwjafv8k", "reviewer": "Reviewer_egYJ", "summary": "", "questions": "Major questions:  \nA: how are metrics aggregated for categories? If averaged over [tasks/examples], do [smaller/larger] tasks have an outsized effect on performance or does the metric aggregation take into account the number of examples per task?  \nB: how is class imbalance handled? For instance, the [implicit-hate dataset](https://github.com/SALT-NLP/implicit-hate) is heptuplicated from a single 7-class dataset to seven binary classification datasets. As each class has 10-20% coverage of the dataset, this would imply that the implicit-hate datasets all have 80-90% \"no\" labels. However, the majority vote results in Table 7 show ~50% performance across these datasets instead of the 80-90% I would have expected from the random split described in A.1?  \nC: Besides presumably handling class imbalance, what are the preprocessing measures alluded to in A.1? Is e.g. dataset size also normalized or are the sizes reported in Table 1 representative of the final benchmark?  \n\nMinor clarifications:  \nD:  what was meant to be in Appendix B2?  \nE: [L098] what is meant by \"social knowledge is communicated via language understanding\"?  \nF: [L103/110] definition of \"appropriacy\"?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors greatly appreciate the attention paid to the work and the clear amount of thought put into this rigorous and nuanced review. We’re excited to be given the opportunity to respond to some of the excellent feedback we have received in this review. Your review excited us, so please excuse the length as we want to deeply engage with your comments and hope our response continues the great conversation.\n\nFraming:\n\nWe fully agree that accurate framing is crucial to the development of work that attempts to explore new grounds for the field, and that the downstream consequences of improper framing are numerous. \n\nThe choice of the term “social knowledge” as framing came from a desire to not be overly prescriptive about the specific nature of signals being recognized by models. The benchmark aims to estimate the capacity of LLMs to pick up on a variety of constructs that literature in Psychology and Linguistics have shown to be central to the social-functional use of language, including norm-adherence, power relations, sentiment analysis, and more. We ultimately selected the term “social knowledge” because it has been used previously as a broad category in psychology (e.g., Turiel 1983; Adolphs 2009) and matched the capabilities we are interested in. We will provide more details and motivation for this broad notational framing in revisions.\n \nWe agree on the need to improve the framed representation for the benchmark. Respectfully, we feel that the term “social cues” alone might be too limiting of a framework due to a perceived emphasis on in-domain syntax. However, we do agree that the framing of the benchmark of the text should be adjusted—especially following insights gained from the Bowman and Dahl article supplied by the reviewer. We are happy to take suggestions for revisions that can more accurately represent the capabilities and limitations of the benchmark. \n\nThe reviewer notes a gap between the Theory described in Section 2 and Tasks motivated in Section 3. We appreciate this feedback and will work to close the gap. Our definition of “social knowledge” is broad and we highlight key aspects of social communication pointed to in research from Communications, Psychology, and Linguistics (Lines 95-112; throughout Section 3). Of course, we fully admit these dimensions are not the totality of all social knowledge (lines 709-814), but they are a start. We also fully agree with the reviewer’s notion that “a new sentiment analysis benchmark would not really help our understanding of ‘social knowledge’” but point out that the goal is not proposing five _separate_ categories but five _interrelated_ categories (Lines 191-214). Our categories have predicted interactions from theory and by including all in the same benchmark, we can test whether models benefit from these implicit relationships—in fact, we don’t see this (Section 5), pointing to a clear gap in current performance!\n\nAggregation:\n\nWe agree with the reviewer’s perceived tension around data aggregation. Too much aggregation obscures real progress and differences between models, while too little aggregation can lead to too many scores and uncertainty around which scores need to go up, making comparison difficult—or worse, excessive benchmark hill-climbing around many disaggregated scores.\nOur choice to aggregate was based on current practice from previous NLP benchmarks such as Glue (Wang et al., EMNLP 2018), SuperGLUE (Wang et al., NeurIPS 2019), and MTEB (Muennighoff et al., EACL 2023). All of these benchmarks aggregate dissimilar tasks into a single metric that is a proxy for performance at some larger construct like “Language Understanding”. For example, in the MTEB paper the authors average the performances of tasks such as classification, clustering, semantic textual similarity (STS), and summarization into a single metric despite their dissimilarity. While such aggregating does minimize disparities between these tasks when only the single aggregate metric is reported, the single metric does provide utility for comparing models.\nSoCKET presents a challenging case for aggregation as it has more than 50 different tasks. We decided it would be inconvenient to report the performance of every combination in the main paper. Also, emphasizing the comparisons between different models at an individual task level can wrongly introduce the notion of promoting “leaderboard chasing” where teams can highlight how their models achieve #1 for a particular set of tasks. Our aggregation of tasks for each social category is intended to instead focus on the aspect of measuring a model’s general social knowledge capacity.\n\nHowever, we fully agree with the spirit of the reviewer’s comment. In Table 2, we report category-level aggregated information, e.g., for “Trustworthy” tasks, and for tasks for each type, e.g., pair comparison. We intend that any users of the benchmark would report these slightly disaggregated metrics as well to show which types of social knowledge are recognized. We will certainly add more details around the experiments from Table 7 to highlight the importance of per-task learning. \nWe also agree that the addition of finer-grained analyses on where models vary in performance and behave expectedly/unexpectedly can improve the quality of our paper. For instance, one experiment that we did not include in the main paper was that LLMs have varying capabilities in understanding the instructions of the tasks (Figure 3). When reporting the performance of LLMs on tasks, it could be beneficial to differentiate the performance of (1) where the model failed to understand the task and (2) where the model made an incorrect prediction. While such additional experiments are out of the scope of this paper, we believe subsequent research using this benchmark can be very useful.\n\nQuestions For The Authors:\n\nMajor Questions\nA: Metrics are aggregated for categories as averages of F1 scores. Each task is scored independently with a macro F1, and the unweighted mean of those scores makes up the aggregate score for a category. \n\nB: In an attempt to stay consistent and ensure fair comparison across several different task types—some of which, like the implicit-hate-dataset, emphasize realistic ratios of positively-labeled hate speech and negative labels—we use a unified metric (macro F1) to measure performance on each task. Thus, for example, in the implicit-hate-dataset, stereotypical_hate makes up 17.9% of all implicit hate samples, which account for 28% of the dataset. This would suggest that ~5% of the implicit-hate-dataset samples in our test set should be positive for stereotypical_hate, and this is what we find. Calculating the macro F1 for a set of ground truth labels with 5% positive and the rest negative and the estimated labels being all negative we find that the score is ~0.49, as reported.\n\nMinor Questions:\nD: B.2 was meant to introduce Table 4 in the appendix and it has been amended. Thank you for bringing that to our attention. \n\nE: By this we mean that information regarding the social-functional aspects of interpersonal communication, which we consider to be under the umbrella term “social knowledge”, are transmitted (perhaps not exclusively) via language use.. This includes those studies, for example, in cognitive science regarding the role of language in emotion perception (Barrett et al., 2007), in interpersonal communication regarding predictability in language being used to assert relationship status in social interaction (Berger, 2005), and in social psychology showing language use conveying implicit racial prejudices of the speaker (Augoustinos and Every, 2007) among others. \n\nF: “Appropriacy” in this context comes specifically from Eggins, 2004 (below) and can be found in other texts (see Allen, 2007) as being the forces which cause language-users to choose certain words or phrases instead of all the possible alternatives. \n\n\n\n\nMissing References:\nThe ACL version of “Making Pre-trained Language Models Better Few-shot Learners” by Gao, Fisch, and Chen has been cited, replacing the preprint previously cited. \n\nTypos Grammar Style And Presentation Improvements:\nWe are sorry for the confusion. Gao’s method was specifically used for fine-tuning the t5 model for regression tasks. The listed prompts without “yes” and “no” was used across all the other zeroshot prompting experiments. We will clarify this in future revisions.\n\nReproducibility: \nIn regard to the reproducibility score, the authors would like to emphasize that all datasets and models used in the paper are open-sourced. Moreover, the benchmark dataset will be publicized when ACL anonymity rules allow, and preprocessing scripts will be made available to reconstruct the benchmark from the original datasets collected. If there are any additional barriers to replication, we would greatly appreciate the suggestions, as we recognize that consistency and reproducibility are at the heart of any useful benchmark. \n\nReferences\n\nAdolphs, Ralph., 2009. The social brain: neural basis of social knowledge. Annual review of psychology, 60, pp.693-716.\n\nAllan, Keith., 2007. The pragmatics of connotation. Journal of pragmatics, 39(6), pp.1047-1057.\n\nAugoustinos, Martha. and Every, Danielle., 2007. The language of “race” and prejudice: A discourse of denial, reason, and liberal-practical politics. Journal of language and social psychology, 26(2), pp.123-141.\n\nBarrett, Lisa F., Lindquist, Kristin A. and Gendron, Maria., 2007. Language as context for the perception of emotion. Trends in cognitive sciences, 11(8), pp.327-332.\n\nBerger, Charles R., 2005. Interpersonal communication: Theoretical perspectives, future prospects. Journal of communication.\n\nEggins, Suzanne., 2004. Introduction to systemic functional linguistics. A&c Black.\n\nMuennighoff et al., 2023. MTEB: Massive Text Embedding Benchmark. EACL\n\nTuriel, Elliot., 1983. The development of social knowledge: Morality and convention. Cambridge University Press.\n\nWang et al., 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. EMNLP\n\nWang et al. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. NeurIPS", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**TLDR** the paper's value to the community lies in (1) an excellent literature review and discussion that is beneficial to the community; (2) a benchmark which may have the potential to drive progress on important model capabilities; and (3) a series of empirical analyses and findings which are scientifically interesting and pave the way for future work. \n\nI weakly believe the paper should not be accepted in its current state, but nevertheless I believe the work is excellent and very close to being publishable. Primarily, I see the following three reasons to accept: \n\n**A1 Literature review and discussion do a great job of contextualizing an important and under-addressed problem**  \nThe problem of social factors/information/knowledge in language and modern NLP systems is a fascinating topic at the intersection of psychology, linguistics and machine learning. The paper does an excellent job of bringing together various threads of research at this intersection in a cohesive discussion that sheds light on an important open problem in our field. \n\n**A2 Benchmark *potentially* provides a valuable and novel measure of progress on important model capabilities**  \nWhile I think our field's relationship with benchmarks is prone to Goodhart's law, I see the value in being able to empirically measure progress on tasks related to capabilities-of-interest; and on a more meta-level, to encourage research in under-explored but important directions. In this regard, the SocKET benchmark provides a compelling reason to accept this paper, insofar as progress on this benchmark is demonstrably tied to progress on the capabilities we care about.\n\n**A3 Empirical analyses raise interesting questions for future research**  \nIn addition to the primary contribution of proposing the SocKET benchmark and evaluating various models on it, the paper also presents several complementary analyses with rigorous experiments and surprising empirical findings (C4). I believe these findings raise open questions that have the potential to spark future research.", "reasons_to_reject": "**TLDR** the paper has several limitations which may undermine the value of the proposed benchmark to the community, potentially even making it detrimental to progress. \n\nWhile the paper and work is excellent, I cannot recommend it for acceptance in its current state. I believe benchmarks can be useful if done right, and harmful if done wrong. As of its current state, the paper has no compelling evidence or arguments that progress on this benchmark corresponds to progress on the capabilities used to motivate it. The paper would greatly benefit by incorporating takeaways from the following works:\n- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://aclanthology.org/2021.naacl-main.385) (Bowman & Dahl, NAACL 2021)\n- [Targeting the Benchmark: On Methodology in Current Natural Language Processing Research](https://aclanthology.org/2021.acl-short.85) (Schlangen, ACL-IJCNLP 2021)\n- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://aclanthology.org/2023.findings-acl.202) (Subramonian et al., Findings 2023)\n\nMore specifically, I would recommend this paper for acceptance if the following issues are resolved or adequately addressed:\n\n\n**R1 Misleading framing of the benchmark**  \nFor me personally, there is a significant disconnect between what is actually measured by the benchmark and how it is framed/motivated in the paper. In particular, I found an important theme in the discussion/references around social information was  language grounded in social interactions (and the associated importance of extra-linguistic context and pragmatic inference). While each category in the socket benchmark is justified and supported by references as being a facet of social information, the relevance of these tasks to the original question of \"social knowledge\" seems tenuous. I'm not convinced the benchmark is theory-driven/grounded as much as it is superficially and post-hoc linked to various theories.  \n\nIn the same way that a new sentiment analysis benchmark would not really help our understanding of \"social knowledge\" (even though sentiment is easily framed as an important facet of social information), I don't see how benchmarks for predicting humor/offensiveness/trustworthiness/etc contribute to the problems motivating this paper. \nI still believe the benchmark is valuable in-and-of-itself, but I feel there was a bait and switch between Sections 1-2 where the problem of \"social knowledge\" is framed, and Section 3 where the benchmark is introduced. \n\nI believe the framing briefly presented in the conclusion (i.e. \"social cue detection\") is much more representative of the benchmark. A discussion situating (the narrower focus of) social cue detection within (the broader scope of) social knowledge/information/factors is missing and would improve the paper I believe. I could see a mischaracterization of this benchmark as \"evaluating sociability\" or \"understanding social knowledge\" may lead to misleading impressions of progress, especially given the chasm between the task formulations (classification, regression, etc on non-conversational examples) and how LLMs are used in practice (natural dialogue). I think the value of the benchmark is compelling without needing to over-hype it.\n\n**R2 Aggregating of results and limited interpretability**   \nWhile I see the value of combining thematically related tasks into aggregate evaluations, I believe there are potential pitfalls to only reporting a single metric per category. A finer grained understanding of where models succeed, fail, or have high variance would be invaluable in contextualizing progress on this benchmark. Table 7 is a good start, but it is only briefly referenced in the context of multi-task training without any discussion of the stratified results. Without this kind of finer grained analysis, I'm worried progress on this benchmark could end up being quite misleading. I'm particularly worried about this, because the benchmark brings together various existing datasets which may have important systematic differences or limitations which skew or bias evaluation results on this benchmark. More generally, this kind of aggregating makes it difficult to interpret what an improvement in any given category actually means. For instance, as a toy example, improving \"humour and sarcasm\" may be improving \"humor\" or \"sarcasm\"; but this is lost in the aggregate metric.", "questions_to_authors": "Major questions:  \nA: how are metrics aggregated for categories? If averaged over [tasks/examples], do [smaller/larger] tasks have an outsized effect on performance or does the metric aggregation take into account the number of examples per task?  \nB: how is class imbalance handled? For instance, the [implicit-hate dataset](https://github.com/SALT-NLP/implicit-hate) is heptuplicated from a single 7-class dataset to seven binary classification datasets. As each class has 10-20% coverage of the dataset, this would imply that the implicit-hate datasets all have 80-90% \"no\" labels. However, the majority vote results in Table 7 show ~50% performance across these datasets instead of the 80-90% I would have expected from the random split described in A.1?  \nC: Besides presumably handling class imbalance, what are the preprocessing measures alluded to in A.1? Is e.g. dataset size also normalized or are the sizes reported in Table 1 representative of the final benchmark?  \n\nMinor clarifications:  \nD:  what was meant to be in Appendix B2?  \nE: [L098] what is meant by \"social knowledge is communicated via language understanding\"?  \nF: [L103/110] definition of \"appropriacy\"?", "paper_topic_and_main_contributions": "**TLDR**: the paper identifies a lack of systematic evaluations targeting social language, proposes a benchmark for detecting social cues to address this gap, evaluates language models on this benchmark, and provides several interesting complementary analyses. \n\n**C1: formulating the problem of social knowledge**  \nThe paper presents an excellent literature review and discussion around challenges of social factors/information/knowledge in NLP systems, motivating this work. \n\n**C2: curating SocKET, a benchmark for evaluating social knowledge**  \nThe paper proposes a benchmark, SocKET, covering 5 categories of social information: humor and sarcasm, offensiveness, sentiment and emotion, trustworthiness, and other social factors (empathy, politeness, intimacy and complaints). \nSocket consolidates datasets from 35 existing datasets into 58 evaluations (formulated as classification, regression, pairwise comparison, or span identification), divided along the 5 categories of social information. \nAn important contribution of this benchmark is its thematic organization of tasks. which enables more comprehensive measures of progress. \n\n**C3: evaluating models and identifying gaps in performance on SocKET**  \nThe paper evaluates several models on SocKET, including zeroshot LLMs on the order of 1-10B parameters (e.g. GPT-J, Alpaca, Bloom, OPT); and finetuned LMs on the order of 10-100M parameters (e.g. bert, roberta). They find that zeroshot models, despite their large number of parameters, perform poorly in contrast to smaller finetuned models. Even finetuned models demonstrate significant room for improvement on this benchmark. Notably, the category of trustworthiness and task of span identification are the most challenging for models. \n\n**C4: presenting interesting findings from complementary analyses**  \n- zeroshot model size does not typically correlate with performance;\n- zeroshot models often do not follow instructions (failing a task by giving an invalid answer rather than a wrong answer);\n- finetuned models exhibit notable task dependencies within and across categories.\n- finetuned models generally perform worse with multi-task training", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "L362: cite ACL version https://aclanthology.org/2021.acl-long.295/"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L428: Figure 1 shows\n\nTable 6: there is a mismatch between the prompt for certain REG tasks and the cited method of  [Gao et al. 2021](https://aclanthology.org/2021.acl-long.295/) (i.e. \"The score should be ranging from 0.0 to 5.0, and can be a decimal\" versus measuring the probability of \"yes\" vs \"no\" as a proxy for regression)."}}, {"review_id": "Jt7eZJKZWz", "reviewer": "Reviewer_CEQi", "summary": "", "questions": "A: Can you provide more details about finetuning T5 for a regression task? While the prompts are listed in Table 6, other details are incomplete. \n\nB: Unrelated to scores, have you evaluated how GPT3+ models perform on this benchmark? I’m curious to see how LLMs fare (I see you have included BLOOM, OPT etc but it is clear that there is a big gap in performance as compared to GPT models).\n\nC: How can you make sure that cross-task generalization has not been corrupted for large pretrained zero-shot models? What model is used to create Figure 2 (the heatmap)?\n\nD: The results table is unclear and I’m open to revising my review based on clarifications.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and suggestions. We provide our responses to the points addressed to improve the reviewer’s understanding of our work and clarify any ambiguity.\n\nResponse to “Reasons to Reject”\n\nThe AVG score which we present is an average of the scores for each individual task, which can be found in Table 7. \nThe rationale behind using a unified average score is to provide a high-level comparison of the performances of zero-shot and fine-tuned models under various settings, both including task type (regression/classification/pair/span) as well as dimension of social knowledge. Also, it is worth noting that because each task setting has a different number of tasks, it is expected that the numbers in Table 2 when averaged by social dimension or by task type will not equal the overall Avg score. We will clarify the score aggregation in our revised version.\n\nResponse to “Questions For The Authors”\n\nThe regression tasks are evaluated on the correlation between the predicted and true scores using Pearson’s r. Since Pearson’s r ranges between -1 and 1, we further normalize our score to fit into a range of 0-1 to be comparable with other metrics. \nRegarding the finetuning of T5, we follow the method of Gao et al. (2020), which are mentioned in lines 361-364. We will add more details in future revisions.\n\nWhile the original focus of our work was to compare the performance of open-source LLMs to advocate open science, we agree with the authors that a comparison of recent GPT models could provide additional insights into understanding the capabilities of modern LLMs. Therefore, we added an experiment using gpt-3.5-turbo on all of the tasks using the same prompts. Our results reveal that GPT-3.5, despite its large number of parameters, is on par with flan-T5-xxl, showing that performance does not always scale with parameter size. We note that we were not able to run GPT-4 on our benchmark during the rebuttal period due to both time and cost constraints (~$700).\n\nThere are currently no fully accurate ways to know whether a closed source model (or one where the training data is not known) has seen some examples during its pretraining. However, we recognize that this is a risk also shared with several prior dataset papers testing LLMs. Nevertheless, we highlight that we did not observe any great performance increase for cross-generalization on any of our zero-shot LLM settings. This result suggests that existing models have limited ability in predicting social knowledge in a zero-shot setting, which suggests that even if there had been a leakage of the training data, it is not represented in our results.\n\nFigure 2 was created by obtaining pairwise scores for each setting, which we describe in Section B.6 in the Appendix.\n\nResponse to “Typos Grammar Style and Presentation Improvements”\n\nWe thank the reviewer for their keen eye in spotting these errors. We have immediately addressed them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Built from 58 tasks across 35 datasets, this benchmark presents a holistic way to analyze models’ understanding of social knowledge. It contains 4 different types of tasks - classification, regression, pairwise comparison and span identification - which somewhat avoids various problems related to task-related artifacts that models often exploit.\n\nThe paper points to avenues of leveraging cross-task transfer for improved understanding of social knowledge for models.", "reasons_to_reject": "What does the AVG in Table 2 mean? What is it an average of? Unifying all the metrics across datasets as an average poses several problems. It washes away performance differences across different columns. How is regression task performance evaluated? How can it be comparable to the other types? No combination of the numbers in Table 2 (for majority) add up to the AVG number.", "questions_to_authors": "A: Can you provide more details about finetuning T5 for a regression task? While the prompts are listed in Table 6, other details are incomplete. \n\nB: Unrelated to scores, have you evaluated how GPT3+ models perform on this benchmark? I’m curious to see how LLMs fare (I see you have included BLOOM, OPT etc but it is clear that there is a big gap in performance as compared to GPT models).\n\nC: How can you make sure that cross-task generalization has not been corrupted for large pretrained zero-shot models? What model is used to create Figure 2 (the heatmap)?\n\nD: The results table is unclear and I’m open to revising my review based on clarifications.", "paper_topic_and_main_contributions": "This paper presents a benchmark to evaluate social knowledge covering several theoretically grounded categories. Authors use theoretical grounding to first curate and then combine 58 tasks from 35 previously published datasets. Then, they evaluate the performance of a plethora of popular models in a zero-shot setting as well as finetuning. Experiments show that models perform moderately, at best, and there is significant room for improvement. They also show how task dependencies affect performance and point to possibilities to leverage cross-task transfer.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The abstract mentions 5 categories but lists only 4. “Social factors” is not listed.\nSubsection B.2 does not contain any text."}, "extra_text": {}}, {"review_id": "oOBGZofzuH", "reviewer": "Reviewer_ytnT", "summary": "", "questions": "1. As SOCKET is a theory-grounded collection of datasets, what are the theories behind it?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful feedback they have provided. We hope to further address and clarify any ambiguity.\n\n1. We understand the reviewer’s opinion on including more types of NLP tasks in SoCKET for increased generalizability. At the same time, we also would like to emphasize that based on our review, the four types we considered for our benchmark (classification, regression, pairwise comparison, and span detection) were what we found to be most common for NLP tasks related to social aspects. While adding other types of tasks - e.g. summarization, question answering, document ranking - may further enrich our dataset, we believe that the current status of the benchmark is sufficient to test our question of interest for this project: to examine the capability of LLMs for solving NLP tasks requiring social knowledge. We also highlight that most benchmarks contain only a single task type and our aim with SoCKET is to examine the models’ abilities in a wider range of settings than previous work.\n\n2. As the reviewer righteously points out, the effectiveness of knowledge transfer in LLMs across tasks has been previously studied. However, our study actually shows the opposite result from past work: transfer between tasks requiring social knowledge is **not** as strong as expected and often is non-existent. This result is known and surprising, as social theory suggests we should see a transfer between tasks, e.g., between offensiveness and humor. Our new results highlight the current limitation of LLMs in understanding and effectively learning how to solve tasks requiring social knowledge, signaling a need for stronger models that can encode the social aspects required for solving such tasks. We believe this distinction from prior research makes it a valuable contribution.\n\n3. Regarding the reviewer’s comment on the addition of recent GPT-based models, we would first like to address that our evaluations on the benchmark were originally intended to only include open-source LLMs to maximize reproducibility. However, given the strong reasoning capabilities of GPT-3.5/GPT-4 and its impact on modern NLP research, we have added an experiment of evaluating SOCKET using gpt-3.5-turbo from OpenAI's API. Interestingly, we were able to discover that while GPT-3.5 outperforms several baseline LLMs, its performance is still only comparable to Google's flan-t5-xxl. This finding strengthens our claim that zero-shot predictions using LLMs largely underperform fine-tuned models. Please bear in mind that while the estimated cost for running GPT-3.5 on SOCKET was within a reasonable price range, the estimated cost for running GPT-4 on the same benchmark was over $700, and unfortunately out of our budget for the period of the rebuttal.\n\nQuestions For The Authors:\n\nA summary of the theoretical framework for SOCKET can be found in lines 95-112 of the submitted manuscript. While we were not able to provide a more detailed theoretical grounding due to space constraints, our framework was constructed by identifying several NLP tasks that measure social information and grouping them into relevant categories, where each category is known to play a role in shaping social relationships and interactions in language from perspectives of interpersonal communication (Berger, 2005), psychology and cognitive science (Barret and Lindquist, 2007), and critical studies (Augustinos and Every 2007). We plan to provide further details of the framework construction in an updated version. Please also refer to our response to Reviewer egYJ where we have discussed the motivation behind the framing of \"social knowledge\".\n\nReferences\n\nAugoustinos, Martha. and Every, Danielle., 2007. The language of “race” and prejudice: A discourse of denial, reason, and liberal-practical politics. Journal of language and social psychology, 26(2), pp.123-141.\n\nBarrett, Lisa F., Lindquist, Kristin A. and Gendron, Maria., 2007. Language as context for the perception of emotion. Trends in cognitive sciences, 11(8), pp.327-332.\n\nBerger, Charles R., 2005. Interpersonal communication: Theoretical perspectives, future prospects. Journal of communication.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "## Reasons to Accept\n\n1. Comprehensive experiments with a large number of datasets collected on four types of tasks (classification, regression, pairwise, span)\n\n2. Logical writing and clear presentation (well-formatted Tables and Figures)\n\n3. In-depth analysis of experiment results \n\n4. Studies social knowledge transfer which may draw inspiration for the community", "reasons_to_reject": "1. The selection of tasks is limited to only 4 types which may not be enough.\n\n2. The technical contribution of studying task transfer may be limited, as it is explored in other NLP tasks in prior works\n\n3. More recent state-of-the-art models like GPT-3.5 and GPT-4 were not evaluated.", "questions_to_authors": "1. As SOCKET is a theory-grounded collection of datasets, what are the theories behind it?", "paper_topic_and_main_contributions": "This paper introduces **SOCKET**, a new benchmark with 58 diverse NLP tasks to comprehensively evaluate **social knowledge** in LLMs. \n\n- Tasks are grouped into 5 categories grounded in linguistic theories:\n  - Humor & Sarcasm\n  - Offensiveness\n  - Sentiment & Emotion\n  - Social Factors\n  - Trustworthiness\n\nThe paper benchmarks popular LLMs, finding **limited social capabilities** (~0.7 out of 1 score). It analyzes **correlations** between tasks/categories, showing promise for social knowledge transfer.\n\n## Key Contributions\n\n- Introduces a **systematic benchmark** for social knowledge evaluation\n- Provides **empirical analysis** revealing LLMs' weaknesses on social tasks \n- Characterizes **relationships** between social language dimensions though task transfer", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new theory-driven benchmark, SOCKET, that contains 58 NLP tasks testing social knowledge, grouped into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness.
LLMs are evaluated on this benchmark, providing interesting findings.

While not all the more recent LLMS have been evaluated, the benchmark is the focus of the paper and the findings are already interesting.
While very interesting, the sudy of the social capability transfer could be improve, maybe in a future work.
Decision: Accept-Main