Venue: EMNLP 2023

# Do LLMs Understand Social Knowledge? Evaluating the Sociability of

Large Language Models with the SocKET Benchmark

 Minje Choi\({}^{\dagger}\) Jiaxin Pei\({}^{\ddagger}\) Sagar Kumar \({}^{\ddagger}\) Chang Shu\({}^{\sharp}\) David Jurgens\({}^{\dagger}\)

\({}^{\dagger}\)University of Michigan, Ann Arbor, MI, USA

\({}^{\ddagger}\)Northeastern University, Boston, MA, USA

\({}^{\sharp}\) University of Cambridge, Cambridge, UK

\({}^{\clubsuit}\){minje, pedropei, jurgens}@umich.edu

\({}^{\dagger}\)kumar.sag@northeastern.edu cs2175@cam.ac.uk

equal contribution

###### Abstract

Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. While LLMs are increasingly deployed in many forms including conversational agents that interact with humans, we lack a grounded benchmark to measure how well LLMs understand _social_ language. Here, we introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge which we group into five categories: humor & sarcasm, offensiveness, sentiment & emotion, trustworthiness, and other social factors. In tests on the benchmark, we demonstrate that current models attain only moderate performance but reveal significant potential for task transfer among different types and categories of tasks, which were predicted from theory. Through zero-shot evaluations, we show that pretrained models already possess some innate but limited capabilities of social language understanding and training on one category of tasks can improve zero-shot testing on others. Our benchmark provides a systematic way to analyze model performance on an important dimension of language and points to clear room for improvement to build more socially-aware LLMs. The resources are released at [https://github.com/minjechoi/SOCKET](https://github.com/minjechoi/SOCKET).

+
Footnote †: November 6, 2021

## 1 Introduction

Interpersonal communication is more than just what is said. Understanding communication requires reasoning not only about the content of a message but also the social implications drawn from that message Halliday (1995). As NLP systems, particularly Large Language Models (LLMs), are increasingly used in interpersonal settings, these models' abilities to understand social knowledge become critical. However, despite the recognized need for social knowledge Hovy and Yang (2021), the NLP field has limited abilities to test it. Here, we introduce SocKET, a new benchmark for evaluating social knowledge.

Evaluating NLP systems has remained a key component for benchmarking the field's progress. Indeed, the rapid replacement of traditional models by LLM-based approaches was strongly motivated by substantial gains by LLMs on a variety of comprehensive Natural Language Understanding (NLU) benchmarks like SuperGLUE Wang et al. (2019) and Natural Questions Kwiatkowski et al. (2019). However, despite the fundamental social aspect of language, comprehensive benchmarks of social language remain absent. Instead, existing computational studies of social language have built individual datasets and models for specific types of information like empathy Sharma et al. (2020), politeness Danescu-Niculescu-Mizil et al. (2013), and humor Van Hee et al. (2018). While beneficial, these semantic-level tasks omit broader social and narrative-level information Li et al. (2021) and present only a narrow view of model performance.

We introduce SocKEt (**Soci**al **K**nowledge **E**valuation **T**ests), a theory-grounded, systematic collection of 58 social language tasks.1 SocKET covers five categories of social information: sentiment & emotion, trustworthiness, humor & sarcasm, offensiveness, and social factors, each motivated by specific theories. To examine models' generalizability, SocKET includes four task formats: classification, regression, pairwise comparison, and span identification. This construction aims at assessing not only NLP models' performances on individual tasks but their ability to perform multiple task types and to productively benefit from related tasks and task categories during learning.

Footnote 1: The choice of the term “social knowledge” in framing stems from its use for a broad category in psychology e.g., Turiel (1983); Adolphs (2009) that matched the capabilities we are interested in.

Our study offers the fol

tions to the research community. (1) We motivate a theoretically-grounded organization of social tasks (SS2) and subsequently introduce a new easy-to-use benchmark, SocKET, that systematically organizes 58 tasks (SS3). (2) We benchmark multiple current LLM approaches to multitask NLU via standard supervised training and zero-shot LLMs (SS4). Across all tests, our results show that baseline LLMs perform moderately, at best, but offer promising signs of being able to leverage task correlations. (3) We test the abilities of models to make use of cross-task transfer (SS5) showing multi-task training on strongly correlated tasks can maintain or even improve performance in specific tasks, but doing so on weakly correlated tasks can hurt the overall performance of LLMs (SS6). We release our framework code and prepackaged datasets at [https://github.com/minjechoi/SOCKET](https://github.com/minjechoi/SOCKET) and [https://huggingface.co/datasets/Blablablab/SOCKET](https://huggingface.co/datasets/Blablablab/SOCKET).

## 2 Social Information in Natural Language Processing

Language is inherently social, as meaning is constructed through social interactions Wittgenstein (1953). A substantial body of research in linguistic theory and communication studies have examined how social knowledge is communicated via language understanding. Theories of language grounded in interaction and communication systems such as Systemic Functional Linguistics (SFL) by Halliday et al. (1989) assert that the function and appropriacy of language in a given context is the key to our understanding of language and its use Eggins (2004); Allan (2007); Halliday et al. (1989); Halliday (2004). We use these insights to probe linguistic models for their ability to capture _social information_, which we define as information conveyed through text about broader metatextual function and contextual appropriacy of the utterances in conversation.

**NLP Studies on Social Information** Numerous studies have contributed to the development of datasets and models aimed toward identifying nuanced social information in language across diverse contexts. Computational linguists have modeled multiple forms of social information in language like sentiment Buechel and Hahn (2017), politeness Fu et al. (2020), humor Meaney et al. (2021), offensiveness ElSherief et al. (2021), and intimacy Pei and Jurgens (2020), often achieving state-of-the-art results close to human performance in their respective settings. Studies such as Park et al. (2021) have also leveraged explicitly-given norms to train models to be more accurate in context-specific situations.

However, these plausible results may be achievable solely by focusing on the statistical and syntactical instead of the social aspects of language. Whether to make advances in language understanding in research or to ensure reliability and safety in deployment, it is of vital importance to study whether models are truly capable of gaining a generalizable understanding of social factors before employing them for tasks that require such knowledge Hovy and Yang (2021). The necessity for such understanding is exemplified by studies showing that, when measuring the same concept, the performance of a model can vary greatly when tested on a different dataset due to factors such as changes in dialect, speaker demographics, and dataset domain Miller et al. (2020); Blodgett et al. (2016); Wang et al. (2022a).

Despite this importance, efforts towards aggregating and synthesizing various datasets into themes have been less practiced. One notable exception is the work of Kang and Hovy (2021), where the authors combine existing datasets on different linguistic styles to introduce a benchmark that enables them to study cross-style language understanding. Similarly, we present a benchmark curated from over fifty different tasks on different aspects of social information, which we group into five distinctive categories.

**Examining the social knowledge of LLMs** LLMs are ubiquitous in NLP and their success is attributed to the ability to capture language characteristics from the immense amount of text seen in pre-training and to effectively apply this information on downstream tasks, achieving state-of-the-art performances in many language understanding tasks Chung et al. (2022a). LLMs have demonstrated less success when solving tasks directly related to social knowledge. For tasks that require social information such as detecting sarcasm Farha et al. (2022) or patronizing language Perez-Almendros et al. (2022), recent models exhibit only moderate performance. One major challenge is that compared to humans, LLMs have less capability to make predictions outside of the provided input and must perform reasoning only based on their innate social information Sap et al. ( 

2019b; Zhou et al., 2020). Yet, it is this very social knowledge that is crucial for human interactions and conversations and is a milestone that should be reached for LLMs to engage in meaningful communications with humans Mahowald et al. (2023).

More recently, general-purpose LLMs trained with instruction-based prompts have been known to achieve strong performances, putting them to use in several practical domains such as summarization, question answering, and classification Sanh et al. (2022). A newly emerging trend is to use curated prompts to identify the psychological capabilities of instruction-guided LLMs. Ruis et al. (2022) and Hu et al. (2022) examine pragmatic understanding capabilities using prompts. Coupled with additional steps such as chain-of-thought (CoT) reasoning, this prompt-based approach has large potential for understanding whether LLMs can provide reasoning capabilities like humans.

**The Inter-relatedness of Social Information** Social language understanding requires accurately perceiving different dimensions and facets of communication that relate to one another. Interpersonal communication makes frequent use of humor Schnurr (2010), mitigation, also known as hedging Schneider (2010), and swearing as a norm violation Stapleton (2003) in defining the contours of the social context for the speakers. Often, the pragmatics of these different dimensions of social language use are intertwined: communication with one dimension influences the interpretation of another, e.g., politeness and offensive speech Culpeper (2021), humor and politeness Attardo (2008), humor and offensiveness Alberts (1992), and mitigation and empathy LI Hai-hui (2019). Understanding one of these dimensions requires models to have the ability to recognize the related dimensions. While past computational work has largely focused on single dimensions, SocKET fills a key gap by testing whether models can accurately recognize multiple, interrelated social dimensions--and whether models can benefit in their understanding from cross-task transfer.

## 3 The SocKEET Benchmark

Here, we describe the steps taken to curate SocKEET as robust benchmark for identifying social information embedded in language in interpersonal communication contexts.

### Task Selection Process

The task curation process began with a systematic review of literature on social from linguistics, communications, and psychology to identify likely categories of social knowledge. Then, possible datasets and tasks were identified through a systematic review of datasets published at ACL, EMNLP, NAACL, EACL, LREC, and SemEval since 2015. In this first pass, we selected more than 100 datasets and tasks to detect different types of social information in language (cf. Table 11 in Appendix B.9 for all candidate datasets and tasks). Tasks were selected based on membership in five categories of social language (described next) that are motivated as core aspects of social language understanding.

For each category, we include tasks of several distinct objectives: binary and multi-class classification, regression, pairwise similarity detection, and span identification.2 Where possible, we aim for diversity within categories and ensure one task for each objective. Candidate tasks were removed if it was found that training a bert-base-uncased model on the task achieved test performance over 0.95, which would provide little insight into progress at recognizing social information .

Footnote 2: Other task types were initially considered (e.g., generation, paraphrasing) but such tasks were not feasible for all models and often were less standardized in their evaluation, complicating cross-task comparison if included.

While this process identified many candidate tasks in multiple categories, the benchmark still defines only partial progress in social knowledge capabilities. Some abilities recognized by social sciences such as decei have only one or two tasks proposed Ott et al. (2011), providing limited data to measure progress. However, recognizing these as limitations (discussed in more detail in SS8), SocKEET provides a diverse set of tasks and capabilities, described next, for the field to begin to measure progress.

### Task categories

Inspired by theories in interpersonal communication and interpersonal pragmatics, we provide a thematic organization of the tasks in SocKEET into five related categories of social knowledge: Humor & Sarcasm, Offensiveness, Sentiment & Emotion, Social Factors, and Trustworthiness.

**Humor & Sarcasm** The practice of humor in conversations and interactions plays a key role 

in maintaining and forming positive social relations Holmes (2006); Brown et al. (1987); Ziv (2010). We differ Humor & Sarcasm from Trustworthness as a social information category because while both categories consider non-cooperative behaviors Grice (1975), humor is considered to be prosocial Attardo (2008). In instances where the humor is not considered to be prosocial and is instead of a derogatory nature, we consider it to be in the Offensiveness category. By nature, humor is a subjective concept that can differ depending on both demographic and contextual factors Ruch (2010), making humor detection a difficult task for LLMs. SocKET includes a number of tasks on humor that can occur in various contexts such as in social media Meaney et al. (2021), short jokes Meaney et al. (2021), and news headlines Hossain et al. (2020). We also include tasks that require detecting relevant concepts of humor such as sarcasm Khodak et al. (2018) and irony Van Hee et al. (2018).

**Offensiveness** Detecting offensiveness using computational methods has gained large attraction in recent years due to the ubiquity of online communication and the necessity to implement automated content moderation to combat abusive behaviors Spertus (1997). However, most existing studies only focus on limited types of offensive languages Jurgens et al. (2019). In this study, we consider offensiveness to be any explicit or implicit language directed towards individuals, entities, or groups Waseem et al. (2017), and the tasks chosen are representative of this understanding. SocKET includes a list of offensiveness detection tasks covering different levels of harmful content and abusive language including both explicit and implicit hate ElSherief et al. (2021), abuse Vidgen et al. (2021), and humor-related offensiveness Meaney et al. (2021). We also include forms of bias directed towards people and groups, as social bias enforces harmful stereotypes Sap et al. (2020).

**Sentiment & Emotion** Emotion is a core element of interpersonal communication that can be communicated through human language in several aspects Majid (2012); Barrett et al. (2007). Social information is crucial in the ability to not only communicate, but also feel emotion. Theories of discretized emotion Ekman (1992) have been supported by empirical findings that humans use discrete labels learned through language to direct their emotional responses to stimuli Lindquist and Barrett (2008). Moreover, emotional responses have been shown to direct communication with peers Lee et al. (2020), and expressing certain emotional responses--such as anger--have been shown to have social ramifications Keltner et al. (1993). Interpreting emotions from text using computational tools has been a popular research topic across numerous areas in social sciences, enabling new discoveries at unprecedented scale Jackson et al. (2022). In SocKET, we include a wide range of tasks from various domains such as daily dialogue Li et al. (2017), written responses to news stories Bucchel and Hahn (2017), and tweets using textual syntax Mohammad et al. (2018), and also emojis Barbieri et al. (2018).

**Trustworthiness** People can detect cues in language that determine the trustworthiness of a message Newman et al. (2003), leading to studies that aim to quantify the level of trust in text using computational methods Choi et al. (2020). In particu

\begin{table}
\begin{tabular}{l l|c c c c c|c c c c c}  & name & age & sex & sex & sex & sex & sex & sex & sex \\ \hline \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives/Conferences & books & warber & Essence & confectives & books \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose B Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & book & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al., 2015) & humage & 5/17 & 30/0 & Books & Surramed & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books & warber & Essence & confectives & books \\ \multicolumn{14}{l}{hose A Sexan} & baseline Chicago (Abbson et al.

 

lar, this direction has gained attention from NLP communities following increased needs to combat and mitigate potential harms coming from the generation and dissemination of false information in online spaces (Wu et al., 2019). In SocKET we include tasks that require identifying perceived trust from several dimensions: impartiality (Pryzant et al., 2020), deception (Ott et al., 2011), propaganda (Martino et al., 2020), rumor (Ma et al., 2017) and bragging, as it is considered to be "unplain speaking" (Haiman, 1998; Jin et al., 2022).

**Other Social Factors** Finally, we include tasks of a more discursive and rhetorical type, that are understood to be more reliant on the contextual elements of social distance, power, and solidarity. In SocKET, the tasks included are empathy (Buechel et al., 2018), politeness (Hayati et al., 2021; Fu et al., 2020), intimacy (Pei and Jurgens, 2020) and complaints (Preotiuc-Pietro et al., 2019). Politeness, like humor, is understood to be a noncooperative prosocial behavior but unlike humor, is concerned with the act of "saving face" (Brown and Levinson, 1987). Empathy, shown to be closely related to politeness (Fukushima and Haugh, 2014), is heavily reliant on social positions in the context of the conversation (Macagno et al., 2022). Intinacy, however, has been shown to be more dependent on notions of time and space between people in dialogue (Marquez Reiter and Frohlich, 2020).

### Dataset Summary

The final SocKET benchmark contains 58 tasks from 35 datasets, grouped into the five categories shown in Figure 1. We denote multiple tasks from the same dataset by adding the task name as a suffix following the dataset name and # symbol.

The collection of tasks chosen for SocKET makes it a comprehensive benchmark to measure language models' abilities to capture underlying social information. Motivated by theories of systemic functional linguistics and interpersonal pragmatics, SocKET cuts across a number of dimensions of interpersonal communication, allowing it to also be a tool to better understand and interpret co-learning abilities and dependencies in sociolinguistic tasks. Having this ability allows researchers and users to more efficiently and effectively deploy NLP methods by providing empirical results on the limits and affordances of a variety of out-of-domain social language tasks.

In total, SocKET spans 2,616,342 items across all tasks, including 269,246 samples in the test set. However, experimenting with an evaluation set of size can be prohibitive due to model size, available resources, and considerations of the environment. Therefore, we also release a subset of our data as SocKETte (SocKEt but **T**inier) that contains at most 1000 items per task in the test set, reducing the test set to 43,731 samples. In Appendix B.3, we show that performance on SocKETte is highly correlated and we hope that this smaller subset enables more rapid progress.

## 4 Benchmarks on the Social Knowledge Capabilities of LLMs

We first train and evaluate several commonly used multitask LLMs on our datasets to obtain benchmark results, which provide a first glimpse of how good LLMs are at learning social knowledge tasks. Experiment details are described in Appendix SSB.

### Training Methods

* BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)
- as well as two lightweight models known to achieve high performance on finentuning tasks
- DeBERTa-V3 (He et al., 2021) and MiniLM (Wang et al., 2020).

**Prompt-based finetuning** Prompt-based finetuning has emerged as a flexible and effective means of adapting models to downstream tasks (Wei et al., 2021). As a benchmark, we include the performances of a T5 model (Raffel et al., 2020) trained on each task via finetuning. We manually design prompts for each task. For classification tasks, we use verbalizers to map the class to word labels and for regression tasks, we adopt a method similar to Gao et al. (2021) in that we use two anchor words "Yes" and "No" and consider the probability of predicting "Yes" as the final score. For span-based tasks, we train the model to directly generate the sequence outputs. A list of prompts can be found in Table 8 and Table 9 in the Appendix.

**Zero-shot predictions** We further apply our designed prompts to test the performances of LLMs in a zero-shot setting where no further finetuning is performed. Using the same prompts proposed in Table 8, we test SocKET on several widely used LLMs: GPT (Radford et al., 2018), GPT-J 

(Zhang et al., 2022), T5 (Raffel et al., 2020), LLaMA (Touvron et al., 2023a), LLaMA-2 (Touvron et al., 2023b), BLOOM (Workshop et al., 2023), BLOOMZ (Muennighoff et al., 2022), FLAN-T5 (Chung et al., 2022b), RedPajama (Computer, 2023), and Alpaca (Taori et al., 2023; Wang et al., 2022b). We also evaluate the performance of GPT-3.53 using OpenAI's API. Samples for which a model does not provide an appropriate label are automatically marked as incorrect. For each LLM variant, we test zero-shot results for different model sizes ranging between 110M and 13B parameters, which we report in Table 4 in the Appendix.

Footnote 3: [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5)

### Results

We compare model performances across category type and task type as shown in Table 2. Each reported value is the average of the scores on every task within the specified group. The rationale behind using a unified average score is to provide a high-level comparison of the performances of zero-shot and fine-tuned models under various settings, including task type (regression/classification/pair/span) as well as dimension of social knowledge.

DeBERTa-V3 achieves the best overall performance after full training on each of the SocKET datasets, followed by other BERT-based models. The prompt-based finetuning of T5 performs worse than standard finetuning, especially on the pairwise classification and regression tasks. Meanwhile, most zero-shot models perform only slightly better than the baseline, indicating that prompting alone does not elicit correct social knowledge--though two models, google-flan-t5-xxl and GPT3.5, are much closer in performance to supervised models.

* while detecting humor or other social emotions can often be correlated with cues such as sentiment, detecting the level of trust within sentences requires more understanding of the context and may be harder to detect using computational models (Choi et al., 2020). At a task level, we observe that models struggle most in span detection tasks. This is a complex task due to its open-ended nature, and thus BERT-based finetuning does not perform as well as in other types of tasks. We highlight that learning the various aspects of social knowledge is indeed a challenge for current LLMs, and thus call for the need for future models with improved social capabilities.

**Supervised models significantly outperform zero-shot models** Table 2 reveals that despite being much smaller in the number of parameters, fine-tuning supervised models such as MiniLM leads to much better performance than zero-shot models using state-of-the-art LLMs. All the zero-shot LLMs performed poorly, many on par with random baselines, apart from FLAN-T5. Figure 1 shows

\begin{table}
\begin{tabular}{c c r r r r r r r r r r} \hline \hline Category & Model & No. params (B) & Humor \& Sarcasm & Offins. & Sentr \& Eno. & Social Factors & Trust. & CLS & PAIR & REG & SPAN /Agts \\ \hline baseline & majority & & 0.27 & 0.42 & 0.12 & 0.25 & 0.41 & 0.39 & 0.34 & 0.50 & 0.00 & 0.32 \\  & random & & 0.40 & 0.35 & 0.17 & 0.36 & 0.35 & 0.38 & 0.51 & 0.50 & 0.00 & 0.32 \\ \hline \multirow{8}{*}{zero-shot} & Effective AI-gp+db & 6 & 0.39 & 0.35 & 0.29 & 0.33 & 0.28 & 0.32 & 0.26 & 0.50 & 0.08 & 0.32 \\  & alpha-naive & 7 & 0.39 & 0.44 & 0.45 & 0.55 & 0.31 & 0.42 & 0.48 & 0.57 & 0.17 & 0.43 \\  & biospecics-bosom-blm & 71 & 7 & 0.50 & 0.49 & 0.43 & 0.53 & 0.49 & 0.51 & 0.56 & 0.09 & 0.48 \\  & cerebral-cerebnas-Certabs-Ilam-lib & 6 & 6.7 & 0.42 & 0.39 & 0.30 & 0.36 & 0.34 & 0.35 & 0.33 & 0.52 & 0.13 & 0.36 \\  & deogold-research-Islam-lib & 13 & 0.49 & 0.43 & 0.36 & 0.42 & 0.31 & 0.38 & 0.52 & 0.53 & 0.17 & 0.40 \\ zero-shot & deep-std-lib & 13 & 0.31 & 0.40 & 0.19 & 0.22 & 0.28 & 0.31 & 0.25 & 0.49 & 0.13 & 0.31 \\  & geogle-final-ctxl & 11 & 0.66 & 0.56 & 0.52 & 0.60 & 0.36 & 0.64 & 0.63 & 0.17 & 0.55 \\  & (5-3b) & 3 & 0.34 & 0.41 & 0.27 & 0.32 & 0.35 & 0.36 & 0.36 & 0.49 & 0.13 & 0.36 \\  & human-2rb-stat & 7 & 0.39 & 0.27 & 0.33 & 0.34 & 0.24 & 0.25 & 0.38 & 0.56 & 0.18 & 0.29 \\  & GPf-3.5 & & 0.64 & 0.55 & 0.57 & 0.65 & 0.45 & 0.57 & 0.49 & 0.67 & 0.21 & 0.36 \\ \hline \multirow{8}{*}{finetuning} & Bert-base-based & 0.11 & 0.78 & 0.76 & 0.65 & 0.70 & 0.62 & 0.70 & 0.79 & 0.77 & 0.55 & 0.71 \\  & retheta-base & 0.086 & 0.79 & 0.77 & 0.68 & 0.72 & 0.63 & 0.70 & 0.83 & 0.79 & **0.64** & 0.72 \\ \cline{1-1}  & adbert=3 & 0.098 & **0.83** & **0.77** & **0.70** & **0.72** & **0.66** & **0.72** & **0.87** & **0.79** & 0.63 & **0.73** \\ \cline{1-1}  & Minimal M & 0.066 & 0.77 & 0.72 & 0.61 & 0.67 & 0.55 & 0.66 & 0.78 & 0.69 & 0.57 & 0.67 \\ \cline{1-1}  & T5* & 0.25 & 0.68 & 0.72 & 0.55 & 0.59 & 0.47 & 0.65 & 0.66 & 0.45 & 0.54 & 0.62 \\ \hline \hline \end{tabular}
\end{table}
Table 2: A comparison of the benchmark performances of different models and training schemes. Best-performing instances are shown in bold. The best performing parameter size for each zero-shot model is shown (cf. Figure 1) . A full comparison of all models across all settings can be found in Table 4 in the Appendix. The performances on each individual task using a DeBERta-V3 model can be found in Table 10 in the Appendix.

 

a detailed picture of how different LLM parameter sizes influence the ability to comprehend social knowledge tasks in a zero-shot setting. Surprisingly, we find that of the various training schemes FLAN-T5 is by far the most effective for inferring social knowledge, even with relatively small models. We speculate this performance is due to its initial pretraining on more than 1,000 tasks.

**More parameters do not guarantee more social knowledge** Another general trend we observe is a weak correlation between the number of parameters and overall performance within the same model architecture (\(\rho=0.266\), \(p=.08\)). This is to some extent determined by the model's ability to understand the task itself given an instruction prompt as well as a sample input, as larger models are capable of understanding a wider variety of tasks (cf. Appendix Table 6). Of course, it is also possible that larger LLMs could encode a greater amount of social knowledge through their greater parameter sizes. Interestingly, we observe that for some models, larger size does not always guarantee better performance. This is the case especially for BLOOM, T5 and GPT, where the largest model is not always the best performer within the group.

Models varied in the ability to follow instructions (Appendix Table 6). As expected, instruction-tuned models like FLAN-T5 and Alpaca are generally able to follow the prompt instructions, while other models may generate answers that are not provided in the options. For our social tasks, instruction-following was not significantly correlated with model size (\(\rho\)=0.08, p=0.60). Thus, lower model performance in Figure 1 is, in part, due to models being unable to answer questions relating to social knowledge.

**When models are able to answer the question, are they right?** Restricting only to instances in which a model outputs a valid answer reveals heterogeneity among different model groups (Figure 3), showing an interplay between model size, coverage, and performance. For architectures such as FLAN-T5 or BLOOMZ we observe a positive correlation between parameter size and performance, both in its ability to understand instructions and to make correct predictions. On the other hand, for certain architectures having larger parameters can actually make it worse at understanding instructions (e.g. LlaMA) or predicting correctly (e.g. OPT). Recognizing that measuring of instruction understanding and the accuracy of an LLM both depend on how strictly one chooses to map the predictions to an answer, overall, our results suggest that while LLMs do contain the potential for understanding social knowledge, additional steps such as finetuning or instruction tuning are likely needed for better social understanding.

## 5 Do we see Cross-task Transfer of Social Knowledge?

In this section, we examine the relations and dependencies between tasks using the predictions of LLMs trained on different tasks and test for dependencies between tasks that are predicted by theory.

**Quantifying Task Dependency** We quantify the dependency between two tasks as follows. We fine-tune a pretrained LLM on task \(t_{A}\) to obtain a model \(m_{A}\), which is used to make predictions on the test set of another task \(t_{B}\). The correlation between the predicted values from model \(m_{A}\) and the true labels of the test set of \(t_{B}\) is considered as the task

Figure 1: A comparison of LLMs on the aggregated scores tested on SocKET under zero-shot settings. The overall performances vary greatly by model architecture, while larger models do not always guarantee better performance.

 

dependency that \(t_{A}\) has on \(t_{B}\). We report the absolute correlation value, as negatively correlated tasks are still informative. We describe how the correlations are obtained across different task types in the Appendix (SSB.6). Span identification tasks are omitted from this analysis, resulting in \(55\times 55\) scores. We also measure the pairwise correlation between models \(m_{A}\) and \(m_{B}\) as well as task dependency to gain an additional perspective of task similarity. Details for the model correlation can be found in Appendix SSB.6 and Figure 7.

The task dependencies for all task pairs, shown in Figure 2, reveal salient block structures within the category,4 especially for the Offensiveness, Sentiment & Emotion, and Social Factors categories, suggesting the existence of shared knowledge within our thematically grouped tasks. These correlations align with existing findings from interpersonal pragmatics on the relationships between social knowledge. For instance, increased self-disclosure or pain-related interactions are known to promote both intimacy (_questioninitumacy_) and empathy (_empathy_) [13, 8], two elements within the Social Factors category, while the usage of emojis (_tweet_emoji_) as effective symbols are indicative of emotional states such as valence (_embobank#valence_) and arousal (_embank#arousal_) [9], which belong to the Sentiment & Emotion category.

Footnote 4: See Figure 6 for fully labeled version.

The Offensiveness category shows mixed results in comparison with Arango et al. ArangoEtAl09, whose results show that hate speech datasets are often overfit and do not generalize well to other similar datasets . Figures 2 & 6, however, show that of the seven datasets included in SocKET, five of them included at least one task which showed comparable correlations when tested both within and out of domain. Indeed, _PersonDirectedAbuse_, a task labeled for offensive language specifically directed towards an individual, is actually predicted better by models fine-tuned on jigsaw# tasks than it was on its own.

Interestingly, correlations are scarce within the Humor & Sarcasm, and Trustworthiness categories. This is consistent with findings from [12] which show that models without exposure to linguistic forms lack the requisite social information to perform well on non-literal pragmatic phenomena such as humor and deceit.

Another notable individual task is _humor rating_ from the Humor & Sarcasm dataset, which performs well as both the fine-tuning and predicted task alongside a number of tasks from the Emotion & Sentiment category--particularly discretized emotion tasks, as well as _hateoffensive_ in the Offensiveness category--which labels comments as either "hateful," "offensive," or neither. While relationships between offensiveness and humor have been theorized as early as Freud FreudFreud10 and sentiment recognition has been shown to bolster offensive language detection [10], relatively little has been said regarding connections between the three categories and thus, this result presents an opportunity for further research.

We observe that _politeness_ shows strong transfer with many of the offensive and hate speech detection tasks in the SocKET benchmark. In particular, those tasks with high correlation within the offensive category are highly correlated in predicting the politeness classification task. This finding is supported by literature showing that impoliteness can fall under the umbrella of offensive language [1] and, although key differences exist in the pragmastics of the two, the constructs are closely related [13, 14].

Interestingly, regression tasks (from the _habackathon_, _embank_, and _empathy_ datasets) in general have strong correlations with several other tasks. This trend suggests that tasks labeled with continuous variables may have more expressive power compared to ordinal or nominal categorization, and thus have a higher potential for stronger

Figure 2: Heatmap of task dependency among all task pairs, annotated at category level. Each value represents the absolute strength of correlation between the true labels of the test set of a specific task (columns) and the predictions made on that task using a model trained on a different task (rows). We observe strong correlations, especially within the Offensiveness, Sentiment & Emotion, and Social Factors categories. A larger version labeled at the task level is shown in Appendix Figure 6.

 

task dependencies. However, the magnitude of the correlation may be influenced by the relative value distributions of different correlation methods. This finding calls for a need for more datasets with continuous labels, which requires more effort but allows models to capture more fine-grained concepts of social knowledge.

## 6 Can Multi-task Training improve Social Knowledge?

Our findings reveal significant task transfer, both within and across task categories, which hints at shared knowledge among tasks. Linguistics studies of social language also note the interrelated perceptions of different dimensions such as humor and offensiveness Culpeper (2021); Attardo (2008); Alberts (1992); LI Hai-hui (2019). We now examine whether LLMs can learn a more robust sense of social knowledge by training on multiple tasks.

**Experimental Setup** Recent studies have explored the possibility of multi-task training on LLMs, which is training a single model on several different tasks simultaneously, with effects of improving its performance on both seen and unseen tasks Aghajanyan et al. (2021); Padmakumar et al. (2022). We apply multi-task training on SocKET, but make one clear distinction from prior work. Whereas previous studies have shown that multi-task training is especially effective when the grouped tasks are of similar types Padmakumar et al. (2022), we introduce a new setting by grouping tasks instead by our defined categories of social knowledge. We expect that same-category tasks contain social knowledge that can be shared across tasks, resulting in LLMs that learn a more robust concept of the specific dimension than when trained on single tasks.

A popular method for multi-task training is prefinetuning Aghajanyan et al. (2021); Shi et al. (2022), which involves a first stage of finetuning on multiple tasks using task-specific heads on a shared encoder, then re-using the encoder for downstream tasks. We apply pre-finetuning in two different settings: (1) _category-wise tasks_, where we perform pre-finetuning on tasks grouped to the same category, and (2) _all tasks_, where all tasks of SocKET are included in the pre-finetuning stage. Consistent with prior work, we perform the second finetuning stage on individual tasks using the pre-finetuned model as initial weights Aghajanyan et al. (2021). Other training details are identical to SS4.

**Results** Multitask training had little to negative effect on task performance (Table 3). Although some tasks did benefit from being co-trained within category (Appendix Table 10)--particularly the Offensiveness category--when aggregated at the category level, the average performance is worse. In particular, the Humor & Sarcasm and Trustworthiness categories have the lowest levels of within-task and cross-task dependencies (SS5). The performance drop is less strong in categories with high dependency, indicating that while multi-task training on similar tasks may not always improve performance, task-relatedness can help preserve performance when also learning task-specific new concepts. Together, our results suggest multi-task training on unrelated social tasks hurts overall performance--a result contrary to social science expectations of how social information is processed--and points to a need to further investigate cases when applying multi-task training as a practice to improve the social knowledge of LLMs.

## 7 Conclusion

People increasingly interact with LLMs in natural conversation. To what degree are these models able to pick up on the social cues? To help answer this question, we introduce SocKET, an NLP benchmark to evaluate how well models perform at learning and recognizing concepts of social knowledge. We provide benchmark results using several popular models and provide case studies of studying the inherent social capabilities of LLMs in a zero-shot setting. Surprisingly, LLMs perform moderately at best, with even large LLMs (\(>\)10b parameters) varying widely in their abilities. Additionally, we show that there exist significant task dependencies both within and across task categories, and that multi-task training on task categories can affect model performance. Our work contributes to the broader NLP community by fostering future efforts toward building and evaluating more socially responsible and coherent LLMs.

\begin{table}
\begin{tabular}{r c c c} \hline   Category & Single task & Category-wise & All tasks \\ \hline Humor \& Sarcasm & 0.76 & 0.76 & 0.74* \\ Offensiveness & 0.76 & 0.76 & 0.76 \\ Sentiment \& Emotion & 0.64 & 0.64 & 0.62 \\ Social Factors & 0.67 & 0.67 & 0.66 \\ Trustworthiness & 0.66 & 0.64* & 0.62* \\ \end{tabular}
\end{table}
Table 3: The performances of different multi-task settings aggregated at category level. Numbers with * indicate cases where the prediction results significantly differ from the single task setting (paired t-tests).

 



## 8 Limitations

Cross-cultural and multilingual expansions Culture is an important aspects of understanding language, especially within the broader setting of multilingual NLP. In this study, however, we make a clear distinction between cultural knowledge and social knowledge, the latter of which is our focus for this study. Our work is grounded in social-psychological theory and the sociolinguistics of interpersonal communication, especially dyadic communication. Such studies are often aimed at phenomena that are widely shared across cultures while recognizing that cultural variation exists within how those phenomena are perceived. In contrast, work in anthropology or cultural studies provides a different perspective and grounding. Such work frequently focuses on cross-cultural perspectives and what is or is-not shared across cultures. For example, in language, the interpretation of whether something is polite can depend on gender norms Mills (2004) and cultural Lorenzo-Dus and Bou-Franch (2003), highlighting the potential context sensitivity. Similarly, the perception of toxicity can depend on the cultural identities of the reader Sap et al. (2019); Ghosh et al. (2021). While highly valuable to study, cultural knowledge is a separate construct from social knowledge (though interrelated) and not the focus of this benchmark, though we hope that our work inspires other benchmarks to help assess such differences.

Regarding multilingual data, SocKET currently contains tasks based in English due to the limited availability of tasks in non-English. While there are a few datasets such as HAHA Chiruzzo et al. (2020) in Spanish and DeTox Demus et al. (2022) in German, we were not able to find sufficient numbers yet to provide a meaningful grouping. This highlights the importance of constructing datasets and frameworks capable of capturing social knowledge for a wide variety of languages, which we consider an important future step.

Additional dimensions and forms of social knowledgeInterpersonal communication conveys a richness of different social information and despite our extensive literature review and data curation process, we fully acknowledge that other dimensions of social knowledge are not included in our current benchmark. In creating SocKET, our aim was to focus on diverse categories of social knowledge that have multiple tasks in order to get a more robust assessment of model capabilities, e.g., multiple tests of a model's ability to recognize humor, in order to avoid the pitfalls of ascribing progress on the basis of a single task alone Subramonian et al. (2023). Nevertheless, SocKET omits several notable dimensions or forms of social knowledge. Some social aspects of language such as pragmatic polysemy Carston (2021); Apresjan (1974) and idioms Strassler (1982) either had too few similar datasets to form a theory-backed category, or there were no existing NLP datasets to test the construct. The latter is the case, especially in the case of linguistic techniques unique to recognize when a speaker is adopting community-specific dialects such as African-American English Hyter et al. (2015); Rivers et al. (2012); Allan (2007) and Queer Language Barrett (2006); Huebner (2021); Harvey (2000).

Social language understanding happens within a static, unspecified context for the current tasks in SocKET. However, the social context in which a message is said can dramatically alter its meaning. NLP is just beginning to incorporate the social context into language understanding Hovy and Yang (2021). While a handful of datasets have begun to explore modeling context explicitly, such as through the preceding conversation Pavlopoulos et al. (2020); Menini et al. (2021), the identity of the speaker Almagro et al. (2022), the social relationship between speakers Jurgens et al. (2023), or explicit social norms Park et al. (2021), there are currently too few of such tasks to compose a comprehensive benchmark with which to measure progress. Future datasets and benchmarks will be needed to study understanding social knowledge when controlling for context.

Thus, SocKET represents a starting point for modeling models' abilities and provides room for improvement via the addition of new categories or constructs, as additional data becomes available. Further inclusion of other dimensions and corresponding tasks should be an ongoing goal.

Benchmarks as markers of progressSocKET fills a current gap for assessing the capabilities of LLMs on understanding social language. However, benchmarks as constructs have been rightly critiqued as markers of progress in NLP (e.g., Bowman and Dahl (2021); Schlangen (2021); Subramanian et al. (2023), due to aspects such as changing or narrowing the field's definition of a task, overemphasizing or overselling progress in a particular

area, or encouraging leaderboard chasing. In designing SocKET, we aimed to directly address the pitfalls of benchmark design by selecting a diverse set of social language understanding tasks that mirrored human capabilities recognized in social science studies; this selection helps ensure a broad measure of performance and that "progress" is not due to improved performance on one type of task. However, the benchmark itself does not capture all of social knowledge (nor do we claim as such) and we view it only as a starting point--a yardstick by which to measure current systems--with a need for new tasks and benchmarks as models advance in their social reasoning capabilities.

The use of a single metric to measure progress in an area or task can mask meaningful insight and fail to contextualize performance. While we follow common practice in NLP (e.g., Wang et al., 2018, Muennighoff et al., 2023, 2019) and report a single mean score in Table 2, the design of SocKET includes specific task categories and types designed to easily and meaningfully inspect what is ultimately contributing to the single score--e.g., are models performing well in classification but poorly in span recognition? Nevertheless, this design is a trade-off: A single score can and likely does promote leaderboard chasing by setting a clear goal to pursue, while completely disaggregated scores like those in Table 4 become unwieldy and make it hard to assess whether meaningful progress is being made when comparing two models. Here, we have opted to report both the overall average and averages for each category and type (10 scores total) in an attempt to balance these two tensions.

Technical limitationsOne major limitation of the current benchmark is we only tested LLMs that have up to 13B parameters. Recent studies show that the LLMs may start to show emergent abilities when they are scaled up above a certain threshold (Wei et al., 2022). Due to limited computational and financial resources, we are not able to test all very large language models, though we welcome future researchers to work on our benchmark and evaluate the sociability of new and larger LLMs.

Finally, our zero-shot model performance used curated prompts on pretrained models without any further finetuning. While it is widely known that instruction-based finetuning specific to downstream tasks can greatly improve performance, we deliberately chose not to do so. Finetuning LLMs with billions of parameters can leave a large carbon footprint, which we avoid for both financial and environmental reasons (Hu et al., 2021, Liu et al., 2022, Lester et al., 2021).

## 9 Ethical Considerations

The interpretation of social information in communication is highly subjective in that it can largely vary depending on demographic and contextual factors. Nevertheless, several NLP datasets are created via crowdsourcing, which raises concerns on whether the dataset's labels are truly representative of our society (Talat et al., 2022). Even within our benchmark, there is the possibility that for tasks such as offensiveness or humor the crowdsourced labels may undermine phrases that might disregard a specific demographic group, which may be inevitably picked up by LLMs that are trained and evaluated on these datasets. Improved versions of our benchmark should include datasets that are more inclusive in such contexts, which we call for future work.

There has been increasing concern over the amount of computing resources required for conducting deep learning research at scale, especially regarding LLMs where task performance is improved through larger datasets, increased model parameters, and longer training hours. The time and amount of computing resources required for training LLMs has become nontrivial (Bender et al., 2021), and it has been increasingly aware among machine learning practitioners to consider the carbon footprint of models and computing methods to minimize risks of global warming. This, combined with limited transparency of experiment results, may harm the very concept of open science. Keeping this in mind, we focused on conducting easily reproducible experiments that can be run on a single GPU within the time frame of hours or a couple of days at the longest. Some of our findings contribute towards this rightful direction, as can be seen in our investigation on multi-task training.

More importantly, we highlight the fact that the main contribution of our study is a thoroughly designed public framework of tasks for examining the social knowledge of LLMs. While it is indeed important to develop and improve LLMs that can perform better on several tasks, we believe that correctly evaluating the level of social knowledge engraved in these models is an equally important task. Without such scrutiny, the users of LLMs deployed in practical settings may be vulnerable to 

socially undesirable or unethical content. We sincerely hope that our efforts in producing SocKET can ease difficulties of conducting future studies that aim to examine and improve the social understanding of LLMs.

## Acknowledgments

The authors thank reviewers for their timely and valuable feedback on the paper, with a special shout-out to R1 for their very detailed feedback which certainly made this paper better. We also thank the members of the Center for Social Media Responsibility, especially Paul Resnick and James Park for their support which enabled the initiation of this project. This work was supported by the National Science Foundation under Grant Nos. IIS-2007251, IIS-2143529, and 2137469. The third author was partially supported by grant SES-2200228 from the National Science Foundation.

## References

* Adolphs (2009) Ralph Adolphs. 2009. The social brain: neural basis of social knowledge. _Annual review of psychology_, 60:693-716.
* Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5799-5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Alberts (1992) JK Alberts. 1992. Teasing and sexual harassment: Double-bind communication. _Constructing and reconstructing gender: The links among communication, language, and gender_, 10:185.
* Allan (2007) Keith Allan. 2007. The pragmatics of connotation. _Journal of Pragmatics_, 39(6):1047-1057.
* Almagro et al. (2022) Manuel Almagro, Ivar R Hannikainen, and Neftali Villanueva. 2022. Whose words hurt? contextual determinants of offensive speech. _Personality and Social Psychology Bulletin_, 48(6):937-953.
* Apresjan (1974) Ju D Apresjan. 1974. Regular polysemy.
* Arango et al. (2019) Ayme Arango, Jorge Perez, and Barbara Poblete. 2019. Hate speech detection is not as easy as you may think: A closer look at model validation. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR'19, page 45-54, New York, NY, USA. Association for Computing Machinery.
* Attardo (2008) Salvatore Attardo. 2008. Semantics and Pragmatics of Humor. _Language and Linguistics Compass_, 2(6):1203-1215.
* Barbieri et al. (2018) Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemEval 2018 Task 2: Multilingual Emoji Prediction. In _Proceedings of the 12th International Workshop on Semantic Evaluation_, pages 24-33, New Orleans, Louisiana. Association for Computational Linguistics.
* Barrett et al. (2007) Lisa Feldman Barrett, Kristen A Lindquist, and Maria Gendron. 2007. Language as context for the perception of emotion. _Trends in cognitive sciences_, 11(8):327-332.
* Barrett (2006) Rusty Barrett. 2006. Queer talk. _Encyclopedia of Language & Linguistics_, 10:316-323.
* Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabette Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immirants and Women in Twitter. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 54-63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrotts: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_, FACcT '21, page 610-623.
* Birke and Sarkar (2006) Julia Birke and Anoop Sarkar. 2006. A Clustering Approach for Nearly Unsupervised Recognition of Nonliteral Language. In _11th Conference of the European Chapter of the Association for Computational Linguistics_, pages 329-336, Trento, Italy. Association for Computational Linguistics.
* Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1119-1130, Austin, Texas. Association for Computational Linguistics.
* Bowman and Dahl (2021) Samuel R Bowman and George E Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In _2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021_, pages 4843-4855. Association for Computational Linguistics (ACL).
* Brown and Levinson (1987) Penelope Brown and Stephen C. Levinson. 1987. _Politeness: some universals in language usage_. Number 4 in Studies in interactional sociolinguistics. Cambridge University Press, Cambridge [Cambridgeshire] ; New York 

* Brown et al. (1987) Penelope Brown, Stephen C Levinson, and Stephen C Levinson. 1987. _Politeness: Some universals in language usage_, volume 4. Cambridge university press.
* Buechel et al. (2018) Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and Joao Sedoc. 2018. Modeling empathy and distress in reaction to news stories. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018)_.
* Bucchel and Hahn (2017) Sven Bucchel and Udo Hahn. 2017. EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 578-585, Valencia, Spain. Association for Computational Linguistics.
* Impoliteness, insults and responses to insults on Twitter. _Topics in Linguistics_, 22(2):62-84.
* Cano and Williams (2010) Annmarie Cano and Amanda C de C Williams. 2010. Social interaction in pain: Reinforcing pain behaviors or building intimacy? PAIN(r), 149(1):9-11.
* Carston (2021) Robyn Carston. 2021. Polysemy: Pragmats and sense conventions. _Mind & Language_, 36(1):108-133.
* Chiruzzo et al. (2020) Luis Chiruzzo, Santiago Castro, and Aiala Rosa. 2020. HAHA 2019 dataset: A corpus for humor analysis in Spanish. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 5106-5112, Marseille, France. European Language Resources Association.
* Choi et al. (2020) Minje Choi, Luca Maria Aiello, Krisztian Zsolt Varga, and Daniele Quercia. 2020. Ten social dimensions of conversations and relationships. In _Proceedings of The Web Conference 2020_, pages 1514-1525.
* Chung et al. (2022a) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022a. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Chung et al. (2022b) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xueziu Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyu Chen, Akakshca Chowdhrey, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Qucc V. Le, and Jason Wei. 2022b. Scaling instruction-finetuned language models.
* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. _CoRR_, abs/1412.3555.
* Computer (2023) Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset.
* Cramer (1999) Harald Cramer. 1999. _Mathematical methods of statistics_, volume 43. Princeton university press.
* CrowdFlower (2016) CrowdFlower. 2016. The emotion in text, published by crowdflower. [https://data.world/crowdflower/](https://data.world/crowdflower/) sentiment-analysis-in-text. Accessed: 2023-01-14.
* Trouth (2016) CrowdTruth. 2016. Short text corpus with focus on humor detection. Original-date: 2016-05-10T12:48:54Z.
* Culpeper (2021) Jonathan Culpeper. 2021. Impoliteness and hate speech: Compare and contrast. _Journal of Pragmatics_, 179:4-11.
* Da San Martino et al. (2020) Giovanni Da San Martino, Alberto Barron-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 1377-1414, Barcelona (online). International Committee for Computational Linguistics.
* Danescu-Niculescu-Mizil et al. (2013) Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Daniel Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. In _Annual Meeting of the Association for Computational Linguistics_.
* Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. _Proceedings of the International AAAI Conference on Web and Social Media_, 11(1):512-515. Number: 1.
* Demus et al. (2022) Christoph Demus, Jonas Pitz, Mina Schutz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. Detox: A comprehensive dataset for German offensive language and conversation analysis. In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_, pages 143-153, Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Eggins (2004) Suzanne Eggins. 2004. _Introduction to systemic functional linguistics_. A&c Black.
* Ekman (1992) Paul Ekman. 1992. An argument for basic emotions. _Cognition and Emotion_, 6(3-4):169-200. Publisher: Routledge _eprint: [https://doi.org/10.1080/02699939208411068](https://doi.org/10.1080/02699939208411068) 

Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munnun De Choudhury, and Diyi Yang. 2021. Latent Harted: A Benchmark for Understanding Implicit Hate Speech. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 345-363, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Farkha et al. (2022) Ibrahim Abu Farha, Silvia Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. Semeval-2022 task 6: isarcasmreal, intended sarcasm detection in english and arabic. In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_, pages 802-814.
* Fischer and Herbert (2021) Brigitte Fischer and Cornelia Herbert. 2021. Emoji as affective symbols: affective judgments of emoji, emoticons, and human faces varying in emotional content. _Frontiers in psychology_, 12:645173.
* Freud (1960) Sigmund Freud. 1960. _Jokes and their relation to the unconscious_. WW Norton & Company.
* Fu et al. (2020) Liye Fu, Susan Fussell, and Cristian Danescu-Niculescu-Mizil. 2020. Facilitating the Communication of Politeness through Fine-Grained Paraphrasing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5127-5140, Online. Association for Computational Linguistics.
* Fukushima and Haugh (2014) Saeko Fukushima and Michael Haugh. 2014. The role of emit understandings in theorizing im/politeness: The metapragmatics of attentiveness, empathy and anticipatory inference in Japanese and Chinese. _Journal of Pragmatics_, 74:165-179.
* Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3816-3830, Online. Association for Computational Linguistics.
* Ghazi et al. (2015) Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting emotion stimuli in emotion-bearing sentences. In _International Conference on Intelligent Text Processing and Computational Linguistics_, pages 152-165. Springer.
* Ghosal et al. (2020) Deepanway Ghosal, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2020. Utterancelevel Dialogue Understanding: An Empirical Study. ArXiv:2009.13902 [cs].
* Ghosh et al. (2021) Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodokumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media. In _Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)_, pages 313-328.
* Price (1975) Herbert Price. 1975. Logic and conversation. In _Speech acts_, pages 41-58. Brill.
* Haiman (1998) John Haiman. 1998. _Talk is cheap: sarcasm, alienation, and the evolution of language_. Oxford University Press, Oxford. OCLC: 252598275.
* Halliday (2004) Michael AK Halliday. 2004. Introduction: How big is a language? On the power of language. _The language of science_, 5:19-32.
* Kirkwood Halliday (1995) Michael Alexander Kirkwood Halliday. 1995. _Discourse in society: Systemic functional perspectives_. 50. Greenwood Publishing Group.
* Halliday et al. (1989) Michael Alexander Kirkwood Halliday, Ruqaiya Hasan, et al. 1989. _Language, context, and text: Aspects of language in a social-semiotic perspective_. Oxford University Press Oxford.
* Harvey (2000) Keith Harvey. 2000. Describing camp talk: language/pragmatics/politics. _Language and Literature: International Journal of Stylistics_, 9(3):240-260.
* Hayati et al. (2021) Shirley Anugrah Hayati, Dongyeop Kang, and Lyle Ungar. 2021. Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6323-6331, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing. _CoRR_, abs/2111.09543.
* Holmes (2006) Janet Holmes. 2006. Sharing a laugh: Pragmatic aspects of humor and gender in the workplace. _Journal of Pragmatics_, 38(1):26-50. Special Issue: Gender and Humor.
* Hossain et al. (2020) Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEval-2020 Task 7: Assessing Humor in Edited News Headlines. In _Proceedings of the Fourteenth Workshop on Semantic Evaluation_, pages 746-758, Barcelona (online). International Committee for Computational Linguistics.
* Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In _The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
* Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
* Hu et al. (2022a) Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022a. A fine-grained comparison of pragmatic language understanding in humans and language models. _arXiv preprint arXiv:2212.06801_.

 

Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022b. A fine-grained comparison of pragmatic language understanding in humans and language models. ArXiv:2212.06801 [cs].
* Huebner (2021) Daniel R Huebner. 2021. Anachronism: The queer pragmatics of understanding the past in the present. _The American Sociologist_, 52(4):740-761.
* Hyter et al. (2015) Yvette D Hyter, Kenyata O Rivers, and Glenda Dejarnette. 2015. Pragmatic language of african american children and adolescents. _Topics in Language Disorders_, 35(1):8-45.
* Jackson et al. (2022) Joshua Conrad Jackson, Joseph Watts, Johann-Mattis List, Curtis Puryear, Ryan Drabble, and Kristen A. Lindquist. 2022. From text to thought: How analyzing language can advance psychological science. _Perspectives on Psychological Science_, 17(3):805-826. PMID: 34606730.
* Jiang et al. (2021) Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Edizioni, Maarten Sap, Regina Rini, and Yejin Choi. 2021. Can Machines Learn Morality? The Delphi Experiment. Publication Title: arXiv e-prints ADS Bibcode: 2021arXiv211007574J.
* Jigsaw (2017) Jigsaw. 2017. Toxic Comment Classification Challenge.
* Jigsaw (2019) Jigsaw. 2019. Unintended Bias in Toxicity Classification.
* Jin et al. (2022) Mali Jin, Daniel Preotiuc-Pietro, A. Seza Dogruoz, and Nikolaos Aletras. 2022. Automatic Identification and Classification of Bragging in Social Media. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3945-3959, Dublin, Ireland. Association for Computational Linguistics.
* Jurgens et al. (2019) David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019. A just and comprehensive strategy for using NLP to address online abuse. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3658-3666, Florence, Italy. Association for Computational Linguistics.
* Jurgens et al. (2023) David Jurgens, Agrima Seth, Jackson Sargent, Athena Aghighi, and Michael Geraci. 2023. Your spouse needs professional help: Determining the contextual appropriateness of messages through modeling social relationships. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10994-11013.
* Kang et al. (2019) Dongyeop Kang, Varun Gangal, and Eduard Hovy. 2019. (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1696-1706, Hong Kong, China. Association for Computational Linguistics.
* Kang and Hovy (2021) Dongyeop Kang and Eduard Hovy. 2021. Style is NOT a single variable: Case studies for cross-stylistic language understanding. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2376-2387, Online. Association for Computational Linguistics.
* Keltner et al. (1993) Dacher Keltner, Phoebe C Ellsworth, and Kari Edwards. 1993. Beyond simple pessimism: effects of sadness and anger on social perception. _Journal of personality and social psychology_, 64(5):740.
* Khodak et al. (2018) Mikhail Khodak, Nikunji Saunshi, and Kiran Vodrahalli. 2018. A Large Self-Annotated Corpus for Saracasm. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Korner et al. (2021) Erik Korner, Gregor Wiedemann, Ahmad Dawar Hakimi, Gerhard Heyer, and Martin Potthast. 2021. On Classifying whether Two Texts are on the Same Side of an Argument. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10130-10138, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lee et al. (2020) Kent M. Lee, Kristen A. Lindquist, Nathan L. Arbuckle, Samantha M. Mowrer, and B. Keith Payne. 2020. An indirect measure of discrete emotions. _Emotion_, 20(4):659-676.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2021) Yan Li, Manoj a Thomas, and Dapeng Liu. 2021. From semantics to pragmatics: where IS can lead in Natural Language Processing (NLP) research. _European Journal of Information Systems_, 30(5):569-590 

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Hai-hui [2019] LI Hai-hui. 2019. Mitigation and Pragmatic Empathy. _Journal of Literature and Art Studies_, 9(2).
* Lindquist and Barrett [2008] Kristen A. Lindquist and Lisa Feldman Barrett. 2008. Constructing Emotion: The Experience of Fear as a Conceptual Act. _Psychological Science_, 19(9):898-903.
* Liu [2012] Bing Liu. 2012. Sentiment analysis and opinion mining. _Synthesis lectures on human language technologies_, 5(1):1-167.
* Liu et al. [2022] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 61-68, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Lorenzo-Dus and Bou-Franch [2003] Nuria Lorenzo-Dus and Patricia Bou-Franch. 2003. Gender and politeness: Spanish and british undergraduates' perceptions of appropriate requests. _Genero, lengauje y traduccion_, pages 187-199.
* Ma et al. [2017] Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 708-717, Vancouver, Canada. Association for Computational Linguistics.
* Macagno et al. [2022] Fabrizio Macagno, Chrysi Rapanta, Elisabeth Mayweg-Paus, and Merce Garcia-Mila. 2022. Coding empathy in dialogue. _Journal of Organatics_, 192:116-132.
* Mahowald et al. [2023] Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective. ArXiv:2301.06627 [cs].
* Majid [2012] Asifa Majid. 2012. Current emotion research in the language sciences. _Emotion Review_, 4(4):432-443.
* Martino et al. [2020] G. Da San Martino, A. Barron-Cedeno, H. Wachsmuth, R. Petrov, and P. Nakov. 2020. SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles. ArXiv:2009.02696 [cs].
* Protein Structure_, 405(2):442-451.
* Meaney et al. [2021] J. A. Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor Offense. In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_, pages 105-119, Online. Association for Computational Linguistics.
* Menini et al. [2021] Stefano Menini, Alessio Palmerco Aprosio, and Sara Tonelli. 2021. Abuse is contextual, what about nlp? the role of context in abusive language annotation and detection. _arXiv preprint arXiv:2103.14916_.
* Miller et al. [2020] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020. The effect of natural distribution shift on question answering models. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 6905-6916. PMLR.
* Mills [2004] Sara Mills. 2004. Class, gender and politeness. _Multilingua_, 23.
* Mittal et al. [2021] Anirudh Mittal, Pranav Jeevan P, Prerak Gandhi, Diptesh Kanoja, and Pushpak Bhattacharyya. 2021. "So You Think You're Funny?": Rating the Humour Quotient in Standup Comedy. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10073-10079, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Mohammad et al. [2018] Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In _Proceedings of the 12th international workshop on semantic evaluation_, pages 1-17.
* Moudgil [2023] Abhinav Moudgil. Short Jokes.
* Muemighoff et al. [2023] Niklas Muemighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Metb: Massive text embedding benchmark. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2006-2029.
* Muennighoff et al. [2022] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Safiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. _arXiv preprint arXiv:2211.01786_.
* Reiter and Frohlich [2020] Rosina Marquez Reiter and David M. Frohlich. 2020. A pragmatics of intimacy. _Internet Pragromatics_, 3(1):1-33.
* Nakov et al. [2018] Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barron-Cedeno, Ruben Miguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram 

Hasanain, Nikolay Babulkov, Alex Nikolov, Gautam Kishore Shahi, Julia Maria Struss, and Thomas Mandl. 2021. The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News. In _Advances in Information Retrieval_, Lecture Notes in Computer Science, pages 639-649, Cham. Springer International Publishing.
* Newman et al. (2003) Matthew L. Newman, James W. Pennebaker, Diane S. Berry, and Jane M. Richards. 2003. Lyning words: Predicting deception from linguistic styles. _Personality and Social Psychology Bulletin_, 29(5):665-675. PMID: 15272998.
* Olsson et al. (1982) Ulf Olsson, Fritz Drasgow, and Neil J Dorans. 1982. The polyserial correlation coefficient. _Psychometrika_, 47(3):337-347.
* Ott et al. (2011) Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. Finding Deceptive Opinion Spam by Any Stretch of the Imagination. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 309-319, Portland, Oregon, USA. Association for Computational Linguistics.
* Padmakumar et al. (2022) Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, and George Karypis. 2022. Exploring the role of task transferability in large-scale multi-task learning. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2542-2550, Seattle, United States. Association for Computational Linguistics.
* Park et al. (2021) Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, and Yulia Tsvetkov. 2021. Detecting Community Sensitive Norm Violations in Online Conversations. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3386-3397, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Parks (1981) Malcolm R. Parks. 1981. Ideology in interpersonal communication: Off the couch and into the world. _Annals of the International Communication Association_, 5(1):79-107.
* Parvaresh (2023) Vahid Parvaresh. 2023. Covertly communicated hate speech: A corpus-assisted pragmatic study. _Journal of Pragmatics_, 205:63-77.
* Pavloopoulos et al. (2020) John Pavloopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4296-4305.
* Pavloopoulos et al. (2021) John Pavloopoulos, Jeffrey Sorensen, Leo Laugier, and Ion Androutsopoulos. 2021. SemEval-2021 Task 5: Toxic Spans Detection. In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_, pages 59-69, Online. Association for Computational Linguistics.
* Pei and Jurgens (2020) Jiaxin Pei and David Jurgens. 2020. Quantifying intimacy in language. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5307-5326, Online. Association for Computational Linguistics.
* Pei and Jurgens (2021) Jiaxin Pei and David Jurgens. 2021. Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9959-10011, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Perez-Almedros et al. (2022) Carla Perez-Almedros, Luis Espinosa-Anke, and Steven Schockaert. 2022. SemEval-2022 task 4: Patronizing and condescending language detection. In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_, pages 298-307, Seattle, United States. Association for Computational Linguistics.
* Peskov et al. (2020) Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Barrow, Cristian Danescu-Niculescu-Mizil, and Jordan Boyd-Graber. 2020. It Takes Two to Lie: One to Lie, and One to Listen. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3811-3854, Online. Association for Computational Linguistics.
* Potthast et al. (2017) Martin Potthast, Tim Gollub, Matti Wiegmann, Benno Stein, Matthias Hagen, Kristof Komlossy, Sebastian Schuster, and Erika P. Garces Fernandez. 2018. Webbis Clickbait Corpus 2017 (Webis-Clickbuit-17).
* Pougue-Biyong et al. (2021) John Pougue-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, and Doyne Farmer. 2021. DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates.
* Preotiuc-Pietro et al. (2019) Daniel Preotiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically Identifying Complaints in Social Media. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, ACL.
* Pryzant et al. (2020) Reid Przyant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically Neutralizing Subjective Bias in Text. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(01):480-489. Number: 01.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67 

Sudha Rao and Joel Tetreault. 2018. Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana. Association for Computational Linguistics.
* Reuel et al. (2022) Ann-Katrin Reuel, Sebastian Peralta, Joao Sedoc, Garrick Sherman, and Lyle Ungar. 2022. Measuring the Language of Self-Disclosure across Corpora. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland. Association for Computational Linguistics.
* Rivers et al. (2012) Kenyatta O Rivers, Yvette D Hyter, and Glenda DeJarnette. 2012. Parsing pragmatics. _The ASHA Leader_, 17(13):14-17.
* Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 502-518, Vancouver, Canada. Association for Computational Linguistics.
* Ruch (2010) Willibald Ruch. 2010. _The sense of humor: Explorations of a personality characteristic_, volume 3. Walter de Gruyter.
* Ruis et al. (2022) Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktaschel, and Edward Grefenstette. 2022. Large language models are not zero-shot commutators.
* Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Linntang Sutawika, Zaidi Alyafaei, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Naguk, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trisahala Neerai, Jos Rozen, Abheesh Sharma, Andrea Santilli, Thihulet Feury, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multitask prompted training enables zero-shot task generalization. In _International Conference on Learning Representations_.
* Sap et al. (2019a) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019a. The risk of racial bias in hate speech detection. In _Proceedings of the 57th annual meeting of the association for computational linguistics_, pages 1668-1678.
* Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5477-5490, Online. Association for Computational Linguistics.
* Sap et al. (2019b) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Commonsense reasoning about social interactions. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4463-4473, Hong Kong, China. Association for Computational Linguistics.
* Scherer and Wallbot (1994) Klaus R. Scherer and Harald G. Wallbot. 1994. "Evidence for universality and cultural variation of differential emotion response patterning": Correction. _Journal of Personality and Social Psychology_, 67(1):55-55. Place: US Publisher: American Psychological Association.
* Schlangen (2021) David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 670-674.
* Schneider (2010) Stefan Schneider. 2010. Mitigation. _Handbooks of pragmatics_, pages 253-269.
* Schnurr (2010) Stephanie Schnurr. 2010. 13. humour. _Interpersonal pragmatics_, 6:307.
* Sharma et al. (2020) Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020. A computational approach to understanding empathy expressed in text-based mental health support. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5263-5276, Online. Association for Computational Linguistics.
* Shi et al. (2022) Yiwen Shi, Taha ValizadehAslani, Jing Wang, Ping Ren, Yi Zhang, Meng Hu, Liang Zhao, and Hualou Liang. 2022. Improving imbalanced learning by pre-finetuning with data augmentation. In _Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications_, pages 68-82. PMLR.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Spertus (1997) Ellen Spertus. 1997. Smokey: Automatic recognition of hostile messages. In _AAai/iaai_, pages 1058-1065.
* Stapleton (2003) Karyn Stapleton. 2003. Gender and swearing: A community practice. _Women and Language_, 26(2):22 

Gerard Steen, Aletta G. Dorst, and J. Berenke Herrmann, editors. 2011. _A method for linguistic metaphor identification: from MIP to MIPVU_. Number 14 in Converging evidence in language and communication research. Benjamins, Amsterdam.
* Strassler (1982) Jurg Strassler. 1982. _Idioms in English: A pragmatic analysis_, volume 183. Gunter Narr Verlag.
* Subramonian et al. (2023) Arjun Subramonian, Xingdi Yuan, Hal Daume III, and Su Lin Blodgett. 2023. It takes two to tango: Navigating conceptualizations of NLP tasks and measurements of performance. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3234-3279, Toronto, Canada. Association for Computational Linguistics.
* Suman and Jain (2021) Thakur Ashutosh Suman and Abhinav Jain. 2021. ASTarTwice at SemEval-2021 task 5: Toxic span detection using RoBERTa-CRF, domain specific pretraining and self-training. In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_, pages 875-880, Online. Association for Computational Linguistics.
* Talat et al. (2022) Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lacchaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas BECher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiboglu, Jude Fernandes, Jeremy Fu, Wenju Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkee, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Likovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihalyo, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jereyn Reicenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciallom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
* Turiel (1983) Elliot Turiel. 1983. _The development of social knowledge: Morality and convention_. Cambridge University Press.
* Van Hee et al. (2018) Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In _Proceedings of the 12th International Workshop on Semantic Evaluation_, pages 39-50, New Orleans, Louisiana. Association for Computational Linguistics.
* Vidgen et al. (2021) Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tremble. 2021. Introducing CAD: the Contextual Abuse Dataset. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2289-2303, Online. Association for Computational Linguistics.
* Wang et al. (2019) Alex Wang, Yada Prukschatchu, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. _Advances in neural information processing systems_, 32.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics.
* Wang and Komatasuzaki (2021) Ben Wang and Aran Komatasuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingofolz/mesh-transformer-jax](https://github.com/kingofolz/mesh-transformer-jax).
* Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_, NIPS'20, Red Hook, NY, USA. Curran Associates Inc.
* Wang et al. (2022a)Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022a. Measure and improve robustness in NLP models: A survey. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4569-4586, Seattle, United States. Association for Computational Linguistics.
* Wang et al. (2022b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions 

Zijian Wang and Christopher Potts. 2019. TalkDown: A Corpus for Condescension Detection in Context. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3711-3719, Hong Kong, China. Association for Computational Linguistics.
* Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In _Proceedings of the First Workshop on Abusive Language Online_, pages 78-84, Vancouver, BC, Canada. Association for Computational Linguistics.
* Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In _Proceedings of the NAACL Student Research Workshop_, pages 88-93, San Diego, California. Association for Computational Linguistics.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _arXiv preprint arXiv:2109.01652_.
* Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. _arXiv preprint arXiv:2206.07682_.
* Wittgenstein (1953) Ludwig Wittgenstein. 1953. _Philosophical Investigations_. Basil Blackwell, Oxford.
* Workshop et al. (2021) BigScience Workshop, ; Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasaanka Ammannachi, Thomas Wang, Benoit Sagot, Niklas Muenninghoff, Albert Villanova del Moral, Olatuji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurencon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokalsan, Adi Simhi, Atitor Soroa, Alham Fikri Aiji, Amit Affassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo Gonzalez Ponferrada, Efrat Levkovizh, Ethhan Kim, Eyal Bar Natan, Francesco De Toni, Gerard Dupont, German Kruszewski, Giada Pistilli, Hardy Elashhar, Hamza Besymann, Hieu Tran, Ian Yu, Idris Abdulumin, Isaac Johnson, Itziar Gonzalez-Dios, Juxier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Munoz, Mariam Masoud, Maria Grandury, Mario Sasko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lobest, Rheza Harliman, Rishi Bomnasani, Roberto Luis Lopez, Ruiz Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastiast Nagel, Shamskie Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilinia Nikolina, Veronika Liappala, Violette Lepercq, Virinda Prabhu, Zaad Alvaraei, Zererak Talat, Arrun Raja, Benjamin Heinzling, Chenglei Si, Davut Emre Tasar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajvoiti Datta, Elizaz Szczechla, Gunjan Chhablami, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Friess, Jose Rozen, Leo Gao, Latnang Stauwika, M Saidi Buri, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teahan, Samuel Albaine, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofr Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjja Zhang, Mohammad Shoeybi, Myriam Peyronente, Nicolas Patty, Nuouamate Nazi, Omar Sanseverio, Patrick von Platen, Pierre Cornette, Pierre Francois Lavallee, Remi Lacroix, Samyam Rajbhandari, Sanchit Gandh, Shaden Smith, Stephane Requena, Suraj PatiI, Tim Detttmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurelie Neveol, Charles Levering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshima, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, JanChristoph Kalo, Jeketarina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Rouchen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatain Shavirina, Thomas Scianion, Tian Yun, Tomasz Limisiewicz, Perena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Proksachatchukun, Yonatan Belinkov, Zachary Bamberger, Zdenek Kasner, Alice Rueda, Amanda Pastana, Amir Feizpour, Ammar Khan, Amy Faranaka, Ana Santos, Anthony Heyvia, Antigona Unldrezj, Arash Aghgool, Arezo Abdollahi, Aycha Tammuur, Azadeh Haji Hosseini, Baharen Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Munoz Ferrandis, Danish Contractor, David Lansky, Davis David, Douwe Kiela, 

Duong A. Nguyen, Edward Tan, Emi Baylor, Ezwinwanne Ozonati, Fatima Mirza, Frankline Ononniwu, Habib Rezanejadh, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholisi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Mariam Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazreen Rajani, Nour Eklott, Nour Fahmy, Olmarewaja Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmard Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieiu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Animna Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Cenari Zhou, Chitrag Jain, Chuxin Xu, Clementine Fourrier, Daniel Leon Pennan, Daniel Molano, Dian Vu, Enrique Manjavacas, Fabio Barth, Florian Fuhrmann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyu Kang, John Giorgi, Jonas Golde, Jose David Posado, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pamies, Maria A Castillo, Marianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Ullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Micho Broad, Nikolaus Mueller, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Calyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Riblawi, Simon Ott, Since Sang-aroonsiri, Frishti Kumar, Stefan Schweter, Susuli Bharati, Tannay Laud, Theo Gigant, Tomoya Kaniuma, Wojciech Kusa, Yahsin Labrak, Yash Shaileks Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023. Bloom: A 176b-parameter open-access multilingual language model.
* Wu et al. (2019) Liang Wu, Fred Morstatter, Kathleen M Carley, and Huan Liu. 2019. Misinformation in social media: definition, manipulation, and detection. _ACM SIGKDD Explorations Newsletter_, 21(2):80-90.
* Zampieri et al. (2019a) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthalal, Norura Farra, and Ritesh Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1415-1420, Minneapolis, Minnesota. Association for Computational Linguistics.
* Zampieri et al. (2019b) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthalal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 75-86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.
* Zhang and Wan (2022) Yunxiang Zhang and Xiaojun Wan. 2022. MOVER: Mask, Over-generate and Rank for Hyperbolic Generation. ArXiv:2109.07726 [cs].

 

Details on dataset processing

### Benchmark construction (SS3)

The SocKET dataset consists of 58 tasks from 35 unique, public datasets. The datasets that make up the benchmark dataset are processed in a way that is meant to balance uniformity across datasets and tasks while minimizing deviations from the original dataset.

For all datasets, key changes from the original dataset are twofold:

* Duplicates and unlabeled items are removed from all datasets. If duplicates occur across data splits, the splits are recombined, reshuffled, and split.
* All datasets are split 80%/10%/10% between train/test/dev splits, respectively. Any datasets not split 80%/10%/10% are recombined, reshuffled, and split 80%/10%/10%.

All datasets were made compatible with the Hugging Face Datasets package.

## Appendix B Experimental Details

### Computational resources (SS4, SS5, SS6)

All of our experiments were conducted on an Ubuntu 22.04.1 machine installed with NVIDIA RTX A5000 and A6000 GPUs. The Python packages used in our experiments include Pytorch 1.13, Transformers 4.21.3, and Pytorch Lightning 1.6.4.

### Comparison of all models

Table 4 contains a detailed version of Table 2, where the scores of every single task are presented.

### Details on the comparison between SocKET and SocKETTe

32 out of 58 tasks contained more than 1,000 test samples, resulting in a disparity between the sizes of the original SocKET and SocKETTe variants. To test that both datasets still offer comparable evaluations for testing models, we compare their scores for a supervised model and compare test set performances. For each task, we train a deberta-v3-base model, evaluate using the test sets of both versions, and compute the correlation between each setting using Pearson's r score. We provide evaluation results of SocKETTe for our models in Table 5. Also, we show through Table 7 and Figure 5 that there exists a strong correlation between the evaluations of both versions, demonstrating that SocKETTe is indeed a representative sample of SocKET.

### Details on language model finetuning (SS4, SS5, SS6)

#### b.4.1 Task-specific heads (SS4, SS5, SS6)

As our benchmark consists of four different task types: classification, regression, sentence pair detection, and span identification - we maintain a unified structure for each task where each sample is fed into the encoder of an LLM, and the output states are then fed into a task-specific head layer. For span detection tasks, we feed the last hidden layer into a bidirectional GRU Chung et al. (2014), and then the output vectors of the GRU into a linear layer that transforms each vector into a dimension of 3, corresponding to the [B,I,O] labels for each token, following earlier work in span identification Suman and Jain (2021). For all other tasks, we feed the last hidden state of the encoder corresponding to the [CLS] token into a separate classifier/regression head consisting of two linear layers of hidden size 768 and a dropout probability of 0.1. We use the mean squared error loss for regression tasks and the cross-entropy loss for all other tasks.

#### b.4.2 Training strategies for language model finetuning (SS4, SS6)

When training models for the benchmark (SS4) and the multi-task (SS6) experiments, the learning rate was linearly increased for 6% of the training steps up to 1e-5 and linearly decreased afterward. All models were trained for a maximum of 10 epochs using three different seeds, with early stopping after validation performance did not increase for three consecutive epochs.

Our multi-task training in SS6 requires two stages of training: (1) a pre-finetuning stage that simultaneously trains a model on multiple different tasks, and (2) a finetuning stage that loads the model trained from (1) and finetunes it to a single task. In the first stage, a single batch can include several different tasks and produce different types of losses. To obtain a unified loss that is differentiable, we aggregated the loss for each sample and sum them up, which we use for backpropagation. For both stages, we use the same aforementioned training steps and learning rate strategy.

For all settings, the training batch size was set to 32 with 16-bit precision enabled. Validation was 





made after each training epoch on the validation set using Pearson's r correlation added by 1 and divided by 2 for regression tasks and macro F1 score for all other tasks. If there were multiple tasks considered due to multi-task training, the average of all task performances was used as the final validation score.

### Details on prompt-based finetuning (SS4, SS5)

We use fix prompts fine-tuning for all the prompt-based models. The batch size was set as 32 for training. For every single task, we set 10 as the max epoch and do early stopping based on the validation loss. The learning rate is set as 5e-5.

For classification tasks, the model is fine-tuned to generate the target label. For regression tasks, we first normalized the scores into (0,1) and then split the labels into two groups. The model is fine-tuned to generate the target label. We also include a few examples from the test dataset.

\begin{table}
\begin{tabular}{c c|c c c c c c c c c c} Model & Accounts & No. perkins [b]: Number & Average & Minimum & Maximum & Maximum \\ \hline  & Accuracy-driven-matrix & 10.69 & 10.68 & 10.78 & 10.99 & 10.89 & 10.89 & 10.89 & 10.99 & 10.99 \\  & bromoans-flame & 10.69 & 10.69 & 10.42 & 6.48 & 9.78 & 10.71 & 0.72 & 0.10 & 10.69 \\  & bromoans-flame & 1.94 & 0.83 & 0.53 & 0.64 & 0.94 & 0.84 & 0.85 & 0.02 & 0.16 & 0.16 \\  & Negative-stream-flow & 1.95 & 0.89 & 0.75 & 0.89 & 0.97 & 0.97 & 0.92 & 0.95 & 0.99 \\  & Negative-stream-flow & 0.74 & 0.87 & 0.75 & 0.78 & 0.79 & 0.99 & 0.99 & 0.93 & 0.95 & 0.99 \\  & Negative-flow-flow & 0.71 & 1.69 & 0.99 & 0.96 & 1.09 & 0.99 & 0.99 & 10.99 & 0.99 & 0.82 \\  & Negative-flow-flow & 1.95 & 1.06 & 1.09 & 0.99 & 1.09 & 0.99 & 1.09 & 1.09 & 0.97 & 0.99 \\  & negative-flow-flow-out & 0.88 & 0.84 & 0.96 & 0.99 & 0.94 & 0.97 & 0.95 & 0.95 & 10.19 & 0.97 & 0.99 \\  & negative-flow-flow-flow & 0.25 & 1.69 & 0.88 & 0.96 & 1.09 & 0.94 & 0.95 & 0.95 & 0.97 & 0.99 \\  & positive-flow-flow-out & 0.78 & 1.69 & 1.09 & 0.97 & 1.09 & 0.97 & 0.98 & 1.09 & 0.98 & 0.99 \\  & negative-flow-out & 1.11 & 1.06 & 1.09 & 0.92 & 1.09 & 0.99 & 0.99 & 1.09 & 0.99 & 0.99 \\  & random-diffractive-off-1st & 0.11 & 0.80 & 0.22 & 0.18 & 0.98 & 0.98 & 0.99 & 0.99 & 0.99 & 0.99 \\  & constant-diffusion-off-1st & 0.11 & 0.80 & 0.22 & 0.18 & 0.98 & 0.99 & 0.99 & 0.99 & 0.99 & 0.99 \\  & constant-diffusion-off-1st & 0.89 & 0.66 & 0.13 & 0.51 & 0.98 & 0.99 & 0.99 & 0.99 & 0.99 & 0.99 \\  & constant-diffusion-off-2nd & 2.70 & 0.73 & 0.88 & 0.86 & 0.92 & 0.98 & 0.97 & 0.99 & 0.99 & 0.99 \\  & constant-diffusion-off-2nd & 0.76 & 0.70 & 0.77 & 0.86 & 0.74 & 0.75 & 0.51 & 0.88 & 0.99 \\  & constant-diffusion-off-2nd & 0.77 & 0.73 & 0.51 & 0.78 & 0.77 & 0.82 & 0.99 & 0.99 & 0.97 & 0.89 \\  & dependent-normal flow-1st + & 1.04 & 0.88 & 0.62 & 0.50 & 0.57 & 0.64 & 0.57 & 0.55 & 0.55 & 0.55 \\  & non-linear-flow-1st & 0.71 & 0.87 & 0.99 & 0.67 & 0.84 & 0.70 & 0.88 & 0.99 & 0.99 & 0.99 \\  & non-linear-flow-1st & 0.71 & 0.87 & 0.99 & 0.67 & 0.84 & 0.70 & 0.88 & 0.99 & 0.99 & 0.99 \\  & non-linear-flow-1st & 0.71 & 0.89 & 0.85 & 0.84 & 0.72 & 0.97 & 0.99 & 0.88 & 0.99 & 0.99 \\  & variable-net-to-1st & 1.30 & 0.80 & 0.85 & 0.84 & 0.82 & 0.97 & 0.99 & 0.99 & 0.99 & 0.99 \\  & feedback-set-to-2nd & 2.70 & 0.82 & 0.84 & 0.85 & 0.83 & 1.09 & 0.88 & 0.95 & 0.87 & 0.87 \\  & feedback-set-to-2nd & 0.80 & 0.82 & 0.84 & 0.84 & 0.84 & 0.84 & 0.84 & 0.84 & 0.84 & 0.99 & 0.99 \\  & feedback-set-to-2nd & 1.35 & 0.88 & 0.82 & 0.80 & 0.80 & 0.73 & 0.80 & 0.82 & 0.94 & 0.99 \\  & feedback-set-to-2nd & 0.84 & 0.86 & 0.77 & 0.86 & 0.86 & 0.86 & 0.86 & 0.86 & 0.86 & 0.86 \\  & feedback-set-to-2nd & 0.86 & 0.88 & 0.88 & 0.84 & 0.84 & 0.84 & 0.84 & 0.84 & 0.80 & 0.99 \\  & feedback-set-to-2nd & 0.88 & 0.88 & 0.84 & 0.84 & 0.84 & 0.

 

tuned to predict "yes" or "no" regarding the prompt question. During inference, the probability of the "yes" token is used as the prediction score. For span tasks, we directly train the model to generate the full answer.

\begin{table}
\begin{tabular}{l r r} \hline task & SocKET & SocKETTE \\ \hline contextual-abuse/fidelityDirectedAbuse & 0.62 & 0.58 \\ contextual-abuse/PersonDirectedAbuse & 0.53 & 0.49 \\ crowdflower & 0.24 & 0.22 \\ dailydiado & 0.38 & 0.38 \\ hashisedimplication & 0.86 & 0.87 \\ hatedfensive & 0.93 & 0.93 \\ humor-pairs & 0.98 & 0.97 \\ implicit-hate/explicit_hae & 0.72 & 0.68 \\ implicit-hate/implicit_hate & 0.71 & 0.72 \\ implicit-hate/intenirectent_hate & 0.68 & 0.70 \\ implicit-hate/inferiority_hate & 0.60 & 0.69 \\ implicit-hate/steorotypical_halte & 0.68 & 0.68 \\ implicit-hate/threatening_hae & 0.63 & 0.67 \\ implicit-hate/white_giveaucance_hate & 0.70 & 0.68 \\ internyn & 0.75 & 0.73 \\ jigsaw@identity_hate & 0.80 & 0.83 \\ jigsaws/transuit & 0.88 & 0.87 \\ jigsaw@obsscene & 0.91 & 0.91 \\ jigsaw/severe\_toxic & 0.73 & 0.74 \\ jigsawa/threat & 0.85 & 1.00 \\ jigsaw/toxic & 0.90 & 0.90 \\ neutralizing-bias-pairs & 0.98 & 0.98 \\ offensiveu & 0.82 & 0.82 \\ sarc & 0.74 & 0.77 \\ sentitebank & 0.97 & 0.97 \\ sexyn & 0.79 & 0.77 \\ toxic-span & 0.68 & 0.69 \\ tweet\_emoji & 0.34 & 0.32 \\ tweet\_emotion & 0.81 & 0.81 \\ tweet\_sentiment & 0.71 & 0.69 \\ two-to-lieft/receiver\_truth & 0.59 & 0.60 \\ two-to-tie/sender\_truth & 0.58 & 0.58 \\ \end{tabular}
\end{table}
Table 7: A comparison of the evaluation scores between the test sets for SocKET versus SocKETTE when evaluated on a DeBERTa-v3 model trained on a single-task setting.

Figure 4: A comparison of the ratio of valid samples which the LLM was able to make an inference given the correct instruction prompt (x-axis) versus the overall scores across every sample in the test dataset where failed predictions are considered incorrect (y-axis).

Figure 5: For each of the 32 tasks in SocKET containing more than 1,000 test samples, we evaluate the performance of a deberta-v3 model trained on a single SocKET task on both the original test set as well as the smaller SocKETE variant. The correlation between the two scores results in a high Pearson’s r score of 0.997, indicating SocKETTe can be reliably deployed for more rapid model testing.

 



### Details on zero-shot predictions (SS4, SS5)

We use manually designed prompts for all the zero-shot prediction tasks and the prompts are shown in Table 8.

#### Computing correlation scores of task dependencies (SS5)

Because our framework consists of several task types, it is challenging to obtain a unified metric of correlation across different task comparisons. We use the following rules to obtain correlation values:

* Regression task & regression task: We compute the Pearson's correlation coefficient of the two arrays.
* Regression task & binary classification task: We compute the point biserial correlation coefficient of a continuous array and a binary array.
* Regression task & multi-class classification task: We set up a linear regression task using the one-hot coded values of the multi-class array as independent variables and the continuous array as the dependent variable. We report the root of the R-squared value of the regression as correlation Olsson et al. (1982).
* Binary classification task & binary classification task: We compute the Matthews' correlation coefficient Matthews (1975) from the two binary arrays.
* Binary or multi-task classification task & multi-class classification task: We compute the Cramer's V score Cramer (1999) from the two arrays of categorical variables.

#### Computing pairwise model similarities (SS5)

We quantify the model similarity between two tasks as follows. We finetune a pretrained LLM on task \(t_{A}\) to obtain a model \(m_{A}\), and another LLM on task \(t_{B}\) to obtain \(m_{B}\). We obtain pairwise model similarities by inferring both models on a sufficiently large dataset--in this case the entire test set of all tasks--and computing the correlation of the two inferred arrays. We construct an undirected graph (Figure 7) where the thickness and color represent absolute correlation strength and polarity between the two models. The addition of polarity enables us to further discover strong negative correlations with task pairs such as politeness and offensiveness.




\begin{table}
\begin{tabular}{l l|p{142.3pt}|p{190.5pt}|c} \hline
**Type** & **Task** & **Question/Options** \\ \cline{2-6}  & AIR talkdown-pairs & For the quote: text, x\({}^{*}\) and its context ‘text’, b\({}^{*}\) is the quote condescending? & No; Yes\({}^{*}\) \\ \cline{2-6} RIG & hhback:attachment/buncr_rating & Determine the degree of humor of the given sentence: ’text’. The score should be ranging from 0.0 to 5.0, and can be a decimal. O\({}^{1}\) are best humours at all, and S\({}^{1}\) is very humorous. \\ \cline{2-6} CLS & hhbase:tabl:achonfines_burner & For the sentence: ’text’, is it hangues? & 



## Appendix A

Figure 6: A detailed heatmap of Figure 2 showing task dependency among all task pairs as well as task labels. Each value represents the absolute strength of correlation between the true labels of the test set of a specific task (columns) and the predictions made on that task using a model trained on a different task (rows).

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Type** & **Task** & **Question** \\ \hline REG & hahackathon/humor_rating & For the sentence:“text”, is it humorous? \\ \hline REG & hahackathon@offense\_rating & For the sentence:“text", is it offensive? \\ \hline REG & embobank@arousal & For the sentence:“text”, is the presented emotion highly arousal? \\ \hline REG & embbank@dominance & For the sentence:“text”, is the presented emotion highly dominant? \\ \hline REG & embbank@valence & For the sentence: “text”, is the presented emotion positive? \\ \hline REG & empathy@distress & For the sentence:“text”, does it show distress? \\ \hline REG & empathy@empathy & For the sentence:“text”, does it show empathy? \\ \hline \hline \end{tabular}
\end{table}
Table 9: The manually designed prompt questions for fine-tuning regression tasks over the t5 model.



Figure 7: Weighted, undirected graph of model correlations. Each edge between nodes \(i\) and \(j\) is weighted by the correlation between predictions from a model fine-tuned on task \(i\) and predictions from a model fine-tuned on task \(j\), evaluated on the entire SocKET dataset. Nodes are sized proportionally to their weighted degree and a Yifan Hu algorithm is applied for layout, with minor adjustments for readability. Refer to §B.8 for details on how the pairwise score for each edge was computed. We observe strong positive correlations similar to Figure 2, especially within the Sentiment & Emotion category and the Offensiveness category. We also see that across-category transfers may happen in a negative direction such as _hayati_policteness_ and several Offensiveness tasks.

 



\begin{table}
\begin{tabular}{l l r|r r r r r r} \hline Dataset & Task name & Task type & Rand. & Maj. & Single task & Categorywise & All tasks \\ \hline _Humor \& Sarcam_ & & & & & & & & \\ hahackathon & humor\_rating & REG & 0.5 & 0.5 & 0.68 0.01 & 0.67 0.01 & 0.66 0.03 \\ hahackathon & is\_humor & CLS & 0.49 & 0.38 & 0.93 0.00 & 0.93 0.00 & 0.91 0.01 \\ humor-pairs & humor-pairs & PAIR & 0.50 & 0.34 & 0.98 0.00 & 0.98 0.00 & 0.97 0.00 \\ tweet\_irony & tweet\_irony & CLS & 0.49 & 0.28 & 0.80 0.01 & 0.80 0.00 & 0.75 0.03 \\ \multicolumn{7}{c}{_Offensiveness_} \\ contextual-abuse & IdentityDirectedAbuse & CLS & 0.35 & 0.50 & 0.61 0.02 & **0.62** 0.00 & **0.63** 0.01 \\ contextual-abuse & PersonDirectedAbuse & CLS & 0.34 & 0.50 & 0.54 0.01 & **0.55** 0.02 & **0.57** 0.02 \\ hathackathon & offense, rating & REG & 0.5 & 0.5 & 0.91 0.01 & 0.91 0.00 & **0.92** 0.01 \\ hashaseismplification & knabisamedmplication & CLS & 0.50 & 0.37 & 0.86 0.00 & 0.86 0.00 & **0.87** 0.00 \\ hateofensive & hatoffensive & CLS & 0.27 & 0.27 & 0.27 & 0.93 0.00 & **0.95** 0.01 & **0.95** 0.01 \\ implicit-hate & explicit\_hae & CLS & 0.37 & 0.49 & 0.72 0.00 & **0.74** 0.01 & **0.73** 0.01 \\ implicit-hate & implicit\_haze & CIDR & 0.47 & 0.43 & 0.71 0.01 & 0.71 0.01 & 0.69 0.01 \\ implicit-hate & incitement\_bate & CIDR & 0.38 & 0.49 & 0.67 0.01 & **0.68** 0.01 & 0.67 0.00 \\ implicit-hate & inferiority\_bate & CIDR & 0.34 & 0.50 & 0.59 0.02 & 0.59 0.01 & 0.58 0.02 \\ implicit-hate & stereotypical\_hate & CIDR & 0.37 & 0.49 & 0.68 0.01 & 0.66 0.01 & 0.68 0.01 \\ implicit-hate & threatening\_bate & CIDR & 0.33 & 0.50 & 0.60 0.02 & **0.64** 0.01 & **0.65** 0.04 \\ implicit-hate & white\_givevance\_hate & CIDR & 0.39 & 0.48 & 0.71 0.01 & 0.71 0.01 & 0.71 0.02 \\ internup & intentyn & TLS & 0.47 & 0.42 & 0.75 0.00 & 0.74 0.01 & 0.74 0.01 \\ jigsaw & identity\_bate & CIDR & 0.34 & 0.50 & 0.79 0.01 & 0.76 0.01 & 0.79 0.01 \\ jigsaw & insult & CDS & 0.38 & 0.49 & 0.88 0.00 & 0.87 0.01 & 0.88 0.01 \\ jigsaw & obscene & CLS & 0.38 & 0.49 & 0.90 0.01 & **0.91** 0.00 & **0.91** 0.01 \\ jigsaw & severe\_toxic & CLS & 0.34 & 0.50 & 0.74 0.01 & 0.73 0.02 & 0.71 0.02 \\ jigsaw & threat & CDS & 0.34 & 0.50 & 0.85 0.02 & 0.81 0.04 & 0.80 0.03 \\ jigsaw & toxic & CLS & 0.41 & 0.47 & 0.91 0.00 & 0.90 0.00 & 0.91 0.01 \\ offensiveyn & offensiveyn & CLS & 0.50 & 0.37 & 0.82 0.00 & **0.83** 0.01 & **0.83** 0.00 \\ sexyn & cksyn & cksyn & 0.37 & 0.49 & 0.80 0.01 & 0.79 0.00 & 0.79 0.00 \\ talkdown-pairs & talkdown-pairs & PAIR & 0.50 & 0.33 & 0.89 0.01 & 0.88 0.01 & 0.88 0.00 \\ toxic-span & toxic-span & SPAN & 0 & 0 & 0.68 0.02 & 0.65 0.05 & 0.67 0.05 \\ tweet\_offensive & tweet\_offensive & CLS & 0.48 & 0.42 & 0.81 0.01 & 0.81 0.01 & 0.80 0.00 \\ \multicolumn{7}{c}{_Sentiment + Emotion_} \\ crowdsflower & crowdflower & CMS & 0.06 & 0.03 & 0.24 0.00 & 0.23 0.01 & 0.22 0.00 \\ dailydialog & dalyldiago & CLS & 0.07 & 0.13 & 0.43 0.04 & **0.47** 0.00 & **0.47** 0.02 \\ embbank & arousal & REG & 0.5 & 0.5 & 0.80 0.02 & **0.81** 0.02 & 0.79 0.02 \\ emobank & dominance & REG & 0.5 & 0.5 & 0.75 0.02 & 0.75 0.01 & 0.73 0.02 \\ emobank & valence & REG & 0.5 & 0.5 & 0.92 0.01 & 0.92 0.00 & 0.90 0.01 \\ emotion-span & emotion-span & SPAN & 0 & 0 & 0.96 0.01 & 0.89 0.00 & 0.86 0.03 \\ empathy & distress & REG & 0.5 & 0.5 & 0.77 0.01 & 0.75 0.03 & 0.72 0.01 \\ empathy & distress\_bin & CLS & 0.50 & 0.31 & 0.68 0.01 & **0.69** 0.03 & 0.65 0.02 \\ same-side-pairs & same-side-pairs & PAIR & 0.48 & 0.35 & 0.66 0.12 & **0.70** 0.09 & **0.76** 0.05 \\ semitebreak & sentitechwalk & CS & 0.50 & 0.31 & 0.97 0.00 & 0.96 0.00 & 0.96 0.01 \\ tweet\_emoij & tweet\_emoij & CLS & 0.04 & 0.02 & 0.34 0.00 & 0.33 0.00 & 0.33 0.00 \\ tweet\_smonotion & tweet\_emoij & CLS & 0.24 & 0.14 & 0.80 0.01 & **0.81** 0.01 & 0.80 0.00 \\ tweet\_sentiment & tweet\_sentiment & CLS & 0.32 & 0.22 & 0.71 0.00 & 0.71 0.01 & 0.69 0.01 \\ \multicolumn{7}{c}{_Social Factors_} \\ complaints & complaints & CDLs & 0.50 & 0.36 & 0.92 0.01 & 0.92 0.00 & 0.91 0.01 \\ empathy & empathy & REG & 0.5 & 0.5 & 0.70 0.04 & **0.71** 0.03 & 0.70 0.01 \\ empathy & empathy\_bin & CLS & 0.50 & 0.33 & 0.63 0.02 & 0.62 0.01 & 0.59 0.03 \\ hayat\_jointness & nayat\_pointness & CLS & 0.47 & 0.41 & 0.87 0.01 & 0.87 0.05 & **0.89** 0.03 \\ questionnaire & questioninitmacry & CLS & 0.16 & 0.06 & 0.49 0.03 & 0.48 0.02 & 0.46 0.02 \\ stantifordpolteness & standfordpolteness & CLS & 0.50 & 0.36 & 0.70 0.02 & **0.71** 0.01 & **0.72** 0.02 \\ \multicolumn{7}{c}{_Trasworthiness_} \\ bragging & brag\_achievement & CLS & 0.36 & 0.49 & 0.74 0.01 & 0.69 0.01 & **0.76** 0.03 \\ bragging & brag\_action & CLS & 0.35 & 0.50 & 0.59 0.02 & 0.57 0.06 & 0.59 0.04 \\ bragging & brag\_possession & CLS & 0.35 & 0.50 & 0.70 0.02 & 0.66 0.03 & 0.52 0.03 \\ bragging & brag\_trait & CLS & 0.34



**Paper/Dataset Title**

\begin{tabular}{p{142.3pt} p{170.7pt} p{85.4pt}} \hline \hline
**Automatic Identification and Classification of Bragging in Social Media Automatic Identification and Classification of Bragging in Social Media Jin et al. (2022) & Bragging (Action) \\ Automatic Identification and Classification of Bragging in Social Media & Bragging (Possession) & Jin et al. (2022) \\ Automatic Identification and Classification of Bragging in Social Media Automatically Identifying Complaints in Social Media Introducing CAD: the Contextual Abuse Dataset Hate & Bridging (Consistent) & Jin et al. (2022) \\ Introducing CAD: the Contextual Abuse Dataset Introducing CAD: the Contextual Abuse Dataset Hate & Complainants Preotiuc-Pietro et al. (2019) & Vidgen et al. (2021) \\  & Identity Based Hate & Vidgen et al. (2021) \\ Introducing CAD: the Contextual Abuse Dataset Hate & Individual Hate Group-Based VideoHate & Vidgen et al. (2021) \\ Introducing CAD: the Contextual Abuse Dataset Sentiment Analysis in Text DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis Detecting Emotion Stimuli in Emotion-Bearing Sentences Measuring the Language of Self-Disclosure across Corpora Measuring the Language of Self-Disclosure across Corpora SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense & Humor Rating Meaney et al. (2021) \\ \end{tabular}



SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense SemEval 2021 Task 7: HaHackathon, Detecting and Rating Humor and Offense Social Bias Frames: Reasoning about Social and Power Implications of Language Social Bias Frames: Reasoning about Social and Power Implications of Language Social Bias Frames: Reasoning about Social and Power Implications of Language Social Bias Frames: Reasoning about Social and Power Implications of Language Social Bias Frames: Reasoning about Social and Power Implications of Language Vower Implications of Language Automated Hate Speech Detection and the Problem of Offensive Language Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica Does BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica SemEval-2020 Task 7: Assessing Humor in Edited News Headlines MOVER: Mask, Over-generate and Rank for Hyperbole Generation Latent Hatred: A Benchmark for Understanding Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Inferiority ElSherief et al. (2021) Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Stereotyping ElSherief et al. (2021) Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Threat ElSherief et al. (2021) Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Offensive ElSherief et al. (2021) Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Irony ElSherief et al. (2021) Implicit Hate Speech Latent Hatred: A Benchmark for Understanding Other Hate ElSherief et al. (2021) Implicit Hate Speech 

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Toxic Comment Classification Challenge

Automatically Neutralizing Subjective Bias in Text SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles

Quantifying Intimacy in Language

Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning

On Classifying whether Two Texts are on the Same Side of an Argument

A Large Self-Annotated Corpus for Sarcasm

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Facilitating the Communication of Politeness

through Fine-Grained Paraphrasing

TalkDown: A Corpus for Condescension Detection in Context

SemEval-2021 Task 5: Toxic Spans Detection

SemEval 2018 Task 2: Multilingual Emoji Prediction

SemEval-2018 Task 1: Affect in Tweets

SemEval-2018 Task 3: Irony Detection in English Tweets

Predicting the Type and Target of Offensive Posts in Social Media

SemEval-2017 Task 4: Sentiment Analysis in Twitter

It Takes Two to Lie: One to Lie, and One to Listen It Takes Two to Lie: One to Lie, and One to Listen "So You Think You're Funny?": Rating the Humour Quotient in Standup Comedy

DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates

The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News

Finding Deceptive Opinion spam by Any Stretch of the Imagination

Finding Deceptive Opinion spam by Any Stretch of the Imagination

A Clustering Approach for Nearly Unsupervised Recognition of Nonliteral Language

Detecting Community Sensitive Norm Violations in Online Conversations

Can Machines Learn Morality? The Delphi Experiment 

SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) CivilComments CivilComments CivilComments (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas Language Dataset with Multiple Personas (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Language Dataset with Multiple Personas Debiased Corpus 2017 VU Amsterdam Metaphor Corpus Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications Dear Sir or Madam, May I Introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer International Survey on Emotion Antecedents and Reactions Short Jokes Short Text Corpus with Focus on Human Detection Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Studying the Dark Triad of Personality through Twitter Behavior Twitter Behavior

[MISSING_PAGE_POST]