Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships
Year: 2023

Abstract:
Understanding interpersonal communication requires, in part, understanding the social context and norms in which a message is said. However, current methods for identifying offensive content in such communication largely operate independent of context, with only a few approaches considering community norms or prior conversation as context. Here, we introduce a new approach to identifying inappropriate communication by explicitly modeling the social relationship between the individuals. We introduce a new dataset of contextually-situated judgments of appropriateness and show that large language models can readily incorporate relationship information to accurately identify appropriateness in a given context. Using data from online conversations and movie dialogues, we provide insight into how the relationships themselves function as implicit norms and quantify the degree to which context-sensitivity is needed in different conversation settings. Further, we also demonstrate that contextualappropriateness judgments are predictive of other social factors expressed in language such as condescension and politeness.

heading: 1 Introduction
text: Interpersonal communication relies on shared expectations of the norms of communication (Hymes et al., 1972). Some of these norms are widely shared across social contexts, e.g., racial epithets are taboo, enabling NLP models to readily identify certain forms of offensive language (Fortuna and Nunes, 2018). Yet, not all norms are widely shared; the same message said in two different social contexts may have different levels of acceptability (Figure 1). While NLP has recognized the role of social context as important (Hovy and Yang, 2021; Sheth et al., 2022), few works have directly
3These authors contributed equally to this work ?These authors contributed equally to this work
incorporated this context into modeling whether messages violate social norms. Here, we explicitly model relationships as the social context in which a message is said in order to assess whether the message is appropriate.
NLP models have grown more sophisticated in modeling the social norms needed to identify offensive content. Prior work has shown the benefits of modeling context (Menini et al., 2021), such as the demographics of annotators and readers (Sap et al., 2019; Akhtar et al., 2021) and the online community in which a message is said (Chandrasekharan et al., 2018; Park et al., 2021). However, these works overlook normative expectations within people’s relationships.
In this paper, we introduce a new dataset of over 12,236 instances labeled for whether the message was appropriate in a given relationship context. Using this data, we show that computation models can accurately identify the contextual appropriateness of a message, with the best-performing model attaining a 0.70 Binary F1. Analyzing the judgments of this classifier reveals the structure of the shared norms between relationships. Through examining a large corpus of relationship-labeled conversations, we find that roughly 19% of appropriate messages could be perceived as inappropriate in another context, highlighting the need for models
10994
that explicitly incorporate relationships. Finally, we show that our model’s relationship-appropriate judgments provide useful features for identifying subtly offensive language, such as condescension.

heading: 2 Social Norms of Appropriateness
text: Relationships are the foundation of society: most human behaviors and interactions happen within the context of interpersonal relationships (Reis et al., 2000). Communication norms vary widely across relationships, based on the speakers’ social distance, status/power, solidarity, and perceived mutual benefit (Argyle et al., 1985; Fiske, 1992). These norms influence communication in content, grammar, framing, and style (Eckert and McConnell-Ginet, 2012) and help reinforce (or subvert) the relationship between speakers (Brown and Levinson, 1987). Prior computational work mostly frames appropriateness as exhibiting positive affect and overlooks the fact that, in some relationships, conversations can be affectively negative but still appropriate (King and Sereno, 1984). For example, swearing is often considered a norm violation (Jay and Janschewitz, 2008), but can also be viewed as a signal of solidarity between close friends (Montagu, 2001) or co-workers (Baruch and Jenkins, 2007). In such cases, the violation of taboo reinforces social ties by forming a sense of in-group membership where norms allow such messages (Coupland and Jaworski, 2003).
In sociolinguistics, appropriateness is a function of both context and speech. Trudgill (1997) argues that “different situations, different topics, different genres require different linguistic styles and registers,” and Hymes (1997) argues that the extent to which “something is suitable, effective or liked in some context” determines its appropriateness. Whether a discourse is appropriate depends strongly on the social context in which it is produced and received (Fetzer, 2015), making the assessment of appropriateness a challenging task due to the need to explicitly model contextual norms. Behavioral choices are subject to the norms of “oughtness” (Harré and Secord, 1972; Shimanoff, 1980), and Floyd and Morman (1997) suggest relationship types as an important factor influencing the normative expectations for relational communication. For example, while it may be considered appropriate for siblings to discuss their past romantic relationships in detail, the topic is likely to be perceived as taboo or inappropriate between
romantic partners (Baxter and Wilmot, 1985).

heading: 3 Building a Dataset of Contextual Appropriateness
text: Prior work has shown that interpersonal relationships are a relevant context for the appropriateness of content (Locher and Graham, 2010). While not all messages differ in this judgment—e.g., “hello” may be appropriate in nearly all settings—building a dataset that embodies this context sensitivity remains a challenge. Here, we describe our effort to build a new, large dataset of messages rated for contextual appropriateness, including how we select relationships and operationalize appropriateness. Due to the challenge of identifying and rating these messages, our dataset is built in two phases.
Selecting Relationships Formally categorizing relationships has long been a challenging task for scholars (Regan, 2011). We initially developed a broad list of relationships, drawing from 1) folk taxonomies (Berscheid et al., 1989), e.g., common relationship types of friends (Adams et al., 2000), family (Gough, 1971), or romantic partners (Miller et al., 2007); and 2) organizational and social roles (Stamper et al., 2009), e.g., those in a workplace, classroom, or functional settings, as these frequently indicate different social status, distance, or solidarity between individuals in the relationship. Using this preliminary list, four annotators performed a pilot assessment of coverage by discussing quotes from movie scripts, social media, or their imagination and identifying cases where an excluded relationship would have a different judgment for appropriateness. Ultimately, 49 types of relationships were included, shown in Table 1.
Defining Appropriateness Appropriateness is a complex construct that loads on many social norms (Fetzer, 2015). For instance, in some relationships, an individual may freely violate topical taboos, while in other relationships, appropriateness depends on factors like deference due to social status. Informed by the theory of appropriateness (March and Olsen, 2004), we operationalize inappropriate communication as follows: Given two people in a specified relationship and a message that is plausibly said under normal circumstances in this relationship, would the listener feel offended or uncomfortable? We use plausibility to avoid judging appropriateness for messages that would likely never be said, e.g., “would you cook me a ham-
burger?” would not be said from a doctor to a patient. We constrain the setting to what an annotator would consider normal circumstances for people in such a relationship when deciding whether the message would be perceived as appropriate; for example, having a teacher ask a student to say something offensive would be an abnormal context in which that message is appropriate. Thus, during annotation, annotators were asked to first judge if the message would be plausibly said and only, if so, rate its appropriateness.
Judging appropriateness necessarily builds on the experiences and backgrounds of annotators. Culture, age, gender, and many other factors likely influence decisions on the situational appropriateness of specific messages. In making judgments, annotators were asked to use their own views and not to ascribe to a judgment of a specific identity.
Raw Data Initial conversational data was selectively sampled from English-language Reddit. Much of Reddit is not conversational in the sense that comments are unlikely to match chit-chat. Further, few comments are likely to be contextsensitive. To address these concerns, we filter Reddit comments in two ways. First, we train a classifier to identify conversational comments, using 70,949 turns from the Empathetic dialogs
data (Rashkin et al., 2019) and 225,907 turns from the Cornell movie dataset (Danescu-NiculescuMizil and Lee, 2011) as positive examples of conversational messages, and 296,854 turns from a random sample of Reddit comments as nonconversational messages. Full details are provided in Appendix B. Second, we apply our conversational classifier to comments marked by Reddit as controversial in the Pushshift data (Baumgartner et al., 2020); while the decision logic for which comments are marked as controversial is proprietary to Reddit, controversial-labeled comments typically receive high numbers of both upvotes and downvotes by the community—but are not necessarily offensive. These two filters were applied to identify 145,210 total comments gathered from an arbitrary month of data (Feb. 2018).

heading: 3.1 Annotation Phase 1
text: In the first phase of annotation, four annotators individually generated English-language messages they found to differ in appropriateness by relationship.1 Annotators were provided with a website interface that would randomly sample conversational, controversial Reddit comments as inspiration. Details of the annotation instructions and interface are provided in Appendix A. The annotation process used a small number of in-person annotators rather than crowdsourcing to allow for task refinement: During the initial period of annotating, annotators met regularly to discuss their appropriateness judgments and disagreements. This discussion process was highly beneficial for refining the process for disentangling implausibility from inappropriateness. Once annotation was completed, annotators discussed and adjudicated their ratings for all messages. Annotators ultimately produced 401 messages and 5,029 total appropriateness ratings for those messages in the context of different relationships.

heading: 3.2 Annotation Phase 2
text: Phase 2 uses an active learning approach to identify potentially relationship-sensitive messages to annotate from a large unlabeled corpus. A T5 prompt-based classifier was trained using OpenPrompt (Ding et al., 2022) to identify whether a given message would be appropriate to say to a per-
1This process was developed after pilot tests showed the random sample approach was unlikely to surface interesting cases, but annotators found it easier to ideate and write their own messages after being exposed to some example communication.
son in a specific relationship. Details of this classifier are provided in Appendix C. This classifier was run on all sampled data to identify instances where at least 30% of relationships were marked as appropriate or inappropriate; this filtering biases the data away from universally-appropriate or inappropriate messages, though annotators may still decide otherwise.
Two annotators, one of which was not present in the previous annotation process, completed two rounds of norm-setting and pilot annotations to discuss judgments. Then, annotators rated 30 messages each, marking each for plausibility and, if plausible, appropriateness; they met to adjudicate and then rated another 41 messages. This produced 2,159 appropriateness ratings across these messages. Annotators had a Krippendorff’s α of 0.56 on plausibility and, for messages where both rated as plausible, 0.46 on appropriateness. While this agreement initially seems moderate, annotators reviewed all disagreements, many of which were due to different interpretations of the same message, which influenced appropriate judgments rather than disagreements in appropriateness itself. Annotators then revised their own annotations in light of consensus in message meaning, bringing the plausibility agreement to 0.72 and appropriateness to 0.92. We view these numbers as more
reliable estimates of the annotation process, recognizing that some messages may have different judgments due to annotators’ values and personal experiences. We mark the 2,159 ratings in this data as Adjudicated data for later evaluation.
Both annotators then independently annotated different samples of the Reddit data in order to maximize diversity in messages. Annotators were instructed to skip annotating messages that they viewed as less context-sensitive (e.g., offensive in all relationship contexts) or where the message did not appear conversational. Annotators provided 5,408 ratings on this second sample. We refer to this non-adjudicated data as Phase 2 data.

heading: 3.3 Dataset Summary and Analysis
text: The two phases produced a total of 12,236 appropriateness judgments across 5299 messages. Of these, 7,589 of the judgments were appropriate, and 4647 were inappropriate. Table 2 shows examples of annotation judgments. In line with prior cultural studies of appropriateness (Floyd and Morman, 1997; Fetzer, 2015), three themes emerged during training. First, annotators noted the perception of the role of teasing in deciding appropriateness. Teasing messages are directed insults (mild or otherwise) aimed at the other party; comments such as “you are so dumb” are likely made in jest
within close relationships such as best friends or siblings but inappropriate in many others. Second, messages’ appropriateness depended in part on whether the relationship was perceived to be supportive; for example, the message “At least you called him by his correct name” could be one of encouragement in the face of a mistake (e.g., if said by a spouse) or a subtle insult that implies the listener should have known more about the third party. Third, differences in the power/status in the relationship influenced appropriateness, where very direct messages, e.g., “you made a mistake there.” were often perceived to be inappropriate when said to a person of higher status, a known violation of politeness strategies (Brown and Levinson, 1987). Ultimately, appropriateness was judged through a combination of these aspects.
As an initial test of regularity in how the relationship influence perceived appropriateness, we measured the probability that a message appropriate for relationship ri is also appropriate for rj using all the annotations, shown in Figure 2 and grouped by thematic categories. Clear block structure exists with some categories, e.g., ORGANIZATION, indicating shared norms of appropriateness for relationships within the same category. In contrast, the FAMILY and SOCIAL categories contain relationships with different power (e.g., parent) and social distance (e.g., friend vs. stranger), leading to varied judgments. Figure 2 also reveals the asymmetry in which message themes are appropriate: While much of what is said for ROLE-BASED relationships is also appropriate in SOCIAL or ROMANCE, the reverse is not true.

heading: 4 Identifying Contextual Inappropriateness
text: Given the high potential agreement of annotators, we test whether models can similarly recognize whether a given message is appropriate if said in the context of a specific relationship.
Experimental Setup Two classes of models were trained. Given the recent successes of promptbased models, we build models using the OpenPrompt library (Ding et al., 2022) and, to support larger models, using the PEFT library (Liu et al., 2022). The OpenPrompt library was used to train t5-base and gpt2-med models using the prompt “Is it appropriate for person1 to say "quote" to person2, "yes" or "no"? [MASK]” using the verbalization “yes’ or ”no” for the masked
Family
Social
Romance
Organizational Peer group Parasocial
Role-Based
Antagonist
Fa mi ly So cia l
Ro ma
nc e
Or ga
niz ati
on al
Pe er
gro up
Pa ras
oc ial
Ro le-
Ba sed
An tag
on ist
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: The probability that, given an appropriate message for the relationships represented by a row, the message will also be appropriate in another relationship listed in the column. Probabilities are calculated across the entire data.
token. Here, we fill in person1 and person2 to refer to the two parties in the relationship. Examples of filled-in templates and other prompts are reported in Appendix D, though performance varied only slightly across different prompts. The PEFT library was used to train the large and xl variants of the flan-t5 model (Chung et al., 2022). This model has been pretrained for instruction following; therefore, based on suggested guidelines from Ziems et al. (2023), we use the following prompt: “Rate whether it is inappropriate for this message to be said in the following social setting?\n setting: relationship description \n message: "quote"\n answer (yes or no):” Due to the resources required for training these larger models, no additional prompts were rigorously evaluated outside of initial pilot testing.
The second class of models uses masked language model (MLM) fine-tuning on the [CLS] token from an MLM to predict appropriateness. Here, we frame the instance using the same language as the OpenPrompt-based models but fill in the MASK with “yes” (i.e., indicating that the message is appropriate to say in the relationship). The classification model is then fine-tuned to classify whether this hard-coded judgment is correct or not. We test two recent MLMs, MiniLM (Wang et al., 2020), a small distilled model, and DeBERTav3 (He et al., 2021), a much larger model. These two models reflect extremes among relatively small MLMs and allow us to assess whether more social
relationship knowledge might be embedded within a larger parameter space.
Annotated data was split at the message level 70:10:20 into train, development, and test sets, resulting in 9,107 train, 1,100 development, and 2,029 test instances. We frame the task similar to offensive language detection and use Binary F1 as our metric where inappropriate is the positive class. Model performance is reported as the average across five random runs. Additional training details and per-seed performance are provided for all systems in Appendix E.
Two baseline systems are included. The first is random labels with respect to the empirical distribution in the training data. The second uses Perspective API (Lees et al., 2022) to rate the toxicity of the message, labeling it as toxic if the rating is above 0.7 on a scale of [0,1]; the same label is used for all relationships. While this baseline is unlikely to perform well, it serves as a reference to how much explicit toxicity is in the dataset, as some (though not all) of these messages are inappropriate to all relationships.
Results Models accurately recognized how relationships influence the acceptability of a message, as seen in Table 3. Prompt-based models were largely equivalent to MLM-based models, though both approaches far exceeded the baselines. The largest model, flan-t5-xl, ultimately performed best, though even the MiniLM offered promising performance, despite having several orders of magnitude fewer parameters. In general, models were more likely to label messages as inappropriate even when appropriate for a particular setting (more false positives). This performance may be more useful in settings where a model flags potentially inappropriate messages which are then reviewed by a human (e.g., content moderation). However, the performance for models as a whole suggests there is substantial room for improvement in how relationships as social context are integrated into the model’s decisions.
Error Analysis Different relationships can have very different norms in terms of what content is acceptable, as highlighted in Figure 2. How did model performance vary by relationship? Figure 3 shows the binary F1 score of the flan-t5-xl model by relationship, relative to the percent of training instances the model saw that were inappropriate; Appendix Table 11 shows full results
per relationship. Model performance was highly correlated with the data bias for inappropriateness (r=0.69; p<0.01). The model had trouble identifying inappropriate comments for relationships where most messages are appropriate (e.g., friend, sibling) in contrast to more content-constrained relationships (boss, student, doctor). These lowperformance relationships frequently come with complex social norms—e.g., the boundary between appropriate teasing and inappropriate hurtful comments for siblings (Keltner et al., 2001)—and although such relationships have among the most training data, we speculate that additional training data is needed to model these norms, especially given the topical diversity in these relationships’ conversations.

heading: 5 Generalizing to Unseen Relationships
text: Through their pretraining, LLMs have learned semantic representations of relationships as tokens. Our classification experiments show that LLMs can interpret these relationship-as-token representations to effectively judge whether a message is appropriate. To what extent do these representations
allow the model to generalize about new relationships not seen in training? In particular, are models able to generalize if a category of relationship, e.g., all family relations, was never seen? Here, we conduct an ablation study where one of our folk categories is held out during training.
Setup The flan-t5-xl model is trained with the same hyperparameters as the best-performing system on the full training data. We use the same data splits, holding out all training examples of relationships in one category during training. We report the Binary F1 from the test set on (1) relationships seen in training and (2) relationships in the held-out category. Note that because training set sizes may change substantially due to an imbalance of which relationships were annotated and because categories have related norms of acceptability, performance on seen-in-training is likely to differ from the full data.
Results Ablated models varied substantially in their abilities to generalize to the unseen relationship types, as well as in their baseline performance (Figure 4). First, when ablating the larger categories of common relationships (e.g., FAMILY, SOCIAL), the model performs well on seenrelationships, dropping performance only slightly, but is unable to accurately generalize to relationships in the unseen category. These unseen categories contain relationships that span a diverse range of norms with respect to power differences, social distance, and solidarity. While other categories contain partially-analogous relationships along these axes, e.g., parent-child and teacherstudent both share a power difference, the drop in performance on held-out categories suggests the model is not representing these social norms in a way that allows easy transfer to predicting appropriateness for unseen relationships with similar norms. Second, relationships in three categories improve in performance when unseen: ORGANIZATIONAL, ROLE-BASED, and PARASOCIAL. All three categories feature relationships that are more topically constrained around particular situations and settings. While the categories do contain nuance, e.g., the appropriateness around the power dynamics of boss-employee, the results suggest that models may do well in zero-shot settings where there is strong topic-relationship affinity— and messages outside of normal topics are inappropriate. Viewing these two trends together, we posit
that the semantic representations of relationships in flan-t5-xl currently capture only minimal kinds of social norms—particularly those relating to topic—and these norms are not represented in a way that lets the model easily generalize to reasoning about relationships not seen in training.

heading: 6 How Much of Conversation is Context Sensitive in Appropriateness?
text: Our annotation and computational models have shown that the relationship context matters in determining appropriateness. However, it is unclear how often conversations are sensitive to this context. For example, the majority of conversation may be appropriate to all relationships. Here, we aim to estimate this context sensitivity by testing the appropriateness of a message in counterfactual settings using an existing dataset labeled with relationship types.
Experimental Setup To estimate context sensitivity, we use our most accurate model to label a large selection of dialog turns from the PRIDE dataset (Tigunova et al., 2021). PRIDE consists of 64,844 dialog turns from movie scripts, each annotated for the relationship between the speaker and receiver, making it ideal as a high-plausibility conversational message said in relationships. However, some turns of the dialog are explicitly grounded in the setting of the movie, e.g., “How’s it going, Pat?” which makes the turn too specific to that particular setting to accurately estimate appropri-
ateness. Therefore, we run SpaCy NER (Honnibal and Montani, 2017) on the dialog and remove all turns containing references to people, companies, countries, and nationalities in order to keep the dialog generic and maximally plausible in many different relationship contexts. Further, we remove turns with only a single token or over 100 tokens. This filtering leaves 47,801 messages for analysis.
PRIDE contains 18 unique relationships, 16 of which were already included in our categories (cf. Table 1); the two previously-unseen relationship types, described as “religious relationships” and “client/seller (commercial),” were also included since our model can accommodate zero-shot prediction.2
To text for context sensitivity, we apply our flan-t5-xl model and measure the appropriateness of the actual relationship context and then the counterfactual cases as if the message had been said in an alternative relationship context seen in their data. This setup allows us to assess whether if a message was appropriate in its intended relationship context, would it still be appropriate in another.
Results Considering only appropriate messages and excluding the unusual enemy relationship from consideration, we find that roughly 19% of the appropriate-as-said messages in the data would be inappropriate if said in the context of a different relationship. Figure 5 shows the probability that a message acceptable in some other relationship context would also be acceptable in the given context; the striking decrease in the likelihood of acceptability follows the increasingly constrained social norms around a relationship. For example, while friends and loved ones have broad latitude to discuss sensitive topics (Hays, 1984), ROLE-BASED relationships and those with larger power differences are more constrained in what is considered acceptable conversation. While the movie dialog in the PRIDE dataset likely differs from a natural dialog, these results point to relationships as important contexts in natural language understanding.
More generally, we suggest a need for sociallyaware models to identify offensive language. While substantial effort has been put into identifying explicit toxic or abusive language (Vidgen et al., 2021), few models, if any, incorporate the context
2These relationships were phrased as “from a person to someone in their church” and “from a person to a commercial associate” in our prompt model testing.
in which the message is said. These models typically rely on previous conversation turns (Zhang et al., 2018) or modeling community-level social norms (Chandrasekharan et al., 2018) to understand how the context may shift whether the message is perceived as appropriate. Our result suggests that the social context—and particularly social relationships—are highly influential in measuring appropriateness. Indeed, together with the result showing the (expected) low performance of the Perspective API toxicity detector, these results suggest NLP models deployed in social settings are likely missing identifying many offensive messages due to their lack of explicitly modeling of social relations. As NLP tools make their way into the workplace setting, which frequently features a mix of ORGANIZATIONAL, SOCIAL, and ROMANCE ties, explicitly modeling context will likely be necessary.

heading: 7 Identifying Subtle Offensiveness using Contextual Appropriateness
text: Prior NLP studies of subtly inappropriate language often omit the social context in which a statement is said (Breitfeller et al., 2019; Pérez-Almendros et al., 2022), yet it is often this context that makes a statement inappropriate. For example, a teacher asking a student “Do you need help writing that?” is appropriate, whereas a student asking a teacher the same question may seem rude. We hypothesize that modeling the relative appropriateness of a message across relationships can help identify types of subtly offensive language. We test this hypothesis using datasets for two phenomena: condescen-
sion (Wang and Potts, 2019) and (im)politeness (Danescu-Niculescu-Mizil et al., 2013).
Experimental Setup The flan-t5-xl model is used to predict the appropriateness of each message in the training data in the TalkDown dataset for condescension (Wang and Potts, 2019), and the Stanford Politeness Corpus (Danescu-NiculescuMizil et al., 2013). Each message is represented as a binary vector of inappropriateness judgments for each relationship. TalkDown is based on Reddit comments, which our model has seen, whereas the politeness data is drawn from Wikipedia and StackExchange conversations. We adopt the same train and test splits as in the respective papers and fit a logistic regression classifier for each dataset to predict whether a message is condescending or impolite, respectively, from the per-relationship appropriateness vector. The logistic regression model uses Scikit-learn (Pedregosa et al., 2011); for each task, we adopt the evaluation metric used in the respective paper. Appendix F has additional details.
Results The relationship appropriateness scores were meaningfully predictive of subtle offensiveness, as seen in Table 4 for condescension and Table 5 for impoliteness. In both settings, the appropriateness features provide a statistically significant improvement over random performance, indicating that adding relationships as context can help identify subtly offensive messages. Further, despite the classifier’s relative simplicity, the appropriateness features alone outperform the bert-large classifier used in Wang and Potts (2019) in the balanced setting, underscoring how explicitly modeling relationships can still be competitive with LLMbased approaches. Performance at recognizing (im)politeness from relationship-appropriateness was lower than the hand-crafted or purely bagof-words approaches. Yet, this gap is expected given that dataset’s design; Danescu-NiculescuMizil et al. (2013) focus on identifying discourse moves, and the politeness classification task comes from messages at the top and bottom quartiles of their politeness rating. Messages in the bottom quartile may be less polite, rather than impolite, and therefore appropriate in more context, thereby making relationship-appropriate judgments less discriminating as features.

heading: 8 Conclusion
text: “Looking beautiful today!”, “You look like you need a hand with that”, and “When can I see you again?”—in the right contexts, such messages can bring a smile, but in other contexts, such messages are likely to be viewed as inappropriate. In this paper, we aim to detect such inappropriate messages by explicitly modeling the relationship between people as a social context. Through a largescale annotation, we introduce a new dataset of over 12,236 ratings of appropriateness for 49 relationships. In experiments, we show that models can accurately identify inappropriateness by making use of pre-trained representations of relationships. Further, through counterfactual analysis, we find a substantial minority of content is contextually-sensitive: roughly 19% of the appropriate messages we analyzed would not be appropriate if said in some other relationship context. Our work points to a growing need to consider meaning within the social context, particularly for identifying subtly offensive messages. All data and code are released at https://github.com/ davidjurgens/contextual-appropriateness.

heading: Acknowledgments
text: The authors thank Aparna Anathasubramaniam, Minje Choi, and Jiaxin Pei for their timely and valuable feedback on the paper. This work was sup-
ported by the National Science Foundation under Grant Nos. IIS-1850221, IIS-2007251 and IIS2143529.

heading: 9 Limitations
text: This paper has three main limitations worth noting. First and foremost, while our paper aims to model the social context in which a message is said, the current context is limited to only the parties’ relationship. In practice, the social context encompasses a wide variety of other factors, such as the sociodemographics of the parties, the culture and setting of the conversation, and the history of the parties. Even relationships themselves are often much more nuanced and the appropriateness may vary widely based on setting, e.g., statements said between spouses may vary in appropriateness when made in public versus private settings. These contextual factors are likely necessary for a full account of the effect of social context on how messages should be perceived. Our work provides an initial step in this direction by making the relationship explicit, but more work remains to be done. Future work may examine how to incorporate these aspects, such as by directly inputting the situation’s social network as context using graph embedding techniques (Kulkarni et al., 2021), where the network is labeled with relationships (Choi et al., 2021), or by modeling relationships particular types of settings such as in-person, phone, texting, or other online communication, which each have different norms.
Second, our data includes annotations on a finite set of relationships, while many more unique relationships are possible in practice, e.g., customer or pastor. Our initial set was developed based on discussions among annotators and aimed at high but not complete coverage due to the increasing complexity of the annotation task as more relationships were added. Our results in Section 5 suggest that our best model could be able to generalize to new types of relationships in some settings and zeroshot results on two new relationship types not seen in training (a fellow church member and a commercial relationship) match expectations of context sensitivity, (cf. Figure 5 . However, performance is likely limited for less-common relationships without additional training data to describe the norms of appropriateness in this context; and, based on the error analysis in Section 4, models are currently unlikely to generalize to unseen relationships that
have complex sensitivity norms. In addition, new settings such as online spaces may require additional definitions of relationships as individuals interact with each other anonymously.
Third, our judgments of appropriateness were drawn from five annotators total, each of whom had different views of appropriateness based on their values and life experience. While our analysis of agreement with the Adjudicated data (Section 3.2) suggests that when annotators can reach a consensus on a message’s meaning, they are highly likely to agree on appropriateness, we nonetheless view that our annotations are likely to primarily reflect the values of the annotators and may not generalize to other social or cultural contexts where the norms of relationships differ. Future work is needed to explore how these norms differ through additional annotation, and we hope that our dataset will provide a reference for comparison to these judgments. For example, future work may make use of annotation schemes that explicitly model disagreements (Fornaciari et al., 2021) or personalized judgments (Plepi et al., 2022); such approaches may be able to better represent common factors influencing appropriateness judgments.

heading: 10 Ethical Considerations
text: We note three points on ethics. First, we recognize that appropriateness is a value judgment, and therefore our data is limited here by the viewpoints of the annotators. Multiple works on offensive language have shown that the values and identities of annotators can bias the judgments and potentially further marginalize communities of practice whose views and norms are not present (Sap et al., 2019; Garg et al., 2022). We have attempted to mitigate this risk by adding diversity to our annotator pool with respect to gender, age, and culture, yet our limited pool size necessitates that not all viewpoints will be present. Given that we show relationships do matter in judging appropriateness, we hope that future work will add diversity through new additions and data to study relationships. We will also release demographic information on annotators as a part of our dataset to help make potential biases more explicit and more easily addressed.
The annotators themselves were authors of the study and were compensated as a part of their normal work with a living wage. Due to the nature of our filtering, the vast majority of our content was not explicitly toxic. Nonetheless, some comments
did contain objectionable messages, and annotators were provided guidance on how to seek self-care if the messages created distress.
With any new tool to identify offensive or abusive language comes a dual use by an adversarial actor to exploit that tool to find new ways to harass or abuse others while still “abiding by the rules.” Our work has shown that relationships are effective context (and features) for identifying previouslyunrecognized inappropriateness. This new capability has the benefit of potentially recognizing more inappropriate messages before they reach their destination. However, some adversaries could still use our data and model to screen their own messages to find those that still are classified as appropriate (while being inappropriate in practice) to evade detection. Nevertheless, given the new ability to identify context-sensitive offensive messages—which we show can represent a substantial percentage of conversation (Section 6)—we view the benefits as outweighing the risk.

heading: A Annotation Details
text: This section describes the details of the annotation process. Annotators were the authors of this paper and were compensated for their work as a part of their normal duties; no additional payments were provided.
The annotation interface was designed using POTATO (Pei et al., 2022), shown in Figure 6, and was accessed through a browser, which allowed annotators to start and stop their labeling at any time. Annotators were allowed to revise their annotations at any time.
During annotation, annotators were presented with the message to be annotated and collapsible instructions for annotation. Figure 7 shows the full written instructions shown to annotators. The instructions were refined through an iterative process throughout the project, and annotators regularly communicated about ambiguity. The instructions were designed to let the annotators know the intent of the study and the downstream tasks that data would be used for.

heading: B Conversation Classifier Details
text: The conversational classifier was used during the initial data sampling phase to identify comments on Reddit that could plausibly have been said in a conversation. This classifier is intended only as a filter to improve data quality by reducing the number of non-conversation comments (e.g., those with Reddit formatting, long monologues, and comments written in a non-conversational register). We have two datasets of known conversations: 70,949 turns from the Empathetic dialogs data (Rashkin et al., 2019) and 225,907 turns from the Cornell movie dataset (Danescu-Niculescu-Mizil and Lee, 2011) as positive examples of conversational messages. We then sample an equivalent number of 296,854 turns from a random sample of Reddit comments as non-conversational messages. While some of these Reddit messages are likely conversational, this classification scheme is only a heuristic aimed at helping filter data. A held-out set of 74,212 instances was used for evaluation, balanced between conversational and not.
A MiniLM classifier (Wang et al., 2020) was trained using Huggingface Transformers (Wolf
et al., 2019) for five epochs, keeping the model with the lowest training loss at any epoch; Epoch 5 was selected. The model attained an F1 of 0.94 for the held-out data indicating it was accurate at distinguishing the conversational turns from the random sample of Reddit comments. We apply this classifier to 1,917,346 comments from Reddit during the month of February 2018 and identify 145,210 whose probability of being a conversation is >0.5. We retain these comments as potential comments to annotate in Phase 2 (Section 3.2).
B.1 Computational resources
All of our experiments were conducted on an Ubuntu 16.04.7 LTS machine installed with NVIDIA RTX A5000 and RTX A6000 GPUs having CUDA 11.3. The Python packages used in our experiments include Pytorch 1.17.0, Transformers 4.25.1, PEFT 0.3.0, OpenPrompt 1.0.1, pandas 1.1.4, spacy 3.3.2, and Sci-kit learn 1.2.0.
B.2 Specification of LLMs
The LLMs used in this paper were downloaded from huggingface.co. The model and their parameter sizes are listed in Table 6.
B.3 Classifiers from Sklearn
For the classification of politeness and condescension tasks, we used logistic regression from sklearn with the solver as ‘lbfgs’ and max_iter set to 400.

heading: C Phase 1 Classifier
text: The phase-1 LLM classifier was trained using the pilot training data and the OpenPrompt framework. In this framework, we use a batch size of 4, the maximum sequence length was set to 256, decoder_max_length=3, truncate_method="head", and teacher_forcing and predict_eos_token were set to default values. The prompt used for the model was framed as a yes/no question - "is it appropriate for PERSON1 to say QUOTE to PERSON2?".

heading: D Additional Prompt-based Model Details
text: We train gpt2-base and t5-base using the OpenPrompt framework. In this framework, we use a batch size of 16, the maximum sequence length was set to 256, decoder_max_length=3, truncate_method="head", and teacher_forcing and predict_eos_token were set to default values. The model was trained using early stopping and the AdamW optimizer with a learning rate set to 1e-4. The different prompts that we used before finaliz-
ing the prompt as "Is it appropriate for PERSON1 to say "QUOTE" to PERSON2?, "yes" or "no"? are reported in table 10.
We train the flan-t5-large and flan-t5-xl models using the PEFT library. Models were trained with a batch size of 96 and 32, respectively. Both models used a maximum sequence length of 192 and learning rate of 1e-2 with AdamW, using all other default library parameters. The model was trained for 20 epochs, keeping the best-performing model by binary F1 on the development dataset for each seed.

heading: E Additional Results
text: E.1 Development Set Performance
The performance of the different models on the development dataset is reported in Table 8 and performance on the test set with standard errors is
reported in Table 9.
E.2 Analysis of Relationship Predictions The data annotation process showed clear associations between pairs of relationships in terms of how often a message would be appropriate (Figure 2). However, the training data for that figure only includes annotations on relationships annotators selected. What structure or regularity might we see from analyzing similarities between all our relationships through model predictions?
As a qualitative experiment, we use the flan-t5-xl model to label the subset of the PRIDE dataset (Section 6) for the appropriateness of all 49 relationships in our training data. This produces a binary matrix of 49 × 47,801. We use PCA to capture regularity and then project relationships onto a 2D visualization using t-SNE (van der Maaten and Hinton, 2008), which is aimed at preserving local similarity in the spatial arrangement. If model predictions are capturing shared norms, we view t-SNE as potentially more useful than a PCA projection, as we want to visualize which re-
lationships with similar judgments as being nearby (what t-SNE does) rather than optimizing the visualization to the global structure of distances (what PCA does). The t-SNE projection was designed using guidance from Wattenberg et al. (2016); a perplexity of 40 was used.
The resulting visualization, shown in Figure 8, captures expected regularity. While the projection is only a visual tool, and aspects such as distance are not meaningful in t-SNE visualizations, the grouping and neighbors suggest the model is sensitive to power/status and social distance in how it decides appropriateness based on the relationship.
E.3 Per Relationship Results
Table 11 shows the peformance of the flan-t5-xl model on the test set, broken down by relationship

heading: F Additional Experimental Setup Details for Identifying Subtly Offensive Messages
text: For experiments with both the TalkDown corpus (Wang and Potts, 2019) and Stanford Politeness Corpus (Danescu-Niculescu-Mizil et al., 2013), the trained flan-t5-xl model was used in a zeroshot setting with no additional training. For the Politeness corpus, politeness ratings are made at the utterance level, outside of any dialog context. As a result, the existing prompt was used (§4, Experimental Setup) to assess relationship-specific appropriateness.
Two modifications were necessary for the TalkDown corpus. First, the TalkDown corpus’s data is rated at the turn level, with condescension judgments based on the interpretation of a reply to a specific piece of quoted text. Wang and Potts (2019) note that incorporating both the quote and reply into the input resulted in better performance. Therefore, we modify our initial prompt slightly as follows: “Rate whether it is inappropriate for message A to be said in response to the message B in the specified social setting: \n A: quoted text \n B: reply text \n setting: relationship description \n answer (yes or no):”. Since the flan-t5-xl model was trained specifically for instruction following (Chung et al., 2022), we expected the model to generate similar outputs as our original prompt. Second, some of the quoted and reply text in TalkDown can be quite long (hundreds of words). Since the adapted prompt contains both quote and reply, we use an flexible truncation process to maximize the content that can still fit within the maximum input token sequence length (196). First, quoted text
over 50 tokens is truncated to the first 50, using the flan-t5-xl tokenizer to segment words. Then, if the full input (with prompt instructions) still exceeds the maximum input length, we truncate both the quoted text and reply evenly, still keeping at least the first then 10 tokens of each.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
9
3 A2. Did you discuss any potential risks of your work? 10
3 A3. Do the abstract and introduction summarize the paper’s main claims? 0,1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3,4,5,6,7
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3,4
C 3 Did you run computational experiments? 4,5,6,7
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? B, C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4, B, C
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? A
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? A
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? The authors annotated the data so demographics were held out during submission to preserve double-blind status.

heading: Future Work
text: Given that we show relationships do matter in judging appropriateness, we hope that future work will add diversity through new additions and data to study relationships. We will also release demographic information on annotators as a part of our dataset to help make potential biases more explicit and more easily addressed. The annotators themselves were authors of the study and were compensated as a part of their normal work with a living wage. Due to the nature of our filtering, the vast majority of our content was not explicitly toxic. Nonetheless, some comments
did contain objectionable messages, and annotators were provided guidance on how to seek self-care if the messages created distress. With any new tool to identify offensive or abusive language comes a dual use by an adversarial actor to exploit that tool to find new ways to harass or abuse others while still “abiding by the rules. ” Our work has shown that relationships are effective context (and features) for identifying previouslyunrecognized inappropriateness. This new capability has the benefit of potentially recognizing more inappropriate messages before they reach their destination. However, some adversaries could still use our data and model to screen their own messages to find those that still are classified as appropriate (while being inappropriate in practice) to evade detection. Nevertheless, given the new ability to identify context-sensitive offensive messages—which we show can represent a substantial percentage of conversation (Section 6)—we view the benefits as outweighing the risk