Venue: ACL 2023

# NormBank: A Knowledge Bank of Situational Social Norms

Caleb Ziems\({}^{\clubsuit}\)  Jane Dwivedi-Yu   Yi-Chia Wang

Alon Y. Halevy    Diyi Yang

Stanford University, Meta AI

{cziems, diyiy}@stanford.edu, {janeyu, yichaw, ayh}@fb.com

###### Abstract

We present NormBank, a knowledge bank of 155k situational norms. This resource is designed to ground flexible normative reasoning for interactive, assistive, and collaborative AI systems. Unlike prior commonsense resources, NormBank grounds each inference within a multivalent sociocultural frame, which includes the setting (e.g., _restaurant_), the agents' contingent roles (_vaiter_, _customer_), their attributes (_age_, _gender_), and other physical, social, and cultural constraints (e.g., the _temperature_ or the _country of operation_). In total, NormBank contains 63k unique constraints from a taxonomy that we introduce and iteratively refine here. Constraints then apply in different combinations to frame social norms. Under these manipulations, norms are _non-monotonic_ -- one can cancel an inference by updating its frame even slightly. Still, we find evidence that neural models can help reliably extend the scope and coverage of NormBank. We further demonstrate the utility of this resource with a series of transfer experiments. For data and code, see [https://github.com/SALT-NLP/normbank](https://github.com/SALT-NLP/normbank)

+
Footnote †: Work done at Meta AI Research

+
Footnote †: Work done at Meta AI Research

## 1 Introduction

As AI systems continually evolve for human assistance and collaboration, they will increasingly operate within cultural and social spaces, and require increasingly robust and flexible knowledge of social norms Carlucci et al. (2015). From dialogue systems Molnar and Szuts (2018); Vaidyam et al. (2019); Bavaresco et al. (2020); Grossman et al. (2019) to socially interactive robots Fong et al. (2003); Deng et al. (2019) and augmented or mixed reality technologies Anderson and Rainie (2022), each could benefit from understanding how humans effectively communicate, make decisions, engage with requests, and broadly interact with others Sunstein (1996); Sherif and Sherif (1953).

Figure 1: **What is special about NormBank?** Norms are grounded by _situational constraints_—environmental and personal attributes, as well as roles and other behaviors. In this example, drinking coffee is an encouraged activity in its prototypical context, for a _customer_ in a _cafe_, but it is counternormative for a working barista to do so in the same cafe, or for a child-age student do so in a classroom. These represent only some of the non-monotonic normative inferences that are represented in NormBank.



Natural language is flexible and highly expressive; thus it is a promising medium for encoding knowledge of social norms (Sap et al., 2019). The goal of this work is to construct NormBank, a natural language bank of social norms that will allow AI systems to reason about social situations under complex constraints. NormBank encodes 155k norms via scalable human annotation, bootstrapped with implicit knowledge from large pre-trained language models (LLMs).

NormBank factors in two important considerations that have been previously overlooked. First: norms are not rigid truths; they are flexibly assumed standards that may be updated with new information from the social context (Blass and Horswill, 2015). Second: this social context is not a flat list of facts but a matrix of hierarchical dependencies (Hovy and Yang, 2021). These two considerations have important design implications for norm representations and reasoning in AI, inspiring two objectives for this work.

**Objective 1** is to support _non-monotonic_ reasoning (Reiter, 1981) over _defeasible_(Pollock, 1987) norms. This means inferences that hold under most cases can be updated or even retracted based on new information. For example, _dancing_ is a positive behavior that is generally permitted in many casual settings and in many cultures. We can still strengthen or cancel this inference. On the one hand, dancing is expected from a professional dancer. But in an Islamic cultural context, individuals are forbidden from publicly dancing with members of the opposite sex. In a hospital setting, a young child is allowed to dance in the waiting room, but this behavior would not be expected from an adult visiting a dying relative For more examples, see Figure 1. This kind of reasoning will not always follow straightforward compositional logic (Klimczyk, 2021), and we expect it to be a challenge for AI systems.1 NormBank is the first data resource to support non-monotonic normative reasoning by encoding _contrasting_ situations under which the _same behavior_ could alternatively be expected or considered taboo (see SS4).

Footnote 1: Typically, it’s okay to drink soda while actively working and it’s okay for a waiter to drink soda; yet the intersection of these conditions is not typical. It is NOT okay for a waiter to drink soda while actively working.

There is a combinatorially explosive space of situational contexts, each with non-compositional and thus unpredictable norms. Enumerating the set of all possible constraints is intractable. To efficiently learn norms in this space, models can rely on the regularizing effects of hierarchical organization and social theory. NormBank introduces hierarchical organization (**Objective 2**) by means of a rich taxonomy over the relevant contextual signals that inform behaviors.

Our new SCENE taxonomy is the first to use Goffman's (1959) dramaturgical theory of social life. We operationalize the theory with _settings_ that have additional _environmental_ constraints. In each setting, there are agents with different _roles_ and _attributes_, who then perform _behaviors_. Norms apply to behaviors in certain situations. For example, in Figure 1, norms around _drinking hot coffee_ differ for agents with different roles (e.g., _barista, customer_) and attributes (e.g., _adult, child_), in different settings (e.g., _cafe, classroom_).

Having addressed the objectives above, we train neural models to expand NormBank with automatic knowledge completion. Experiments show promising results: these models can extrapolate social commonsense to new behaviors in new situations, leveraging similarities in analogous roles across different situations. Finally, we demonstrate how to transfer knowledge via sequential finetuning from NormBank to social reasoning tasks. Together, knowledge completion and transfer learning suggest that our dataset will serve as a useful resource for advancing neural models toward situationally-grounded social reasoning.

## 2 Related Work

Commonsense knowledge bases(CSKBs) are sets of structured knowledge about everyday life. They capture broad taxonomic relationships Liu and Singh (2004); Speer et al. (2017); Elsahar et al. (2018), logical relations Lenat (1995); Zhang et al. (2018), and universal laws of causality and physical mechanics Talmor et al. (2019); Bisk et al. (2020). More recent datasets encode _social mechanics_, like broad human values Ziems et al. (2022); Hendrycks et al. (2021), situation-agnostic norms Forbes et al. (2020); Fung et al. (2022), and prototypical rules of social behavior and motivation Sap et al. (2019); Huang et al. (2019). General purpose CSKBs like ATOMIC Sap et al. (2019) and Social-Chemistry-101 Forbes et al. (2020) can be converted into tasks like Social IQa Sap et al. (2019) and \(\delta\)-NLI Rudinger et al. (2020), plus injected into language models for downstream applications Chang et al. (2020); Mitra et al. (2019); Ji et al. (2020, 2020).

 

Norm discoveryis an emerging method inspired by automatic knowledge base construction (Mitchell et al., 2018; Weston et al., 2013; Craven et al., 2000) and extracting social knowledge from LLMs via prompting Trinh and Le (2018); Petroni et al. (2019); Wang et al. (2019); Sakaguchi et al. (2020). In concurrent work, Fung et al. (2022) propose NormSAGE, which automatically discovers mandated or conventional behaviors from dialogues. Their prompts resemble our bootstrapping efforts in SS3, with the added step of automatic self-verification. NormBank differs from NormalSAGE in that we rely on human annotation to collect more creatively non-prototypical situations to challenge and expand normative reasoning models.

Normative reasoningsystems like Delphi Jiang et al. (2021), and UNICORN Lourie et al. (2021) are pre-trained on existing social knowledge bases Forbes et al. (2020); Emelin et al. (2021); Hendrycks et al. (2021); Lourie et al. (2021); Sap et al. (2020), which contain more conventional social behaviors from narrative contexts. Until Pyatkin et al. (2022), in work concurrent to our own, normative reasoning systems have been framed as universal oracles, making forced-choice judgments about human behaviors _in a vacuum_Talat et al. (2022). These models lack the capacity for defeasible reasoning Madaan et al. (2021); Rudinger et al. (2020). Oracles instead tend to assume the most prototypical contexts Boratko et al. (2020). Many of these predictions will appear reasonable if we pragmatically infer a conventional narrative, but for systems to achieve robust social intelligence, they must account for the long tail of the distribution. We can easily find unconventional contexts in which the correct inference contained in NormBank is misunderstood by current models.2

Footnote 2: For example, Delphi believes _yelling_ and _clenching your fists_, _breathing heavily_, or _asking someone personal questions about their sex life_ are all conventionally inappropriate. NormBank gives acceptable contexts for each: _guests riding a roller coaster, athletes running track, and doctors performing routine checkups_, respectively.

## 3 SCENE: A Dramaturgical Framework

The self... is a dramatic effect arising diffusely from a SCENE.  -- **Erving Goffman** (1959)

To help models efficiently learn non-monotonic normative reasoning over a seemingly unbounded set of possible contexts, and to test this understanding in LLMs, we will need to establish a more tractable set of elements to represent this social matrix. For this purpose, we construct a hierarchical taxonomy of constraints, which we call _Situational Constraints for social Expectations, Norms, and Etiquette_ (SCENE for short). SCENE follows Goffman's (1959) dramaturgical model of social life. According to this model, people are like actors trying to maintain a social performance in front of an audience. Each actor performs a particular _role_ as if in a scene from a movie. The scene is grounded in a particular _setting_, which includes aspects of the _environment_ that inform the performance. Each scene also has a script Schank and Abelson (1977), which tells the actor what kinds of _behaviors_ will be perceived as in-character or out-of-character. Additionally, the actor will embody socially meaningful _attributes_ like age, gender, status, etc. These attributes may be relevant to the scene and the actors place in it. In Figure 2, the example setting is a _restaurant_ where the environment is _uncrowded_ and the hour is _night_. There are two primary roles of _customer_ and _server_, and for norm formation, some relevant attributes include their respective _genders_, _sexualities_, and _ages_, which parameterize the behaviors that are appropriate for this dinner, such as _dating_ and _drinking alcohol_.

Settings\(\leavevmode\hbox{\small 8}\mskip-4.0mu\) (e.g., banks, classrooms, homes, hospitals) are the loci of scripted social interactions Schank and Abelson (1977), and they frame all subsequent elements of NormBank, so we begin here with 129 distinct settings like _amusement park, bus_, and _elevator_. Settings derive from two popular knowledge resources. First, there are 80 settings from from ConceptNet Speer et al. (2017), a broad knowledge base of the words and phrases that people commonly use.3 There are another 255 settings from the "movie scene" label in the MovieGraphs Vicol et al. (2018) resource--a collection of social situations that were depicted in movie clips.

Footnote 3: Settings were defined by head-entities with an IsA relationship to some tail in [place, location, area]. Manual inspection proved the usefulness of this heuristic.

#### The Environment

contains signals that can trigger associative priming of social norms (e.g., the noise level of a study space; Aarts and Dijksterhuis). This portion of the taxonomy is designed to be broad and general-purpose, with a set of attributes that can refine any setting. Our taxonomy is based on a broad review 

formation and its relevant factors (van Rijswijk and Haans, 2018; Janicik and Bartel, 2003; Boyce et al., 2000; Russell and Ward, 1982; Durkheim, 1915). Importantly, the taxonomy is further refined through crowdsourced feedback (SS4). Ultimately, our taxonomy grows to contain \(404\) environmental constraints. An extensive overview of the environmental constraints is given in Appendix B.1, but we summarize them here.

In the environment, there are important taxonomic subclasses of factors that inform norms. One subclass is **time** constraints, like seasonality (Janicik and Bartel, 2003), holidays and special customary observances (Durkheim, 1915), and another is the **country of operation**, which serves as a proxy for regional cultural differences (Meyer, 2014). We also include factors from **environmental psychology**(Bell et al., 2001) that involve the agent's comfort and ease in the environment (e.g., noise level, privacy, and cleanliness). Additionally, **physical conditions** include factors like weather, which impact visibility, coordination, safety, and comfort (Boyce et al., 2000; van Rijswijk and Haas, 2018; Cunningham, 1979). In addition to the imposed taxonomy, annotator feedback (SS4) lead us to add a subclass called **restrictions** that formally limit attendance, participation, and behavior, due to notions of formality, religiosity, or exclusiveness.

Roles2 may be ubiquitous, but it is challenging to collect reliable, setting-specific roles with high coverage. Our solution is to use the powerful associative knowledge of LLMs to automatically enumerate roles for each setting via prompting, inspired by Trinh and Le (2018), Petroni et al. (2019), and others (Wang et al., 2019; Sakaguchi et al., 2020). Specifically, we prompt GPT-3 (Brown et al., 2020) text-davinci-002 in a zero-shot manner with the phrase "Some roles <preposition> <determiner> <setting>:" where the preposition and determiner are manually configured to match the setting (for example, "some roles at a casino:" or "some roles on the beach:"). On average, we generate 5.5 roles per setting, with a total of 928 unique roles.

Attributes1are properties of individual agents that determine their social norms. Here again, the goal is to derive a general purpose taxonomy from the literature. Some attributes are basic demographic categories like the person's **age bracket**, **gender**, **race**, **religion**, and **sexuality**(Thompson Jr and Pleck, 1986; Dempsey and De Vaus, 2004; Helgeson, 2016). Related demographic categories include **education** level, **employment**, and **marital status**. Since food is a focal point for culture and morality, we include **diet**. We also include material constraints like **medical condition** and **social class**. Finally, we increase the coverage of this set by including generic descriptors of two types: **condition or state** adjectives, which describe a temporary condition (e.g., _dizzy_), and **characteristic** adjectives that describe more permanent attributes (e.g., _blonde_). In total, our taxonomy defines \(578\) attribute constraints.

Behaviors2are the primary target of analysis for social norms. As with roles, we co-opt GPT-3 to enumerate behaviors for each setting and role, but the approach here is augmented in two ways. First, we include a norm _expectation_ in the prompt. By querying for _unexpected_ behaviors, we can begin to shift the distribution of behaviors away from the prototypical. Second, we increase the diversity of generations by conditioning on

Figure 2: **An example of the SCENE Dramaturgical Framework used to constrain NormBank. The restaurant setting is specified by the attendance (_not crowded_) and time of day (_night_) in the environment. The two agent roles, customer and server; the latter is specified by the agebracket (_adult_) and gender (_male_) attributes. The former are engaged in the behaviors _drinking alcohol_ and _going on a date_. _Note:_ Graphics are for illustration. NormBank is a text dataset and does not contain any images.

 

the agent's attribute. This further reduces the number of conventional behaviors in our set. The prompt is "Some things you would (never) do <preposition> <determiner> <setting>  (if you were <attribute>):" where elements in parentheses are optional elements. In this way, we generated an average of 776.5 behaviors per setting, which was filtered down to 112.6 behaviors per setting, via programmatic methods described in Appendix B.2.

## 4 Building NormBank

Section 3 gave us a high-recall set of constraint variables for explaining situational social norms. Our end goal is to build a resource that contains reliable norms to ground, train, and test automatic normative reasoning systems. We want these norms to describe challenging, non-prototypical examples, and to depend on subtly contrasting situations that, when shifted, change the norm label non-monotonically. This motivates us to use human annotation over the rich SCENE taxonomy.

Our process is essentially the reverse of the current paradigm established by prior work, which starts with a basic narrative context and subsequently extracts Fung et al. (2022) or annotates Forbes et al. (2020) the expected behaviors. Instead, we start with behaviors and ask annotators to provide us with different dramaturgical contexts (SCENE constraints) under which that behavior could be variously seen as _expected_, _okay_, or _unexpected_. Thus we obtain richer and less prototypical instances--examples not mentioned in standard dialogue, which will significantly challenge models. The approach is inspired by contrast sets Gardner et al. (2020) and counterfactual augmentation Kaushik et al. (2019) as means of reducing spurious correlations in model inferences.

### Annotation Task

For the annotation task, we recruit experienced English-speaking Mechanical Turk annotators who have \(\geq\)98% acceptance with \(\geq\)100 HITs and are located in the United States. The task requires human creativity over a large combinatorial space. For a given setting \(s\) and a behavior \(b\), an annotator will tell us distinct situational contexts under which \(b\) is alternatively _expected_ (required by duty or anticipated with high probability), _okay_ (permitted or anticipated with moderate probability), or _unexpected_ (forbidden, stigmatized, taboo, or otherwise anticipated with very low probability).

These _expected, okay_, and _unexpected_ categories are called "norm labels." The language of expectation is useful for describing behavioral patterns--the focus of this work--rather than prescribing rules for what _ought_ to be done, as in prior datasets Ziems et al. (2022); Emelin et al. (2021); Lourie et al. (2021); Forbes et al. (2020); Sap et al. (2020). Importantly, we do not impose any ethical or moral philosophy Hendrycks et al. (2021), but instead, encourage annotators to find norms that merely describe observable social life Cialdini et al. (1991).

The annotator fully specifies the appropriate situational context by means of disjunctions and conjunctions of constraints. For example, "spit at a dentist's office" can be unexpected when (PERSON's roleis 'dentist') or when ((PERSON's roleis 'patient') AND (PERSON's behaviour is 'checking in')). Annotators select SCENE constraints using drop down menus that follow the hierarchy of SS3 (for details on the HIT interface, see Appendix C.2). They are also free to insert their own custom constraints into the hierarchy. In this way, we iteratively expand the taxonomy.

### Dataset Quality

Quality Control.Manual inspection of over 2.5k data points reveals that the open-ended and creative aspects of the task are natural incentives for high-quality work Chandler et al. (2015); Sheehan (2018). To further ensure the quality of NormBank, we trained annotators with careful instruction, a qualification test, a staging round, personalized feedback, programmatic filtering, and finally, a series of random audits Litman et al. (2015); Sheehan (2018). The instructions included at least 3 fully-worked examples for each norm label, plus suggestions and explanations for a total of 24 constraints. We administered a six-question qualifier, which tested workers' knowledge of the taxonomy, definitions, free text response, and how to properly indicate constraint conjunctions and disjunctions through the task interface. If the worker passed at least five questions correctly on the first try, she would gain access to the staging round - a small-scale version of the task in which each submission would receive detailed and personalized feedback.

We invested a significant amount of time to feedback, offering \(75\) to \(200\) words of review for each of \(2,502\) staging HITs. Once a worker submitted 

high-quality HITs in the staging round, he or she could move to the full task. To identify poor work here, we programmatically flagged workers with extremely low variation in their annotations. Finally, we periodically performed a total of three random audits, sampling \(250\) annotations in each audit, to confirm the quality of the annotations. Workers were paid a base rate, plus an additional itemized bonus for every additional constraint they added, which incentivized workers to be more expressive and creative. Annotators received a median of $30 per hour for this task.

Quality Metrics.The above methods all proved remarkably successful in generating a creative and high-quality resource. Because our task is creative and subjective, data quality is not easily measured by inter-annotator agreement. We instead report human evaluations over the Gold NormBank data in the bottom row of Table 3 (alongside model generations from SS5.1). Annotations are considered sensible (82.5%), relevant (82.17%), and normative (72.9%). Still, it is important to note that around half of gold annotations are marked by third-party evaluators as fully correct by majority vote.

With regard to the correctness metric, annotator disagreements can be traced to differences in the annotators' models of the world, which likely stem from their own personal differences, including age, profession, and worldview. For example, an annotator likely familiar with the Cambodian tradition of "Pithi Strang Preah" marked that "honoring your ancestors" is normal for Cambodians on Cambodian New Year, while an annotator unfamiliar with this practice marked it as unexpected. Furthermore, we administered political leaning and the moral foundations surveys to all annotators, which we release alongside NormBank to help explain how these personal differences informed the probabilities they assigned to events. This resource will be of interest to computer scientists and social scientists, since NormBank contains not only commonsense facts, but also culturally-conditioned distributions over behavior and expectations about behavior.

Dataset Summary Statistics.Table 1 gives the summary statistics for the annotated dataset. NormBank contains a total of 408k constraints, applied to 155k norms, for an average of 2.63 constraints per norm. The SCENE taxonomy broadly captures the kinds of constraints annotators were looking for 94% of the time, and they were able to find their exact con

Figure 3: **Distribution of NormBank constraints where the area of each cell is proportional to the distribution. Constraints are dominated most by the agent’s attributes and roles, with a smaller and even split between behaviors and the environment.**



straint value from a pre-populated list in 69% cases. For concurrent behavior and attribute constraints, annotators had to input their own values in 59% and 33% of cases respectively, followed by 27% and 9% of cases for the environment and roles. Overall, this indicates that our GPT-3 prompting method achieved high recall, especially for roles, and least so for behaviors, which is unsurprising, given the almost unbounded space of viable human behavior.

Figure 3 gives the distribution of constraints in NormBank. Constraints are dominated most by the agent's attributes and roles. Age, condition, and characteristic are the most popular attributes, while roles vary. There is an even split between behaviors and the environment. In the environment, there is a notable focus on _time_ constraints, and slightly lesser but more even attention towards the remaining subcategories.

Links to Existing Knowledge.NormBank's SCENE taxonomy has close links to existing knowledge resources. ConceptNet directly seeded 80 settings in SCENE. Beyond this, we successfully link over 90% of taxonomic items from the setting, environment, roles, and attributes directly with concepts in ConceptNet. These taxonomic items cover 93.6% of all constraint categories and 70.0% of all constraint values. ConceptNet is further linked to WordNet, DBPedia, Umbel, Cyc, and Wiktionary, so by extension, NormBank can be coupled to these resources.

## 5 Experiments: How to use NormBank

NormBank is not designed for any particular narrow task; it is designed as a general-purpose knowledge resource that can ground social reasoning through downstream tasks (compare ATOMIC Sap et al. (2019) and ConceptNet Liu and Singh (2004)). Towards this end, NormBank should contain richly organized knowledge that can be learned by neural models and applied for non-monotonic reasoning in new settings. In this way, it should be possible to automatically expand the NormBankresource. The knowledge contained here should also be applicable across a range of social reasoning tasks. Thus our experiments aim to demonstrate two things: (SS5.1) that we can automatically expand NormBank using neural methods, and (SS5.2) that NormBank is a useful resource with relevant knowledge for downstream applications. For all experiments in the following subsections, we use an 80%-10%-10% train-dev-test split in which <setting, behavior> tuples in one set are never seen in another.

### Automatic Knowledge Completion

How can we expand NormBank? We considered two different methods of knowledge bank completion Weston et al. (2013); Craven et al. (2000), which rely on different assumptions. Results from both methods indicate that NormBank is rich enough to support its own automatic expansion. Classification is the simpler case, where we assume a closed world Bordes et al. (2013); Lin et al. (2015); Socher et al. (2013), while generation assumes an open world Shi and Weninger (2018) with a modifiable set of constraints.

Classification.Here, our known constraints and behaviors (SS3) will remain fixed Shi and Weninger (2018), but we can discover new relationships by classifying unseen behavior and constraint combinations as _expected, okay,_ or _unexpected_. The advantage of this approach is that it is straightforward, and the disadvantage is that evaluating classifiers over the power set of the entire constraint space would be intractable; thus more efficient search methods will be needed.

For all norm classification tasks, we fine-tune three popular transformer models: BERT-base-uncased Devlin et al. (2019), RoBERTa-base Liu et al. (2019), and ALBERT-base-v2 Lan et al. (2020), with hyperparameters in Appendix A. Results in Table 2 appear promising. Given the scope and scale of NormBank, models are capable of learning non-monotonic inferences, achieving F1 scores as high as 72.1% on the test set. This shows that classification is a reliable method for NormBank knowledge expansion.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline Model & P & R & F & Acc \\ \hline ALBERT & 68.0 & 66.6 & 67.1 & 71.0 \\ BERT & 72.1 & 70.7 & 71.3 & 74.6 \\ RoBERTa & **73.3** & **71.4** & **72.1** & **75.4** \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Classification Results** for {_expected, okay,_ or _unexpected_} show that standard transformer models can learn to make normative inferences with an accuracy that is adequate for expanding NormBank.

 

Generation.The model is trained with a forward language modeling objective over the string \(g\):

\[\begin{split}&\mathbf{g}=\{\texttt{[SETTING]},s_{1},s_{2},...,s_{n},\\ &~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}

 

'hydrate' while the temperature is 'warm.' BART, however, correctly discerns that it's unexpected for athletes whose behavior is 'playing sports.' In general, GPT-3 appears more likely to underspecify the situation. For example, GPT-3 responds that it's _expected_ for a homeowner to 'leave the gate open' in the 'backyard,' and BART agrees, but BART further specifies that the owner might be 'working outside' to justify the _expectation_.

Both quantitative and qualitative analyses indicate that prompting methods can certainly complement, but may not fully replace, fine-tuned generation approaches to NormBank expansion. A mixed approach may be most desirable due to coverage and correctness, while generation errors may be fixed using self-correction via classification (above) or further prompting Fung et al. (2022).

Finally, the middle pane of Table 3 shows the proportion of generated constraints that fall into our taxonomy (Tax. Constr.) and the proportion contained in NormBank (Pre-pop. Constr.). The former shows that our taxonomy broadly captures the relevant axes (80-90% of our best models' generations are taxonomic). The latter shows that between one third and one half of generations 'link' prior constraints to new situations; the rest of generated constraints are brand new.

### Transfer Learning for Downstream Tasks

Finally, we conduct transfer learning experiments to demonstrate the utility of the data for downstream applications, further indicating the scope and power of NormBank as a general-purpose resource for social reasoning. Concretely, we follow the sequential training paradigm Pratt et al. (1991), which has proven better than multitask training and fine-tuning on a broad range of commonsense tasks Lourie et al. (2021a). Specifically, we initialize a RoBERTa model with weights from our best-performing norm classifier from Section 5.1 and fine-tune on the target set for 7 epochs.

We evaluate on two specifically _moral_ reasoning tasks, Anecdotes and Dilemmas, both from the Scruples benchmark Lourie et al. (2021b). We also consider two multiple-choice commonsense QA datasets. Social IQa Sap et al. (2019) is designed to test social intelligence (e.g., inferring motivations, emotional reactions), while CosmosQA Huang et al. (2019) tests cause and effect and counterfactual reasoning in everyday situations.

All results in Table 4 are averaged over five separate train-test runs, and significance is given by the paired bootstrap test. NormBank's utility is seen by comparing the accuracy of models with transfer learning from NormBank against those with task-only fine-tuning (Base Model). Results show that NormBank improves situational moral classification (Anecdotes; +0.4%) and forced choice binary moral judgments Dilemmas +6.8%) with significance. Also consider NormBank utility as compared to transfer learning from either CosmosQA (CQA) or SocialIQa (SIQa). The only task on which transfer from NormBank does not achieve the best performance is on CosmosQA evaluation. Here, we find that transfer from the more structurally related Social IQa task is preferred. We conclude that NormBank is a useful resource for a range of downstream applications in moral, social, and emotional reasoning in context.

## 6 Conclusion

Social norms are the foundation of culture and society McDonald and Crandall (2015); Hogg and Reid (2006), and an understanding of these norms is crucial for assistive and collaborative AI. In this work, we introduced SCENE a new scheme for hierarchically organizing the seemingly unbounded space of situational contexts that determine social norms. With this framework, we built NormBank, the first social knowledge bank to leverage such contextual information for contrast sets of richly conditioned defeasible social norms. We found that NormBank supports its own automatic expansion via classification, generation, and prompting methods. Finally, we demonstrated the utility of NormBank for situational social reasoning tasks.

\begin{table}
\begin{tabular}{l|c|c|c|c} Eval & Base & \multicolumn{3}{c}{w/ Transfer Learning from} \\ Task & Model & CQA & SIQa & **NormBank** \\ \hline ANEMODTES & 68.3 & 68.3† & 68.0 & **68.7***� \\ DIELEMMAS & 64.3 & 67.4 & 70.9* & **71.1*** \\ SOCIALIQA & 59.9 & 64.1 & 59.9 & **64.2** \\ COSMOSQA & 59.8 & 59.8 & **63.5***� & 61.2 \\ \end{tabular}
\end{table}
Table 4: **Transfer Learning Accuracies** demonstrate the utility of NormBank. By sequential finetuning on NormBank, we improve performance over baseline on all tasks, and transfer performance from NormBank exceeds transfer performance from CosmosQA and from SocialIQa in three cases. Best performance is **bolded**. Star* results indicate significant improvements over the Base Model, while † marks significance over CSA, and ‡ marks significance over SIQa.

 



## 7 Limitations

At its core, NormBank is a collection of logical operations on unique constraints. Consequently, one practical limitation stems from the issue that some situations cannot be reasonably expressed as a set of constraints. While theoretically all logic can be decomposed into AND and OR operations, the logic may be too challenging for an individual to formulate, or the set of constraints themselves might be too large and unwieldy. The latter is problematic, because language models have a finite input token capacity, and for the set of constraints to be digestible, they must fit within that capacity. Relatedly, if the logic to encode constraints become more sophisticated, ensuring that logic is not unnecessarily duplicated will pose a greater challenge. Additionally, certain properties of NormBank like the role and behaviors may be challenging to succinctly describe. Further work will be needed to ascertain how these can be incorporated or to more clearly define situations that are out of scope.

Due to limitations on time and computational resources, we have not exhaustively evaluated all downstream applications of NormBank, and in future work, we will test additional transfer tasks beyond the moral and social classification tasks considered in this work. Since NormBank is the first to encode non-monotonic situational norms, there was no other available _benchmark_ that is directly analogous to ours. Instead, our primary evidence for NormBank's utility is in Table 3, where human evaluators confirm that models trained on NormBank can reliably learn to make new inferences about non-monotonic situational norms.

Other follow up studies should consider training larger normative reasoning models, and/or engineering better prompts for expanding NormBank. Relatedly, we have no data to speculate about the long-term evolution of real-world norms relative to this resource, nor the rate of decay in the reliability of NormBank. Future work should also expand this resource with perspectives from cultures other than our available annotator pool. The pool was not representative of all cultures and people groups, as we discuss further in the Ethics section.

## 8 Ethics

**Ethical Assumptions.** First, to set proper boundaries on this resource and the tasks it can facilitate, we will outline the ethical assumptions of this work and address some potential misconceptions. We want to stress that NormBank represents a collection of situational norms that we do not treat as prescriptive, but rather descriptive. Unlike prior _moral_/_ethics_ datasets (Ziems et al., 2022; Emelin et al., 2021; Lourie et al., 2021; Forbes et al., 2020; Sap et al., 2020), we use the neutral language of _expected, okay,_ and _unexpected_ behaviors to focus on empirically observed patterns and avoid an over-emphasis on the ethical grey area of what _ought_ to be done. Unlike tricky moral dilemmas, the situational social norms of NormBank have an answer that a majority can agree is descriptively observable as the expectation under the respective conditions and/or cultural context. Nevertheless, normative judgments can vary between individuals in different social groups and time periods (Haidt et al., 1993; Shweder, 1990; Bicchieri, 2005; Culley and Madhavan, 2013; Amaya et al., 2021). NormBank can and should be expanded via automatic or manual methods that can incorporate these axes of variation. Our annotator pool was limited to English-speaking individuals living in the United States in the year 2022. Future expansion efforts could be crowdsourced from other cultures and geographic regions and in future decades.

We reiterate that the norms in NormBank should _not_ be used for prescriptive advice or personal guidance in any way. Our work intends to unlock future work in the capacity to imbue language models with situational commonsense and enable them to jointly reason with the situational contexts. Language models which ignore situational contexts altogether may be just as hazardous, if not more.

Finally, there are likely biases towards certain roles and values in NormBank. We have taken steps to mitigate some forms, such as gender bias, by neutralizing constraints (e.g., [PERSON]'s role is 'cowboy or cowgirl' and [PERSON]'s role is 'ball boy or girl'). Our SCENE taxonomy, with the standardized structure of its role and attribute constraints, will allow practitioners to further analyze specific axes of prejudice and thus implement targeted mitigation strategies. Specific identity attributes like gender, ethnicity, and religion are represented in 24% of norms.5 Stakeholders can invest a smaller but more concerted effort towards mitigating bias in these constraints. We encourage

stakeholders to give auditing control over a given norm to those who are affected by it. Previous norm-datasets encode norms in free-text annotations which lack a hierarchical taxonomy of contexts, but our taxonomy can be used to interpret, diagnose, and mitigate prejudice, and to return power to those affected by these prejudices.

Risks in deployment.Before starting any annotation, the resources and findings presented in this work were thoroughly reviewed and approved by an internal review board. Prior to being put into production, the method would also need to be re-evaluated when applied to a new domain to ensure reliable performance in order to prevent unintended consequences. To help mitigate risks in deployment from misunderstandings about the ethical assumptions above, we require users of this data to complete a Data Use Agreement. The user will check that they understand the ethical assumptions above: especially that NormBank is not to be taken for advice. Practitioners will also agree not to use NormBank for malicious purposes "including (but not limited to): mockery, discrimination, and hate speech."

## Acknowledgements

We are thankful to Julia Kruk, William Held, Albert Lu, Camille Harris, and the anonymous ACL reviewers for their helpful feedback. Coleb Ziems is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655.

## References

* L. Aarts and P. Dijksterhuis (2003)The silence of the library: environment, situational norm, and social behavior. Journal of personality and social psychology84 (1), pp. 18. External Links: Document Cited by: SS1.
* I. Altman (1975)The environment and social behavior: privacy, personal space, territory, and crowding. External Links: Link Cited by: SS1.
* A. Amaya, R. Bach, F. Keusch, and F. Kreuter (2021)New data sources in social science research: things to know before working with reddit data. Social science computer review39 (5), pp. 943-960. External Links: ISSN 0021-921X, Link, Document Cited by: SS1.
* J. Anderson and L. Rainie (2022)The metaverse in 2040. Few Research Center. External Links: Link Cited by: SS1.
* R. Bavaresco, D. Silveira, E. Reis, J. Barbosa, R. Righi, C. Costa, R. Antunes, M. Gomes, C. Gatti, M. Vanzin, et al. (2020)Conversational agents in business: a systematic literature review and future research directions. Computer Science Review36, pp. 100239. External Links: ISSN 0021-921X, Link, Document Cited by: SS1.
* P. A. Bell, T. Green, J. D Fisher, and A. Baum (2001)Environmental psychology. New Jersey. External Links: ISBN 978-0-030-3095-3 Cited by: SS1.
* C. Bicchieri (2005)The grammar of society: the nature and dynamics of social norms. Cambridge University Press. External Links: ISBN 978-0-030-3095-3 Cited by: SS1.
* Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. (2020)Piqa: reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, Vol. 34, pp. 7432-7439. External Links: Link, Document Cited by: SS1.
* J. A. Blass and I. D. Horswill (2015)Implementing injunctive social norms using defeasible reasoning. In Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference, Cited by: SS1.
* M. Boratko, X. Li, T. O'Gorman, R. Das, D. Le, and A. McCallum (2020)ProtoQA: a question answering dataset for prototypical common-sense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1122-1136. External Links: Link, Document Cited by: SS1.
* A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013)Translating embeddings for modeling multi-relational data. Advances in neural information processing systems26. Cited by: SS1.
* P. R. Boyce, N. H. Eklund, B. J. Hamilton, and L. D. Bruno (2000)Perceptions of safety at night in different lighting conditions. International Journal of Lighting Research and Technology32 (2), pp. 79-91. External Links: ISSN 0021-921X, Link, Document Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 0021-921X, Link, Document Cited by: SS1.
* F. M. Carlucci, L. Nardi, L. locchi, and D. Nardi (2015)Explicit representation of social norms for social robots. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4191-4196. External Links: ISBN 978-0-030-3095-3 Cited by: SS1.
* J. Chandler, G. Paolacci, E. Peer, P. Mueller, and K. A. Ratliff (2015)Using nonnaive participants can reduce effect sizes. Psychological science26 (7), pp. 1131-1139. External Links: ISSN 0021-921X, Link, Document Cited by: SS1.
* Ting-Y. Chang, Y. Liu, K. Gopalakrishnan, B. Hedayatina, P. Zhou, and D. Hakkani-Tur (2020)Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks. In Proceedings of Deep Learning Inside Out 

(DeeLO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 74-79, Online. Association for Computational Linguistics.
* Cialdini et al. [1991] Robert B Cialdini, Carl A Kallgren, and Raymond R Reno. 1991. A focus theory of normative conduct: A theoretical refinement and reevaluation of the role of norms in human behavior. In _Advances in experimental social psychology_, volume 24, pages 201-234. Elsevier.
* Cialdini et al. [1990] Robert B Cialdini, Raymond R Reno, and Carl A Kallgren. 1990. A focus theory of normative conduct: Recycling the concept of norms to reduce littering in public places. _Journal of personality and social psychology_, 58(6):1015.
* Craven et al. [2000] Mark Craven, Dan DiPasquo, Dayne Freitag, Andrew McCallum, Tom Mitchell, Kamal Nigam, and Sean Slattery. 2000. Learning to construct knowledge bases from the world wide web. _Artificial intelligence_, 118(1-2):69-113.
* Culley and Madhavan [2013] Kimberly E Culley and Poornima Madhavan. 2013. A note of caution regarding anthropomorphism in hci agents. _Computers in Human Behavior_, 29(3):577-579.
* Cunningham [1979] Michael R Cunningham. 1979. Weather, mood, and helping behavior: Quasi experiments with the sunshine samaritan. _Journal of personality and social psychology_, 37(11):1947.
* Dempsey and Vaus [2004] Ken Dempsey and David De Vaus. 2004. Who cohabits in 2001? the significance of age, gender, religion and ethnicity. _Journal of Sociology_, 40(2):157-178.
* Deng et al. [2019] Eric Deng, Bilge Mutlu, Maja J Mataric, et al. 2019. Embodiment in socially interactive robots. _Foundations and Trends(r) in Robotics_, 7(4):251-356.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Durkheim [1915] E Durkheim. 1915. The elementary forms of the religious life: A study in religious sociology.
* Elsahar et al. [2018] Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Emelin et al. [2021] Denis Emelin, Ronan Le Bras, Jena D Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 698-718.
* Fong et al. [2003] Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. 2003. A survey of socially interactive robots. _Robotics and autonomous systems_, 42(3-4):143-166.
* Forbes et al. [2020] Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 653-670.
* Fung et al. [2022] Yi R Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2022. NormSage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. _arXiv preprint arXiv:2210.08604_.
* Gardner et al. [2020] Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models' local decision boundaries via contrast sets. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1307-1323.
* Goffman [1959] Ervin Goffman. 1959. _The presentation of self in everyday life_. Doubleday Anchor Books.
* Grossman et al. [2019] Joshua Grossman, Zhiyuan Lin, Hao Sheng, Johnny Tian-Zheng Wei, Joseph J Williams, and Sharad Goel. 2019. Mathbot: Transforming online resources for learning math into conversational interactions. _AAAI 2019 Story-Enabled Intelligence_.
* Haidt et al. [1993] Jonathan Haidt, Silvia Helena Koller, and Maria G Dias. 1993. Affect, culture, and morality, or is it wrong to eat your dog? _Journal of personality and social psychology_, 65(4):613.
* Helgeson [2016] Vicki S Helgeson. 2016. _Psychology of gender_. Routledge.
* Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch Critch, Jerry Li Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. In _International Conference on Learning Representations_.
* Hogg and Reid [2006] Michael A Hogg and Scott A Reid. 2006. Social identity, self-categorization, and the communication of group norms. _Communication theory_, 16(1):7-30.
* Hovy and Yang [2021] Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 588-602.
* Huang et al. [2019] Liftu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.

 

In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2391-2401.
* Janick and Bartel (2003) Gregory A Janick and Caroline A Bartel. 2003. Talking about time: Effects of temporal planning and time awareness norms on group coordination and performance. _Group Dynamics: Theory, Research, and Practice_, 7(2):122.
* Ji et al. (2020a) Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, and Minlie Huang. 2020a. Generating commonsense explanation by extracting bridge concepts from reasoning paths. In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_, pages 248-257.
* Ji et al. (2020b) Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020b. Language generation with multi-hop reasoning on commonsense knowledge graph. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 725-736.
* Jiang et al. (2021) Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. _ArXiv preprint_, abs/2110.07574.
* Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In _International Conference on Learning Representations_.
* Klimczyk (2021) Joanna Klimczyk. 2021. Compositional semantics and normative 'ought'. _Axiomathes_, 31(3):381-399.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Lenat (1995) Douglas B Lenat. 1995. Cyc: A large-scale investment in knowledge infrastructure. _Communications of the ACM_, 38(11):33-38.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Lin et al. (2015a) Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015a. Modeling relation paths for representation learning of knowledge bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 705-714.
* Lin et al. (2015b) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning entity and relation embeddings for knowledge graph completion. In _Twenty-ninth AAAI conference on artificial intelligence_.
* Litman et al. (2015) Leib Litman, Jonathan Robinson, and Cheskie Rosenzweig. 2015. The relationship between motivation, monetary compensation, and data quality among us-and india-based workers on mechanical turk. _Behavior research methods_, 47(2):519-528.
* Liu and Singh (2004) Hugo Liu and Push Singh. 2004. Conceptnet--a practical commonsense reasoning tool-kit. _BT technology journal_, 22(4):211-226.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bertraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Lourie et al. (2021a) Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021a. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 13480-13488.
* Lourie et al. (2021b) Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021b. Scruples: A corpus of community ethical judgments on 32, 000 real-life anecdotes. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 13470-13479.
* Madan et al. (2021) Aman Madan, Dheeraj Rajagopal, Niket Tandon, Yingang Yang, and Eduard Hovy. 2021. Could you give me a hint? generating inference graphs for defeasible reasoning. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 5138-5147.
* Mathews and Karpatrick Canon (1975) Kenneth E Matthews and Lance Kirkpatrick Canon. 1975. Environmental noise level as a determinant of helping behavior. _Journal of Personality and Social Psychology_, 32(4):571.
* McDonald and Crandall (2015) Rachel I McDonald and Christian S Crandall. 2015. Social norms and social influence. _Current Opinion in Behavioral Sciences_, 3:147-151.
* Meyer (2014) Erin Meyer. 2014. _The culture map: Breaking through the invisible boundaries of global business_. Public Affairs 

Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018. Never-ending learning. _Communications of the ACM_, 61(5):103-115.
* Mitra et al. [2019] Arindam Mitra, Pratyay Banerjee, Kuntal Kumar Pal, Swaroop Mishra, and Chitta Baral. 2019. How additional knowledge can improve natural language commonsense question answering? _arXiv preprint arXiv:1909.08855_.
* Molnar and Szints [2018] Gyorgy Molnar and Zoltan Szints. 2018. The role of chatbots in formal education. In _2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY)_, pages 000197-000202. IEEE.
* Petroni et al. [2019] Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick Lewis, Anton Bachthin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
* Pollock [1987] John L Pollock. 1987. Defeasible reasoning. _Cognitive science_, 11(4):481-518.
* Pratt et al. [1991] Lorien Y Pratt, Jack Mostow, Candace A Kamm, Ace A Kamm, et al. 1991. Direct transfer of learned information among neural networks. In _Aaai_, volume 91, pages 584-589.
* Pyatkin et al. [2022] Valentina Pyatkin, Jena D Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. 2022. Reinforced clarification question generation with defeasibility rewards for disambiguating social and moral situations. _arXiv preprint arXiv:2212.10409_.
* Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Reiter [1981] Raymond Reiter. 1981. On closed world data bases. In _Readings in artificial intelligence_, pages 119-140. Elsevier.
* Rudinger et al. [2020] Rachel Rudinger, Vered Shwartz, Jena D Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4661-4675.
* Russell and Ward [1982] James A Russell and Lawrence M Ward. 1982. Environmental psychology. _Annual review of psychology_, 33(1):651-689.
* Sakaguchi et al. [2020] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8732-8740.
* Sap et al. [2020] Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5477-5490.
* Sap et al. [2019a] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019a. Atomic: An atlas of machine commonsense for ifthen reasoning. In _Proceedings of the AAAI conference on artificial intelligence_, volume 33, pages 3027-3035.
* Sap et al. [2019b] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Commonsense reasoning about social interactions. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4463-4473.
* Schank and Abelson [1977] R. C. Schank and R. P. Abelson. 1977. _Scripts, plans, goals, and understanding: An enquiry into human knowledge structures_. Erlbaum.
* Scott et al. [2007] Alister Scott, Alana Gilbert, and Ayele Gelan. 2007. _The urban-rural divide: Myth or reality?_ Macaulay Institute Aberdeen.
* Sheehan [2018] Kim Bartel Sheehan. 2018. Crowdsourcing research: data collection with amazon's mechanical turk. _Communication Monographs_, 85(1):140-156.
* Sherif and Sherif [1953] Muzafer Sherif and Carolyn W Sherif. 1953. Groups in harmony and tension; an integration of studies of intergroup relations.
* Shi and Weninger [2018] Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 32.
* Shweder [1990] Richard A Shweder. 1990. In defense of moral realism: Reply to gabennesch. _Child Development_, 61(6):2060-2067.
* Socher et al. [2013] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. _Advances in neural information processing systems_, 26.
* Speer et al. [2017] Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _Thirty-first AAAI conference on artificial intelligence_ 

Cass R Sunstein. 1996. Social norms and social roles. _Colum. L. Rev._, 96:903.
* Talat et al. (2022) Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 769-779, Seattle, United States. Association for Computational Linguistics.
* Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4149-4158.
* Thompson Jr and Pleck (1986) Edward H Thompson Jr and Joseph H Pleck. 1986. The structure of male role norms. _American Behavioral Scientist_, 29(5):531-543.
* Trinh and Le (2018) Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. _arXiv preprint arXiv:1806.02847_.
* Vaidyam et al. (2019) Aditya Nrusimha Vaidyam, Hannah Wisniewski, John David Halamka, Matcheri S Kashavan, and John Blake Torous. 2019. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. _The Canadian Journal of Psychiatry_, 64(7):456-464.
* van Rijswijk and Haans (2018) Leon van Rijswijk and Antal Haas. 2018. Illuminating for safety: Investigating the role of lighting appraisals on the perception of safety in the urban environment. _Environment and behavior_, 50(8):889-912.
* Vicol et al. (2018) Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler. 2018. Moviegraphs: Towards understanding human-centric situations from videos. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 8581-8590.
* Vilnai-Yavetz and Gilboa (2010) Iris Vilnai-Yavetz and Shaked Gilboa. 2010. The effect of services escape cleanliness on customer reactions. _Services Marketing Quarterly_, 31(2):213-234.
* Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaoran Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4020-4026, Florence, Italy. Association for Computational Linguistics.
* Weston et al. (2013) Jason Weston, Antoine Bordes, Oksana Yakhnenko, and Nicolas Usunier. 2013. Connecting language and knowledge bases with embedding models for relation extraction. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1366-1371.
* Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension.
* Ziems et al. (2022) Caleb Ziems, Jane A. Yu, Yi-Chia Wang, Alon Y. Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing_, Dublin, Ireland. Association for Computational Linguistics.

 

Models & Hyperparameters

Classification.We use the base versions of BERT Devlin et al. (2016); 768-hidden, 12-heads, 110M parameters), RoBERTa Liu et al. (2015), 125M parameters), and ALBERT-v2 Lan et al. (2016), 768-hidden, 12-heads, 11M parameters). For each model, we fine-tune using AdamW Loshchilov and Hutter (2019) for 7 epochs with a batch size of 16 and a learning rate of \(1e-5\). These hyperparameters were chosen by hyperparameter search on the dev set over {\(1e-5,2e-5,3e-5,5e-5\}\) and the number of epochs in {\(1..8\)}, with \(\epsilon=1e-8\) and the batch size set to 16.

Generation.We trained BART-large (406M parameters), GPT-2 (768-hidden, 12-heads, 117M parameters), and T5-small (512-hidden, 8 heads, 60M parameters) for 1 epoch using a batch size of 8 and a learning rate of \(3e-5\). We also prompted GPT-3 Davinci-002 and Davinci-003 in a few-shot manner via the OpenAI API, using the following prompt, which contains seven representative datapoints from the NormBank train set:

```
It is UNWEPTED to "talk about sex" at an "cafe" when: + PRESON's role is "barlist"
```

It is OAX to "flatten to music" in an "arcade" when all of  * the following are true: + PERSON's role is not "cook" + PERSON's roles is not "anittenance worker' + PRESON's behavior is "word headphones" + it is EXOECton to "hang Film to dry" at the "darkroom" when  * all of the following are true: + PERSON's role is "draftroom technician" + PRESON's behavior is "partraining routine tasks" ```

It is OAX to "smoke a cigar" in a "limousine" when all of  * the following are true: + OTHER's age bracket is not "gradaschoolor younger" + ORTH's attitude towards smoking is not "bothered by smoke" 

redundant behaviors per setting. The filtering techniques are as follows:

1. Remove any conditional form "if you were..." as well as any mention of the role or the setting in the behavior itself
2. Remove elaborations on behaviors "<behavior> because ... <elaboration>"
3. Normalize the logical form by removing words that negate behaviors (do not, never, not, forget to, refuse to, fail to, anything, something, anyone, someone, in any way, any)
4. Remove biased terms like _properly, should, would, try to, be able to, and see someone_.
5. Remove any bullet points or stray characters
6. Require that the behavior has an active (not passive) verb in it and that the verb does not have an explicit subject and that there is no dependent clause (as indicated by the marker [mark] dependency).

## Appendix C Annotation Task Details

### Qualification Task

To qualify for the HIT, workers were required to pass the following qualifying test, answering at least 5 out of 6 questions correctly.

1. **True or False:** you are allowed to add your own Constraints by typing them directly into the box. [_Answer_: **True**]
2. **True or False:** "carry a gun" can be an "Expected" behavior on an Airplane. [_Answer_: **True**]
3. **True or False:** "read a book" is an "Expected" behavior for a passenger on an Airplane. [_Answer_: **False**]
4. **True or False:** it is possible for a BEHAVIOR to be both "Expected" and "Okay" under the same Constraints. [_Answer_: **False**]
5. Let's say you are adding some Constraints for when "eating shrimp" is "Unexpected" in the SETTING: restaurant. You know that shellfish are disallowed in both Hinduism and Judaism, as well as by the vegan and vegetarian diets. You are thinking of adding these in the following **Incorrect**] (PERSON's religion is Judaism) AND (PERSON's religion is Hindiusism) AND (PERSON's diet is vegenan) AND (PERSON's diet is vegetarian)
6. Let's say you are adding some Constraints for when "eating shrimp" is "Okay" in the SETTING: restaurant. You know that shellfish are disallowed in both Hinduism and Judaism, as well as by the vegan and vegetarian diets. You are thinking of adding these in the following Constraint table. Is this correct? [_Answer_: **Correct**] (PERSON's religion is NOT Juhaism) AND (PERSON's religion is NOT Hinduism) AND (PERSON's diet is NOT vegan) AND (PERSON's diet is NOT vegetarian)

### HIT Interface

For each HIT, the annotator is presented with a setting \(s\in{\cal S}\) and a behavior \(b\in{\cal B}\) that we generated for the given \(s\). The annotator helps us describe when this behavior would be _expected_; then describes when it is merely _okay_; and finally _unexpected_. Annotators describe each norm with the conjunction and disjunction of SCENE constraints. The annotator appends each constraint to its conjunction as a 4-tuple consisting of a (1) _category_, (2) _name_, (3) _relation_, and (4) _value_. These are shown with examples in the HIT Instructions (4) and HIT Interface (Figure 5) screenshots. The _category_ helps annotators search for constraints and organize their thoughts. The _category_ is a high-level designation of where the constraint is organized: according to the _environment_, _role_, _attribute_, or _behavior_. The _name, relation_ and _value_ constitute a standard semantic triple. The _name_ designates the subject of the constraint, and it a specification of the _category_, like the "temperature of the environment." The _relation_ is a logical type that includes equality and inequality. The _value_ designates the predicate of the constraint (e.g., "freezing").

Annotators can build constraint 4-tuples from drop-down menus that enumerate our hierarchical taxonomy (Section 3). Annotators can also freely edit the above fields and contribute novel constraints. Finally, annotators compose constraints into disjunctive normal form (DNF), the OR of ANDs, to describe when behaviors are _expected_, _okay, or unexpected_ in a given setting 

Figure 4: HIT Instructions.

 

Figure 5: HIT Interface





**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

A For every submission: A1. Did you describe the limitations of your work? _Section 7 (Page 9)_ A2. Did you discuss any potential risks of your work? _Section 8 (Page 9)_ A3. Do the abstract and introduction summarize the paper's main claims? _Abstract and Section 1_ A4. Have you used AI writing assistants when working on this paper? _Left blank_. B Did you use or create scientific artifacts? _Sections 3-4_ B1. Did you cite the creators of artifacts you used? _Sections 3-4_ B2. Did you discuss the license or terms for use and / or distribution of any artifacts? _Section 8_ B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? _Section 8_ B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? _Section 8_ B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? _Section 4, 8_ B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. _Table 1_ C Did you run computational experiments? _Section 5_ C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? _Appendix D_

C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Appendix D_ C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Section 5; Table 4_ C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Section 5_
* Did you use human annotators (e.g., crowdworkers) or research with human participants? _Section 4_ D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _Appendix B_ D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _Section 4_ D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _Section 4_ D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _Section 8_ D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _Section 4_