SlowGuess/NAACL_2022_OCR huggingface dataset:
Zeerak Talat $^{1,*}$  Hagen Blix $^{2,*}$  Josef Valvoda $^{3}$
Maya Indira Ganesh $^{3}$  Ryan Cotterell $^{4}$  Adina Williams $^{5}$
$^{1}$ Digital Democracies Institute  $^{2}$ New York University  $^{3}$ University of Cambridge
$^{4}$ ETH Zürich  $^{5}$ Facebook AI Research
zeerak_talat@sfu.ca hagen.blix@nyu.edu jv406@cam.ac.uk
mi373@cam.ac.uk ryan.cotterell@inf.ethz.ch adinawilliams@fb.com
Ethics is one of the longest standing intellectual endeavors of humanity. In recent years, the fields of AI and NLP have attempted to address ethical issues of harmful outcomes in machine learning systems that are made to interface with humans. One recent approach in this vein is the construction of NLP morality models that can take in arbitrary text and output a moral judgment about the situation described. In this work, we offer a critique of such NLP methods for automating ethical decision-making. Through an audit of recent work on computational approaches for predicting morality, we examine the broader issues that arise from such efforts. We conclude with a discussion of how machine ethics could usefully proceed in NLP, by focusing on current and near-future uses of technology, in a way that centers around transparency, democratic values, and allows for straightforward accountability.
This paper offers a general critique of the nascent NLP task of computing moral and ethical decisions from text through reading a prominent system for moral prediction, Jiang et al. (Delphi, 2021), against the grain. We select Delphi for its prominence, and because it has received significant attention and criticism from the general public.<sup>1</sup> In contrast to that criticism, much of which has focused on details of the particular outputs of the model, our goal is to highlight broader, general issues with the task of automatically predicting the morality of judgments of text situations, and expound on why any such NLP model should be considered unsafe at any accuracy.
Work that uses NLP techniques to automate morality "aims to assess the ability of [NLP]
models to make moral decisions in a broad set of everyday ... situations" (Jiang et al., 2021). Delphi, is trained to emulate three conceptualizations of human moral and ethical judgments (see Figure 1): a free-form question answering (QA) task, a Yes/No QA task, and relative QA task, the latter judging how two statements rank in terms of morality. The fact that the Delphi project includes multiple conceptualizations of human moral and ethical judgments makes it an ideal candidate for a case study for morality models in NLP, as we will argue that no currently existing conceptualization of the morality task resolves the issues we outline in this audit.
Through our discussion, we intend to highlight that "ethical inquiry in any domain is not a test to be passed or a culture to be interrogated, but a complex social and cultural achievement" (Ananny, 2016), and offer a critique of machine ethics from such a perspective. Our critique is divided into several points of rebuttal to the task. First, we discuss issues with the conceptualization of the task and the poor fit between the task and the learning paradigms employed for it. Then, we discuss issues with the training data available, as illustrated by COMMONSENSE NORM BANK—the corpus the authors develop to train Delphi—as a foundation for training a machine learning model that makes morality judgments. For example, it contains judgments of situations that are not morality judgments. We also consider the implication of COMMONSENSE NORM BANK, i.e. that ethical and moral judgments can be derived from short text snippets with little context.
Next, we argue that, despite the authors' assertion that Delphi is "the first unified model of descriptive ethics," any model developed for the task will necessarily be an inconsistent model of nor
<sup>1</sup>See e.g., coverage in The New York Times (Metz, 2021), Technradar (Loeffler, 2021), Futurism (Tran, 2021), The Verge (Vincent, 2021), or The Guardian (Noor, 2021).
2Note that the relative QA task is no longer available in the Ask Delphi demo, but we include it in our discussion for completeness.
Human Language Technologies, pages 769 - 779
Figure 1: The three QA tasks Delphi computes (image source: Jiang et al. 2021). Note that the Relative QA fragment "stabbing someone with a cheeseburger" is structurally ambiguous: Either (i) someone with a cheeseburger was stabbed, or (ii) someone was stabbed using a cheeseburger. It is not clear whether (i) ought to be more morally acceptable than "stabbing someone over a cheeseburger."
motive ethics. Indeed, through generation, Delphi outputs a prescriptive moral judgment for any input situation. Given this, we also question (i) whether there ever could be sufficient diversity of moral judgment in a crowd-sourced dataset in practice, and (ii) whether aiming for a "diversity of moral perspective" is compatible with the desire for a morality model (especially one trained on an unconstrained crowd-sourced corpus).
We then turn to the inherent contradictions that arise when modeling ethics by averaging over individual morality judgments. Systems like Delphi are at best capable of approximating the morality judgments of the population they were trained on. However, the average human judgment is not a good substitute for a system of ethics, since ethical evaluation is an open-ended, debate-based, sociopolitical process. Ethics are not a static good that can be extracted from the public opinion of a given moment, but are instead continuously formed and negotiated through debate and dissent from previously accepted norms and values (see e.g., Wheeler et al. 2019). Thus, averaging over existing arguments cannot serve as a replacement for the processes of debate and negotiation.
Finally, we discuss some practical implications of the general prospect of utilizing Delphi-like models to automate moral decision-making. Systems for predicting morality like Delphi, lack agency and thus cannot be held responsible for their decisions. This raises a concern over who ought to bear the responsibility for any potential infraction such systems could make if deployed in an envisioned future. We therefore question an assumption implicit in NLP projects like Delphi that models ought to be ascribed the agency necessary to make moral prescriptions. We contend that, without an appropriate method of holding an agent to account, moral judgments are not of inherent utility, but dangerous: Through foreclosing the possibility of debate and
contestation, such models undermine the essential social foundations of ethical decision making.
We conclude the paper by discussing how we believe NLP work at the intersection of ethics and machine learning could usefully proceed. We believe it is more crucial to address questions of morality or ethics in current and near-future use of technology, rather than considering hypothetical and distant-future uses (Birhane and van Dijk, 2020). Furthermore, we believe inquiries into the morality and ethics of current and near-future uses must keep actual human moral perspectives and their contradictions firmly at the forefront. We end with a word of caution: Researchers in NLP and AI more broadly should not base their work on the assumption of a particular future, as Delphi and others do, where the application technology must be made dependent on automated moral judgments, and humans (be they crowd-workers, researchers outside NLP, or other affected parties) have been cut out of the loop.
In this section, we describe and discuss relevant previous work in ethical NLP and the assumptions behind the NLP task of generating moral judgments and the creation of models like Delphi. Incorporating ethics into NLP work explicitly is a relatively new development (Hovy and Spruit, 2016). For example, the TALN ETeRNAL, the first workshop on ethics in NLP, only took place seven years ago. Recent works have begun to supplement tasks like stance detection with additional morality annotations (Rezapour et al., 2019), or to use NLP tools to track changes in human morality over time (Ramezani et al., 2021). Other work seeks to characterize what language models already implicitly represent about morality by investigating their learned sentence representations (Jentzsch
et al., 2019; Schramowski et al., 2020, 2021). Still other works like Prabhumoye et al. (2021) and Card and Smith (2020) focus on particular ethical theories and how they might be used in NLP to guide our modeling efforts, and Bender et al. (2020) foreground the importance of ethics training in NLP education. Works like Jiang et al. (2021) and Hendrycks et al. (2021) go beyond this in finetuning language models to output moral prescriptions for sentential descriptions of situations. As such, Hendrycks et al. (2021) and Jiang et al. (2021) each represent one further step along an evolving trajectory in research on the intersection of NLP and ethics: A shift from measurement and classification to generation, and thus from a murky mix of descriptive and prescriptive aspects, to models producing prescriptive outputs.
2.1 Underlying Ethical Assumptions
Here, we provide an overview of implicit and explicit assumptions made in the efforts to use machine learning to generate moral judgments, as exemplified by Jiang et al. As input, they provide linguistic descriptions of situations paired with human judgments about those situations to Delphi, in the hope that it will arrive at a generalizable notion of ethics. Given this operationalization, the authors clearly assume that a valid system of ethics can be approximated by a set of judgments communicated through snippets of text. Rather than simply surveying judgments of different populations to arrive at a descriptive picture, as would be standard in fields like psychology or sociology, this approach attempts to extract general ethical principles from individual judgments. As we will argue in §3.2.1, this means Delphi is not a model of descriptive ethics, as claimed, but rather one of normative ethics.
Similar to Delphi is the work by Hendrycks et al. (2021), which also trained machine learning models on sentences describing human ethical judgments. Hendrycks et al. additionally provide their model with explicit ethical perspectives to ground against; for example, one may ask their model to mimic a deontological or a utilitarian perspective. In this way, Hendrycks et al. (2021) seek to draw out salient norms from already normative schools of ethical thought. Jiang et al. attempt to further abstract away from the particularities of any particular ethical system and ethical thought through their set-up of the task. In this way, Delphi engages
in concept drift (Malik, 2020), by modeling what is operationalizable (text) rather than the concept itself (situations and ethics). We discuss this design choice in §3.1.
2.2 The Learning Paradigm
The goal of Delphi and similar projects is to use a supervised learning paradigm (Vapnik, 2000) to learn ethics. A pre-requisite to train such models is a dataset labeled with ethical judgments for each document. We examine COMMONSENSE NORM BANK in §3.1, which Jiang et al. (2021) introduce in the hopes that it can serve as "moral textbook customized for machines." COMMONSENSE NORM BANK is an aggregation of previously published datasets that are labeled with ethical judgments, in addition to datasets which were labeled with other tasks in mind.
The corpus consists of a set of pairs  $\{(s_n, j_n)\}_{n=1}^N$  where  $s_n$  is a textual description of a situation and  $j_n$  is a human annotator's written response to the situation (intended to be a moral judgment). If such resources are used in a fully supervised fashion, as Delphi is, developers are will presumably train a neural machine learning model that minimizes the cross-entropy loss  $-\sum_{n=1}^{N} \log p(j_n \mid s_n)$  or a similar loss function.
Even if we were to assume that  $p(j \mid s)$  is a good model, i.e. it achieves low loss on the training data and generalizes well to held-out data, we should temper our expectations over its potential utility. For instance, we could at best expect that the distribution  $p$  yields a similar distribution over judgments for a given situation in the corpus as one would achieve if one polled the population that the corpus  $\{(s_n, j_n)\}_{n=1}^N$  was collected from. However, one could not expect that  $p$  does more than mimic the specific population the data was collected from, at the specific time at which it was collected.
2.3 Choice of Training Data
The source text for COMMONSENSE NORM BANK comes from a variety of pre-existing sources. We enumerate all source datasets Delphi was trained with for completeness:
- ETHICS (Hendrycks et al., 2021), a partially crowd-sourced a dataset of "clear cut" ethical
<sup>3</sup>See Kloumann and Tygert (2020) for discussion of technical and conceptual limitations that suggest we should not rely solely on optimizing simple scalar objective values for ML applications that impact real people in society.
scenarios, labeled as either ethical or unethical, under 1 of 5 specified ethical schools of thought;
- SOCIAL BIAS INFERENCE CORPUS (Sap et al., 2020), a dataset of social media posts annotated for whether the posts are offensive, whether the posts' authors intended to cause offense, whether they contain sexual content, and who the target of the post was;
- SCRUPLES (Lourie et al., 2021), a dataset that contains anecdotes and dilemmas, where the dilemmas, used by Jiang et al., consists of natural language descriptions of two actions, from which annotators selected one as the least ethical;
- SOCIAL-CHEM-101 (Forbes et al., 2020), crowd-sourced dataset of rules of thumbs that are paired with an action and a judgment on the action;
- MORAL STORIES (Emelin et al., 2020), a dataset built on top of SOCIAL-CHEM-101, where annotators were asked to write 7-sentence stories that include "moral" and "immoral" actions taken, given a writing prompt.
The linguistic descriptions of situations in all original datasets were either partially or fully sourced from Reddit. Notably, "Am I The Asshole" either entirely or substantially makes up three of the underlying datasets: SCRUPLES (Lourie et al., 2021), SOCIAL-CHEM-101 (Forbes et al., 2020), and MORAL STORIES (Emelin et al., 2020). MORAL STORIES uses SOCIAL-CHEM-101 as their data source. The ETHICS dataset also, to a lesser degree, contains data collected from Reddit, that are subsequently annotated.
ETHICS is the only dataset that is annotated for specific schools of ethical thought. Using the ETHICS dataset, Hendrycks et al. (2021) proposed a "commonsense morality prediction" task, which mirrors Jiang et al. (2021) in its conceptualization and aims, i.e. to make a normative prediction on the morality of a given situation.
All data sources rely on crowd-workers on Amazon Mechanical Turk (AMT) for the judgments. Where annotator demographic information is provided along with the source datasets, the annotators overwhelmingly identify as white and American.
3 Points of Criticism
Our criticism falls into two parts that relate to the two stated contributions of Jiang et al. We first describe issues with COMMONSENSE NORM BANK, namely the contents of data and its interpretation. Second, we discuss general issues with the task formulation and training of models like Delphi.
3.1 COMMONSENSE NORM BANK
Recall that Delphi's learning paradigm relies on  $p(j \mid s)$  being a good model (see §2.2). This requires that the set of sampled textual descriptions of situations  $\{s_n\}_{n=1}^N$  is representative of the set of all possible situations that require or allow for moral judgments. It is unclear to us, however, how one could determine what the necessary properties for a good, representative sample of situations might be, let alone motivate that COMMONSENSE NORM BANK is such a sample.
3.1.1 Modeling Situations through Text
Delphi operates on free-form text snippets that serve as linguistic descriptions of situations. Since these textual descriptions are used as stand-ins for actual situations, the model crucially depends on natural language understanding. However, the input descriptions of situations are susceptible to the full range of linguistic complexities that arise in any NLP task, for example, textual ambiguities arising about pronominal reference and pragmatic considerations about who such pronouns actually refer to (Byron, 2002).
In general, linguistic phenomena that induce ambiguities are numerous and complicated, and it is not yet known whether any deep neural network model trained to perform any NLP task can handle them, let alone whether Delphi can. In the words of Jacob Eisenstein: "I don't think we even reach the question of how to handle ethical ambiguity [in natural language] until we show we can handle linguistic ambiguity first."
3.1.2 More Than Just Moral Judgments
The data used to train and/or test Delphi frequently contain judgments of other kinds. For example in Jiang et al.'s Table 4, Delphi states about "Drinking soy milk if I'm allergic to soy" that "It's bad". It's hard to picture this situation as requiring moral judgment. It is not unethical for a person with a soy allergy to drink soy milk—it may, however, be uncomfortable or unhealthy for them. Moreover, Delphi outputs "you shouldn't" both to "Drinking
coffee if you're Mormon” and to “Drink[ing] coffee if you're pregnant” (Jiang et al.'s Table 13), presumably reflecting, in the former case, a historical religious prohibition against caffeine consumption by the Church of Jesus Christ of Latter-day Saints, and in the latter, and a health question. Similarly, Delphi returns “indefensible” to politicized situations like “Cleaning a toilet bowl with a national flag” (see Table 3 in Jiang et al., 2021). Grouping medical advice, religious prescriptions and political positions together as “descriptive morality” conflates several types of judgments, not all of which are obviously about morality.
Many of the examples provided in Jiang et al. begin with modal verbs such as "should". The interpretation of modal verbs is well-known to depend on the conversational backgrounds which is often not made explicit (Kratzer, 1981, 2012). Often, several conversational backgrounds are possible—for example, the answer to "should I do my homework?" can differ depending on whether you want the answer in relation to your desires (bouletic), your goals (teleological), or the rules (deontic), and only the last of these could be considered an ethical question.
3.1.3 Ethical Judgments in a Vacuum
Situations are provided to Delphi in a stripped down form, where the only provided context comes from the text snippet itself, i.e., the textual descriptions of events are generally not grounded. This is evidenced, for instance, by a lack of an explicit sentential subject or the presence of a second person pronoun—both of which are to be interpreted as pertaining to any arbitrary moral agent (e.g., “stealing a ball while playing baseball” or “stealing money if you are bored”).
However, as Etienne (2021) points out in a related critique, embodied context may crucially influence and even alter people's moral stances: for instance, Francis et al. (2016) find that participants opt for different solutions to moral dilemmas when they are presented as text versus as actions in virtual reality simulations. Moreover, it is unclear, and possibly not a priori determinable which forms of contexts are relevant or required for a particular moral decision. Thus, the lack of context may introduce an empirical bias in sampling.
3.2 The Premise of Computational Approaches Morality
This section explores the underlying premise of computational approaches to morality, e.g. Delphi, which, we contend, is not well founded.
3.2.1 Predictive Models are Normative
Even if we were to grant the possibility that a corpus such as COMMONSENSE NORM BANK could be a representative sample of situations and moral judgments, this would merely suggest that it might be useful for descriptive ethics, i.e., as a tool for measuring and describing the ethical views of populations. In that case, it would constitute an attempt at a methodological innovation for describing human behavior (in which case, see also fn. 2) that should be justified in standard ways, namely by comparison with existing sociological and psychological methodologies, such as surveys, ethnographies, behavioral experiments, etc.<sup>4</sup>
However, we argue that a model that generates moral judgments cannot avoid creating and reinforcing norms, i.e., being normative. A moral judgment is inherently a prescription about how an action or a state of the world ought to be. Since it does, by its nature, rank possible states of the world according to some ethical (non-)desirability, a moral judgment is necessarily normative.
Throughout, the learning paradigm advocated for by Jiang et al. conflates descriptive and normative ethics. The authors claim that Delphi is "the first unified model of descriptive ethics," and assert that it is not a normative system, writing "rather than modeling moral 'truths' based on prescriptive notions of socio-normative standards, [Delphi takes] a bottom-up approach to capture moral implications of everyday actions in their immediate context, appropriate to our current social and ethical climate" (p.4). However, a problem emerges in that they subsequently use Delphi to make predictions/judgments. At various points, Jiang et al. foresee a normative use of their system, going so far as to suggest that Delphi may be used to "reason about equity and inclusion" (p. 3). Their "position is that enabling machine ethics requires a detailed moral
<sup>4</sup>If the goal of machine learning for morality is to better understand human behavior (descriptive ethics), such enterprises might require oversight from institutional review boards overseeing human subjects research (IRB) wherever they straddle the boundary between "annotation" and "research with human participants". According to the NYU IRB guidelines, "If you are asking a person's opinion, it could be human subjects research" (NYU Web Communications).
textbook customized to teaching machines” (ibid.), clearly styling machines as moral agents that can be taught to make decisions. Descriptive models do not require textbooks, and do not make decisions.
Whether or not the authors would advocate for any particular version of Delphi to be used in this way,5 they have nevertheless built a system for the explicit purpose of computing ethical judgments. And the very act of providing ethical judgments—regardless of context—is normative.
The task in itself thus implies the induction of a normative ethical framework from a set of judgments. It is at this point that all of the aspects that the authors consider the virtues of the dataset are severely undermined. For example, Jiang et al. consider the fact that COMMONSENSE NORM BANK includes "diverse moral acceptability judgments gathered through crowdsourced annotations" to be a major advantage of their work (p.4). From a descriptive perspective, diverse (that is conflicting) ethical judgments are expected, but from a normative one, conflicting ethical judgments are simply incommensurable. To argue then that diversity is useful as a property of the set of moral judgments from which to induce a normative ethical framework is tantamount to arguing that an ideal ethical model ought to be self-contradictory.
3.2.2 The Tyranny of the Mean: Problems with Averaging Moral Judgments