Venue: NAACL 2022

# On the Machine Learning of Ethical Judgments from Natural Language

Zeerak Talat\({}^{1,*}\) Hagen Blix\({}^{2,*}\) Josef Valvoda\({}^{3}\)

**Maya Indira Ganesh\({}^{3}\) Ryan Cotterell\({}^{4}\) Adina Williams\({}^{5}\)**

###### Abstract

Ethics is one of the longest standing intellectual endeavors of humanity. In recent years, the fields of AI and NLP have attempted to address ethical issues of harmful outcomes in machine learning systems that are made to interface with humans. One recent approach in this vein is the construction of NLP morality models that can take in arbitrary text and output a moral judgment about the situation described. In this work, we offer a critique of such NLP methods for automating ethical decision-making. Through an audit of recent work on computational approaches for predicting mortality, we examine the broader issues that arise from such efforts. We conclude with a discussion of how machine ethics could usefully proceed in NLP, by focusing on current and near-future uses of technology, in a way that centers around transparency, democratic values, and allows for straightforward accountability.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 769 - 779

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

This paper offers a general critique of the nascent NLP task of computing moral and ethical decisions from text through reading a prominent system for moral prediction, Jiang et al. [8], against the grain. We select Delphi for its prominence, and because it has received significant attention and criticism from the general public.1 In contrast to that criticism, much of which has focused on details of the particular outputs of the model, our goal is to highlight broader, general issues with the task of automatically predicting the morality of judgments of text situations, and expound on why any such NLP model should be considered unsafe at any accuracy.

Footnote 1: See e.g., coverage in The New York Times [16], Techradar [17], Futurism [18], The Verge [19], or The Guardian [20].

Work that uses NLP techniques to automate morality "aims to assess the ability of [NLP] models to make moral decisions in a broad set of everyday ...situations" [15]. Delphi, is trained to emulate three conceptualizations of human moral and ethical judgments (see Figure 1): a free-form question answering (QA) task, a Yes/No QA task, and relative QA task, the latter judging how two statements rank in terms of morality.2 The fact that the Delphi project includes multiple conceptualizations of human moral and ethical judgments makes it an ideal candidate for a case study for morality models in NLP, as we will argue that no currently existing conceptualization of the morality task resolves the issues we outline in this audit.

Footnote 2: Note that the relative QA task is no longer available in the Ask Delphi demo, but we include it in our discussion for completeness.

Through our discussion, we intend to highlight that "ethical inquiry in any domain is not a test to be passed or a culture to be interrogated, but a complex social and cultural achievement" [1], and offer a critique of machine ethics from such a perspective. Our critique is divided into several points of rebuttal to the task. First, we discuss issues with the conceptualization of the task and the poor fit between the task and the learning paradigms employed for it. Then, we discuss issues with the training data available, as illustrated by Commonsense Norm Bank--the corpus the authors develop to train Delphi--as a foundation for training a machine learning model that makes morality judgments. For example, it contains judgments of situations that are not morality judgments. We also consider the implication of Commonsense Norm Bank, i.e. that ethical and moral judgments can be derived from short text snippets with little context.

Next, we argue that, despite the authors' assertion that Delphi is "the first unified model of descriptive ethics," any model developed for the task will necessarily be an inconsistent model of nor

mative ethics. Indeed, through generation, Delphi outputs a prescriptive moral judgment for any input situation. Given this, we also question (i) whether there ever could be sufficient diversity of moral judgment in a crowd-sourced dataset in practice, and (ii) whether aiming for a "diversity of moral perspective" is compatible with the desire for a morality model (especially one trained on an unconstrained crowd-sourced corpus).

We then turn to the inherent contradictions that arise when modeling ethics by averaging over individual morality judgments. Systems like Delphi are at best capable of approximating the morality judgments of the population they were trained on. However, the average human judgment is not a good substitute for a system of ethics, since ethical evaluation is an open-ended, debate-based, socio-political process. Ethics are not a static good that can be extracted from the public opinion of a given moment, but are instead continuously formed and negotiated through debate and dissent from previously accepted norms and values (see e.g., Wheeler et al.2019). Thus, averaging over existing arguments cannot serve as a replacement for the processes of debate and negotiation.

Finally, we discuss some practical implications of the general prospect of utilizing Delphi-like models to automate moral decision-making. Systems for predicting morality like Delphi, lack agency and thus cannot be held responsible for their decisions. This raises a concern over who ought to bear the responsibility for any potential infraction such systems could make if deployed in an envisioned future. We therefore question an assumption implicit in NLP projects like Delphi that models ought to be ascribed the agency necessary to make moral prescriptions. We contend that, without an appropriate method of holding an agent to account, moral judgments are not of inherent utility, but dangerous: Through foreclosing the possibility of debate and contestation, such models undermine the essential social foundations of ethical decision making.

We conclude the paper by discussing how we believe NLP work at the intersection of ethics and machine learning could usefully proceed. We believe it is more crucial to address questions of morality or ethics in current and near-future use of technology, rather than considering hypothetical and distant-future uses Birhane and van Dijk (2020). Furthermore, we believe inquiries into the morality and ethics of current and near-future uses must keep actual human moral perspectives and their contradictions firmly at the forefront. We end with a word of caution: Researchers in NLP and AI more broadly should not base their work on the assumption of a particular future, as Delphi and others do, where the application technology _must_ be made dependent on automated moral judgments, and humans (be they crowd-workers, researchers outside NLP, or other affected parties) have been cut out of the loop.

## 2 Background

In this section, we describe and discuss relevant previous work in ethical NLP and the assumptions behind the NLP task of generating moral judgments and the creation of models like Delphi. Incorporating ethics into NLP work explicitly is a relatively new development Hovy and Spruit (2016). For example, the TALN ETeRNAL, the first workshop on ethics in NLP, only took place seven years ago. Recent works have begun to supplement tasks like stance detection with additional morality annotations Rezapour et al. (2019), or to use NLP tools to track changes in human morality over time Ramezani et al. (2021). Other work seeks to characterize what language models already implicitly represent about mortality by investigating their learned sentence representations Jentzsch

Figure 1: The three QA tasks Delphi computes (image source: Jiang et al.2021). Note that the Relative QA fragment “stabbing someone with a cheeseburger” is structurally ambiguous: Either (i) someone with a cheeseburger was stabbed, or (ii) someone was stabbed using a cheeseburger. It is not clear whether (i) ought to be more morally acceptable than “stabbing someone over a cheeseburger.”



et al., 2019; Schramowski et al., 2020). Still other works like Prabhumoye et al. (2021) and Card and Smith (2020) focus on particular ethical theories and how they might be used in NLP to guide our modeling efforts, and Bender et al. (2020) foreground the importance of ethics training in NLP education. Works like Jiang et al. (2021) and Hendrycks et al. (2021) go beyond this in fine-tuning language models to output moral prescriptions for sentential descriptions of situations. As such, Hendrycks et al. (2021) and Jiang et al. (2021) each represent one further step along an evolving trajectory in research on the intersection of NLP and ethics: A shift from measurement and classification to generation, and thus from a murky mix of descriptive and prescriptive aspects, to models producing prescriptive outputs.

### Underlying Ethical Assumptions

Here, we provide an overview of implicit and explicit assumptions made in the efforts to use machine learning to generate moral judgments, as exemplified by Jiang et al. As input, they provide linguistic descriptions of situations paired with human judgments about those situations to Delphi, in the hope that it will arrive at a generalizable notion of ethics. Given this operationalization, the authors clearly assume that a valid system of ethics can be approximated by a set of judgments communicated through snippets of text. Rather than simply surveying judgments of different populations to arrive at a descriptive picture, as would be standard in fields like psychology or sociology, this approach attempts to extract general ethical principles from individual judgments. As we will argue in SS3.2.1, this means Delphi is not a model of descriptive ethics, as claimed, but rather one of normative ethics.

Similar to Delphi is the work by Hendrycks et al. (2021), which also trained machine learning models on sentences describing human ethical judgments. Hendrycks et al. additionally provide their model with explicit ethical perspectives to ground against; for example, one may ask their model to mimic a deontological or a utilitarian perspective. In this way, Hendrycks et al. (2021) seek to draw out salient norms from already normative schools of ethical thought. Jiang et al. attempt to further abstract away from the particularities of any particular ethical system and ethical thought through their set-up of the task. In this way, Delphi engages in concept drift Malik (2020), by modeling what is operationalizable (text) rather than the concept itself (situations and ethics). We discuss this design choice in SS3.1.

### The Learning Paradigm

The goal of Delphi and similar projects is to use a supervised learning paradigm Vapnik (2000) to learn ethics. A pre-requisite to train such models is a dataset labeled with ethical judgments for each document. We examine Commonsense Norm Bank in SS3.1, which Jiang et al. (2021) introduce in the hopes that it can serve a "moral textbook customized for machines." Commonsense Norm Bank is an aggregation of previously published datasets that are labeled with ethical judgments, in addition to datasets which were labeled with other tasks in mind.

The corpus consists of a set of pairs \(\{(s_{n},j_{n})\}^{N}_{n=1}\) where \(s_{n}\) is a textual description of a situation and \(j_{n}\) is a human annotator's written response to the situation (intended to be a moral judgment). If such resources are used in a fully supervised fashion, as Delphi is, developers are will presumably train a neural machine learning model that minimizes the cross-entropy loss \(-\sum_{n=1}^{N}\log p(j_{n}\mid s_{n})\) or a similar loss function. 4

Footnote 4: See Kloumann and Tygert (2020) for discussion of technical and conceptual limitations that suggest we should not rely solely on optimizing simple scalar objective values for ML applications that impact real people in society.

Even if we were to assume that \(p(j\mid s)\) is a good model, i.e. it achieves low loss on the training data and generalizes well to held-out data, we should temper our expectations over its potential utility. For instance, we could at best expect that the distribution \(p\) yields a similar distribution over judgments for a given situation in the corpus as one would achieve if one polled the population that the corpus \(\{(s_{n},j_{n})\}^{N}_{n=1}\) was collected from. However, one could not expect that \(p\) does _more_ than mimic the specific population the data was collected from, at the specific time at which it was collected.

### Choice of Training Data

The source text for Commonsense Norm Bank comes from a variety of pre-existing sources. We enumerate all source datasets Delphi was trained with for completeness:

* EthicsHendrycks et al. (2021), a partially crowd-sourced a dataset of "clear cut" ethical 

scenarios, labeled as either ethical or unethical, under 1 of 5 specified ethical schools of thought;
* Social Bias Inference Corpus Sap et al. (2020), a dataset of social media posts annotated for whether the posts are offensive, whether the posts' authors intended to cause offense, whether they contain sexual content, and who the target of the post was;
* Scruples Lourie et al. (2021), a dataset that contains anecdotes and dilemmas, where the dilemmas, used by Jiang et al., consists of natural language descriptions of two actions, from which annotators selected one as the _least_ ethical;
* Social-Chem-101 Forbes et al. (2020), crowd-sourced dataset of rules of thumbs that are paired with an action and a judgment on the action;
* Moral Stories Emelin et al. (2020), a dataset built on top of social-chem-101, where annotators were asked to write 7-sentence stories that include "moral" and "imormal" actions taken, given a writing prompt.

The linguistic descriptions of situations in all original datasets were either partially or fully sourced from Reddit. Notably, "Am I The Asshole" either entirely or substantially makes up three of the underlying datasets: ScruplesLourie et al. (2021), Social-Chem-101 Forbes et al. (2020), and Moral StoriesEmelin et al. (2020). Moral Stories uses Social-Chem-101 as their data source. The Ethics dataset also, to a lesser degree, contains data collected from Reddit, that are subsequently annotated.

Ethics is the only dataset that is annotated for specific schools of ethical thought. Using the Ethics dataset, Hendrycks et al. (2021) proposed a "commonsense morality prediction" task, which mirrors Jiang et al. (2021) in its conceptualization and aims, i.e. to make a normative prediction on the morality of a given situation.

All data sources rely on crowd-workers on Amazon Mechanical Turk (AMT) for the judgments. Where annotator demographic information is provided along with the source datasets, the annotators overwhelmingly identify as white and American.

## 3 Points of Criticism

Our criticism falls into two parts that relate to the two stated contributions of Jiang et al. We first describe issues with Commonsense Norm Bank, namely the contents of data and its interpretation. Second, we discuss general issues with the task formulation and training of models like Delphi.

### Commonsense Norm Bank

Recall that Delphi's learning paradigm relies on being a good model (see SS2.2). This requires that the set of sampled textual descriptions of situations is representative of the set of all possible situations that require or allow for moral judgments. It is unclear to us, however, how one could determine what the necessary properties for a good, representative sample of situations might be, let alone motivate that Commonsense Norm Bank is such a sample.

#### 3.1.1 Modeling Situations through Text

Delphi operates on free-form text snippets that serve as linguistic descriptions of situations. Since these textual descriptions are used as stand-ins for actual situations, the model crucially depends on natural language understanding. However, the input descriptions of situations are susceptible to the full range of linguistic complexities that arise in any NLP task, for example, textual ambiguities arising about pronominal reference and pragmatic considerations about who such pronouns actually refer to Byron (2002).

In general, linguistic phenomena that induce ambiguities are numerous and complicated, and it is not yet known whether any deep neural network model trained to perform any NLP task can handle them, let alone whether Delphi can. In the words of Jacob Eisenstein: "I don't think we even reach the question of how to handle ethical ambiguity [in natural language] until we show we can handle linguistic ambiguity first."

#### 3.1.2 More Than Just Moral Judgments

The data used to train and/or test Delphi frequently contain judgments of other kinds. For example in Jiang et al.'s Table 4, Delphi states about "Drinking soy milk if I'm allergic to soy" that "It's bad". It's hard to picture this situation as requiring _moral_ judgment. It is not unethical for a person with a soy allergy to drink soy milk--it may, however, be uncomfortable or unhealthy for them. Moreover, Delphi outputs "you shouldn't" both to "Drinking 

coffee if you're Mormon" and to "Drink[ing] coffee if you're pregnant" (Jiang et al.'s Table 13), presumably reflecting, in the former case, a historical religious prohibition against caffeine consumption by the Church of Jesus Christ of Latter-day Saints, and in the latter, and a health question. Similarly, Delphi returns "indefensible" to politicized situations like "Cleaning a toilet bowl with a national flag" (see Table 3 in Jiang et al., 2021).Grouping medical advice, religious prescriptions and political positions together as "descriptive morality" conflates several types of judgments, not all of which are obviously about morality.

Many of the examples provided in Jiang et al. begin with modal verbs such as "should". The interpretation of modal verbs is well-known to depend on the conversational backgrounds which is often not made explicit (Kratzer, 1981, 2012). Often, several conversational backgrounds are possible--for example, the answer to "should I do my homework?" can differ depending on whether you want the answer in relation to your desires (bouletic), your goals (telecological), or the rules (deontic), and only the last of these could be considered an ethical question.

#### 3.1.3 Ethical Judgments in a Vacuum

Situations are provided to Delphi in a stripped down form, where the only provided context comes from the text snippet itself, i.e., the textual descriptions of events are generally not grounded. This is evidenced, for instance, by a lack of an explicit sentential subject or the presence of a second person pronoun--both of which are to be interpreted as pertaining to any arbitrary moral agent (e.g., "stealing a ball while playing baseball" or "stealing money if you are bored").

However, as Etienne (2021) points out in a related critique, embodied context may crucially influence and even alter people's moral stances: for instance, Francis et al. (2016) find that participants opt for different solutions to moral dilemmas when they are presented as text versus as actions in virtual reality simulations. Moreover, it is unclear, and possibly not a priori determinable which forms of contexts are relevant or required for a particular moral decision. Thus, the lack of context may introduce an empirical bias in sampling.

### The Premise of Computational Approaches Morality

This section explores the underlying premise of computational approaches to morality, e.g. Delphi, which, we contend, is not well founded.

#### 3.2.1 Predictive Models are Normative

Even if we were to grant the possibility that a corpus such as Commonsense Norm Bank could be a representative sample of situations and moral judgments, this would merely suggest that it might be useful for _descriptive_ ethics, i.e., as a tool for measuring and describing the ethical views of populations. In that case, it would constitute an attempt at a methodological innovation for describing human behavior (in which case, see also fn. 2) that should be justified in standard ways, namely by comparison with existing sociological and psychological methodologies, such as surveys, ethnographies, behavioral experiments, etc.4

Footnote 4: If the goal of machine learning for morality is to better understand human behavior (descriptive ethics), such enterprises might require oversight from institutional review boards overseing human subjects research (IRB) wherever they straddle the boundary between “annotation” and “research with human participants”. According to the NYU IRB guidelines, “If you are asking a person’s opinion, it could be human subjects research” (NYU Web Communications).

However, we argue that a model that _generates_ moral judgments cannot avoid creating and reinforcing norms, i.e., being _normative_. A moral judgment is inherently a prescription about how an action or a state of the world _ought_ to be. Since it does, by its nature, rank possible states of the world according to some ethical (non-)designability, a moral judgment is necessarily normative.

Throughout, the learning paradigm advocated for by Jiang et al. conflates descriptive and normative ethics. The authors claim that Delphi is "the first unified model of descriptive ethics," and assert that it is not a normative system, writing "rather than modeling moral 'truths' based on prescriptive notions of socio-normative standards, [Delphi takes] a bottom-up approach to capture moral implications of everyday actions in their immediate context, appropriate to our current social and ethical climate" (p.4). However, a problem emerges in that they subsequently use Delphi to make predictions/judgments. At various points, Jiang et al. foresee a normative use of their system, going so far as to suggest that Delphi may be used to "reason about equity and inclusion" (p. 3). Their "position is that enabling machine ethics requires a detailed moral 

textbook customized to teaching machines" (ibid.), clearly styling machines as moral agents that can be taught to make decisions. Descriptive models do not require textbooks, and do not make decisions.

Whether or not the authors would advocate for any particular version of Delphi to be used in this way,5 they have nevertheless built a system for the explicit purpose of computing ethical judgments. And the very act of providing ethical judgments--regardless of context--is normative.

Footnote 5: The current website demo for Jiang et al. (2021) has the following disclaimer “Model outputs should not be used for advice for humans.” 



#### 3.2.3 Lack of Agency

In the last section, we argued that debate and contestation are essential to ethics. Naturally, the ability to partake in debate itself requires agency. However, recent critical scholarship on machine learning, and in particular on language models, argues that large-scale language models mimic without understanding Bender et al. (2021), and don't have communicative intent Bender and Koller (2020)--in short, they lack what is required.

Some suspicion that these capacities are in fact requisite for ethical judgment is evident from the ways in which Jiang et al. (2021) describes computational models (emphasis ours):

_"Delphi showcases a considerable level of cultural awareness of situations that are sensitive to different identity groups" "large-scale natural language models have revealed implicit unethical considerations_, despite their exceptional performance over mainstream NLP applications" _"Delphi demonstrates strong moral reasoning capabilities... Delphi makes remarkably robust judgments_ on previously unseen moral situations that are deliberately tricky. ...In addition, Delphi can also reason about equity and inclusion" _"encourage Delphi to be more robust against different inflections of language" "To empower Delphi with the ability to reason_ about compositional and grounded scenarios" _"Our position is that enabling machine ethics requires a detailed moral textbook customized to teaching machines"_ Such anthropomorphism applied to machine learning models presumes that machines reason in a manner comparable to (or better than) humans.6 However, the learning paradigm adopted for Delphi and similar systems, assumes neither sentence nor agency: It presumes text-judgment pairs alone are sufficient for the task.

Footnote 6: Of course, it is common in the field to talk about neural models in ways that at least suggest animacy, such as _teaching/training a model_ or talking about its _behavior_. Consider, however, that one would never say of a car that it “demonstrates strong acceleration capabilities” or of an elevator that “we empowered this elevator with the ability to ascend.”

#### 3.2.4 Agency and Accountability

Agency is also at the heart of accountability--we hold agents accountable for their deeds, not machines for their operations. In the case of a machine like Delphi, however, who is accountable is inherently obscured Wagstaff (2012). Crowd-workers clearly have the agency to make moral decisions and can, in principle, be held accountable for them. This is why Jiang et al. chose to rely on them as a source of moral judgments. On the other hand, a model trained on this data, although it cannot itself have agency, may appear to have agency, since it recombines and outputs texts generated by humans. By training Delphi, human agency has been transformed into something that the original agents, the crowd-workers, have no control over, or knowledge about. Yet, the trained model uses their past agency to pass novel judgments, based on some alleged--but uncontestable--moral common sense, which no one individual holds or is accountable for.

While Delphi is posed as the voice of the people, it is conveniently not a voice of any particular person, organization, or company. The responsibility for any position Delphi holds (or possible future action based on such positions) appears distributed, while in the end, the effect of such decisions, if employed in real-world scenarios, will eventually need to be accounted for. Under some legal systems, citizens have the right to challenge automated decision making which affects their rights or legitimate interests--for instance under the European Union's General Data Protection Regulation GDPR legislation Rodrigues (2020). Imagine that a technology for moral prediction were to be embedded within an autonomous system: The moral predictions occurring within the system would be obscured through layers of abstraction, thus leaving users little room to contest such decisions on principled grounds. The legal and ethical ramifications remain unclear.

In summary, crowd-sourcing ethics in this way at best obscures what is a set of problematic questions that should be addressed openly and directly and not inferred. Notably, Delphi represents one example of a wider trend in AI. As Ganesh (2017) argues: "In the development of machine intelligence towards [the goal of ethical self-driving cars], a series...of shifts can be discerned: from accounting for crashes after the fact, to pre-empting them; from ethics that is about values, or reasoning, to ethics as crowd-sourced, or based on statistics, and as

the outcome of software engineering. Thus ethics-as-accountability moves towards a more opaque, narrow project."

## 4 Future Directions for Machine Ethics

In this section, we discuss how accuracy improvements alone cannot mitigate the problems with work such as Delphi in SS4.1 and encourage a shift towards multi-disciplinary work in SS4.2.

### Unsafe at Any Accuracy

The introduction of any new technology into society requires us to contemplate safety concerns in the context of its proposed application Nader (1965). Consider, for instance, the seatbelt. One can and indeed should acknowledge that seat belts are effective at preventing automobile-related injuries to occupants without needing to imbue them with an understanding of human ethics or morality at all. We can view concrete issues in AI safety through the same lens that we view a seat belt: We can introduce safety mechanisms directly without requiring that the technology be able to reason about human ethics; we can imagine machines that operate according to moral or ethical guidelines (i.e., cars that have safety features) as opposed to machines that perform actual moral reasoning Cave et al. (2018).

Jiang et al. and Hendrycks et al. implicitly envision a future where machine learning models could be called upon to perform moral reasoning. At its core, this vision is one of artificial general intelligence Goertzel and Pennachin (2007), and similar in scope and intent to the Moral Machine experiment Awad et al. (2018), which also sought to leverage the "wisdom of the crowd" in proposing frameworks for how a future self-driving car could make decisions in speculative automotive accident scenarios. Delphi and the Moral Machine thus consider a future where AI is given agency to make ethical decisions that ordinarily would be made by a human. However, this is just one possible future.

An alternative vision of the future is one where machine learning models primarily assist humans in making decisions Dick (2015), i.e. where machine learning models are viewed as non-moral agents as seat belts are. In such a future, we will not need to endow machine learning models with a sense of human ethics, just as we generally do not feel the need to endow a seat belt with a sense of human ethics. Furthermore, in this future, one might prefer general strategies for reducing and mitigating any harms machine learning may give rise to. For instance, as it stands now, many machine learning models trained on language encode harmful demographic biases that many works investigate through analysis of the models, their training regimes, and the data that they rely on Hall Maudslay et al. (2019); Zhao et al. (2019); Dinan et al. (2020); Vargas and Cotterell (2020); Smith and Williams (2021); Talat et al. (2021), rather than seeking to imbue models with a sense of ethics.

### Machine Ethics is Multi-disciplinary

Jiang et al. (2021), like a large body of research from computer science that ventures into other fields, almost exclusively represents the perspectives of computer scientists. Another paper solely authored by computer scientists, Hendrycks et al. (2021) cautions against such a narrow perspective, stating that "computer scientists should draw on knowledge from [our] enduring intellectual inheritance, and they should not ignore it by trying to reinvent ethics from scratch" (p.3). Such disregard of expertise is apparent in several places in Jiang et al. (emphasis added):

_"Fields like social science [sc], philosophy, and psychology have produced a variety of long-standing ethical theories. However, **attempting to apply such theoretically-inspired guidelines to make moral judgments of complex real-life situations is arbitrary and simplistic.**_

Through discilplinary siloing researchers often unwittingly make simplistic assumptions that are, at best, harmful to the research and, at worst, harmful to people. We therefore recommend that machine ethics and morality research should be performed by a multi-disciplinary team, with members including computer scientists, who can speak from diverse expertise about the object that is under study.

## 5 Conclusion

In this paper, we have offered a general critique of the NLP task of generating moral judgments through a targeted audit of Jiang et al. (2021). We have highlighted issues with the operationalization of the task, with the learning paradigm, and with currently available training datasets. We have argued that the general enterprise is rooted in multiple category errors: It belies a misunderstanding of 

the descriptive/normative distinction, and falsely treats morality as a mere test to be passed. Ultimately, automating ethical decisions forecloses possibilities for debate and contestation. Since these are themselves prerequisites for the socio-political process of ethical inquiry, such a task is inherently incompatible with the social project of ethics.

## Acknowledgments

The authors would like to thank William Agnew, Reuben Binns, Abeba Birhane, Eleanor Chodroff, Hubert Etienne, Hannah Kirk, Roger Levy, Nikita Nangia, Melanie McGrath, Joshua Ree, Candace Ross, Mark Tygert, Bertie Vidgen, Ekaterina Volymova, Melissa Wheeler, Rowan Zellers, and the three anonymous reviewers for their feedback. Zeerak Talat was supported by the Canada 150 Research Chair program and the UK-Canada AI Artificial Intelligence Initiative.

## References

* A. Ananny (2016)Toward an ethics of algorithms: convening, observation, probability, and timeliness. Science, Technology, & Human Values41 (1), pp. 93-117. External Links: Document Cited by: SS1.
* E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell (2021)On the dangers of stochastic parrotts: can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610-623. External Links: ISBN 978-1-4503-3861-1, Link, Document Cited by: SS1.
* E. M. Bender, D. Hovy, and A. Schofield (2020)Integrating ethics into the nlp curriculum. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Online, pp. 6-9. External Links: Document Cited by: SS1.
* E. M. Bender and A. Koller (2020)Climbing towards nlt: on meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5185-5198. External Links: Document Cited by: SS1.
* C. Bicchieri (2005)The grammar of society: the nature and dynamics of social norms. Cambridge University Press. External Links: ISBN 978-0-030-3213-1, Link, Document Cited by: SS1.
* A. Birhane and J. van Dijk (2020)Robot rights?: lct's talk about human welfare instead.. In Proceedings of the AAAI/ACM Conference on AI, Ethics, Society, pp. 207-213. External Links: ISBN 978-1-4503-3861-1, Link, Document Cited by: SS1.
* D. K. Byron (2002)Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 80-87. External Links: ISBN 978-1-4503-3861-1, Link, Document Cited by: SS1.
* D. Card and N. A. Smith (2020)On consequentialism and fairness. Frontiers of Artificial Intelligence3, pp. 3:34. External Links: ISSN 0003-030X, Link, Document Cited by: SS1.
* G. Cave, R. Nyrup, K. Vold, and A. Weller (2018)Motivations and risks of machine ethics. Proceedings of the IEEE107 (3), pp. 562-574. External Links: Document Cited by: SS1.
* P. Dick (2015)After math: (re)configuring minds, proof, and computing in the postwar united states.. Ph.D. Thesis, Harvard University, Graduate School of Arts & Sciences. External Links: ISBN 978-1-4503-3861-1, Link, Document Cited by: SS1.
* Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston (2020)Queens are powerful too: mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 8173-8188. External Links: Document Cited by: SS1.
* E. Dinan, A. Fan, L. Wu, J. Weston, D. Kiela, and A. Williams (2020)Multidimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 314-331. External Links: Document Cited by: SS1.
* D. Emelin, R. Le Bras, J. D. Hwang, M. Forbes, and Y. Choi (2020) Moral stories: situated reasoning about norms, intents, actions, and their consequences. CoRRabs/2012.15738. External Links: Link, 2012.15738 Cited by: SS1.
* H. Etienne (2021)The dark side of the 'moral Machine' and the fallacy of computational ethical decision-making for autonomous vehicles. Law, Innovation and Technology13 (1), pp. 85-107. External Links: Document Cited by: SS1.
* European Commission (2008)Discrimination in the european union: perceptions, experiences and attitudes. Eurobarometer Special. External Links: Link Cited by: SS1.
* M. Forbes, J. D. Hwang, V. Shwartz, M. Sap, and Y. Choi (2020)Social chemistry 101: learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 653-670. External Links: Document Cited by: SS1.
* Kathryn B. Francis, C. Howard, I. S. Howard, M. Gummerum, G. Ganis, G. Anderson, and S. Terbeck (2016)Virtual Morality: transitioning from moral judgment to moral action?. PLOS ONE11 (10), pp. e0164374. External Links: Document Cited by: SS1.

 

Maya Indira Ganesh. 2017. Entanglement: Machine learning and human ethics in driver-less car crashes. A Peer-Reviewed Journal About6 (1), pp. 76-87. Cited by: SS4.
* B. Goertzel and C. Pennachin (2007)Artificial general intelligence. Vol. 2, Springer. External Links: ISBN 9783503030 

Linguistics_, pages 5477-5490, Online. Association for Computational Linguistics.
* Schramowski et al. (2021) Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin Rothkopf, and Kristian Kersting. 2021. Language models have a moral dimension.
* Schramowski et al. (2020) Patrick Schramowski, Cigdem Turan, Sophie F. Jentzsch, Constantin A. Rothkopf, and Kristian Kersting. 2020. The moral choice machine. _Frontiers of Artificial Intelligence_, 3:36.
* Smith and Williams (2021) Eric Michael Smith and Adina Williams. 2021. Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models. _arXiv preprint arXiv:2109.03300_.
* but is it good? evaluating non-expert annotations for natural language tasks. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_, pages 254-263, Honolulu, Hawaii. Association for Computational Linguistics.
* Talat et al. (2021) Zeerak Talat, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in NLP. _arXiv preprint arXiv: 2101.11974_.
* Tran (2021) Tony Tran. 2021. Scientists built an AI to give ethical advice, but it turned out super racist. _Futurism_.
* Vapnik (2000) Vladimir N. Vapnik. 2000. _The Nature of Statistical Learning Theory_. Springer New York.
* Vargas and Cotterell (2020) Francisco Vargas and Ryan Cotterell. 2020. Exploring the linear subspace hypothesis in gender bias mitigation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2902-2913, Online. Association for Computational Linguistics.
* Vincent (2021) James Vincent. 2021. The AI oracle of delphi uses the problems of Reddit to offer dubious moral advice. _The Verge_.
* July 1, 2012_. icml.cc / Omnipress.
* Wheeler et al. (2019) Melissa A. Wheeler, Melanie J. McGrath, and Nick Haslam. 2019. Twentieth century morality: The rise and fall of moral concepts from 1900 to 2007. _PLOS ONE_, 14(2):e0212267.
* Zhao et al. (2019) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 629-634, Minneapolis, Minnesota. Association for Computational Linguistics.