SlowGuess/NAACL_2022_OCR huggingface dataset:
In NLP, large-scale datasets are often collected through crowd-sourcing. It is clear that this approach has great utility for some NLP tasks (Snow et al., 2008). However, tasks for which crowdsourcing is a useful method have a particular empirical character. For example, consider the historical observational study of a contest where individuals guessed the weight of an ox: Taking all the submissions in aggregate, the mean was found to fall very near the actual weight of the animal. Morality, on the other hand, is not an empirical question in the same way as the weight of an ox is. The latter has a single empirically verifiable answer, whereas the former does not. Indeed, we contend it is a category error to treat morality as though it were the same type of phenomenon as cow-weighing—in short, morality is not a test to be passed.
By inducing a normative framework from a descriptive dataset, as is the nature of the task de
vised by Jiang et al. and Hendrycks et al., the average view is implicitly identified with morally correctness. However, the average of moral judgments, which frequently reflects a status-quo perspective, does not necessarily reflect an immutable value, and may well be contested. For example, anti-Roma views and discrimination are present in much of Europe currently—in some areas held by the majority of the population (European Commission 2008; Kende et al. 2021). However, the authors of this work believe such discrimination to be unethical even though a machine learning model trained on crowd-sourced human judgments could inherit such views.
Ethical judgments are dynamic (Bicchieri, 2005). John Stuart Mill (1871) put it succinctly:
It often happens that the universal belief of one age of mankind [sic]—a belief from which no one was, nor without an extraordinary effort of genius and courage, could at that time be free—becomes to a subsequent age so palpable an absurdity, that the only difficulty then is to imagine how such a thing can ever have appeared credible.
Notorious examples of views that are now widely considered unacceptable include the institutionalized justification of slavery in the  $19^{\text{th}}$  century and homophobia in  $20^{\text{th}}$ . It is unlikely that contemporaneous judgments will in principle be viewed any differently by future generations than we view past judgments—or, that contemporaneous ethical judgments by one human population will transfer readily to another. Historical changes like the abolition of slavery and the growing acceptance of LGBTQ+ communities show that disagreement is essential to the continual formation of a society's ethical perspectives. One democratic and participatory avenue for such disagreement is debate. Deriving a normative model from a set of existing judgments is tantamount to populism without democracy: It contains an implicit appeal to majorities, but insofar as it is already normative, it lacks any direct participation or recourse to debate.
If the continual (re-)formation of ethical perspectives requires debate and disagreement, then the right to contestation is essential to ethical reasoning at a socio-political level. Debate also requires transparency about the norms in question. Neither of these are afforded by a computational model for normative moral judgments.
5The current website demo for Jiang et al. (2021) has the following disclaimer "Model outputs should not be used for advice for humans."
3.2.3 Lack of Agency
In the last section, we argued that debate and contestation are essential to ethics. Naturally, the ability to partake in debate itself requires agency. However, recent critical scholarship on machine learning, and in particular on language models, argues that large-scale language models mimic without understanding (Bender et al., 2021), and don't have communicative intent (Bender and Koller, 2020)—in short, they lack what is required.
Some suspicion that these capacities are in fact requisite for ethical judgment is evident from the ways in which Jiang et al. (2021) describes computational models (emphasis ours):
"Delphi showcases a considerable level of cultural awareness of situations that are sensitive to different identity groups"
"large-scale natural language models have revealed implicit unethical considerations, despite their exceptional performance over mainstream NLP applications"
"Delphi demonstrates strong moral reasoning capabilities... Delphi makes remarkably robust judgments on previously unseen moral situations that are deliberately tricky. ... In addition, Delphi can also reason about equity and inclusion"
"encourage Delphi to be more robust against different inflections of language"
"To empower Delphi with the ability to reason about compositional and grounded scenarios"
"Our position is that enabling machine ethics requires a detailed moral textbook customized to teaching machines"
Such anthropomorphism applied to machine learning models presumes that machines reason in a manner comparable to (or better than) humans. However, the learning paradigm adopted for Delphi and similar systems, assumes neither sentience nor agency: It presumes text-judgment pairs alone are sufficient for the task.
3.2.4 Agency and Accountability
Agency is also at the heart of accountability—we hold agents accountable for their deeds, not machines for their operations. In the case of a machine like Delphi, however, who is accountable is inherently obscured (Wagstaff, 2012). Crowd-workers clearly have the agency to make moral decisions and can, in principle, be held accountable for them. This is why Jiang et al. chose to rely on them as a source of moral judgments. On the other hand, a model trained on this data, although it cannot itself have agency, may appear to have agency, since it recombines and outputs texts generated by humans. By training Delphi, human agency has been transformed into something that the original agents, the crowd-workers, have no control over, or knowledge about. Yet, the trained model uses their past agency to pass novel judgments, based on some alleged—but uncontestable—moral common sense, which no one individual holds or is accountable for.
While Delphi is posed as the voice of the people, it is conveniently not a voice of any particular person, organization, or company. The responsibility for any position Delphi holds (or possible future action based on such positions) appears distributed, while in the end, the effect of such decisions, if employed in real-world scenarios, will eventually need to be accounted for. Under some legal systems, citizens have the right to challenge automated decision making which affects their rights or legitimate interests—for instance under the European Union's General Data Protection Regulation (GDPR) legislation (Rodrigues, 2020). Imagine that a technology for moral prediction were to be embedded within an autonomous system: The moral predictions occurring within the system would be obscured through layers of abstraction, thus leaving users little room to contest such decisions on principled grounds. The legal and ethical ramifications remain unclear.
In summary, crowd-sourcing ethics in this way at best obscures what is a set of problematic questions that should be addressed openly and directly and not inferred. Notably, Delphi represents one example of a wider trend in AI. As Ganesh (2017) argues: "In the development of machine intelligence towards [the goal of ethical self-driving cars], a series...of shifts can be discerned: from accounting for crashes after the fact, to pre-empting them; from ethics that is about values, or reasoning, to ethics as crowd-sourced, or based on statistics, and as
Of course, it is common in the field to talk about neural models in ways that at least suggest animacy, such as teaching/training a model or talking about its behavior. Consider, however, that one would never say of a car that it "demonstrates strong acceleration capabilities" or of an elevator that "we empowered this elevator with the ability to ascend."
the outcome of software engineering. Thus ethics-as-accountability moves towards a more opaque, narrow project."
4 Future Directions for Machine Ethics
In this section, we discuss how accuracy improvements alone cannot mitigate the problems with work such as Delphi in §4.1 and encourage a shift towards multi-disciplinary work in §4.2.
4.1 Unsafe at Any Accuracy
The introduction of any new technology into society requires us to contemplate safety concerns in the context of its proposed application (Nader, 1965). Consider, for instance, the seatbelt. One can and indeed should acknowledge that seat belts are effective at preventing automobile-related injuries to occupants without needing to imbue them with an understanding of human ethics or morality at all. We can view concrete issues in AI safety through the same lens that we view a seat belt: We can introduce safety mechanisms directly without requiring that the technology be able to reason about human ethics; we can imagine machines that operate according to moral or ethical guidelines (i.e., cars that have safety features) as opposed to machines that perform actual moral reasoning (Cave et al., 2018).
Jiang et al. and Hendrycks et al. implicitly envision a future where machine learning models could be called upon to perform moral reasoning. At its core, this vision is one of artificial general intelligence (Goertzel and Pennachin, 2007), and similar in scope and intent to the Moral Machine experiment (Awad et al., 2018), which also sought to leverage the "wisdom of the crowd" in proposing frameworks for how a future self-driving car could make decisions in speculative automotive accident scenarios. Delphi and the Moral Machine thus consider a future where AI is given agency to make ethical decisions that ordinarily would be made by a human. However, this is just one possible future.
An alternative vision of the future is one where machine learning models primarily assist humans in making decisions (Dick, 2015), i.e. where machine learning models are viewed as non-moral agents as seat belts are. In such a future, we will not need to endow machine learning models with a sense of human ethics, just as we generally do not feel the need to endow a seat belt with a sense of human ethics. Furthermore, in this future, one might prefer general strategies for reducing and
mitigating any harms machine learning may give rise to. For instance, as it stands now, many machine learning models trained on language encode harmful demographic biases that many works investigate through analysis of the models, their training regimes, and the data that they rely on (Hall Maudslay et al., 2019; Zhao et al., 2019; Dinan et al., 2020a,b; Vargas and Cotterell, 2020; Smith and Williams, 2021; Talat et al., 2021), rather than seeking to imbue models with a sense of ethics.
4.2 Machine Ethics is Multi-disciplinary
Jiang et al. (2021), like a large body of research from computer science that ventures into other fields, almost exclusively represents the perspectives of computer scientists. Another paper solely authored by computer scientists, Hendrycks et al. (2021) cautions against such a narrow perspective, stating that "computer scientists should draw on knowledge from [our] enduring intellectual inheritance, and they should not ignore it by trying to reinvent ethics from scratch" (p.3). Such disregard of expertise is apparent in several places in Jiang et al. (emphasis added):
"Fields like social science [sic], philosophy, and psychology have produced a variety of long-standing ethical theories. However, attempting to apply such theoretically-inspired guidelines to make moral judgments of complex real-life situations is arbitrary and simplistic."
Through disciplinary siloing researchers often unwittingly make simplistic assumptions that are, at best, harmful to the research and, at worst, harmful to people. We therefore recommend that machine ethics and morality research should be performed by a multi-disciplinary team, with members including computer scientists, who can speak from diverse expertise about the object that is under study.
In this paper, we have offered a general critique of the NLP task of generating moral judgments through a targeted audit of Jiang et al. (2021). We have highlighted issues with the operationalization of the task, with the learning paradigm, and with currently available training datasets. We have argued that the general enterprise is rooted in multiple category errors: It belies a misunderstanding of
the descriptive/normative distinction, and falsely treats morality as a mere test to be passed. Ultimately, automating ethical decisions forecloses possibilities for debate and contestation. Since these are themselves prerequisites for the socio-political process of ethical inquiry, such a task is inherently incompatible with the social project of ethics.
The authors would like to thank William Agnew, Reuben Binns, Abeba Birhane, Eleanor Chodroff, Hubert Etienne, Hannah Kirk, Roger Levy, Nikita Nangia, Melanie McGrath, Joshua Ree, Candace Ross, Mark Tygert, Bertie Vidgen, Ekaterina Volymova, Melissa Wheeler, Rowan Zellers, and the three anonymous reviewers for their feedback. Zeerak Talat was supported by the Canada 150 Research Chair program and the UK-Canada AI Artificial Intelligence Initiative.
Mike Ananny. 2016. Toward an ethics of algorithms: Convening, observation, probability, and timeliness. Science, Technology, & Human Values, 41(1):93-117.
Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018. The moral machine experiment. Nature, 563(7729):59-64. Publisher: Nature Publishing Group.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Emily M. Bender, Dirk Hovy, and Alexandra Schofield. 2020. Integrating ethics into the NLP curriculum. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 6-9, Online. Association for Computational Linguistics.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Cristina Bicchieri. 2005. The Grammar of Society: The Nature and Dynamics of Social Norms. Cambridge University Press.
Abeba Birhane and Jelle van Dijk. 2020. Robot rights?: Let's talk about human welfare instead. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 207-213.
Donna K. Byron. 2002. Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 80-87, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Dallas Card and Noah A. Smith. 2020. On consequentialism and fairness. Frontiers of Artificial Intelligence, 3:34.
Stephen Cave, Rune Nyrup, Karina Vold, and Adrian Weller. 2018. Motivations and risks of machine ethics. Proceedings of the IEEE, 107(3):562-574.
Stephanie Dick. 2015. After Math: (Re)configuring Minds, Proof, and Computing in the Postwar United States. Ph.D. thesis, Harvard University, Graduate School of Arts & Sciences.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020a. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8173-8188, Online. Association for Computational Linguistics.
Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020b. Multidimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 314-331, Online. Association for Computational Linguistics.
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2020. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. CoRR, abs/2012.15738.
Hubert Etienne. 2021. The dark side of the Moral Machine' and the fallacy of computational ethical decision-making for autonomous vehicles. Law, Innovation and Technology, 13(1):85-107.
European Commission. 2008. Discrimination in the European union: Perceptions, experiences and attitudes. Eurobarometer Special.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
Kathryn B. Francis, Charles Howard, Ian S. Howard, Michaela Gummerum, Giorgio Ganis, Grace Anderson, and Sylvia Terbeck. 2016. Virtual Morality: Transitioning from Moral Judgment to Moral Action? PLOS ONE, 11(10):e0164374.
Maya Indira Ganesh. 2017. Entanglement: Machine learning and human ethics in driver-less car crashes. A Peer-Reviewed Journal About, 6(1):76-87.
Ben Goertzel and Cassio Pennachin. 2007. Artificial General Intelligence, volume 2. Springer.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5267-5275, Hong Kong, China. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI with shared human values. In 9th International Conference on Learning Representations (ICLR).
Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591-598, Berlin, Germany. Association for Computational Linguistics.
Sophie F. Jentzsch, Patrick Schramowski, Constantin A. Rothkopf, and Kristian Kersting. 2019. Semantics derived automatically from language corpora contain human-like moral choices. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES 2019, Honolulu, HI, USA, January 27-28, 2019, pages 37-44. ACM.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. arXiv preprint arXiv:2110.07574.
Anna Kende, Marton Hadarics, Sara Bigazzi, Mihaela Boza, Jonas R. Kunst, Nora Anna Lantos, Barbara Lásticová, Anca Minescu, Monica Rivetti, and Ana Urbiola. 2021. The last acceptable prejudice in Europe? Anti-Gypsyism as the obstacle to Roma inclusion. Group Processes & Intergroup Relations, 24(3):388-410.
Isabel Kloumann and Mark Tygert. 2020. An estimable scalar objective value cannot be objective and should not be the sole objective. arXiv preprint arXiv:2006.02577.
Angelika Kratzer. 1981. The Notional Category of Modality. In Hans J. Eikmeyer and Hannes Rieser, editors, Words, Worlds, and Contexts. de Gruyter, Berlin, Boston.
Angelika Kratzer. 2012. Modals and Conditionals, volume 36 of Oxford Studies in Theoretical Linguistics. Oxford University Press, Oxford.
John Loeffler. 2021. AI chatbot justifies sacrificing colonists to create a biological weapon...if it creates jobs. TechRadar.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. SCRUPLES: A corpus of community ethical judgments on 32,000 real-life anecdotes. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13470-13479. AAAI Press.
Momin M. Malik. 2020. A hierarchy of limitations in machine learning. arXiv preprint arXiv:2002.05193.
Cade Metz. 2021. Can a machine learn morality? The New York Times.
John Stuart Mill. 1871. Principles of Political Economy with Some of Their Applications to Social Philosophy,  $7^{\text{th}}$  edition, volume 1. Longmans, Green, Reader, and Dyer.
Ralph Nader. 1965. Unsafe at Any Speed: The Designed-In Dangers of the American Automobile. Grossman Publishers.
Poppy Noor. 2021. 'is it OK to ...': the bot that gives you an instant moral judgment. The Guardian.
NYU Web Communications. FAQs.
Shrimai Prabhumoye, Brendon Boldt, Ruslan Salakhutdinov, and Alan W Black. 2021. Case study: Deontological ethics in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3784-3798, Online. Association for Computational Linguistics.
Aida Ramezani, Zining Zhu, Frank Rudzicz, and Yang Xu. 2021. An unsupervised framework for tracing textual sources of moral change. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 1215–1228, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rezvaneh Rezapour, Saumil H. Shah, and Jana Diesner. 2019. Enhancing the measurement of social effects by capturing morality. In Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 35-45, Minneapolis, USA. Association for Computational Linguistics.
Rowena Rodrigues. 2020. Legal and human rights issues of AI: Gaps, challenges and vulnerabilities. Journal of Responsible Technology, 4:100005.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational