SlowGuess/NAACL_2021_OCR huggingface dataset:
Shrimai Prabhumoye *, Brendon Boldt *, Ruslan Salakhutdinov, Alan W Black
{sprabhum, bboldt, rsalakhu, awb}@cs.cmu.edu
Recent work in natural language processing (NLP) has focused on ethical challenges such as understanding and mitigating bias in data and algorithms; identifying objectionable content like hate speech, stereotypes and offensive language; and building frameworks for better system design and data handling practices. However, there has been little discussion about the ethical foundations that underlie these efforts. In this work, we study one ethical theory, namely deontological ethics, from the perspective of NLP. In particular, we focus on the generalization principle and the respect for autonomy through informed consent. We provide four case studies to demonstrate how these principles can be used with NLP systems. We also recommend directions to avoid the ethical issues in these systems.
The 21st century is witnessing a major shift in the way people interact with technology, and natural language processing (NLP) is playing a central role. A plethora of NLP applications such as question-answering systems (Bouziane et al., 2015; Gillard et al., 2006; Yang et al., 2018) used in diverse fields like healthcare (Sarrouti and Ouatik El Alaoui, 2017; Zweigenbaum, 2009), education (Godea and Nielsen, 2018; Raamadhurai et al., 2019), privacy (Ravichander et al., 2019; Shvartzshanider et al., 2018); machine translation systems (Cherry et al., 2019; Barrault et al., 2019; Nakazawa et al., 2019; Liu, 2018), conversational agents (Pietquin et al., 2020; Serban et al., 2018; Liu et al., 2016), recommendation systems (Alharthi and Inkpen, 2019; Greenquist et al., 2019) etc. are deployed and used by millions of users. NLP systems have become pervasive in current human lifestyle by performing mundane tasks like setting reminders and alarms to complex tasks like
replying to emails, booking tickets and recommending movies/restaurants. This widespread use calls for an analysis of these systems from an ethical standpoint.
Despite all the advances in efficiency and operations of NLP systems, little literature exists which broadly addresses the ethical challenges of these technologies. Ethical theories have been studied for millennia and should be leveraged in a principled way to address the questions we are facing in NLP today. Instead, the topic of "ethics" within NLP has come to refer primarily to addressing bias in NLP systems; Blodgett et al. (2020) provides a critical survey of how bias is studied in NLP literature. The survey finds that research on NLP systems conceptualize bias differently and that the techniques are not well tied with the relevant literature outside of NLP. This creates a gap between NLP research and the study of ethics in philosophy which leaves a rich body of knowledge untapped.
Our work bridges this gap by illustrating how a philosophical theory of ethics can be applied to NLP research. Ethics (or ethical theory), is a theoretical and applied branch of philosophy which studies what is good and right, especially as it pertains to how humans ought to behave in the most general sense (Fieser, 1995). As NLP research qualifies as a human activity, it is within the purview of ethics. In particular, we are using a prescriptive, rather than descriptive, theory of ethics; prescriptive theories define and recommend ethical behavior whereas descriptive theories merely report how people generally conceive of ethical behavior.
We select two ethical principles from the deontological tradition of ethics and focus on how these principles are relevant to research in NLP. Namely we look at the generalization principle and respect for autonomy through informed consent (Johnson and Cureton, 2019; Kleinig, 2009). We select deontology because it is reasonable, provides clear ethical rules and comports with the legal idea of the
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3784-3798 June 6-11, 2021. ©2021 Association for Computational Linguistics
rule of law in the sense that these ethical rules bind all persons equally, rather than shifting standards to effect a certain outcome.
We find that there are two main ways in which ethical guidelines can be applied in NLP (or to any other area of technology):
1. An ethical guideline can aid in deciding what topics within a field merit attention; that is, it answers the question "which tasks have important ethical implications?".
2. An ethical guideline can aid in determining how to address a problem; that is, it answers the question "what factors and methods are preferable in ethically solving this problem?"
We primarily address (1) and briefly touch on (2) by presenting four case studies relevant to NLP. In each case study we use an ethical principle to identify an area of research that could potentially conflict with it, and suggest NLP directions to mitigate it. Although we have selected two principles from a deontological perspective, we are not intimating that these principles can address all ethical issues nor that deontological ethics is the only ethical framework in which our rules and case studies could function (§6). Instead, we present the following as a starting point for NLP researchers less familiar but interested in applicable ethical theory.
- Providing an overview of two deontological principles along with a discussion on their limitations with a special focus on NLP.
- Illustrating four specific case studies of NLP systems which have ethical implications under these principles and providing a direction to alleviate these issues.
2.1 Ethics
While there are a number of categories of prescriptive ethical theories, including deontology (Kant, 1785), consequentialism (e.g., utilitarianism) (Bentham, 1843), and virtue ethics (Aristotle, 350 B.C.E.), we are only addressing deontology. We do not take a stance in this paper as to whether or not there exists an objectively correct ethical theory, but we offer a brief sketch of deontological ethics and our reasons for using it. Deontology or deontological ethics refers to a family of ethical theories
which hold that whether an act is ethically good or bad is determined by its adherence to ethical rules (Alexander and Moore, 2016). These rules can be agent-focused duties (e.g., duty to care for one's children) or patient-focused rights (e.g., right to life). Such rules can also be formulated in modal logic, allowing for more precise reasoning over sets of rules (Hooker and Kim, 2018).
Deontology stands in contrast to another popular framework of ethics: consequentialism. Consequentialism holds the ultimate consequences of an action to be the deciding factor regardless of the nature of the actions taken to get there. We can illustrate the difference between them by observing how each of them might condemn something like racially biased hiring in academia.<sup>1</sup> A deontologist might say that this practice is wrong because it violates the human right to equal treatment regardless of race. A consequentialist on the other hand, would argue that this is wrong because its effect is stymieing academic creativity by reducing intellectual diversity.
We ultimately select the deontological framework in this work for the following reasons:
1. We find deontology to be convincing in its own right, namely, its ability to delineate robust duties and rights which protect the value of each and every person.
2. The universally applicable rules $^{2}$  of deontology provide a good basis for providing recommendations to researchers. Since rights and duties (at their core) are not situation dependent, they are tractable to address in NLP applications. $^{3}$
3. The focus on rights and duties which apply to everyone equally fits well with the widespread legal concept of the rule of law which states that every person is subject to the same laws.
2.2 Ethics in NLP
We appeal to the fact that problems should be analyzed with a systematic framework, and ethical
<sup>1</sup>Note that we are presenting generic examples of deontological and consequentialist frameworks and that a variety of nuanced theories in each category exist.
2While determining rules which apply universally across all cultures is a difficult task, the existence of organizations, such as the United Nations, presuppose the achievability of identifying internationally applicable norms.
3In contrast to (action-based) utilitarianism which mandates evaluating the full consequences of each action.
theories provide precisely these frameworks. Research should not be based on preconceived notions of ethics which can be overly subjective and inconsistent. To more rigorously determine what is right and wrong, we rely on ethical theories. Card and Smith (2020) present an analysis of ethics in machine learning under a consequentialist framework. This paper is a kindred spirit in that we both seek to make a philosophical theory of ethics concrete within machine learning and NLP, yet the methods of the paper are somewhat orthogonal. Card and Smith (2020) provide a comprehensive overview of how the particular nature of consequentialist ethics is relevant to machine learning whereas we intend to provide tangible examples of how deontological ethical principles can identify ethically important areas of research. Saltz et al. (2019); Bender et al. (2020) advocate for explicitly teaching ethical theory as a part of machine learning and NLP courses; the case studies in this paper would be a logical extension of the material presented in such a course.
NLP research on ethics has primarily focused on two directions: (1) exploring and understanding the impact of NLP on society, and (2) providing algorithmic solutions to ethical challenges.
Hovy and Spruit (2016) started the conversation about the potential social harms of NLP technology. They discussed the concepts of exclusion, overgeneralization, bias confirmation, topic under- and overexposure, and dual use from the perspective of NLP research. A lot of work followed this discussion and made contributions towards ethical frameworks and design practices (Leidner and Plachouras, 2017; Parra Escartín et al., 2017; Prabhumoye et al., 2019; Schnoebelen, 2017; Schmaltz, 2018), data handling practices (Lewis et al., 2017; Mieskes, 2017) and specific domains like education (Mayfield et al., 2019; Loukina et al., 2019), healthcare (Suster et al., 2017; Benton et al., 2017) and conversational agents (Cercas Curry and Rieser, 2018; Henderson et al., 2018). Our paper does not focus on a particular domain but calls for attention towards various NLP systems and what ethical issues may arise in them.
Most of the work providing algorithmic solutions has been focused on bias in NLP systems. Shah et al. (2020); Tatman (2017); Larson (2017) aim to study the social impact of bias in NLP systems and propose frameworks to understand it better. A large body of work (Bolukbasi et al., 2016; Sun et al., 2019; Zhao et al., 2019, 2017; Sap et al.,
2019; Hanna et al., 2020; Davidson et al., 2019) directs its efforts to mitigate bias in data, representations, and algorithms. Blodgett et al. (2020) provide an extensive survey of this work and point out the weaknesses in the research design. It makes recommendations of grounding work analyzing bias in NLP systems in the relevant literature outside of NLP, understanding why system behaviors can be harmful and to whom, and engaging in a conversation with the communities that are affected by the NLP systems. Although issues with bias are certainly within the scope of the principles we present, we do not specifically write on bias because it has already received a large amount of attention.
3 Deontological Ethics
There is a variety of specific deontological theories which range from having one central, abstract principle (Kant, 1785) to having a handful of concrete principles (Ross, 1930). Rather than comprehensively addressing one theory, we select two rules, one abstract and one concrete, which can fit within a variety of deontological theories. The generalization principle is an abstract, broad-reaching rule which comes from traditional Kantian ethics. The respect for autonomy is concrete and commonly seen in politics and bioethics.
3.1 Generalization Principle
The generalization principle has its roots in Immanuel Kant's theory of deontological ethics (Kant, 1785). The generalization principle states the following (Johnson and Cureton, 2019).
An action  $\mathcal{A}$  taken for reasons  $\mathcal{R}$  is ethical if and only if a world where all people perform  $\mathcal{A}$  for reasons  $\mathcal{R}$  is conceivable.
It is clearer when phrased in the negative.
An action  $\mathcal{A}$  taken for reasons  $\mathcal{R}$  is unethical if and only if a world where all people perform  $\mathcal{A}$  for reasons  $\mathcal{R}$  logically contradicts  $\mathcal{R}$ .
The main utility of the generalization principle is that it can identify unethical actions that may seem acceptable in isolated occurrences but lead to problems when habitually taken by everyone.
For example, let us take making and breaking a legal contract (the action) whenever it is convenient (the reasons); implicit in the reasons for making a
<sup>4</sup>It is also referred to as the "universal law" formulation of Kant's categorical imperative.
contract is that the other person believes we will follow through (Johnson and Cureton, 2019). If we universalize this and conceive of a world where everyone makes contracts which they have no intent of keeping, no one would believe in the sincerity of a contract. Hence, no one would make contracts in the first place since they are never adhered to. This is the sort of contradiction by which the generalization principle condemns an action and the rationale behind it.
Another example is plagiarism of research papers in conference submissions. Let us assume that a top tier conference did not check for plagiarism because they trust in the honesty of the researchers. In this case, a researcher  $\mathbf{G}$  decides to take an action  $\mathcal{A}$  of plagiarising a paper due to the following set of reasons  $\mathcal{R}$ : (1)  $\mathbf{G}$  believes that they would not get caught because the conference does not use plagiarism detection software, (2) publishing this paper in the said conference would boost  $\mathbf{G}$ 's profile by adding 100 citations, and (3) this would increase  $\mathbf{G}$ 's chances of getting a job. Plagiarism in this case would be ungeneralizable and hence unethical. If all researchers who want to boost their profile were to submit plagiarised papers, then every researcher's profile would be boosted by 100 citations, and 100 citations would lose their value. Hence, this would not increase  $\mathbf{G}$ 's chances of getting a job, contradicting  $\mathcal{R}3$ . Thus,  $\mathbf{G}$ 's reasons for plagiarism are inconsistent with the assumption that everyone with same reasons plagiarises.
3.2 Respect for Autonomy
Respect for autonomy generally addresses the right of a person to make decisions which directly pertain to themselves. One of the primary manifestations of this is the concept of informed consent, whereby a person A proposes to act in some way  $\mathbb{X}$  on person B which would normally infringe on B's right to self-govern. Specifically, we use the formulation of informed consent given by Pugh (2020) based on Kleinig (2009):
1. B must be sufficiently informed with regards to the relevant facts concerning  $\mathbb{X}$  to understand what  $\mathbb{X}$  is (and what consequences are likely to occur as a result of  $\mathbb{X}$ ).
2. On the basis of this information,  $\mathbf{B}$  herself makes the decision to allow  $\mathbf{A}$  to do  $\mathbb{X}$ .
Informed consent is an important idea in bioethics where it typically applies to a patient's
right to refuse treatment (or certain kinds of treatment) by medical personnel. In routine medical treatments this informed consent might be implicit, since one would not go to the doctor in the first place if they did not want to be treated at all, but in risky or experimental medical procedures, explaining the risks and benefits and obtaining explicit consent would be mandatory. In this case, the patient's autonomy specifically refers to opting out of medical procedures, and informed consent is a concrete method by which to respect this autonomy.
A non-medical example of respect for autonomy and informed consent would be hiring an interpreter A for a language that the user B does not speak. Under normal circumstances, B's autonomy dictates that she and only she can speak for herself. But if she is trying to communicate in a language she does not speak, she might consent to A serving as an ad hoc representative for what she would like to say. In a high-stakes situation, there might be a formal contract of how A is to act, but in informal circumstances, she would implicitly trust that A translates what she says faithfully (X). In these informal settings, A should provide necessary information to B before deviating from the expected behaviour X (e.g., if the meaning of a sentence is impossible to translate). Implicit consent is a double-edged sword: it is necessary to navigate normal social situations, but it can undermine the respect for autonomy in scenarios when (1) the person in question is not explicitly informed and (2) reasonable expectations do not match reality.
4 Applying Ethics to NLP systems
We apply the generalization principle in  $\S 4.1$  and  $\S 4.2$  and respect for autonomy in  $\S 4.3$  and  $\S 4.4$ .
4.1 Question-Answering Systems
Question-answering (QA) systems have made a huge progress with the recent advances in large pre-trained language models (Devlin et al., 2019; Radford et al., 2019; Guu et al., 2020). Despite these improvements, it is difficult to know how the model reached its prediction. In fact, it has been shown that models often obtain high performance by leveraging statistical irregularities rather than language understanding (Poliak et al., 2018; Geva et al., 2019; Gururangan et al., 2018). The result is that when a QA system is wrong it is difficult for an end user to determine why it was wrong. Presumably, the user would not know the answer
"I like to imagine you as a girl but your sentence structure and rhetoric is so concise and to the point which points to the contrary (nothing against women, simply factual)."
Figure 1: Examples of flagging micro-aggression comments by different NLP systems.
(a) Micro-aggressive comment and its scores by state-of-the-art hate speech detection and sentiment analysis tools (Breitfeller et al., 2019).
(b) NLP system flagging the micro-aggressive comment as offensive and generating the reasoning for flagging it (Sap et al., 2020).
to the question in the first place, and so it would be difficult to determine even that the QA system was wrong.
The act of widely deploying such a QA system is in conflict with the generalization principle. For example, a QA system  $\mathbf{G}$  is unsure of its prediction  $\mathcal{A}$  and does not know how it arrived at the answer. Instead of notifying the user about its inability to reach the prediction,  $\mathbf{G}$  decides to return the prediction  $\mathcal{A}$  due to the following reasons  $\mathcal{R}$ : (1)  $\mathbf{G}$  believes that the user does not know the answer and hence (2)  $\mathbf{G}$  believes that the user will trust its answer and not ask for reasons for giving the prediction. If all QA systems operate like this, users will lose trust in QA systems being able to answer their questions reliably and no longer use them. This contradicts assumption  $\mathcal{R}2$ , violating the generalization principle. This issue goes deeper than a matter of the (in)accuracy of the answer; explainability is still important for a near-perfect QA system. First, the source of an answer could be fallible (even if the content was interpreted correctly), in which case it is important to be able to point which sources were used. Second, answers can often be ambiguous, so a user might naturally ask for clarification to be sure of what the answer means. Finally, it is natural for humans to build trust when working with a system, and explainability is an important step in this process.
Attention weights have been widely used for explaining QA predictions. Attention weights learnt by neural models denote the words or phrases in a sentence that the model focuses on. Hence, words or phrases with high attention weights are considered as explanations to the QA predictions. But these weights do not reliably correlate with model
predictions, making them unsuitable for explainability (Pruthi et al., 2020; Serrano and Smith, 2019; Jain and Wallace, 2019). Recently, generating natural language explanations (Rajani et al., 2019; Latcinnik and Berant, 2020) for predictions has gained traction. These methods train a language generation model to generate explanations for the QA predictions. Using a black-box model for text generation, though, pushes the same problem further down the line. Part of the issue with both of the aforementioned methods is that the "reasoning" for the answer is determined after the answer has been generated (i.e., reasoning should inform the answer, not vice-versa).
The way forward: A method which reaches the prediction through reasoning would be more in line with the generalization principle. For example, reaching the prediction through traversal of a knowledge graph. This has been used in scenarios where a knowledge base exists (Han et al., 2020; Jansen et al., 2018) for a QA system as well as in dynamic graph generation to reach the prediction (Liu et al., 2020; Rajagopal et al., 2020; Bosselut and Choi, 2019). In these methods, the reasoning is part of the process to generate the final answer, which is more suitable in failing gracefully and building user trust.
4.2 Detecting Objectionable Content
Social media platforms have made the world smaller. At the same time, the world has seen a surge in hate-speech, offensive language, stereotype and bias on online platforms. These online platforms have traffic in the millions of textual comments, posts, blogs, etc. every day. Identifying such objectionable content by reading each item is in
tractable. Hence, building an NLP system which can read textual data and flag potential objectionable content is necessary. These systems can reduce the burden on humans by reducing the number of posts that need to be seen by human eyes.
The pivotal role NLP systems play in flagging such content makes the ethical considerations important. Fig. 1a shows a microaggressive comment and its scores by a state-of-the-art (1) hate speech detection system and (2) sentiment analysis system. Since these systems rely on surface level words or phrases to detect such (overt) comments, they tend to miss subtle (covert) objectionable content (Breitfeller et al., 2019). If such NLP systems are used universally, then the users of hate speech will discover ways to phrase the same meaning with different words (as illustrated above). Thus, the NLP content flagging system will not be able to detect objectionable content, and there will be no point in deploying it. This contradiction suggests that NLP systems must not make their predictions based only on superficial language features but instead seek to understand the intent and consequences of the text presented to them. Hence, they should generate reasons for flagging posts to facilitate the decision making of the human judges and also to provide evidence about the accuracy of their predictions.