SlowGuess/NAACL_2022_OCR huggingface dataset:
Chenhao Tan  
Department of Computer Science & Harris School of Public Policy  
The University of Chicago  
chenhao@uchicago.edu
A growing effort in NLP aims to build datasets of human explanations. However, it remains unclear whether these datasets serve their intended goals. This problem is exacerbated by the fact that the term explanation is overloaded and refers to a broad range of notions with different properties and ramifications. Our goal is to provide an overview of the diversity of explanations, discuss human limitations in providing explanations, and ultimately provide implications for collecting and using human explanations in NLP.
Inspired by prior work in psychology and cognitive sciences, we group existing human explanations in NLP into three categories: proximal mechanism, evidence, and procedure. These three types differ in nature and have implications for the resultant explanations. For instance, procedure is not considered explanation in psychology and connects with a rich body of work on learning from instructions. The diversity of explanations is further evidenced by proxy questions that are needed for annotators to interpret and answer "why is [input] assigned [label]". Finally, giving explanations may require different, often deeper, understandings than predictions, which casts doubt on whether humans can provide valid explanations in some tasks.
With the growing interest in explainable NLP systems, the NLP community have become increasingly interested in building datasets of human explanations. These human explanations can ideally capture human reasoning of why a (correct) label is chosen. If this is indeed the case, they are hypothesized to aid models with additional supervision, train models that explain their own predictions, and evaluate machine-generated explanations (Wiegreiffe and Marasovic, 2021). In fact, DeYoung et al. (2020) already developed a leaderboard, where the
implicit assumption is that humans can provide valid explanations and these explanations can in turn be uniformly considered as groundtruths.
However, are these assumptions satisfied and can human explanations serve these goals? In this work, we aim to introduce prior relevant literature in psychology to the NLP community and argue against abusing the term explanations and prematurely assuming that human explanations provide valid reasoning for inferring a label.
First, we point out the rich diversity in what the NLP community refers to as explanations and how researchers collect them. The term "explanation" is overloaded in the NLP and AI community: it often refers to many distinct concepts and outcomes.<sup>1</sup> For example, procedural instructions are different from explanations that attempt to convey proximal causal mechanisms. The diversity of explanations is further evidenced by the variety of proxy questions that researchers ask to collect explanations, e.g., "highlight the important words that would tell someone to see the movie" vs. "highlight ALL words that reflect a sentiment". These proxy questions are necessary because the question of "why is [input] assigned [label]" is too open-ended. It follows that these "human explanations" are supposed to answer different questions in the first place and may not all be used for the same goals, e.g., serving as groundtruth labels.
In addition to the diversity, we highlight two insights from psychology on whether humans can provide valid explanations: 1) prediction does not entail explanation (Wilson and Keil, 1998), i.e., although humans may be able to provide valid labels, they may not be able to provide explanations that capture the reasoning process needed or used to infer a label; 2) everyday explanations are necessarily incomplete (Keil, 2006; Lombrozo, 2006), because they seldom capture the complete deduc
$^{1}$ Broniatowski et al. (2021) further argues that interpretation and explainability are distinct concepts.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173 - 2188 July 10-15, 2022 ©2022 Association for Computational Linguistics
tive processes from a set of axioms to a statement.
In summary, not all explanations are equal and humans may not always be able to provide valid explanations. We encourage the NLP community to embrace the complex and intriguing phenomena behind human explanations instead of simply viewing explanations as another set of uniform labels. A better understanding and characterization of human explanations will inform how to collect and use human explanations in NLP.
2 Types of Human Explanations in NLP
To understand whether datasets of human explanations can serve their intended goals, we first connect current human explanations in NLP with existing psychology literature to examine the use of the term "explanation" in NLP. We adapt the categorization in Lombrozo (2006) and group human explanations in NLP into the following three categories based on the conveyed information:
- Proximal mechanisms. This type of explanation attempts to provide the mechanism behind the predicted label, i.e., how to infer the label from the text, and match efficient cause in Lombrozo (2006). We created E1 in Table 1 to illustrate this type of explanation. Note that E1 does not provide the complete mechanism. For instance, it does not define "year" or "temporal modifier", or make clear that "1997" is a "year". Neither does it cover the axioms of logic. This is a common property of human explanations: they are known to cover partial/proximal mechanisms rather than the complete deduction from natural laws and empirical conditions (Hempel and Oppenheim, 1948).
- Evidence. This type of explanation includes the relevant tokens in the input (e.g., E2 in Table 1) and directly maps to highlights in Wegreffer and Marasović (2021). However, it does not map to any existing definitions of explanations in the psychology literature since the evidence does not provide any information on how evidence leads to the label. In other words, evidence alone does not explain.
- Procedure. Unlike proximal mechanisms, this type of explanation provides step-by-step rules or procedures that one can directly follow, e.g., E3 in Table 1. They are more ex
plicit and unambiguous than proximal mechanisms. In fact, one can write a rule based on E3 to find marriage relation, but one cannot easily do that with E1. Furthermore, the procedures are grounded to the input, so it is related to formal cause, "the form or properties that make something what it is" (Lombrozo, 2006), which is definitional and does not convey the underlying mechanisms. Procedural instructions are only possible for some tasks, while proximal mechanisms are the most common form of everyday explanations.
These three categories empirically capture all the explanations discussed in NLP literature. Lombrozo (2006) also discuss two other categories, final causes (the goal) and material causes (the constituting substance). For instance, a final cause to "why [input] is assigned [label]" can be that "this label is provided to train a classifier". These two categories have been less relevant for NLP.
Implications. This categorization allows us to think about what kind of explanations are desired for NLP systems and help clarify how to use them appropriately. First, proximal mechanisms are best aligned with human intuitions of explanations, especially in terms of hinting at causal mechanisms. However, they can be difficult to collect for NLP tasks. For example, Table 2 shows example explanations in E-SNLI that fail to convey any proximal mechanisms: they either repeat the hypothesis or express invalid mechanisms ("the bike competition" does not entail "bikes on a stone road"). See further discussions on the challenges in collecting explanations in §4. Furthermore, they may be difficult to use for supervising or evaluating a model.
Second, evidence by definition provides little information about the mechanisms behind a label, but it can be potentially useful as additional supervision or groundtruths. We will further elaborate on the nature of evidence in different tasks in §3. However, it may be useful to the community to use clear terminology (e.g., evidence or rationale (Lei et al., 2020; Carton et al., 2020)) to avoid lumping everything into "explanation".
Finally, procedures are essentially instructions, and Keil (2006) explicitly distinguishes explanations from simple procedural knowledge: "Knowing how to operate an automated teller machine or make an international phone call might not entail having any understanding of how either system
Instance	Label	Explanation
Task: whether the query is supported or refuted by the preceding texts (Thorne et al., 2018)
S1: No Way Out is the debut studio album by American hip hop recording artist, songwriter and record producer Puff Daddy . S2: It was released on July 1 , 1997 , by his Bad Boy record label . The label ’s official crediting as “ The Family ”...... Query: 1997 was the year No Way Out was released.	Proximal mechanism (E1): “It” in S2 refers to “No Way Out” and “on July 1, 1997” is the temporal modi-fier of “release”, we can thus infer 1997 was the year that No Way Out was released. Evidence (E2): S1, S2
Task: whether person 1 is married to person 2 (Hancock et al., 2018)
Tom Brady and his wife Gisele Bündchen were spotted in New York City on Monday amid rumors of Brady’s alleged role in Deflategate	True	Procedure (E3): The words “and his wife” are be-tween person 1 and person 2.
Table 1: Types of human explanations and corresponding examples. "S1:" and "S2:" were added to facilitate writing the explanation, which also shows the non-triviality of writing explanations.
P: Men in green hats appear to be attending a gay pride festival. H: Men are attending a festival. E: The men are attending the festival.
P: Several bikers on a stone road, with spectators watching.  
H: The bikers are on a stone road. E: That there are spectators watching the bikers on a stone road implies there is a bike competition.
Table 2: Examples from E-SNLI (Camburu et al., 2018). P: premise, H.: hypothesis, E: explanation.
works". Another reason to clarify the procedure category is that it would be useful to engage with a rich body of work on learning from instructions when human explanations are procedural (Goldwasser and Roth, 2014; Matuszek et al., 2013).
We would like to emphasize that procedures or instructions are powerful and can potentially benefit many NLP problems (e.g., relation extraction). At the same time, it is useful to point out that procedures are different from proximal mechanisms.
3 Proxy Questions Used to Collect Human Explanations
Although explanations are supposed to answer "why is [input] assigned [label]" (Wiegreffe and Marasović, 2021), this literal form is too open-ended and may not induce "useful" human explanations. As a result, proxy questions are often necessary for collecting human explanations. These proxy questions further demonstrate the diversity of human explanations beyond the types of explanation. Here we discuss these proxy questions for collecting evidence. See the appendix for discussions on proximal mechanisms and procedures.
To collect evidence (highlights), researchers adopt diverse questions for relatively simple single-text classification tasks (see Table 3). Consider
the seemingly straightforward case of sentiment analysis, "why is the sentiment of a review positive/negative". A review can present both positive and negative sentiments (Aithal and Tan, 2021), so the label often comes from one sentiment outweighing the other. However, in practice, researchers often ask annotators to identify only words supporting the label. Critical wording differences remain in their questions: Zaidan et al. (2007) ask for the most important words and phrases that would tell someone to see the movie, while Sen et al. (2020) requires all words reflecting the sentiment. Two key differences arise: 1) "the most important" vs. "all"; 2) "telling someone to see the movie" vs. "reflecting the sentiment".
In contrast, personal attack detection poses a task where the negative class ("no personal attack") by definition points to the lack of evidence in the text. It follows that the questions that researchers can ask almost exclusively apply to the positive class (i.e., "highlight sections of comments that they considered to constitute personal attacks").
In comparison, researchers approach evidence more uniformly for document-query classification tasks. They generally use similar proxy questions (e.g., Thorne et al. (2018) and Hanselowski et al. (2019) ask almost the same questions) and ask people to select sentences instead of words. That said, intriguing differences still exist: 1) Lehman et al. (2019) simply ask annotators to provide accompanying rationales; 2) Thorne et al. (2018) aim for "strong" reasons, which likely induces different interpretations among annotators; 3) Khashabi et al. (2018) collect questions, answer, and sentence indices at the same time, among which sentence indices can be used to find the corresponding sentences as evidence. It remains unclear how these
Reference	Task	Questions
Zaidan et al. (2007)	Sentiment analysis	Evidence: single-text classification
To justify why a review is positive, highlight the most important words and phrases that would tell someone to see the movie. To justify why a review is negative, highlight words and phrases that would tell someone not to see the movie.
Sen et al. (2020)	Sentiment analysis	Label the sentiment and highlight ALL words that reflect this sentiment.
Carton et al. (2018)	Personal attack de-tection	Highlight sections of comments that they considered to constitute personal attacks.
Lehman et al. (2019)	Question answering	Evidence: document-query classification
Generators were also asked to provide answers and accompanying rationales to the prompts that they provided.
Thorne et al. (2018)	Fact verification (QA)	If I was given only the selected sentences, do I have strong reason to believe the claim is true (supported) or stronger reason to believe the claim is false (refuted).
Khashabi et al. (2018)	Question answering	Ask them (participants) for a correct answer and for the sentence indices required to answer the question.
Table 3: Questions that prior work uses to collect human explanations. We include the short version of the guidelines here for space reasons. Refer to the appendix for the full text of the relevant annotation guidelines.
differences in annotation processes and question phrasings affect the collected human explanations.
Implications. Our observation on proxy questions aligns with dataset-specific designs discussed in Wegreffe and Marasovic (2021). We emphasize that these different forms of questions entail different properties of the collected human explanations, as evidenced by Carton et al. (2020). For example, the lack of evidence in the negative class in personal attack classification likely requires special strategies in using human explanations to train a model and evaluate machine rationales. Sentence-level and token-level annotations also lead to substantially different outcomes, at least in the forms of explanations. We believe that it is important for the NLP community to investigate the effect of proxy questions and use the collected explanations with care, rather than lumping all datasets under the umbrella of explanations.
We also recommend all researchers to provide detailed annotation guidelines used to collect human explanations. As the area of collecting human explanation is nascent, the goal is not to promote consistent and uniform annotation guidelines but to encourage the community to pay attention to the different underlying questions and characterize the resultant diverse properties of human explanations.
4 Can Humans Provide Explanations?
In order for human explanations to serve as additional supervision in training models and evaluate machine-generated explanations, human explanations need to provide valid mechanisms for
a correct label. Finally, we discuss challenges for humans to provide explanations of such qualities.
Conceptual framework. We situate our discussion in the psychological framework provided by Wilson and Keil (1998) to highlight what may be required to explain. Wilson and Keil (1998) examines where explanation falls in three central notions: prediction, understanding, and theories. They argue that these three notions "form a progression of increasing sophistication and depth with explanations falling between understanding and theories". For instance, we may be able to predict that a car will start when we turn the ignition switch, but few of us are able to explain in detail why this is so. In contrast, if a person is able to explain in detail why a car starts when you turn on the ignition switch, they can likely predict what will happen if various parts of the engine are damaged or removed.
These three central notions are also essential in machine learning. Traditional label annotation is concerned with prediction, however, being able to predict does not entail being able to explain.
Emulation vs. discovery. Next, we gradually unfold the practical challenges in collecting valid explanations from humans. The first challenge lies in whether humans can predict, i.e., assign the correct label. We highlight two types of tasks for AI: emulation vs. discovery (Lai et al., 2020). In emulation tasks, models are trained to emulate human intelligence and labels are often crowdsourced. Labels, however, can also derive from external (social/biological) processes, e.g., the popularity of a tweet and the effect of a medical treatment. Mod-
els can thus discover patterns that humans may not recognize in these discovery tasks. While most NLP tasks such as NLI and QA are emulation tasks, many NLP problems, especially when concerned with social interaction, are discovery tasks, ranging from identifying memorable movie quotes to predicting the popularity of messages (Danescu-Niculescu-Mizil et al., 2012; Tan et al., 2014).
Aligning with our discussion on explanation and prediction, most datasets of human explanations in NLP assume that humans are able to predict and are on emulation tasks. However, we note exceptions such as explanations of actions in gaming (Ehsan et al., 2019), where humans may often choose suboptimal actions (labels).
Cognitive challenges in providing valid explanations. Even conditioned on that humans can predict the label, humans may not be able to provide valid explanations for at least two reasons. First, as Wilson and Keil (1998) suggests, explanation requires more depth than prediction. For instance, we may possess some notions of common sense (e.g., one should not slap a stranger), but it is unclear whether we can explain common sense in detail (e.g., why one should not slap a stranger through theory of morality), similar to the car ignition example. One may argue that theory of morality may not be what NLP researchers seek, but it is critical to consider the desiderata of human explanations, with the limits in mind.
Second, explanation often requires people to report their subjective mental processes, i.e., how our minds arrive at a particular judgement, rather than following objective consensual guidelines such as annotating logical entailment. However, classic work by Nisbett and Wilson (1977) suggests that our verbal reports on our mental processes can be highly inaccurate. For instance, in admission decisions, legitimate information can be used to justify preferences based on illegitimate factors such as race (Norton et al., 2006). Many studies on implicit bias also reinforces that we are not aware of our biases and thus cannot include them (i.e., the actual reasoning in our mind) in our explanations (Greenwald et al., 1998).
Explanations are necessarily incomplete. Finally, there are indeed cases where we believe that humans can provide valid mechanisms. For instance, some question answering tasks boil down to logical inference from evidence to query. In these
cases, NLP researchers need to recognize that human explanations are necessarily incomplete: people do not start from a set of axioms and present all the deductive steps (Keil, 2006; Lombrozo, 2006). Therefore, even for simple tasks such as natural language inference, we may simply give explanations such as repeating the hypothesis without presenting any axiom or deduction required to infer the label.
Implications. We cannot assume that humans are capable of providing explanations that contain valuable proximal mechanisms. The very fact that humans can still provide explanations for incorrect labels and tasks where they do not perform well suggests that one should be skeptical about whether human explanations can be used to train models as additional supervision or evaluate machine-generated explanations as groundtruths.
Note that incomplete explanations can still be very useful for NLP. We believe that recognizing and characterizing this incompleteness (e.g., which proximal mechanism is more salient to humans) is critical for understanding and leveraging human explanations for the intended goals in NLP. To summarize, we argue that human explanations are necessarily incomplete and it is important to understand and characterize this incompleteness, which can inform how we can leverage it for the intended goals in NLP.
Explanations represent a fascinating phenomenon and are actively studied in psychology, cognitive science, and other social sciences. While the growing interest in explanations from the NLP community is exciting, we encourage the community to view this as an opportunity to understand how humans approach explanations and contribute to understanding and exploring the explanation processes. This will in turn inform how to collect and use human explanations in NLP. A modest proposal is that it is useful to examine and characterize human explanations before assuming that all explanations are equal and chasing a leaderboard.
We thank anonymous reviewers for their feedback, and members of the Chicago Human+AI Lab for their insightful suggestions. This work is supported in part by research awards from Amazon, IBM, Salesforce, and NSF IIS-2040989, 2125116, 2126602.