Machine Explanations and Human Understanding
Authors: Chacha Chen
Shi Feng
Amit Sharma
Chenhao Tan
Venue: Accepted by TMLR ()

Abstract: Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even negative results. An open question, therefore, is under what conditions explanations can improve human understanding and in what way. To address this question, we first identify three core concepts that cover most existing quantitative measures of understanding: task decision boundary, model decision boundary, and model error. Using adapted causal diagrams, we provide a formal characterization of the relationship between these concepts and human approximations (i.e., understanding) of them. The relationship varies by the level of human intuition in different task types, such as emulation and discovery, which are often ignored when building or evaluating explanation methods. Our key result is that human intuitions are necessary for generating and evaluating machine explanations in human-AI decision making: without assumptions about human intuitions, explanations may improve human understanding of model decision boundary, but cannot improve human understanding of task decision boundary or model error. To validate our theoretical claims, we conduct human subject studies to show the importance of human intuitions. Together with our theoretical contributions, we provide a new paradigm for designing behavioral studies towards a rigorous view of the role of machine explanations across different tasks of human-AI decision making.

Published in Transactions on Machine Learning Research (04/2023)
Machine Explanations and Human Understanding
Chacha Chen
chacha@uchhioape edu
Deportment of Computer Science University of Chicago
Shi Feng*
shif@uchicago.co edu
Department of Computer Science University of Chicago
Amit Sharma
amsharrmmicroooft.com
Microsoft Research
Chenhao Tan
Chenhao@uchicagee
Department of Computer Science University of Chicago
Reviewed on OpenReview: https: // openreview. net/ forum? id= BFCGGF11ABG
Abstract
Explanations are hypothesized to improve human understanding of machine learning models and achieve a variely of desirable outcomes, ranging from model debugging enhancing human decision making, However, empirical studies have found mixed and even negative results. An open question, therefore, is wnder what conditions erplanations human underslanding und in what way. To address this question, we first identify three core concepts that cover most existing quantitative measures of understanding: task decision boundary, model decision boundary, and model error. Using adapted causal diagrams, we provide a formal characterization of the relationship between these concepts and human approximations (i.e., understanding) of them. The relationship varies by the level of human intuition in different task types, such as emulation and discovery, which are often ignored when building evaluating explanation methods. Our key result is that human intuitions necessary for generating and evaluating machine explanations in human-Al decision making: without assumptions about human intuitions, explanations may improve human understanding of model decision boundary, but cannol improve human understanding of task decision boundary or model error. To validate our theoretical claims, we conduct human subject studies to show the importance of human intuitions. Together with our theoretical contributions, we provide a new paradigm for designing bchavioral studies towards rigorous view of the role of machine explanations across different tasks of human-Al decision making.
improve
1 Introduction
Despite the impressive performance of machine learning (ML) models (He et al., 2015; Mnih et al., 2015; Silver et al., 2017), there are growing concerns about the black-box nature of these models. Explanations are hypothesized to improve human understanding of these models and achieve a variety of desirable outcomes, ranging from helping model developers debug (Hong et al., 2020) to improving human decision making in critical societal domains (Green & Chen, 2019a;b; Lai & Tan, 2019; Lai et al., 2020; Poursabzi- Sarrsabbi-SSngdchh et al., 2021; Zhang et al., 2020). However, the exact definition of human understanding remains unclear. Moreover,
Equual contribution.

---

Published in Transactions on Machine Learning Research (04/2023)
empirical experiments report mixed results the effect of explanations (Adebayo et al., 2022; Green & Chen, 2019a;b; Lai & Tan, 2019; Poursalvi-Sangdeh et al., 2021; Ríbeiro et al., 2016), raising an intriguing puzzle about what factors drive such mixed results. We argue that frameworks of how human understanding is shaped by machine explanations (Alvarcz-Melis et al., 2021; Yang et al., 2022) is key to solving this puzzle. We prove that effective machine explanations are theoretically impossible without incorporating human intuitions.
Roadmap and contributions. We begin by tackling the question of what human understanding we would like to achieve in the context of human-AT decision making in classification tasks (Section 2). We identify three core concepts of interest from existing literature: 1) task decision boundary (dericing the true label in the prediction problem, the key largel for decision making both for humans and models (Bansal et al., 2020; Biran & McKeown, 2017; Buçinca et al., 2020; 2021; Carton et al., 2020; Doshi-Velez & Kim, 2017;
Feng & Boyd-Graber, 2018; Gonzalez et al., 2020; Guo et al., 2019; Kiani ct al., 2020; Lai & Tan, 2019;
Lai et al., 2020; Liu et al., 2021; Nourani et al., 2021; Yoursabzi-Sangdhee et al., 2021; Weerts et al., 2019; Zhang et al., 2020)), 2) model decision boundary (simulating the model predictedd label, evidence of strong human understanding of the model (Alqaraawi et al., 2020; Buçinca et al., 2020; Chandrasekaran et al., 2018; Chromik et al., 2021; Doshi-Veler & Kim, 2017; Friedler et al., 2019; Hase & Bansal, 2020; Lakkaraju et al., 2016; Lipton, 2016; Liu et al., 2021; Lucic et al., 2020; Nguyen, 2018; Nourani et al., 2021; Poursabzi-Sangdeh et al., 2021; Ribeiro et al., 2018; Wang & Yin, 2021)), and 3) model error (recognizing whether a predicted label is wrong, a useful intermediate variable in decision making and central subject in trust (Bansal ct al., 2019; 2020; Biran & McKeown, 2017; Buçinca et al., 2020; 2021; Bussone et al., 2015; Carton et al., 2020; Feng & Boyd-Graber, 2018; Gonzalez et al., 2020; Kiani et al., 2020; Lai & Tan, 2019; Lai et al., 2020; Liu et al., 2021; Nourani et al., 2021; Poursabzi-Sanndeeh et al., 2021; Wang & Yin, 2021; Weerts et al., 2019; Zhang et al., 2020)
To enable a rigorous discussion of how machine explanations can shape human understanding, we develop a theoretical framework of human understanding with adapted causal diagrams (Section 3). Depending upon what shown to the person (e.g., the model's prediction on an instance), different causal relationships emerge among the core variables and their human approximations. To formalize these relationships, we introduce an operator, show, that articulates how assumptions/interve /iteeventttrveosss (henceforth conditions) shape human understanding. We consider two conditions: 1) whether a person has the perfect knowledge of the task and 2) whether machine predicted labels are revealed. Going through these conditions yields a tree of different scenarios for human understanding (Section 4).
We then incorporate machine explanations in the framework to vel. the utility of explanations in improving human understanding (Section 5). Our causal diagrams reveal the critical role of task-specific human intuitions in effectively making sense of explanations, and shed light on the mixed findings of empirical research on explanations. We first point out that existing explanations are mostly derived from model decision boundary. Although explanations can potentially improve human understanding of model decision boundary, they canmot improve human understanding of task decision boundary or model error without assumptions about task-specific intuitions. In other words, complementary performance (i.e., human+Al > lumma;; humann AT > A1) is impossible without assumptions about task-specific human intuitions. To achieve complementary performance, we articulate possible ways that human intuitions can work together with explanations.
Finally, we validate the importance of human intuitions through human-subject studies (Section 6). To allow for full control of human intuitions, use Wizard-of-Oa -z setup (Dahlbäck et al., 1993). Our experimental results show: 1) when we remove human intuitions, humans are more likely to agrec with prodicted labels; 2) participants are more likely to agree with the predicted label when the explanation is consistent with their intuitions. These findings highlight the importance of articulating and measuring human intuitions.
Our key theoretical results emphasize the importance of articulating and measuring human intuition in order to develop machine explanations that enhance human understanding. Our experiments act as a minimal viable product, demonstrating how we can make assumptions on human task-specific intuitions to predict when and in what way explanations can be helpful. Note that our experiment results do not directly substantiate our primary theorems, but they do provide a strong supporting evidence, showing a promising way forward in
For example, which feature is important for the decision; see full definition in Section 3.2.
2

---

Published in Transactions on Machine Learning Research (04/2023)
this emerging field of human-AI interactions. Altogether, our work contributes to the goal of developing rigorous science of explanations (Doshi-Velez & Kim, 2017).
2 Three Core Concepts for Measuring Human Understanding
In this section, we identify three key concepts of interest for measuring human understanding in human- Al decision making: task decision boundary, model decision boundary, and model error. We present high-level definitions of these concepts and formalize them in Section 3.2
We use a two-dimensional binary classification problem to illustrate the three concepts (Figure 1). Task decision boundary, as represented by the dashed line, defines the mapping from imputs to ground-trut tonnh labels: inputs on the left are positive and the ones on the right are negative. Model decision boundary, as represented by the solid line, determincs model prodictions. Consequently, the area between the two boundaries is where the model makes mistakes. This yellow highlighted background caplures model error, iee, where the model prediction is incorrect. With a perfect model, the model decision boundary would be an exact match of the task decision boundary, and model crror is empty.
To the best of our knowledge, we are not aware of any existing quantitative behavioral measure of human under- standing that does not relate to one of these three concepts of interest.* 4 Building on a recent survey (Lai et al., 2021), we identify 32 papers that: 1) use machine learning mod- els and explanations with the goal of improving human understanding; and 2) conduct empirical human studies to evaluate human understanding with quantitative met- rics. Although human-subject experiments can vary in subtle details, the three concepts allow us to organize existing work into congruent categories. We provide a reinterpretation of existing behavioral measures using the three concepts below (a detailed summary of the papers is in Appendix 5)
f()
2(-)
2(-)
Task decision bowndary
Model decision boundary
eodel
Figure 1: Illustration of the three concepts ing a binary classification problem. Task decision boundary (dashed line) defines the ground-truth mapping from imputs to labels. Model decision boundary (solid line) defines the model prodic- tions. Model error (highlighted area) represents where the model's predictions are incorrect.
Measuring human understanding of task decision boundary (and model error) via human+ perfor- mance. Similar to the application-grounder evaluation defined in Deshi-Velez & Kim (2017), the most well-adopted evaluation measurement of human understanding is to measure human understanding of the task decision boundary through human+ Al performance (Bansal et al., 2020; Biran & McKcown, 2017: Buçinca et al., 2020; 2021; Carton et al., 2020; Doshi-Velcz & Kim, 2017; Feng & Boyd-Graber, 2018; Gonzalez et al., 2020; Guo et al., 2019; Kiani et al., 2020; Lai & Tan, 2019; Lai et al., 2020; Liu et al., 2021; Nourani et al., 2021; Saungeh et al., 2021; Weerts et al., 2019; Zhang et al., 2020). In those experiments, participants are presented with machine predictions and explanations, then they asked to give final decision based on the information, with the goal of achieving complementary performance. For example, human decision-makers are asked to predict whether this defendant would re-offend within two years, given machine prediction and explanations (Wang & Yin, 2021). For binary classification problems, measuring human understanding of the model error is equivalent to measuring human understanding of the task decision boundary when machine predictions are shown.
Measuring human understanding of model decision boundary via human simulatability. A straightforward way of evaluating human understanding of model decision boundary is to measure how
PIn this work, omit subjective measures such cognitive load and subjective assessments of explanation usefulness. We present deterministic example for case of understanding and in mind. In general, one can also think of model decision boundary, task decision boundary, and model error probabilistically.
interpret this work with deterministic functions
*As will show later, understanding the mochanism behind model's prodictions (c.g., feature importance) falls under the category of understanding the model decision boundary. 5 Also available https:// //g: thub. cow/Chacha-eeeeeeoeaoaaaeeeaa SaaeechaahhhenBnne
Human- Studies.
3

---

Published in Transactions on Machine Learning Research (04/2023)
well humans can simulate the model predictions, or in other words, the human ability of forward simula- tion/prediction (Doshi-Velez & Kim, 2017). Humans are typically asked to simulate model predictions given an input and some explanations (Alqaraawi et al., 2020; Buçinca et al., 2020; Chandrasekaran et al., 2018; Chromik et al., 2021; Doshi-Velee & Kim, 2017; Friedler et al., 2019; Hase & Bansal, 2020; Lakkaraju et al., 2016; Lipton, 2016; Liu et al., 2021; Lucie al., 2020; Nguyen, 2018; Nourani el al., 2021; Poursabzi-Sangdeel et al., 2021; Ribeiro et al., 2018; Wang & Yin, 2021). For example, given profies of criminal defendants and machine explanations, participants are asked to guess what the Al model would predict (Wang & Yin, 2021).
Measuring human understanding of model decision boundary via counterfactual reasoning. Sometimes researchers measure human understanding of the decision boundary by evaluating participants' counterfactual reasoning abilities (Friedler et al., 2019; Lucic et al., 2020). Counterfactual reasoning investigates the ability to answer the 'what if question. In practice, participants are asked to determine the output of a perturbed input applied to the same ML model (Friedler et al., 2019). Lucic et al. (2020) asked participants to manipulate the input to change the model output.
Measuring human understanding of model decision boundary via feature importance. Ad- ditionally, Wang & Yin (2021) also tested human understanding of model decision boundary via feature importance, specifically by (1) asking the participants to select among list of features which one was most/lcast influential on the model's prodictions and (2) specifying a feature's marginal effect on predictions. Ribeiro et al. (2016) asked participants to perform feature engineering by identifying features to remove, given the LIME explanations. These can be viewed as a coarse inquiry into properties of the model's model decision boundary.
Measuring human understanding of model error through human trust. In some other cases, trust reliance is introduced a criterion reflecting human understanding of model error. Explanations are used to guide people to trust an Al model when it is right and not to trust it when it is wrong. Hence, by analyzing when and how often human follows machine predictions, trust can reflect the human understanding of the model error (Buçinca et al., 2021; Wang & Yin, 2021; Zhang et al., 2020). In other cases, the measure of human understanding of model error can be used as an intermediate mcasurement towards mcasuring task decision boundary (Bansal et al., 2019; 2020; Biran McKeown, 2017; Buçinca et al., 2020; 2021; Bussone et al., 2015; Carton al., 2020; Feng & Boyd-Graber, 2018; Gonzalez al., 2020; Kiani et al., 2020; Lai & Tan, 2019; Lai et al., 2020; Liu al., 2021; Nourani et al., 2021; Poursabzi-Sanglle et al., 2021; Wang & Yin, 2021; Weerts et al., 2019; Zhang et al., 2020), where human subjects are asked whether they agree with machine predictions.
3 A Theoretical Framework of Human Understanding
We introduce theoretical framework to formalize human understanding of the three core concepts in the context of human-Al decision making. This framework enables a rigorous discussion on human understanding as well as the underlying assuptioos/iiterventioo that shape hose understanding, and sets up our discussion about the role of machine explanations.
3.1 Core Functions
Formally, the three core concepts are functions defined w.r.t. a prediction problem and a ML model:
Task decision boundary is a function f: x + Y that represents the groundtruth mapping from an input X to the output Y.
Model decision boundary is another function X + Y that represents our ML model and outputs a prediction Y given an input. is usually trained to be an approximation of f. We assume that we are given a trained model g; the training process of g is not crucial for this work. Another important assumption is that Y is the best prediction that g can provide for X. In other words, g does not contain information that can improve its prediction for X.
4

---

Published in Transactions on Machine Learning Research (04/2023)
Model error is indicator of whether the model prediction differs from the groundtruth for imput: =(X, (.)) I[[(X) # g(X)), vX E X. We use x(X) for short when the omitted arguments f and are clear from context.
We call them core functions as they underpin human understanding.
Our work is concerned with a test instance that is independent from the training data and the given model g. Given X, we refer to the outputs of core functions Y, Y, and Z as the three local core variables. Note that the core functions do not involve any people; they exist even in the absence of human understanding.
3.2 Human Approximation (Understanding) of the Core Functions
We use H to represent task-specific human intuitions." 7 For example, in prostate cancer diagnosis, H may include the knowledge of the location of prostate and the association between darknoss and tumor; in sentiment analysis, H can refer to the ability to understand the semantics of language and interpret its polarity; in recidivism prediction, H may capture the intuition that older people tend to not recidivate. We define as human intuitions that are relevant to the task at hand and distinct from more general intuitions, such as the ability distinguish different colors in interpreting feature importance. Note that H may not always be valid.
We then use f" g. and = to denote the human's subjective approxinations of the core functions, each of them being function with the same domain and codomain as its objective counterpart. These human approximations can be interpreted as mental models, influenced by the human intuitions (H), which can change over time as the human-AI interaction progresses.
Table summarizes the notations for core functions and human understandings.
Given the definition of these concepts, We rephrasc common cooperative tasks in human-Al decision making in terms of the core functions and human understanding grouped by stakeholders:
For decision makers such as doctors, judges, and loan officers, the main goal is to improve their understanding of task decision boundary (f*).
For decision subjects such as patients, defendants, and loan applicants, the object of interest differ even for these three examples. Patients care about the task decision boundary more, while defendants and loan applicants may care about the model decision boundary and especially model error, and would like to figure out how they can appeal model docisions.
Model developers might be most interested model error, and the eventual goal is to change the model decision boundary.
For algorithm auditors, the main goal is to figure out whether the model decision boundary and model error conform to laws/rogulations.
Measuring human understanding empirical studies. The distance between core functions and their human approximations can be used as measure for human understanding. Since human approximations are Cheoretical construets that only exist in the human brain, we need to perform user studies to measure them. For example, we ask human to guess what the model would have predicted for a given input X; the human's 's answer H characterizes their local understanding of the model decision boundary. In the rest of the paper, one can interpret "human understanding" with this particular measurement of human approximations. Perfect human understanding thus refers to 100% accuracy in such measurement.
Training data and g not independent. choice of the word, intuition: (1) use intuition instcad of belicf, because intuition is broader than
7 "Clarification belicf. For example, knowledge is typically not counted belief; (2) use intuition instead of prior, since do not emphasize the difference between prior and posterior. fact, sometimes intuitions can be activated because showing explanations, so certainly not prior that case.
5

---

Published in Transactions on Machine Learning Research (04/2023)
Variable
Function
Description
Input instance
J:X + Y
Task decision boundary
P
9: X + Y
Model decision boundary
2:X +
Model error
Yu
fl X + Y
Human understauding of the f
9 X
Y
Human understanding of the g
2H
2 X
Human understauding the = Task-specific human intuitions Machine explanations
Core functions
Human approximations
II
Figure 2: Visualizing the relations between local vari- ables, core functions, and human approximations of them.
e(g): :0):xx->0
Table 1: A summary of notations.
We assume that the human approximation remains static and examine human's local understanding with our framework in the main paper. We note that improving human global understanding is often the actual goal in many applications, and add discussions on global understanding in Appendix D.
3.3 Causal Graph Framework for Core functions
To reason about human understanding, we need to understand how core functions relate to each other, and how interventions may affect human understanding. To do so, we adapt causal directed acyclic graphs (DAGs) to formalize a causal model for the core functions of human understanding. We explain in more details about our adaptations of the causal diagrams in Appendix C.
Let us first look at core functions the left in Figure 2. As work is concerned with the causal mechanisms between variables, different from standard causal diagrams, we make them explicit by adding a functional node (g in a square) on the edge from X to Y, indicating that g controls the causal link from X to Ý. is thus parent of Ý. X and Y are connected with a dashed line through ʃ since we do not assume the causal direction between them (Veitch et al., 2021). Z is the binary indicator of whether Y and Y are different. From this part of the figure, we have the following observations:
According to d-separation (Pearl, 2009), Ý is independent of Y given X and g-
Z is a collider for Y and Y, so knowing Z and Ý entails Y in binary classification.
On the right in Figure 2, We formulate human understanding of the core functions. Task-specific intuitions drives human mental models of the core functions. As this node generates functions, we use & to highlight. its difference from other nodes. Figure 2 shows base version of how human intuitions relate to human understanding of core variables. For now, we do not make any additional assumptions; we simply connect human intuition (H) with their understanding (JH ",, =H and omit any potential connections between them. In other words, because H has connection with f, 9, and z, fd g , are independent from f.g, Later, we will discuss more realistic instantiations on their relationships in Section 4.
According to d-separation, Figure 2 suggests that human approximation of core variables are independent from core variables given X and H, without extra assumptions about human intuitions. Therefore, our goal is articulate what assumptions we make and how they can improve human understanding.
A new show operator. Without extra assumptions, the causal direction between YH, H 2 unclear. We visualize this ambiguity by connecting nodes with undirected dashod links in Figure 3(a) as the basc diagram. This base diagram is not useful in its current state; in order to use the diagram to reason about human understanding, we need realizations of the base diagram where dashed links are replaced by solid, directional links.
To disambiguate the dashed links in the base diagram, we need to articulate how assumptions and interventions can affect human understanding, ie., H n 2 and their corresponding local variables Y, Y, Z. Thus, we introduce a new operator, show, capture interventions on nodes and odges in causal diagrams. When show is applied to core variable, that information becomes available to the human. For example, show(Y) means that the human can the model prediction Y. This operation draws equivalence between the

---

Published in Transactions on Machine Learning Research (04/2023)
(ee Rase
(b) Emuilaioon
(c) Discovery
Prediction shown Emmulation
(e) Prediction hidden
()) Prediction shown Discovery
Prediction hiddlen Discovery
Emulation
Figure 3: Causal diagrams visualizing the relationship between a human's 's local understanding (Y ÿ zH) With the base diagram at the root, we organize its realizations based on different conditions in two-level tree. Undirected dashed lines represent ambiguous causal links. Directed solid lines represent causal links. The bidirectional dashed line in subfigure (d) represents the correlation between y and Y potentially induced by the prediction model. Shaded nodes and their edges represent show operations.
core variable and the human approximated counterpart, i.e., y That is, show(Y) adds a link from y to y and removes all other edges going into ÿ effectively disambiguating the relation between ÿ Y and Z (Figure 3f)).
We introduce the new show operator. As opposed to the standard do operator, the show operator can change the causal diagram as we reason about human understanding, including adding new edges and disambiguating dashed links. Notation-wise, show allows us to specify the condition for human approximations; c.g., Y show!() denotes the human local understanding of Y given predicted label Y. As result, show provides strong flexibility in updating causal diagrams, and we believe that this flexibility provides an important first step to have a rigorous discussion on human understanding.
4 Human Understanding without Explanations
To illustrate the use of our framework, we start by analyzing human-Al decision making without the presence of explanations. We consider two conditions in Figure 3.
Condition 1 emulation discovery (Effect of sY)w(Y)). The first condition is assumption about human knowledge, i.e., that the human has perfect knowledge about task decision boundary; in other words, f" perfectly matches f and y = Y for all inputs (note that this is an assumption about H). Tasks where human labels are used as ground truth generally satisfy this condition, e.g., topic classification, reading comprehension, and object recognition. We follow Lai et al. (2020) and call them emulation tasks, in the sense that the model designed to emulate humans; by contrast, discovery tasks are the ones humans do not have perfect knowledge of task decision boundary (c.g., deceptive review detection and recidivism prediction).
Figure 3b and illustrate difffernces between emulation and discovery tasks. First, human local understanding of task decision boundary Y collapsed with Y in emulation tasks (Figure 3b), so no edge goes into Y and Y affects and F However, unlike emulation tasks where y = Y, the edge connections remain the same in discovery lasks (Figure 3c), i.e., we are unable rule out any connections for now. Hence, human understanding of task decision boundary is usually not of interest in emulation tasks (Chandrasekaaan et al., 2018; Nguyen, 2018). In comparison, human understanding of both model decision boundary and task decision boundary has been explored in discovery tasks (Bansal et al., 2020; Feng & Boyd-Grabbe 2018; Park et al., 2019; Wang & Yin, 2021).
Emulation and discovery can be ends of continuous spectrum. The emulation discovery categorization determines the set of causal diagrams that applies to the problem; this decision the discretion of practitioners who design and interpret experiments using our framework.
7

---

Published in Transactions on Machine Learning Research (04/2023)
Condition 2 3-prrdiction shown us. hidden (Effect of show(YY)) An alternative condition is an intervention that shows the model prediction Y to the human. In this way, human would then gain a perfect understanding of the local model decision boundary and always predict y =.
We start with emulation tasks, where the relationships are relatively straightforward because the human understanding of task decision boundary is perfect (Y = Y). When & shown (Figure 3d), human =
understanding of local predicted label becomes perfect, i.e, = Y. It follows that z - + "*)= I(Y # Y) Z. This scenario happens in debugging for emulation tasks, where model developers know the true label, the predicted label, and naturally whether the predicted label is incorrect for the given instance. In this case, the desired understanding is not local, but about global model decision boundary. Refer to Appendix D for discussions on global understanding. In comparison, when Y is not shown (e.g, an auditor tries to extrapolate the model prediction), recall Y " = Y in emmlation tasks, so Y can affect y and z" As shown Figure 3e, the connection between ÿ and 7H remains unclear.
In discovery tasks, when Y is shown (Figure 3f), 8 P. However, the relationships between y and 2H remain unclear and can be potentially shaped by further information such machine explanations. When Y is not shown (Figure 3g), no information can be used to update the causal diagram in discovery tasks. Therefore, Figure 3g is the same as the base diagram where all interactions between local understandings are possible, which highlights the fact. that, no insights about human understandings can be derived without any assumption or intervention.
Implications. Our framework reveals the underlying mechanism of human local understanding with two important conditions: 1) knowing the task decision boundary; and 2) showing machine predictions Y. Such conditions allow us to disambiguate connections between human understanding of core variables. For example, in emulation with prediction shown (Figure 3d), the relationship between all variables is simplified to a deterministic statc.
Another implication is that we need to make explicit assumptions in order to make claims such as human performance improves because human understanding of the model error is better (i.e., humans place appropriate trust in model prodictions). As there exist ambiguous links between variables, for example, in discovery tasks with prediction shown (Figure 3f), we can not tell whether it is y + 2H or 2H - Y nor can we tell from experimental data without making assumptions. The alternative hypothesis to "appropriate trust- + improved task performance" is that y directly improves human understanding of the task decision boundary directly (i.e., "model predictions + improved task performance"). In these ambiguous cases, explanations may shape which scenario is more likely, and it is critical to make the assumptions explicit to support causal claims drawn from experiments.
5 Machine Explanations and Human Intuitions
In this section, we discuss the utility and limitations of machine explanations using our framework. We first show that without assumptions about human intuitions, explanations can improve human understanding of model decision boundary, but not task decision boundary or model error. As a result, complementary performance in discovery iasks is impossible. We then discuss possible ways that human intuitions can allow for effective use of explanations and lay out several directions for improving the effectiveness of explanations. Our analyses highlight the importance of articulating and measuring human intuitions in using machine explanations to improve human understanding
5.1 Limitations of Machine Explanations without Human Intuitions
Existing explanations are generated from g (Figure 4(a)). We first introduce explanation (E) to our causal diagram. Since the common goal of explanation in the existing literature is to explain the underlying mechanism of the model, E is derived from a function, e(g), which is independent from f given g For example, gradient-based methods use gradients from g to generate explanations (Bach et al., 2015; Sundararajan et al., 2017). Both LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) use local surrogate models to compute importance scores, and the local surrogate model is based on g. Counterfactual
8

---

Published in Transactions on Machine Learning Research (04/2023)
explanations (Mothilal et al., 2019; Wachter et al., 2017) typically identify examples that lead to different predicted outcome from g. As a result, e(g) and E do not provide information about or beyond g.
In addition, we emphasize that there should be no connection between E and task-specific intuitions, H. Conceptually, only task-agnostic human intuitions are incorporated by existing algorithms of generating explanations. It is well recognized that humans cannot understand all parameters a complex model, so promoting sparsity incorporates some human intuition. Similarly, the underlying assumption for transparent models is that humans 2016) or generalized linear models (Caruana et al., 2015; Nelder & Wedderburn, 1972). However, none of these assumptions on human intuitions are task-spocife, i..., related to task decision boundary, model error, or human understanding of them.
fully comprehend certain class of models, c.g., decision sets (Lakkaraju et al.,
Now we discuss the effect of explanations on human understanding without assuming any task-specific human intuitions (i.c., without adding new edges around H).
Without assumptions about task-specific human intuitions, explanations can improve human understanding of model decision boundary, but cannot improve human undorstanding of task decision boundary or model error. We start with the cases where predicled labels are not shown.
Theorem When predicted labels are not shown for an instance (X), explanation (E) that derived from g alone can improve human understanding of the model decision boundary (i.e., increasing the correlation between H and )).
Proof. Figure 4(b1) shows the subgraph related to y and Y from Figure 2. Without explanations, Y and ÿ- are independent given X. Figure 4(b2) demonstrates the utility of machine explanations. Because of the connection with y through and e(g), the introduction of E can improve human understanding of model decision boundary, Y.H.
Note that our discussion on improvement is concerned with the upper bound of understanding assuming that humans can rationally process information if the information is available. This improvement holds regardless of the assumption about human knowledge of Y (i.e., both in emulation and discovery tasks).
When predicted labels are shown, improving local human understanding of model decision boundary is irrelevant, so we focus on task decision boundary and model crror.
Thcorem 2. When predicted labels are shown for an instance (X), explanation (E) that is derived from only canmot improve humnan underslanding of lask decision boundory or model error (L.e., increasing the correlation between and H or the correlation between Z and z beyond the improvement from showing predicted labels.
Proof. First, in emulation lasks, when provided with predicted labels (i.e., show humans would achieve perfect accuracy at approximating the three core variables. That is, the perfect local understanding already holds in emulation tasks without machine explanations, explanations have no practical utility in this setting. Therefore, machine explanations cannot help humans achieve better local approximation than showing predicted labels in local understanding.
In comparison, Figure 4(c1) shows the diagram for the more interesting case, discovery tasks. According d-separation, given the prediction y and X, the explanation E is independent. of Y and Z. That is, E cannot. provide any extra information about Y (task decision boundary) and Z (model error) beyond the model (g)- As our definition about g and Y in Section 3 entails, the model cannot provide any better approximation of Y than Y.9 For a person who has no intuitions about a task at all, the best course of action is to follow machine predictions. Therefore, if we do not make assumptions about human intuitions, E cannot bring any additional utility over showing Y.
9In other words, g cannot provide bad & while offering useful information E that he directly used improve y for g itself.
9

---

Published in Transactions on Machine Learning Research (04/2023)
sway
22
AE
8
8
g
+
460
y2
600 (N))
(02)
(cl)
(2))
Figure 4: (a) E generated from g. (b1) Y and ÿ are independent given X. (b2) The utility E: E can improve human understanding of model decision boundary. (cl) E cannot improve human understanding of lask decision boundary and model error without human intuitions. (c2) Combined with human inluitions, E improve task decision boundary and model error. We use orange lines to highlight the links that lead to the utility of E.
An important corollary thus Corollary 2.1. Complementary performance is impossible in discovery tasks without extra assumptions about human inluilions.
Our results suggest that the common hypothesis that explanations improve human decision making, iee,, bringing closer to Y, hinges on assumptions about H. Note that the scope of these two theorems is for local human understanding, please refer to Appendix D for discussion global understanding.
As concrete example, consider the case of topic calegorization with an alien who does not understand human language (i.c., guaranteeing that there is no intuition about the task). Machine explanation such feature importance cannot provide meaningful information about model error to the decision maker (the alien). For instance, highlighting the word "basketball" as an important feature can help the alien understand the model decision boundary and learn that "basketball" associated with the sports category from the model's perspective (Theorem 1). However, due to the lack of any intuition about human language, the alien interpre(s "baske(ball" as string of bits and cannot achieve any improved understanding in what constitutes the sports category (i.e., the task decision boundary) and model error (Theorem 2). This extreme example illustrates how the lack of relevant task-specific intuitions may limit the utility of explanations in deceptive review delection (Lai & Tan, 2019). Our experiments in Section 6 will simulate a case without human intuitions.
In comparison, machine explanations have been shown to be useful model dcbugging scenarios (Ribeiro et al., 2016). In this case, the implicit assumption about human intuition that developers know the groundtruth (task decision boundary) and explanations derived from g can improve the understanding of model decision boundary, which in turn improves the understanding of model error. This improved understanding of model error is essential to model debugging.
5.2 Machine Explanations + Human Intuitions
Next, we discuss how explanations can be integrated with human intuitions to achieve an improved under- standing in discovery tasks (recall that and y are entailed emulation tasks when Yis shown). We have seen that E itself does not reveal more information about or Z beyond g. Therefore, an important role of E in shaping human intuitions and influencing the connection between f.g.2 with H. We present two possible ways.
Activating existing knowledge about model error. E can activate existing human knowledge that can reveal information about model error (Figure 4(c2)). We examine two sources of such knowledge that are concerned with what information should be used and how. First, human intuitions can evaluate relevance, iee., whether the model leverages relevant information from the input based on the explanations. For example, human intuitions recognize that "Chicago" should not be used for detecting deceptive reviews or that race should not be used for recidivism prediction, so model prediction relying on these signals may be more
10

---

Published in Transactions on Machine Learning Research (04/2023)
likely wrong. The manifestation of relevance depends on the explanation's form: feature importance directly operates on pre-defined features (e.g,, highlighting race for tabular data or word in review), example-based or counterfactual explanations narrow the focus of attention to a smaller (relevant) area of the input.
Second, human intuitions can evaluate mechanism, i.., whether the relationship between the input and the output is valid. Linear relationship a simple type of such relation: human intuitions can decide that education negatively correlated with recidivism, and thus that a model making positive predictions based education is wrong. In general, mechanisms can refer to much more complicated (non-linear) relations between (intermediate) imputs and labels.
Figure 4(c2) illustrates such activations in causal diagrams. The link from E' to H highlights the fact that human intuitions when E is shown are different from H without E because these intuitions about relevance and mechanism would not have been useful without machine explanation. We refer to in Figure 4(c2) as (E) - If (E) is correlated with 2 (indicated by the dash link), then z is no longer independent from Z (e.g,, incorrect connection between education and recidivism is correlated with model error) and can thus improve z" and then Y because Z is a collider for Y and Y, leading to complementary performance. It is important to emphasize that this potential improvement depends on the quality of Hahow(E); Note that in this case, the improvement in z" also disambiguates the relationship between Y" and 2" (i.e., appropriate trust + improved decision making). The lack of useful task-specific human intuitions can explain the limited human- AI performance in deceptive review detection (Lai & Tan, 2019).
Expanding human intuitions. Another way that explanations can improve human understanding is by expanding human intuitions (see Figure 10 in the Appendix). As an example of counterintuitive explanations, the word "Chicago" is highlighted as an important factor for deceptive review (Lai et al., 2020) for two reasons: 1) deceptive reviews in this dataset are written by crowdworkers for hotels in Chicago; 2) people are less likely to provide specific details (such which street/speciff neighborhood) when they write fictional texts. Highlighting the word for "Chicago" (relevance) and its connection with deceptive reviews (mechanism) is counterintuitive to humans because this is not part of existing human knowledge. But if machine explanations can expand human intuitions and help humans derive theory I, this can lead to improvement in human understanding of task decision boundary (i.e., humans develop new knowledge from machine explanations) Formally, the key change in the diagram for this scenario is that E influences human intuitions in the next. time step H4+1-
5.3 Towards Effective Explanations for Improving Human Understanding
Machine explanations are only effective if we take into account human intuitions. We encourage the research community to advance our understanding of task-specific human intuitions, which are necessary for effective human-Al decision making. We propose the following recommendations.
Articulating and measuring human intuitions. It is important to think about how machine explanations can be tailored to either leveraging existing human knowledge or expanding human intuitions, or other ways that human intuitions can work together with explanations.
First, we need to make these assumptions about human intuitions explicit so that the research community can collectively study them rather than repeating trial-and-error with the effect of explanations an end outcome such as task accuracy. We recommend the research community be precise about the type of tasks, the desired understanding, and the required human intuitions to achieve success with machine explanations.
Second, to make progress in experimental studies with machine explanations, we need to develop ways to either control or measure human intuitions. This can be very challenging in practice. We will present an experiment where we control and measure human intuitions in human-Al decision making as an initial step.
Incorporate o and z into explanations. An important premise for explanations working together with human intuitions is that machine explanations capture the mechanism or the relevance underlying the model. Indeed, faithfulness receives significant interest from the machine learning community for the sake of
11

---

Published in Transactions on Machine Learning Research (04/2023)
Person 1/8:
Person 1/8:
predicts
B3K
The All predicts that the person's income below 50k person's 23.
mainty because the personts Feature is 0.07.
Feature
0.85
Education: Masters
Festure
0.07
Age:
Please select predictions
Please select your predistion:
50K
(a) Anonymized.
(b) Regular
Figure Screenshots of the interfaces for the anonymized group (a) and the regular group (b).
explaining the mechanisms of a model. However, faithfulness to g alone is insufficient to improve human understanding of task decision boundary and the model crror.
In order to effectively improve human understanding of and 2, would be useful to explicitly incorporate to report the error rate in a development set. In the case of deceptive review detection, it could be when "Chicago" is used important feature, the model is 90% accurate. This allows humans to have access to part of model error and have more accurate z
f and z into the generation process of E. For example, basic way incorporate
To summarize, we emphasize the following three lakeaways:
Current machine explanations are mainly about the model and its utility for human understanding of the task decision boundary and the model error is thus limited.
Human intuitions are a critical component to realize the promise machine explanations in improving human understanding and achieving complementary performance.
We need to articulate assumptions about human intuitions and measure human intuitions, and incorporate human intuitions, f. and into gencrating machine explanations.
6 Experiments
In light of our formal framework, emphasize the importance of incorporating human intuition for effective explanations. We designed our experiment as an example of applying our theoretical framework, to demonstrate how our framework can inform experimental design and analysis. While our experiment doesn't: directly validate the theorems, it provides strong supporting evidence for the converse, illustrating that by explicitly articulating assumptions about human intuition, we can begin to predict situations and mechanisms in which explanations can be beneficial.
Our experiments test two hypotheses about the impact of human intuition on their interaction with machine explanations. First, when people do not have sufficient intuition to judge whether the model is correct, they are more likely to agree with the model (H1; see discussion on Figure 4(c1)). Second, when people do have intuition, they are more likely to agree with the model when model explanations are consistent with their intuitions (H2; see discussion on Figure 4(c2)).
To simulate real-world decision making, we follow the standard cooperative setting where the participants make predictions with the assistance of both model prediction and feature importance explanation. We use a synthetic model that allows to create feature importance explanations such that either the participant has intuition about the highlighted feature or the highlighted feature importance agrees/ agrees/disagress disagrees with the participant's intuition.
6.1 Experiment Design
We focus the key considerations here and more details can be found in Appendix F.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Task: income prediction. Our hypotheses are about how people's decisions are affected by the aligmment between their intuitions and the information presented about the model. So it crucial that we can manipulate this aligmment and control for confounders. Inspired by the Adult Income dataset (Blake, 1998), we choose the task of predicting a person's anmual income based on their profile because people generally have intuitions about what factors determine income but are unlikely to know every person's income (hence a discovery task). We simplify the available features so that we can control for confounders. In our version, each profile contains two attributes, age and education, and the participant makes a binary prediction: whether the person's 's annual income is above below $50K.
Human intuition in income prediction. Following Section 5.2, we consider human intuitions on relevance and mechanism. We define relevance (R) as the intuition that education is more important than age in determining income, and mechanism (M) as the intuition that income positively correlates with both features.
To better understand what these intuitions entail, let's consider Alice who believes R and M. For a profile with high education and low age, as Alice believes that education is more important than age (R), she will rely more on education; as she believes that education positively correlates with income, she will likely predict high income without any AI assistance. In Figure 6, We use the background color to represent Alice's likely predictions: blue for high income and red for low income. When machine prediction and explanation are shown, these intuitions can be used to make predictions.
Alignment with human intuition. Next, we explain how we implement explanation alignment, still using participant Alice as the example. Intuitively, aligument with R is determined by whether the explanation highlights the right feature. Since R specifies that education is more important than age, feature importance explanation that's aligned with R should highlight education. Similarly, aligmment with M is determined by whether the model prediction is consistent with the highlighted feature: if the highlighted feature has a high value, the model should predict high income. In Figure 6, we use a cross to visualize the violation R and border for the violation of M. Note that even if the explanation violates R (i.e., highlighting age instead of oducation), M can be supported if high age + high income or low age + low income.
Measuring human intuitions. Participants may not necessarily hold these assumed intuitions (i.e., R and M), we thus measure human intuitions by asking participants about which feature more important. and whether the correlation is positive (see Appendix Section F.3).
Removing human intuition. To study H1, we need to "remove" human intuitions. To do that, we anonymize the fcatures (education and age) to feature and feature B. Figure 5a provides an example profile with removed intuitions. As result, participants should have limited intuitions about this task, especially relevance.
Synthetic data. To understand human decision making with the information provided from the model, we consider four groups of data in the table in Figure 6: AB are consistent. with both R and M, CD violates R. EF violates M, and GH violates R and M. Figure 6 also presents pictorial view. A-H enunerates all possible combinations of logically consistent explanations and predictions in the two off-diagonal quadrants. We focus on these two quadrants because 1) relevance does not matter for profiles that are high education & high age or low education & low age, and 2) participants are more likely to ignore information from our models in the other two quadrants.
Agrcement as the evaluation metric. Since our hypotheses are formed around the agreement between human and model predictions, the focus is on model error. We infer z by measuring the agreement between human prediction with AT assistance HY and model prediction (Y).
Summary. To summarize, there are two groups in our experiment, regular and anonymized. Each participant sees eight (ABCDEFGH) randomly shuffled examples. Since we evaluate two intuitions, we extend our two main hypotheses into the following four. We use agreement(.) represent the average agreement on multiple data groups.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Identifier
Education
Age
Prediction
Explanation
R education relevant
A
High
Low
>50K
Education
income than
B
Low
High
<50K
Education
M - age and education are positively correlated with Income
C
High
Low
<00K
Age
Edu
x
x
D
Low
High
>50K
Age
E
High
Low
<50K
Education
Prediction
/
X
K
Low
High
>50K
Education
Prediction 50k
G
High
Low
>5K
Age
M is violated
Low
High
<50K
Age
Note: columns and M, (()) means type of human intuition supported (violated). columa y, v(X) means the label given by Alice's intuition same as (different from) the predicted label.
Age
Figure 6: We present our instances on the left and visualize them on the right. The background color represents Alice's likely predictions: blue (red) means $50K (< $50K).
H1 (Over-reliance without intuition): Without sufficient intuition, people tend to follow model predictions: agreement anonymized) > agroement(regularr)
H2a (Alignment with R & M increases agreement): In the regular group, people are more likely to agree with model predictions when their inluitions are supported: agreement(ABB > agreement(GH) This hypothesis is an ideal test on "explanation + improved understanding of model error" because model predictions are the same in GH as AB, any observed difference should be attributed to the effect of explanations.
H2b (Alignment with R correlates with agreement): In the regular group, people more likely to agree with model predictions when R is supported: agreement (AB) agreement(CD),
H2c (Alignment with M correlates with agreement): In the regular group, people are more likely to agree with model predictions when M is supported: agreement(AB)) (AB) > agreement(EF).
For H1, we use t-test to compare subjects in different conditions. For H2, we have a within-subject design and use paired (-test to compare the agreement rate of the same participants for instances with different identifiers.
Limited validity of H2b, H2c for evaluating explanation consistency. We emphasize that although H2b and H2c reasonable hypotheses, they cannol. support the causal effect. of explanation consistency. As shown in Figure 3(f), the link between Y and z is unclear in discovery tasks when predicted labels are shown. With our simplified setup, when only R or M is violated, y is alrcady different from Ý without any assistance. Even if explanation ion does not shape H (Figure 4(c1)) (i.e., humans ignore machine explanations) H2b and H2c can hold. In contrast, H2a can only hold when explanation works together with human intuition and z + y (Figure 4(c2)). In other words, as predictions are the same in GH as AB, any observed difference should be attributed to the effect of explanations. This discussion showcases the utility of our framework in experiment. design.
6.2 Study Details
We use crowdsourcing platform Prolific and recruit 242 participants; 136 in the regular group, 106 in the anonymized group, following our power analysis. Participants' age range from 18 to 79 with an average of 38. There are 121 female and 121 male participants. Participation language is English. We also prescreen participants based on their approval rate (97% to 100%). The median time taken for each participant is 3.32 minutes and they are paid S0.60, which leads to an hourly payment of $10.85. Among the regular, 70 hold the assumned intuitions.
limited to adults in the US and first
Participants are first presented with brief information about the study and a consent form. Next, for the regular group, to mcasure their intuitions, they are askod about which feature is more important and
1htttps://prolfficc.co
14

---

Published in Transactions on Machine Learning Research (04/2023)
whether the correlation is positive. For anonymized group, they skip this step. Then, participants proceed to complete the main part of the study, in which they answer 8 adult income prediction questions. Figure 5 shows screenshots of the question answering page for the two groups. The order of instances is randomized across participants. As a final step, they complete an exit survey.
Data. Specifically, we construct 'high education low age' samples as masters with age varying from 23 to 26. 'low education high age' samples are middle school with age varying from 46 to 49. To avoid the task being too artificial, we use different. ages for A-H for participant. To further remove the potential effect due to the differences between ages, create two groups of data so that the average age is the same for ABCD and EFGH. The full dataset can be found in Appendix Table 3. We randomly assign participants to a data group and ensure that each group is assigned to similar number of participants.
6.3 Results
H1 (Over-reliance without intuition). In order to compare the agreement across two conditions (anonymized vs. regular group), we compute user-level agreement in each condition and run the independent t-test betweon users in the two conditions. Consistent with H1, our results show that users in the anonymized group are more likely to agree with AI compared with users in the regular group (t= 7.29,p < 0.001). The average agreement rate for users in the anonymized group is as high as 70.64%, compared to 54.32% in the regular group. In other words, without any strong intuitions about the underlying task, humans tend to rely on AI predictions.
H2a (Alignment with R & M; explanation consistency). Consistent with the hypothesis, the average agreement rate of AB is 90.71% and GF is 83.57%, and the difference is statistically significant (t. = p p- = 0.017). This result is consistent with early work on explanation coherence (Thagard, 1989).
H2b (Aligmment of R; explanation consistency). We use paired t-test on the users that hold the assumed intuitions in Figure 6. We first investigate the agreement of R. Consistent with our hypothesis, the agreement of AB is 90.71%, much higher than CD (25.00%). The difference is statistically significant. (t 14.25.p p 0.001).
H2c (Alignment with M; explanation consistency). Similarly, we investigate agreement for alignment with M. Consistent with our hypothesis, the agreement with AB is 90.71%, much higher than that with EF 14)). The difference is statistically significant (t = 15.01, p < 0.001).
In summary, results from our experiment have confirmed all the hypotheses, and demonstrate how task-specific human intuitions shape the outcome of a study on the effect of machine explanations.
7 Conclusion
In this work, we propose the first theoretical work to formally characterize the interplay between machine explanations and human understanding. We theoretically prove the necessity of incorporating human intuition for effective explanations. Effective human-AI decision making will require formalization of both human and AT's contribution in decision making. To this end, we recommend the research community explicitly articulate human intuitions involved in research hypotheses and experimental design. Hypotheses such as "explanations improve human decisions" can only provide limited insights because they are dependent on specific human intuitions and may not be generalizable to other contexts. In algorithm development, in order to effectively improve human understanding of f and 2, would be useful to explicitly incorporate f and z into generating machine explanations.
Acknowledgement
We thank the anonymous reviewers for their insightful comments. We also thank Victor Veitch and Vera Q. Liao for their insight.ful comments and suggestions, Giang Nguyen for suggesting relevant citations, and
15

---

Published in Transactions on Machine Learning Research (04/2023)
members of the Chicago Human+AAl lab for their thoughtful feedback. This paper is supported by in part by CDAC discovery grant at the University of Chicago and an NSF grant, IIS-2040989.
References
Ashraf Abdul, Christian von der Weth, Mohan Kankanhalli, and Brian Y Lim. Cogam: Measuring and moderating cognitive load in machine learning model explanations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2020.
Julius Adebayo, Michael Muelly, Harold Abelson, and Been Kim. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations, 2022.
Ahmed Alqaraawi, Martin Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Berthouze. Evaluating saliency map explanations for convolutional neural workorks a user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 275-285, 2020.
David Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, and Jennifer Wortman Vaughan. From human explanation to model interpretability: A framework based on weight of evidence. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), volume 8, pp. 3, 2021.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Plo$ one, 10(7):e0130140, 2015.
Gagan Bansal, Besmira Nushi, Ece Kamar, Walter Lasecki, Daniel S Weld, and Eric Horvitz. Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Compulation and Crowdsourcing, pp. 2 2-,1,- 2019.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S. Weld. Does the whole exceed its parts? the effect of ai explanations complementary team performance. arXiv preprint vxXi:2006.147772 2020.
Or Biran and Kathleen McKeown. Human-centric justification of machine learning predictions. In IJCAL, volume 2017, pp. 1461-1467, 2017.
Catherine Blake. Uci repository of machine lcarning databases. Mitturn/MMReposttoryy html, 1998.
http://www/ ies. uci. edu/-
Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 454-464, 2020.
Zana Buçinea, Maja Barbara Malaya, and Krzysztof Gajos. To trust or think: Cognitive forcing functions can reduce overreliance on in ai-assisted decision-making arXiv preprint Xix::21102 09692, 2021.
Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. The role of explanations on trust and reliance in clinical docision support systems. In 2015 International Conference on Healthcare Informatics, pp. 160-169. IEEE, 2015.
Samuel Carton, Qiaozhu Mei, and Paul Resnick. Feature-based explanations don't help people detect misclassifications of online toxicity. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pp. 95-106, 2020.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-1730, 2015.
Arjun Chandrasckaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make vqa models more predictable to human? Xiv preprint arxio: 12366, 2018.
16

---

Published in Transactions on Machine Learning Research (04/2023)
Michael Chromik, Malin Eiband, Felicitas Buchner, Adrian Krüger, and Andreas Butz. I think get your point, ai! the illusion of explanatory depth in explainable ai. In 26th International Conference Intelligent User Interfaces, pp. 307 317, 2021.
Julien Colin, Thomas Fcl, Rémi Cadène, and Thomas Serre. What cannot prodict, do not understand: A human-centered evaluation framework for explainability methods. Advances in Newral Information Processing Systems, 35:2832-255 2845, 2022.
Nils Dahlbück, Arne Jönsson, and Lars Ahrenberg. Wizard of oz studies studies-why why and how. Knowledge--asee systemss 6(4):258 266, 1993.
Kate Donahue, Alexandra Chouldechova, and Krishnaram Kenthapadi. Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. arXiv preprinl arXiv:2202. 08821, 2022.
Finale Doshi-Velez and Been Kim. Towards a rigorous science interpretable machine learning. arXiv preprint NE4::1770.06000 2017.
Shi Feng and Jordan Boyd-Graber, Graber. What can ai do for me: Evaluating machine learning interpretations in cooperative play. Xiv preprint arXiv:1810.096.1 2018.
Sorelle A Friedler, Chitradeep Dutta Roy, Carlos Scheidegger, and Dylan Slack. Assessing the local inter- pretability of machine learning models. orXiv preprint arXiv: 19:2:990.0000 0020000111, 2019.
Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, and Srinivasan lyer. Human evaluation of spoken vs. visual explanations for open-domain qa. arXiv preprint 2.:::0012.1015: 2020.
Ben Green and Yiling Chen. Disparate interactions: An algorithm-inn-tte-loop analysis of fairness in risk assessments, In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 90-99. ACM, 2019a.
Ben Green and Yiling Chen. The principles and limits of algorithm-in- in-the-loop decision making, Proceedings of the ACM on Human- Human-Computer Computer Interaction, 3(CSCW):50, 2019b.
Shunan Guo, Fan Du, Sana Malik, Eunyee Koh, Sungchul Kim, Zhicheng Liu, Donghyun Kim, Hongy uan Zha, and Nan Cao. Visualizing uncertainty and alternatives in event sequence predictions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1 1-12, 2019.
Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprinl arXiv:2005.0183 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 1034, 2015.
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explainable ai: Challenges and prospects. arxiv preprinl urxiv:1814 04608, 2018
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. Human factors in model interpretability: Industry practices, challenges, and needs. Proceedings of the ACM on Human- Contuuerrrrterrr Interaction, 4(CSCW1): 1 26, 2020.
Amirhossein Kiani, Bora Uyumazturk, Pranav Rajpurkar, Alex Wang, Rebecca Gao, Erik Jones, Yifan Yu, Curtis P Langlotz, Robyn Ball, Thomas J Montine, et al. Impact of a deep learning assistant on the histopathologic classification of liver cancer. NPJ digital medicine, 3(1):1-8, 2020.
Sunnie SY Kim, Nicole Meister, Vikram Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: evaluating the human interpretability of visual explanations. In Computer Vision-ECCT 2022: 17th Europeon Conference, Tel Awir Israel, October 23 27, 2022, Proceedings, Part XII, pp. 280 298. Springer, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2023)
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(())2377229))) 2018.
Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman, and Finale Doshi- Velez. Human evaluation of models built for interpretability. In Proceedings of the AAAT Conference on Human Computation and Crowdsoureing, volume 7, pp. 59 67, 2019.
Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Fairness, Accoumlability, and Transporency, pp. 29-38, 2019.
Vivian Lai, Han Liu, and Chenhao Tan. why is' chicago' deceptiveee towards building model-driven tutorials for humans. In Proceedings of the 2020 CHI Conference Human Factors in Computing Systems, pp. 1-13, 2020.
Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards a science of human-ai decision making; A survey of empirical studies. orXiv preprint arXiv:2112. 11471, 2021.
Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1675 1684, 2016.
Brian Y Lim, Anind K Dey, and Daniel Avrahami. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, pp. 2119-2211 2128, 2009.
Zachary C Lipton. The mythos model interpretability. arXiv preprint 1xXi::606.034000 2016.
Han Liu, Vivian Lai, and Chenhao Tan. Understanding the effect of out-of-distribution examples and interactive explanations human-ai decision making. arXiv preprint arXi::2101.0553 2021.
Ana Lucic, Hinda Haned, and Maarten de Rijke. Why does my model fail? contrastive local explanations for retail forecasting. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 90 90-9999999 98, 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of NTPS, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Ramaravind Kommiya Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. arXiv preprint arXiv: 07697, 2019.
John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. Journal of the Royal Statistical Society: Series A (Geneml), 135(3):370-355 1972.
Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies, Volume (Long Papers), pp. 1069 1078, 2018
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Newral Information Processing Systems, 34: 26422-26436, 2021.
Mahsan Nourani, Chiradecp Roy, Jeremy Block, Donald Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. Anchoring bias affects mental model formation and user reliance in explainable ai systems. Tn 26th International Conference Intelligent User Interfaces, pp. 340-350, 2021.
18

---

Published in Transactions on Machine Learning Research (04/2023)
Joon Sung Park, Rick Barber, Alex Kirlik, and Karrie Karahalios. A slow algorithm improves users' assessments of the algorithm's accuracy. Proceedings of the ACM on Human- Compuler Interaction, 3 (CSCW):1-BB 15, 2019.
Judea Pearl. Causality. Cambridge university press, 2009.
Ellen Peters, Daniel Västffälll, Paul Slovic, CK Mertz, Ketti Mazzocco, and Stephan Dickert. Numeracy and decision making. Psychological science, 17(5):407 413, 2006.
Forough Poursabzz-SSngdehh Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Faclors in Compuling Systems, pp. 1 1-, 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of KDD, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Auchors: High-precision model-agnostic explanations.
In AAAL, volume 18, pp. 1527-1535, 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, Toannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319 3328. PMLR, 2017.
Mohammad Reza Tacsiri, Giang Nguyen, and Anh Nguyen. Visual correspondenco- basod explanations improve ai robustness and human-ai team accuracy. Advances in Neural Information Processing Systems, 35:34287-34301 2022.
Paul Thagard. Explanatory coherence. Behavioral and brain sciences, 12(3):435 467, 1989.
Victor Veitch, Alexander D' D'Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. In Advances in Neural Information Processing Systems, 2021.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
Xinru Wang and Ming Yin. Are explanations helpful? a comparative study the effects of explanations ai-assisted decision-making, In 26rd International Conference on Intelligent User Interfaces, 2021.
Hilde JP Wecrts, Werner van Ipenburg, and Mykola Pechenizkiy. A human-grounded evaluation of shap for alert processing. XrXiv preprint arXic: 03322,, 2019.
John Winn. Causality with gates. In Artificial Intelligence and Statistics, pp. 1314 1322. PMLR, 2012.
Scott Cheng-Hsin Yang, Nils Erik Tomas Folke, and Patrick Shafto. A psychological theory of explainability. In International Conference on Machine Learning, pp. 25007 25021. PMLR, 2022.
Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making, arXiv preprint 2rXiv:2001.02) 2020.
19

---

Published in Transactions on Machine Learning Research (04/2023)
A Limitations and Potential Negative Societal Impacts
Limitations. We present a deterministic example for the three core concepts in the paper for case of understanding. In general, one can also think of model decision boundary, task decision boundary, and model eтror probabilistically. Our three core concepts can be applied in most classificaiton tasks. However, for other tasks like generation, the notion of model decision boundary and model error require substantial future work.
Our theoretical framework only first step towards understanding the interplay between machine expla- nations and human understanding. For instance, do not consider the effect of showing Y on human intuitions. Our discussions are mostly based information entailed by causal diagrams without accounting for psychological biases in human intuitions (e.g., issues related to numeracy (Lai & Tan, 2019; Peters et al., 2006)). Our work focuses on understanding the role of machine explanations, while another recent study provides a deep dive related to our discussion in Section 4 by examining human-AI collaboration in different. distributions (regimes) of instances (Donahue et al., 2022).
Our work addesses the quantitative measures of human understanding in the context of human-Al decision making in classificaiton tasks. More generally, Hoffman et al. (2018) aims to summarize varicty of psychometric measurements for explainable AI.
Potential negative socictal impacts. Our work is highly thcoretical, and can shed light on both positive and negative societal impacts of interpretable machine learning. The potential negative impacts mainly from the possible downstream applications. The implications drawn from our work that researchers should incorporate human intuitions into explanation methods can affect many AI applications in the rcal-world. The risks arise especially in high-stake domains, such as recidivism task and healthcare. Black box machine learning models have shown bias in lot of applications. The way that human intuition should be included in generating explanations can further pose Al bias into human decision making process if used inappropriately, thus exacerbating existing incqualities. For example, when human intuition are exploited in way that the human decision maker will more likely to follow AT predictions regardless, any existing algorithmic bias in this case may exacerbates social inequities.
B A summary of recent empirical studies
We surveyed literature of recent empirical studics that include quantifiable metric for evaluating human understanding with machine explanations. In Table 2 we present a summary recent empirical studies and provide a reinterpretation with the three core concepts: model decision boundary g, model error 2, and task decision boundary f, as we defined in the main paper.
C Notation Clarification & Gate notation for the show operator
x
g
H
g
c(g)
(a)
b
(c)
Figure Adapted causal diagrams.
We provide additional clarification and justification for adapted causal diagrams.
Figure 7(a): g captures the relation between X and Y. This is implicit in standard causal diagrams. However, given that discussion centers around human understanding of these relations, we make them explicit and annotate them edges.
Figure 7(b): a new type of node H indicated by the diamond shape. captures human task-specific intuitions. It generates human approximations of the core concepts. As a result, has directed links to functions in our diagrams. Our goal in this work is to provide formal language to discuss human
20

---

Published in Transactions on Machine Learning Research (04/2023)
Paper
Model
Prediction Explanations
5
Colin et al. (2022)
InceptionV ResNet
Hilden
Local feature Importance (Sallency lapar, Integrated Cradi x ents, Occlusion (OC), SmoothhGred (SG) and Grad-CAM) Coufidence scare, esample- based methodss (nearest neighbors)
Taesiri al. (2023)
ResNet, kNN, other deep Jearn- Shown ing models
x
Kim et al. (2022)
N, HogNNot, ProtoPNed, Pro- Mixed LoTree RosNet
Example-based methods (Nro Proto Troe), local fest Impor- / / fance (GradCAM, BagNet)
Nguyen et al. (2021)
Shown
Model uncertainty (classification confidence probability); Local
/ feature importance (gradient-hused, sulient-objeet. detection model);
Example-tase meibood (protolypes)
Bugines (2021)
Wizard
Shown
Model uncertainty (classification confidence for probabdist))
V
Chromik at al. (2021)
Docision troes frandom forests
Shown
Local feature importance [perturhan partuurattoonbaaee SHIPP)
X X
Nourani al. (2021)
Other deep learuing modele
Shown
Local festure importannce (video jestures)
VVV
Liu et al. (2021)
vector machines (SVMs)
Shown
Local feature importance (coefficients)
Wang & Yin (2021)
Logistic regression
Shown
Examplo- beadd methods (Nearvet neighbor or similar training, in- / /
stances; Counterfactual explanations (counterfactual examples);
Global feature importance (permutationn
Poursabot-Sangteh ot al. (2021) Linear regrossion
Shown
Presentation of simple models (Tinesur regression;; Information about / / festures information the model considers)
training
Bansal et al. (2020)
RoBERTa, Geveralized addicive Shown models
Model uncertainty (classifictton: confidence (or probability)) Local x // feature Importance (pertirbation-haso (I.ME))) Natural language
explanations (expert-peneraled ralioundes);
Zhang et al (2020)
Decision trees/ trees/raandon /random forests
Shown
Model uncertainty (classification: confidence (or probability)): Local // feature importance [perturbation-bseed SIAPP); Information about training, data (input. festures information the model considers) Global feature importance (shape function of GAMs))
Abdul et aL (2020)
Generalized additive models
Shown
X
Litele et al. (2020)
Decision trees /random forests
Hidden
Counterfactual explanations (contrastive or sensitive features)
/
X
Lai (2020)
BRRI; Support-vector machines
Shown
Local festure importance [attention]; Model perfornance (socuracy)
Global example- based explanations (model tutorial)
Algazaawi et al. (2020)
Convolution Neural Networks
Hidden
Local feature importance (propagation basegtttioo--seedd based (LRP), perturbation- / x based (LJMF)
Carlon et. al. (2020)
Recurrent Neural Networks Other deep learning models
Shown
Local feolure imporiance (xllention)
Hase& & Bansal (2020)
Shown
Local feature importance (perturbation- based (LIME)); Rule-based based / xxx explanations (anchors); Example-based methods (Nearest nelgisoo similar texining inslances;; Partial decision boundary (Lraversing Uhe latent space azound data imput)
Buginea al. (2020)
Wizard Oz
Mixed
1. Modell uncertainty (chsseflestion confidence (or probabiityy); Local feature importance (gradient-baeet extractive evidence
Example-based methods (Nearest neighbor similar training instances)
Klani et al. (2020)
Other deep learning models
Shown
Gonzalez al. (2020)
Other deep learning models Whoedd
Shown
Lago et al. (2011))
Mixed
Rule-besed explonations (dectston ants)
Weests et al. (2019)
Decision trees/ /random forests
Hiddea
Model uncerlainty (classificstion confidence (ur probability)): Local X /v
feature importance (perturbation-bess SHAP)
Guo et al. (2019)
Recurrent Nearal Networkss
Shown
Model uncertalnty (classifleation confidence (or pmbaaillty))
x
Friedler et al. (2019)
Logislic regression; Decision trees rradomn foresta; Shallow (I- to 2-layer) neural networkss
Hidden
Counterfact und explunal ious ious (counterfactual examples): Presentation /
of simple models (decision
logistic regression, layer MLP)
Lai & Tan (2019)
Supporl-vector machines
Shown
Example beeed methods (Neatret neighbor or similar teaining in- x / stances); Model performance (accuracy)
Feng Doyd-Craber (2018)
Generalized additive models
Shown
Model uacertintty (elassification confidence (or probability)); Global / example-based explonations (prototypes)
Nguyen (2018)
Logistic regression: Shallow (1- Hidden
Local feature importance (gradient- based, pertarbstion--aaee (LIME)) / X X
to 2-layer) neural networks
Ribetro et al. (2018)
VQA model (hyhrid L.T M and Hidden CNN)
Rule-basod explanaions (anchors)
x X
Chandrasekarm al. (2018)
Convolution Neural Networks
Hidden
Local feature importance (attention, gradient- based)
Biran & McKeown (2017)
Logistic regression
Shown
Natural language explanations (model-generated rationales)
X / /
Lakkaraju es al. (2016)
Bayeian decision liste
Hidden
Rule-besed explanations (decision sete)
X x
Ribeiro et al. (2016)
Support vectur machines; Incep- tion neural network
Shown
Local feature importance (perturbation cettuurbationnbbeeee baaeedoo--bddddedeeeeeeeee (LIME))
Bussone ot al. (2018)
Wisard of Oz
Shown
Model uncertainty (classification con findennce (or prolubiity))
X /.Xx X based explanations (tree- based explamation); Counterfactual ex- planations (comnterfactual examples)
Lim (2009)
Decision tzees/ /random forests
Shown
Table 2: A summary of recent empirical studies measuring human understanding with machine explanations. The papers are sorted by time, starting from the newest. Note: columns g, 2, and f mean model decision boundary, model error, and task decision boundary respectively. v(or x) means the study measures (or does not measure) the corresponding type of human understanding. This table is also maintained online at https://giithnn wwt//oo////aeecce Chen/ wwweeeeheeeeeeeeeeeeieeee -omainnnnn---------
Human- Stuuies..
21

---

Published in Transactions on Machine Learning Research (04/2023)
mental models in the context of human-Al decision making and articulate assumptions about H in different conditions.
Figure 7(c): We relation between functions to indicate that explanations are generated with function (e(g)) derived from g. Although the relations between functions, they act in the same way as causal relations between variables, so d-separation still applics in the function space.
Our show operator can be rewritten using gate notation (Winn, 2012). For example, Figure 8 shows the diagram under the emulation condition; this diagram is equivalent to Figure 3(b) which uses our show notation. Since the gate is just an intervention written differently, a show operation can also rewritten as a regular intervention. As a corollary, show does not represent conditional probability: Ey Y|show((Y) (Y) and IE[y Y], y is generated by two distinct processes.
Z"
PH
yH
show(Ý)
F
show
The partial realization of the base diagram under the emulation condition, visualized using gate notation; the semantics of the diagram equivalent to Figure 3.b which uses our show operation.
Figure 8:
D Global Understanding
Similar Section 4, discuss how interventions and assumptions can shape human global understanding We discuss the two representative scttings with concrete examples in (Figure 9) and the other settings are similar to derive.
sbout
showe
(a) Emulation: prediction shown (egg. model debugging)
(b) Discovery: prediction shown (e.g. assisted making recidivism task)
Figure 9: (a): Emulation task with prediction shown; (b): Discovery task with prediction shown. We use the dotted arrow to represent the temporal updates of human global understanding. In each of the setting, we illustrate time steps from to t + 2.
22

---

Published in Transactions on Machine Learning Research (04/2023)
Emulation tasks with machine prediction shown (Figure 9()).. We start with a simple example, emulltton tasks with machine prediction shown. Model debugging for developers is one such example. In this case, human intuition at time step & derives human global understanding g and 2 directly. Since it is an emulation task, f is not of interest, we omit it in the graph. For human local understanding, as Ý is shown, Y. Also, given the emulation assumption, y = We add subscript to indicate time steps.
This scenario captures the dynamies of debugging. At the human local understanding level, Figure (() at each time step looks the same as Figure 3(d). As we have discussed, the relationship between local variables is deterministic. While knowing whether the model is correct for single case may not be particularly useful for debugging, the key difference between the local version (Figure 3(d)) and the global version (Figure 9()) is the edge from observations (Px) to human intuition H₂ and the temporal updates of human intuitions (the edge from He to Hz+1). This allows model developers to update their belief of the model error z" and the decision boundary gd and decide the scope of this bug and whether this is a bug that needs to be fixed or can be fixed. The case of debugging highlights that global understandings are often the targets in human-AI interaction and updates in human belief is critical for understanding changes in global understanding.
For example, in Ribeiro et al. (2016), in topic classification, emulation task where users know task decision boundary, showing LIME explanations can improve human global understanding of model decision boundary (i.e., which features are important) and thus also model error.
Explanation (E) can be incorporated in this figure and increase the connection between g" and g because g is a parent of E.
Discovery task with prediction shown (Figure 9b). A concrete example of this scenario is Al-assisted decision making in the recidivism task (Kleinberg et al., 2018), where judges make bailing decisions giving risk estimation
(f). The update in indicattss the fact that we are interested in improving global human understanding of task decision boundary. Similar to the local casc, do not know the exact relationship between n and foo the local variable. Therefore, alternative path to improve y by improving global human understanding of model error (z"), then n" leading to "'ihhth with sshown
Similar to our discussion in Scction 5, by showing machine predictions, human global undorstanding of gd can be updated. However, it is impossible lo see updales on = and f beyond g since human understanding is bounded by the ML model without assumptions about human intuitions. Combined with human intuitions and machine explanations E, ² and fd can be potentially updated.
Implications. The first takeaway is that improving global understanding is often the actual target of interventions. For instance, in emulation tasks with machine predictions shown, although local understandings have deterministic relations with each other, the key open question lies in how understanding local error relates to updates in the global understanding of model error.
Second, in order to improve global understanding, it is critical to understand the updates in human global understanding. Interventions are ideally designed so that fd ~ f. g" ~ 9: z ~ u, depending on the application. Therefore, we observe that showing predicted labels only reveals information about g, and the information related to f and constrained by g. This observation highlights the limited utility of showing predicted labels.
Finally, our framework shows that there does not have to be links between f" g" and " Although we may often be interested in conclusions such as calibrating trust improves human performance, the causal link may be unidentifialle and this relationship can also change over time. This observation reveals a fundamental limit in understanding the effect of machine explanations on human understanding. That said, it is possible to identify causal relation between H and H by making additional assumptions on the diagrams (c.g., there is no temporal updates in human mental models, in other words, all time steps are independent of each other).
23

---

Published in Transactions on Machine Learning Research (04/2023)
E
Expanding Human Intuitions
H
H1+1
show
E,
show
Er+
show
Y
H
show
y
y++1
7+
P
Z
Pif
ZH
1+1
YH
pH 7+1
Figure 10: Explanations E help expanding human intuitions by influencing human intuitions in the next time step.
F
Additional Experiment Details
F.1
Data
Table 3 shows the data instances we constructed and used in our experiment. Specifically, we construct 'high education low age' samples as masters with age varying from 23 to 26. 'low education high age' samples are middle school with age varying from 46 to 49, To avoid the task being too artificial, we use different ages for A-H for a participant. To further remove the potential effect due to the differences between ages, we create two groups of data that the average age is the same for ABCD and EFGH. We randomly assign participants to data group and ensure that each group is assigned to a similar number of participants.
Instance ID
Education
Age
Feature A
Feature
Predicted label 850K)
Explanation
Identifier
1
Masters
25
0.85
0.13
Education
A1
2
Masters
0.85
0.10
Education
A2
3
Middle school
46
0.15
0.83
Education
B1
4
Middle school
0.15
0.93
Education
B2
5
Masters
26
0.85
0.17
Age
C1
6
Masters
0.85
0.07
Age
C2
7
Middle school
48
0.15
0.90
Age
D1
8
Middle school
0.15
0.87
Age
D2
9
Masters
23
0.85
0.07
Education
E1
10
Masters
0.85
0.17
VAAVVVAR
VVAAAA
Education
B2
11
Middle school
47
0.15
0.87
Education
F1
12
Middle school
0.15
0.90
Education
13
Masters
24
0.85
0.10
Age
C1
14
Masters
0.85
0.13
Age
G2
15
Middle school
49
0.15
0.93
Age
H1
16
Middle school
0.15
0.83
Age
H2
Table 3: The data instances used in experiment.
24

---

Published in Transactions on Machine Learning Research (04/2023)
F.2 Participants' Agreement Report
In table 4, we report participants" average agreement across different groups and different categories (identifiers) of instances. We average over participants. Each participant did one question from each data instance catogory (identifier).
Identifier Agreement (anonymized)
Agreement (regular)
40.75%
85.71%
45.26%
95.71%
41.41%
32.86%
D
39.31%
17.14%
47.25%
28.57%
48.70%
15.71% %
48.45%
74.29%
48.70%
92.86%
Table 4: Participants agreement by the identifier in anonymized group
F.3 User interface design
Task description and attention check. After participants agree with the online consent form (Figure 11), they will be re-directed to a page including a description of the task and an attention check, as shown Figure 12 and Figure 13.
Measuring human intuitions. For regular group, participants re-directed to the preliminary questionnaire page after they read the task description and passed the attention check. We calibrate human intuition in terms of relevance (R) and mechanism (M) using the 4 questions as shown in Figure 14.
Exit survey. After participants have done with all the prediction problems, they will proceed to an exit survey page, as shown in Figure 15.
25

---

Published in Transactions on Machine Learning Research (04/2023)
Please read the consent form below carefully before proceeding.
To begin the study, please click "Agree" at the bottom and you will proceed to the detailed task instructions. At the end of the study, you be redirected back to the Prolific website validate that you have completed the study.
Consent Form
doing research study evaluate the effectiveness of Al assistance for humans to predict adult Income. We invite you to take part In this research study because you reside in US and your native language is English. We expect that you will this research study for 2-3 minutes depending on the experiment type. Your participation is voluntary.
Description: We researchers
Incentives: You will be asked main questions. The expected duration is 2-3 minutes. You will get cents for each decision. you guess correctly, you get additional cents. In the event of an incomplete study, you will NOT receive compensation for answering partial questions.
study contains attention checks to make sure that participants are finishing tasks honestly and completely. As long as read the instructions and complete the tasks with adequate attention, your HIT will be epproved. you fail these checks or finish the hastily, your HIT will rejected. This study also contains an eligibility check make sure there duplicate participants.
PLEASE NOTE:
Risks and Benefits: Your participation in this study does not involve any to you beyond that everyday life. This study may benefit society by improving the understanding of how Al assistance can improve humans' ability to predict income.
Confidentiality: Your Worker ID used to distribute payment to you. Please aware that your Worker can potentially be linked to information about you your Prolific public profile page, depending the settings you have for your Amazon profile. We will not be accessing any personally identifying Information about you that you may have put on your platform public profile page.
If you decide to withdraw, data collected up until the point of withdrawal may still be included in analysis.
identifiable (your worker
shared outside the research team.
De-identified information from this study may be used for future research studies shared with other researchers for
future research without your additional informed consent.
Contacts & Questions: If you have questions or concerns about the study, you may email questions to
you any questions about your rights as participant this research, you have harmed, or wish discuss other study-related concerns with someone who is not part of the research team, you can contact
Participation is voluntary. Refusal participate withdrawing from the research will involve penalty lass of benefits to which you might otherwise be entitled. By clicking "Agree" below, you confirm that you have read the consent form, least years for your records. You have not waived any legal rights otherwise would have as participant in research study.
Consent
agree to participate the research. Please print or save copy this page
Agree
Do not agree
Figure 11: Consent form.
26

---

Published in Transactions on Machine Learning Research (04/2023)
Explaration
Predictng phane.
task, you go through Income is above below SS0K.
people. each profile, you predict
Each profile contains facts about
Education (from middle school masters). Roughly half of the profiles are above high school (a g. masters) and
q.. school). About half the profilles
equal
Age (from to 50 years old), Roughly half of the profiles are abeve 30 and half are below.
of
assist you For each profile, will see the Al's prediction and explanation which points information used to make the prediction. The Al not perfect, and hopefully explanation can help you identify its mistakes.
intelliaanos (Al)
age, many other
It important that you complete the exit survey receive unique code that is required to be compensated. Please unique code after complating the
Please enswer the questions below carefully. The questions test your understanding experiment. disqualified study if question answered incorrectly.
your
study?
To find the level of education of person.
To pr redict w/ whether personfs income above S0K or not.
To find the age of highest income person.
performance on income
22. have to answer an exit survey after the study get unique code, which should be submitted order to get
compensation.
Yes. lagres
No. don't need submit the code get comperisated.
Submit
Figure 12: Task description and attention check for participants in the regular group. The user is required to select the correct answers before they are allowed to proceed to the training phase. The correct answers can inferred from the given text on this page.
27

---

Published in Transactions on Machine Learning Research (04/2023)
Step
Step
Step
Cap enation task
Preckccing phane.
Giee us some feediacii
this task, you will go through the profiles of people. For each profile, you will predict whether the persoris annual
About half of the profilee have an annual income above $50k.
An Artificial Intelligence model will assist this task, The model considers education on, and many other factors. The uses neural network to combine these inputs into feeture A and feature which the then uses prediction,
explanation which points festure used to make the prediction. The Al is perfect, and hopefully the explenation help identify its mistakes.
is important that you complete the submit the unique code after completing the HIT.
receive code that required compensated, Please
Please the questions below carefully: The questions your understanding of this experiment You will be disqualiffed from the study any question answered incorrectly.
*1 What is task this study?
To explain feeture the model.
predict whathar person's income
predict
complex networks.
evaluste Alls performance prediction.
survey study get unique code, submitted
compensation.
Yes. aggre..
No. don't need to submit code get compenseted.
Submit
Figure 13: Task description and atlention check for participants in the anonymized group. The user is required to select the correct answers before they allowed to proceed to the training phase.
28

---

Published in Transactions on Machine Learning Research (04/2023)
Step
Step
Step
Explanation of the task.
Predicting
Give feedback!
each of person's profiles, there two features: of education age.
11. In your opinion, what's the relationship between education and income?
Positively correlated. In other words, people with higher education usually have higher incomes.
Negatively correlated. In other words, people with higher education usually have lower incomes.
Not correlated.
*Q2. In your opinion, what's the relationship between age and income?
Positively correlated. In other words, older people usually have higher incomes.
Negatively correlated. In other words, older people usually have lower incomes.
Not correlated.
+Q3. Which feature you think the more indicative of income within following two?
Education
Age
Please rate your agreement with the following statement: am confident about my answer to the Q3 with respect to which feature is more indicative of income.
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
Submit
Preliminary questions calibrate human intuitions. If participant agrees that both education and age are positively correlated with income and he/she thinks education is more important, the participant holds intuitions consistent with both R and M.
Figure 14:
29

---

Published in Transactions on Machine Learning Research (04/2023)
Step
Step
Step 3
Explanation the task.
Predicting phase.
Give us some feedback!
Thank you for participating this survey, Please answer the following questions.
"Which feature you think important after the study?
Education
Age
What is your gender?
Female
Male
prefer not to answer
What is your age?
18-25
26-40
41-60
61 and above
prefer not to answer
What is the highest degree or level of school you have completed? If currently enrolled, select the highest degree received.
Some high school, no diploma, and below
High school graduate, diploma or the equivalent (for example: GED)
Some college credit, no degree
Trade/technccll/ooattio training Bachelor's degree or above
prefer not to answer
Please explain how you leveraged Al assistance in making predictions.
Do you have any suggestions to our experiment design and web interface?
Submit
Figure 15: Exit survey.
30


Reviews:
[{"review_id": "JqCYEs25ts", "reviewer": "Reviewer_SMPN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper poses an interesting framing of how model explanations help user understanding of ML models.\n- Section 2 is thorough and well-written. \n- Table 2 (and much of the Appendix) is a very helpful resource for the explanation community.\n\nWeaknesses\n- None come to mind.\n- I would have preferred a more robust human subject experiment. Since only 16 instances were used, I'm not sure we can safely conclude much (even though the number of participants was more than sufficient). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the interplay between human understanding and model explanations, which had not been given a thorough treatment to date. Grounded in an exhaustive search over existing literature, the authors note three factors that govern this relationship: task decision boundary, model decision boundary, and model error. The proposed notion that human intuition is important to explanations is another contribution, which was mere folklore until now. The authors conclude with a human subject experiment of how human intuition impacts a participants understanding of a model explanation.\n\nThis is a fine piece of work that I recommend accepting as is. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Minor: when listing multiple citations for a given fact/statement, please try to ensure the papers are listed in chronological order; this is a stylistic thing but surely improves readability\n- If possible, could you scale the human subject experiments to at least 30 instances? This can hopefully permit stronger statements about how participants perceive explanations and how it affects their understanding, as it were.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Is1yfSWmrc", "reviewer": "Reviewer_JmbV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think this is a quite well-written paper. The framework is presented and contextualized well, and it is easy to follow.\n\nAfter reading the paper, I'm still a bit confused however about the claim in the theorem \"Explanations can improve human understanding of model decision boundary, but cannot improve human understanding of task decision boundary or model error.\" In particular, I'm not entirely sure how explanations couldn't help improve human understanding of the task decision boundary. If I have something like a counterfactual explanation and model fits the task quite well, cfe's should at least provide a noisy approximation of the task decision boundary, even if the features are totally anonymized. I suppose that the claim in the theorem assumes the predicted label is not shown, but it is very hard to separate predicted labels for certain types of explanations, like counterfactual explanations. Perhaps my concern rests with the assumption of the causal model, or this theorem is not applicable to certain types of explanations that are inseparable with the predicted label, like cfe's. Overall, I'm having trouble fully grasping how understanding the task decision boundary from explanations categorically is not true, as it is currently presented in the theorem, and I would love some clarification here.\n\nMinor:\nThe line: \"To the best of our knowledge, we are not aware of any existing quantitative behavioral measure of human understanding that does not belong to one of these three concepts of interest.\" is a bit confusing. Perhaps this is true  for understanding model/task predictions, but there are other aspects of model understanding (e.g., just understanding the mechanism by which a model makes a prediction) that are not included in these definitions. At least, I think this should be slightly qualified to better suite to problem at hand, and it is a bit confusing to read currently.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a causal framework for modeling the relationship between ML explanations, based on the idea that human intuitions have a key role in the relationship between humans and understanding explanations. They find that without intuitions surrounding explanations, humans can't understand certain aspects of ML prediction problems, like the task decision boundary or the error of the model. They validate their framework with experimental results using humans.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Currently, I believe the statement of theorem 2 should be addended to specify that it is applicable for explanations which do not require the use of the predicted label. However, if I'm mistaken, I would greatly appreciate some clarification here.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SVY6hzhokh", "reviewer": "Reviewer_hQ3d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n- Formalizing the three core functions and their interactions seems helpful.  \n- Incorporating Z into E is an interesting idea that is largely unexplored in work on explainability.  \n\nI found many weaknesses with the submission (listed bellow), but would like to make two higher level comments first:  \n- These weaknesses can be summarized as follows:  this framework treats both \"explanations\" and \"human intuitions\" as if they are binary features that are either present or absent and, consequently, it is not clear what the framework tells us about the typical setting where users have some degree of human intuition and we are interested in the effects of different types of explanations.  \n- While the I am very familiar with explainability, I am not particularly familiar with causality.  So I found the submission fairly difficult to read and may have missed something important.  \n\nWeaknesses:\n- In the third to last paragraph of Section 1, the authors claim that \"Our causal diagrams reveal.... , and shed light on the mixed findings of empirical research on explanations.\"  This claim is not supported because there is no analysis that looks at whether or not these past empirical findings considered human intuitions.  The example of \"deceptive review detection\" is interesting, but there should be a more systematic study.  \n- The framework is based on a novel operator (the show operator) whose affects on the causal graph are defined, but not justified.  \n    - For example, showing the user $\\hat{Y}$ does not seem to be equivalent to showing them $E(X, g)$ because $E$ may contain information about $g$ for points $X'$ that are similar to $X$ when $E$ is a local approximation-based explanation.  This is not the same as considering some notion of \"global understanding\".  \n- One of the primary implications of the framework, that \"complementary performance is impossible without assumptions about task-specific human intuitions,\" is not particularly interesting and commonly considered in work on explainability:  if the user has no \"task-specific human intuitions\" (ie, domain knowledge) it is clear that they cannot help make better predictions.  For example, no one expects a non-expert to be able to make medical decisions because they were shown a model's explanation. \n- It is unclear how to extend the framework to model the effects of a specific human intuition rather than simply showing that, without any human intuition, it is impossible to gain certain types of understanding.   \n- The results of the user study do not support the claims of either Theorem 1 or 2:\n    - The participants were asked to make a prediction, so the experiment does not test Theorem 1. \n    - There was no measurement presented showing that users made better predictions in the \"regular\" setting than in the \"anonymized\" setting.  This result would have supported Theorem 2. \n- The results of the user study are not of general interest to the community:\n    - The results pertaining to H1 are expected.  The authors note this themselves when they say \"For a person who has no intuitions about a task at all, the best course of action is to follow machine predictions.\"\n    - The results pertaining to H2 have been identified by past works.  \n- The setup considered by the user study is very simplistic. \n- The claim that, for discovery tasks, \"if we do not make assumptions about human intuitions, E can cannot bring any additional utility over showing $\\hat{Y}$\" seemingly contradicts the example giving in the \"Expanding human intuitions\" paragraph.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission works towards a theoretical framework for how a person understands a learned model.  To do this, it starts by identifying three core functions that a person may want to understand:  the model boundary, the task boundary, and the model error.   Then, it models the effects of an input, an explanation, and human intuitions on a person's understanding of these core functions using causal graphs and a new operator.  This framework suggests that, without assumptions about these human intuitions, explanations cannot help a person understand the task boundary or the model error.  The submission supports this claim using a simple user study.   ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "The reviewer thinks that the submission needs a substantial re-write to address the above weaknesses.   \n", "broader_impact_concerns": "Not applicable \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the proposed framework and a focus on the role of human intuition in the effectiveness of explanation are useful notions and the manuscript is well constructed. As said above, I believe there is an audience for this work and that the claims could be adjusted to better align with the presented results. I am recommending acceptance with a minor revision. In the revision, I would like to see author adjust claims on how their work can be used to shed light on prior studies and how the human studies validate elements of their proposed framework. I would also like the bolded claim at the top of page 8 to more clearly include the assumed lack of human intuition. 
Decision: Accept (poster)