Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary
Authors: Zhuoyan Li
Ming Yin
Venue: NeurIPS 2024 poster (2024)
Keywords: human-AI interaction
human-centered evaluation of explanation
Primary area: human-AI_interaction

Abstract: Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions, and utilized these models to improve human-AI team performance. Meanwhile, due to the ``black-box'' nature of AI models, providing AI explanations to human decision makers to help them rely on AI recommendations more appropriately has become a common practice.  In this paper, we explore whether we can quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that  human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign. Furthermore, individuals often fail to detect any anomalies in these  explanations, despite their decisions being affected by them.

Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary
Zhuoyan Li Department of Computer Science Purdue University West Lafayette, IN, 47906 1 1141780purdue. edu
Ming Yin
Department of Computer Science Purdue University West Lafayette, IN, 47906 mingyin@purdue edu
Abstract
Recent advances in Al models have increased the integration of Al-based decision aids into the human decision making process. To fully unlock the potential of AI- assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions, and utilized these models to improve human-AL Al team performance. Meanwhile, due to the "black-box" nature of AI models, providing AI explanations to human decision makers to help them rely on Al recommendations more appropriately has become a common practice. quantitatively model how humans integrate
this paper, explore whether both Al recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign. Furthermore, individuals often fail to detect any anomalies in these explanations, despite their decisions being affected by them.
Introduction
Recent advances in Al models have significantly increased the integration of Al-based decision aids into human decision making process. The widespread adoption of such Al-based decision aids has opened up new paradigm of human-Al collaboration -thiii AI model provides recommendations for given decision making task, while human decision makers are responsible for making the final decisions. To fully unlock the potential of Al-based decision aids in enhancing human decision making, a few studies (I-5) have developed computational models to capture how humans factor Al recommendations into their decision-makin process, and explored how these behavioral models can utilized to improve human-Al team performance. For example, Vodrahalli, Gerstenberg, and Zou (6) developed a human behavior model to characterize the impact of AI predictions and confidence levels on human final decisions. This model was then utilized to adjust the model confidence displayed people, with the objective of calibrating human trust in AI assistance.
Meanwhile, the black-box nature of prevalent models has driven greater integration of model explanations, generated through various explainable AI (XAI) methods 11) into Al-assisted decision making. These explanations seek provide some insights into the the underlying decision rationales of AI models, assisting humans in evaluating the reliability of Al decisions and identifying optimal strategies to rely on Al recommendations. However, many empirical studies [1) which evaluate the effectiveness of current XAI methods for improving people's understanding
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of the AI model [T6 19 20] and supporting their calibrated trust in the model 13 14 20]. have demonstrated that humans often struggle to use the explanations generated by these methods in the optimal way. Thus, despite designers' expectations that XAI methods will positively shape human interaction with Al models, they often fall short of their intended goals, such as fostering appropriate levels of trust and reliance in Al-assisted decision-making. This could be because existing XAI methods do not account for human reactions, making them unadaptive to human cognitive processes. If this is the case, one may naturally wonder if it is feasible quantitatively model how humans incorporate both AI recommendations and explanations into their decision making process. If so, can the quantitative understanding of human behavior obtained from the learned behavior models be utilized to directly manipulate Al explanations, thereby nudging human decision Becision-makers towards making targeted decisions?
To answer these questions, in this paper, we begin by training behavior models that characterize how Al recommendations and explanations are factored into human decisions based on the collected behavior datasets for various decision making tasks. Utilizing these learned models, we then adjust the AI explanations for different purposes through gradient-based optimizations. Our extensive human experiments across various tasks demonstrated that such human behavior modeling can bring forth both benefits and risks when used for benign purposes, the human behavior models can inform the adjustment of Al explanations such that the manipulated explanations could significantly enhance the secissoon-maiing performance of human-A Al teams in most tasks; however, the same behavior models can also be exploited by adversarial parties for adversarial purposes, such as increasing human decision-make biases against certain protected group and significantly decreasing the fairness level of humans' final decisions across different groups. Finally, examining human perceptions of these manipulated explanations reveals the "scary" truth while human decisions are easily swayed by these altered explanations, individuals generally fail to detect any anomalies in the explanations, underscoring a significant vulnerability in human-Al interaction.
2 Related Work
2.1
Computational Modeling of Al-assisted Decision Making
There has been surge of interests among researchers recently in computationally modeling human behavior in Al-assisted decision making (4) The goals of these studies for modeling human behavior are diverse, encompassing improving human- AI team performance through intelligent interventions or model recommendation adjustments (6] 29). deciding when to present Al powered code suggestion in programming (30). evaluating the utility of AI explanations to improve user understanding of Al model behavior (31-33 and deploying adversarial attacks on Al models to reduce human trust (34). In this paper, take more holistic view and explore whether we can model how humans integrate both Al recommendations and explanations into their decision making process, and what the implications of such behavior modeling are.
2.2 Human-centered Evaluation of AI Explanations
With the increasing of AI technologies decision aids for humans, a variety of explainable AI techniques have been developed to increase the interpretability of AI models [7- 33 33 (7). To understand the effectiveness of these explanation methods, a growing body of empirical human- centered evaluations have been conducted to examine how AI explanations would affect the ways that
humans perceive and interact with AI models (12 31 32 38 39 These evaluations look into various aspects of impacts of AT explanations, such as the in influence on people's 's trust and reliance on
the AI model [13 14 20 understanding of AI model [16] 19 20 and the collaboration performance of the human AI team 12 40. 41]. Recently, some research has explored the modification of AI explanations to influence human behavior. For example, Lakkaraju and Bastani (4) demonstrated that handcrafting modifications in Al explanations such as hiding sensitive features can mislead human trust in AI models. Another study (43) found that aligning Al explanations with humans' own decision rationales can increase agreement between human decisions and the AI model's predictions. Different from previous work which required intensive handcrafting of Al explanations, in this paper, we explore whether it is possible to directly exploit the computational human behavior models to manipulate AI explanations, even without the access to Al models, with the goal of nudging human decisions towards targeted directions.
2

---

3 Methodology
3.1 Problem Formulation
In this study, we explore the scenario of Au collaboration within the context of Al-assisted decision making, and we now formally describe it. Consider decision making task represented by n-dimension feature vector E RR", and y is the correct decision to make in this task. Specifically, in this study, we focus on decision making tasks with binary choices of decisions, i.e., y E [-1.]] The Al model's recommendation on the decision task is represented as y (x), y E f- Following the explainable Al methods like LIME or SHAP(8), the AI model could also provide some "explanations" of its decision, e = {x)) E R², by showing the contributions of each feature the decision. With all these information, the human decision maker (DM) needs to make the final decision y² E 1} by either accepting rejecting AI model's decision recommendation y" which can be characterized by nº = H(x, y" e). The goal of our study is to explore whether we can quantitatively model such decision making process seees--scciiillll....oo H(x, y" .e) end whether this quantitative understanding of human behavior can be utilized to adjust AI explanations (i.e., change to e') without accessing to the original Al model (.), thereby nudging human DMs to make the targeted decision 8² E {-1, 1}, denoted as 8 H(x y"","" e').
3.2 Modeling Human Behavior in Al-assisted Decision Making
We first build computational models to characterize how humans integrate both Al recommendations and explanations into their decision process. Following previous works on modeling human behavior in different scenarios of Al-assisted decision making 3T) we adopted a two-layer neural network
as the structure for modeling the human decision this study:
(1) The inputs to the behavior model include the task features æ, the Al model's prediction y" the Al explanation and the interaction term between the task features and the Al explanation x @ e that reflects how humans may redirect their attention the corresponding features highlighted by the Al explanation. Given the human behavior dataset ' { {.:. y," Ci, y Jiml we can employ the maximum log- ligeliill estimation to learn the behavior model Hws.
yd Hur, (ar y) e) = Hun, (læ, y" ,,,"",,,z Gel
3.3 Manipulating Al Explanations through the Behavior Model
We next proceed to explore how the quantitative understanding of human behavior from Hw,, can be utilized to manipulate Al explanations. In particular, given the targeted decision ÿ² for the task instance z, want to identify new AI explanation e' that maximizes the likelihood that human DMs make the targeted decision 1² according to the learned behavior model Ho, In addition, to prevent the case where the manipulated explanations e' has very low level of fidelity (10), such as suggesting a recommendation that is inconsistent with the Al model's prediction y" we also impose a constraint that the new explanation e' should still support the original AI recommendation y"" Since we assume no access to the original AI model, we define Cconsistency (e, y as a measurement of agreement consistency between the manipulated AI explanations and the AI recommendation:
{1
0 if sign (E, e,) = sign(y")),
Leomsistency (e, )
otherwise.
Together, we use the following optimization problem to manipulate AI explanations:
argmin Loehavior Coehaviv (Hwy, (x. y² e'), g²), subj. to Cconsistency (e', <
(3)
where Liehavior is defined as the cross entropy function. Since exactly solving the above optimization problem is intractable, we used the gradient- optimization to approximate
e/p 11 = e'n - V ei (Luchavior (Hw, (x, " e6), 8²)) + Co (E's, ))
(4)
where is the step size, X is the trade-off parameter, and e/ represents the parameterized explanations in the optimization process. We can iteratively optimize manipulated explanations e' until Lrehavior is smaller than threshold 5 or reach the maximum number of rounds T'.
3

---

4 Human Behavior Model Learning
To develop the human behavior model for manipulating Al explanations, we first conduct a human subject experiment to collect human behavior data.
4.1
Decision Making Task and AI Assistance
We consider four decision making tasks in this study:
Census Prediction (Tabular Data) [44]: This task was to determine a person's ss annual income level. In each task, the human DM was presented with a profile with 7 features, including the person's 's gender, age, education level, martial status, occupation, work type, and working hour per week. The subject was asked to decide whether this person's 's annual income is higher or lower than S50k for each task. We trained a random forest model to make the income prediction, and the accuracy of the Al model was 76%.
Recidivism Prediction (Tabular Data) (45): This task was to determine a person's 's recidi- vism risk. In each task, the human DM was presented with profile with 8 features, including their basic demographics (e.g., gender, age, race), criminal history (e.g., the count of prior non-juvenile crimes. juvenile misdemcanor crimes, juvenile felony crimes committed), and information related to their current charge (c.g., charge issue, charge degree). The subject asked to decide whether this person would reoffend within two years. We trained random forest model to make the prediction, and the accuracy of the Al model was 62%.
Bias Detection (Text Data) [46]: In this task, the human DM was presented with text snippet and needed to decide whether it contained any bias. We fine-tuned a BERT [47] model to identify bias in the snippet, and the accuracy of the Al model is 79%.
Toxicity Detection (Text Data) [48]: In this task, the human DM was presented with a text snippet and needed to decide whether it contained any toxic content. We fine-tuned a BERT model to identify the toxic content, and the accuracy of the Al model is 86%.
To understand how people respond various AI explanations, we employed LIME and SHAP to explain the predictions made by the Al model. Additionally, we augment the LIME or SHAP expla- nations by either randomly masking out contributions from some features or amplifying contributions of some features (referred to as the "Augmented" explanations) to see how humans react to them. These explanations are provided with Al recommendations together to humans in decision making.
4.2 Experimental Procedure
We posted our data collection study on the Prolific to recruit human participants. Upon arrival, we randomly assigned each participant to one of the four decision making tasks and they needed to fill in an initial survey to report their demographic information and their knowledge of Al models and explanations. Participants started the study by completing a tutorial that described the decision making task that they needed to work on. To familiarize participants with the task, we initially asked them to complete five tasks independently without AI assistance. During these training tasks, we immediately provided the correct answer at the end of the task. After the completion of training tasks, participants moved on to the formal tasks. the formal tasks, participants would receive one type of Al explanations among SHAP, LIME, or Augmented. Specifically, each participant was asked to complete a total of 15 tasks. In each task, participants were provided with the Al prediction and the explanations along with the task instance. They were then required to make their final decisions. Finally, participants were required to complete an exit survey to report their perceptions of the AI explanations they received during the study. They were asked to rate the alignment of AI explanations with their own rationale, as well as the usefulness, transparency, comprehensibility, satisfaction with the provided explanations, and their trust in the Al models, survey questions, please refer to Appendix A.3 We offered a base payment of $1.2 and potential bonus of $1 if the participant's 's accuracy is above 85%. The study was open to only, and each worker can complete the study once.
5-point Likert scale. For the detailed
workers
latps://www.polfffccomm/

---

Table 1: The number of subjects recruited data collection for training behavior models, and the
average accuracy of the human behavior model in 5-fold cross validation for each task.
Census
Recidivism
Bias
Toxicity
Number of Participants Model Accuracy
78
80
72
42
0.74
0.79
0.65
0.76
4.3 Training Results
After collecting data on human behavior, we developed human behavior models for each type of task. For the human behavior models for two textual misks-Toxicity Detection and Bias Detection-we employed the pretrained BERT encoder to extract features from the original sentences, which were then used as the task feature æ in the human behavior model Hu..- We optimized these behavior models using Adam [49 with an initial learning rate of le 4 and batchsize of each training iteration of 128. The number of training epochs is set as 10. [hooss the number of participants recruited, as well as the average accuracy of the human behavior model evaluated through 5-fold cross validation for each task. We observed that the average accuracy of all human behavior models exceeds 0.65, which considered to be reasonable. Consequently, we utilized these learned human behavior models to manipulate Al explanations in the following evaluations.
5 Evaluation I: Manipulating AI Explanations for Adversarial Purposes
In our first evaluation, we adopted the role of an adversarial party to explore whether they could utilize the learned human behavior model to manipulate Al explanations. The manipulation goal was to nudge human DMs to be biased against certain protected groups in the decision making process. We are particularly interested in comparing the fairess level of human decision outcomes between human DMs who receive original explanations, such as SHAP or LIME, and those who receive manipulated explanations. Notably, all human DMs are provided with the same AI predictions for the same decision making task. Additionally, we also explore differences in human perceptions of original AI explanations versus manipulated Al explanations.
Evaluation Metrics and Manipulating AI Explanations. Following previous work 51, we used the false positive rate difference (i.e., FPRD) and the false negative rate difference (i.e., FNRD) to measure the fairness level of human decision outcomes- -the closer these values are to zero, the more fair the decisions are. To manipulate Al explanations and nudge human DMs toward biasing against certain protected groups, we define the targeted human decision g for each task as follows:
Census Prediction: In this task, we considered a person's 's sex as the protected attribute. The targeted human decision defined as e² exceeds S50K) when Bsex male, and ÿ² -1 (indicating person's 's annual income does not exceed $50K) when Tggx = female. The fairness metrics can be computed as FPRD FPR FPRmalcc and FNRD FNR FNNmmaee
(indicating person's 's annual income
Recidivism Prediction: In this task, we considered the defendant's race as the protected attribute. The targeted human decision is defined as ÿA (indicating the defendant will reoffend) when Frace black, and j = - (indicating the defendant will not reoffend) when Trace white. The two fairness metrics can be computed as FPRD = FPRwhite FPRslack and FNRD FNR FNR
Bias Detection: In this task, we divided text snippets into groups based on their political leaning. The targeted human decision is defindd as g² (indicating the text is biased) when Bleaning democratic, and j" -1 (indicating the text is not biased) when B'eaning republican. The fairness metrics can be computed as FPRD = FPR FPR and FNRD FNR FNR
Toxicity Detection:
In this task, we divided text snippets into groups based on the victim
of the text. The targeted human decision defined as j/² = (indicating the text is toxic)
when Evictim = white, and j = -1 (indicating the text is non-toxic) when Eeicctm = black. The two fairness metries can computed as FPRD FPRptack FPRwhite- and FNRD FNR FNRwhite-
5

---

Table 2: The number of participants we recruited in the evaluation study, categorized according the type of Al explanation they received and the task they were assigned to.
Census
Recidivism
Bias
Toxicity
SHAP
86
89
60
88
LIME
65
71
59
85
Adversarially Manipulated
82
92
71
65
Benignly Manipulated
77
84
69
46
For the Bias Detection and Toxicity Detection tasks, the original datasets [46] 48] provide annotations for the political leanings of the sentences and the targeted victims of the text snippets, respectively. After determining the targeted decision 5/ for each task instance, we then followed the gradient-based optimization procedure (i.e., Equation 4 to identify the manipulated explanation. We the step size as 0.01, the trade-off À as 0.01, the optimization threshold 7 as 0.1, and the maximum optimization number T as 100. For the initial AI explanation Ego the start of the optimization process, we directly initialize eggo as c/p ~ 1). We then repeated this optimization process for 5 times and took the average to use in the following human evaluations. For the examples of manipulated explanations, please refer to Appendix B.3
Data Collection. We followed the experimental procedure described in Section 4.2 to collect data on human responses to and perceptions of different AI explanations. We randomly assigned either SHAP or LIME explanations, or the manipulated explanations, to participants. Participants were required to complete 15 tasks with AI model predictions and the assigned explanations (SHAP or LIME or manipulated). We offered a base payment of $1.2 and a potential bonus of $1 the participant's is accuracy is above 85%. Table reports detailed statistics of the participants in each task. Below, we analyzed how the manipulated explanations affect fairness level of human decisions and how do humans perceive those explanations.
5.1 How do the adversarially manipulated explanations affect fairness level of human decisions?
The fairness levels of participants' decision outcomes under the manipulated explanation, SHAP ex- planation, and LIME explanation are presented in Fi Visually, it appears that when human DMs are provided with manipulated explanations, both FPRD and FNRD scores of their decision outcomes tend to deviate more from zero compared to when DMs receive SHAP or LIME explanations.
To examine whether these differences are statistically significant, we conducted regression analyses. Specifically, the focal independent variable the type of explanation received by participants, while the dependent variables were the participants' FPRD and FNRD scores. To minimize the impact of potential confounding variables. we included set of covariates in our regression models, such as participants' demographic background (c.g., age, race, gender, education level), their knowledge of Al explanations, their trust in Al models, and the FPRD or FNRD scores of the Al model decisions they received in the study. These covariates were selected based on prior HCI research [14] 20
41] which empirically reveal how characteristics human DMs may moderate the impacts of AI explanations on human decisions in Al-assisted decision making.
Our regression results indicate that the adversarial party can significantly increase the level of unfairness in human decision outcomes with manipulated explanations through human behavior modeling. Specifically, when examining FPRD, we found that participants who received manipulated Al explanations made more unfair decisions compared to those who received SHAP or LIME explanations (p <. 0.05) in the Census and Recidivism tasks. The difference was marginally significant (p < 0.1) in the Toxicity task. When examining FNRD, results show that participants who received manipulated explanations made decisions that were significantly more unfair than those who received SHAP or LIME explanations (p < 0.01) in the Bias task.
5.2 How do humans perceive the adversarially manipulated AI explanations?

---

Manipulated
SHAP
LIME
Manipulated
SHAP
LIME
FPR Difference
FNR Difference
0.5
0.5
0.3
0.1
-0.1
-0.1
-0.3
-0.3
-0.5
-0.5
Census
Recidivism
Bias
Tuxicity
Census
Recidivism
Bias
Toxicity
(a) False Positive Rate Difference
(b) False Negative Rate Difference
Figure I: Comparing average FPRD and FNRD of the human decision outcomes under the adversar-
ially manipulated explanation, SHAP explanation, or LIME explanation. Error bars represent the 95% confidence intervals of the mean values. and denote significance levels of 0.1, 0.05, and 0.01, respectively. For both FPRD and FNRD, value closer to zero indicates that the human decisions are more fair.
Manipulated
SHAP
LIME
Manipulated
SHAP NONNS
LIME
Transpar ssparencyy
5.0
5.0
4.5
01 4.5
4.0
4.0
3.5
3.0
0.0
2.5
5 2.5
2.0
2.0
Census
Recidivism
Bias
Toxicity
Census
Recidivism
Blas
Toxicity
(a) Transparency
(b) Usefulness
Figure 2: Comparing the average human perceived transparency and usefulness of the adversarially manipulated explanation, SHAP explanation, and LIME explanation. Error bars represent the 95% confidence intervals of the mean values.
In Section 5.1
we found that the adversarial party can manipulate Al explanations nudge human DMs toward making more unfair decisions compared to those who received the original Al explana- tions, aligning with the adversarial party's intentions. To determine whether DMs could detect any abnormalities in the manipulated explanations, we examined how their perceptions of Al explanations varied among manipulated, SHAP, and LIME explanations.
Figures 2a and compare the average perceived transparency and usefulness three types of AI significant differences in how the explanations perceived by
explanations. Visually, there people who received different explanations. We also applied regression models to predict human perceptions of these explanations by accounting for their demographic background (e.g., age, race, gender, education level), their knowledge of Al explanations and their trust in Al models. The regression results indicate that there usefulness of manipulated explanations compared to SHAP or LIME explanations. Similar patterns were observed for perceptions of alignment, comprehensibility, satisfaction, and trust between the manipulated and unmanipulated explanations. While ad adversarially manipulated explanations signifi- cantly influence human decision making behavior, individuals generally do not detect abnormalities in the manipulated Al explanations across most tasks. For further details, please refer to Appendix B.2
significant differences in the perceived transparency and
6 Evaluation II: Manipulating AI Explanations for Benign Purposes
In the previous section, we found that the adversarial party could use the behavior model to manipulate Al explanations, thereby misleading humans into making unfavorable decisions against specific groups. Naturally, one might wonder could third party also use behavior models to manipulate AI
explanations for benign purposes, such as promoting more appropriate human reliance on Al models? For instance, can manipulated AI explanations lead humans to reject AI recommendations when the
7

---

Monipulated
SHAP
LIME
Manipulated
SHAP
LIME
Manipulated
SHAP
LIME
Overreliance
Underreliance
Accuracy
0.9
0.
0.7
0.3
0.5
0.5
Consus
Rocidivism
Blos
Towicht
0.3
Consus
Rockd vism
Blas
Tonicky
0.1
Census
Recidiviam
Bias
Texiciy
(a) Accuracy
(b) Overreliance
(c) Underreliance
Figure 3: Comparing the average accuracy, overreliance, and the underreliance of human decision outcomes under the benignly manipulated explanation, SHAP explanation, or LIME explanation.
Error bars represent the 95% confidence intervals of the mean values. on and denote significance levels of 0.1, 0.05, and 0.01 respectively.
Al model decision is likely incorrect, and encourage acceptance when the decision is likely correct? We aim to explore the answers to this question in this section.
Evaluation Metrics and Manipulating AI Explanations Following previous work 14 41
we used the accuracy, underreliance, and overreliance to measure human DMs' appropriate reliance level on Al models. Underreliance refers the fraction of tasks where the participant's decision was different from the Al model's decision when the Al model's decision was correct. Overreliance refers the fraction of tasks where the participant's decision was the same the Al model's 's decision when the Al model's ' decision was incorrect. To manipulate AI explanations for the promotion of more appropriate reliance on AI models. is necessary determine the reliability of AI model prediction on each task instance. Recent work (52 53) has proposed methods to leverage the complementary strengths of humans and AI by combining human independent decisions and Al model, which is often shown to result in more accurate decisions than those made by either humans or AI models alone. Specifically, given the human independent decision Vindependent the Al model recommendation y
yd and the task instance æ, these methods learn models to combine Vindependent and y" to produce a combined result:
Jcombine CoombineModellgitteeeee
(fiideeeeeeeeee 3 x))
(5)
To see whether Veembine can yield better decisions compared Al alone or human alone, we evaluated various combination models including the human-Al combination method (53) and several truth inference methods (54 used crowdsourcing. Our results showed that the human-Al combination method (53 generally outperformed AI solo and independent human decision, as well as other combination methods. Thus, Ycombinee produced by the human-AI combination method (53) is defined as the targeted decision jjh for manipulating AI explanations. For detailed information
the evaluations of each combination method, please refer to the Appendix [C.. We again followed Equation 4 to manipulate the AI explanations. We set the step size 7 0.01, the trade-off À as 0.01, the optimization threshold r as 0.1, and the maximum optimization number T as 100, and the initial AI explanation e/o at the start of the optimization process is initialized We repeated this optimization process for 5 times and took the average to use in the following experiments. For the examples of manipulated explanations, please refer to Appendix C.4
ego UU-1, 1).
Data Collection. We recruited participants from Prolific once again to collect behavioral data under the benignly manipulated explanations, following the experimental procedure described in Section 4.2 We offered a base payment of $1.2 and a potential bonus of $1 if the participant's 's accuracy is above 85%. Table ssows the detailed statistics of the participants we recruited for each task. Subsequently, we analyzed whether the benignly manipulated explanations can promote appropriate reliance of human DMs on AI models, as well their perceptions of these AI explanations.
6.1
Can benignly manipulated explanations promote appropriate reliance of human DMs on AI models?
Figures 3a 3b and 3C compare the average accuracy, overreliance, and underreliance of human decision outcomes under manipulated, SHAP, and LIME explanations, respectively. It is clear that providing human DMs with manipulated explanations leads to an increase the accuracy of their decision outcomes for most of tasks. We subsequently conducted regression analyses to determine
8

---

Manipulated
SHAP
LIME
Manipulated
SHAP
LIME
5.0
5.0
4.5
4.5
4.0
4.0
Transp
3.5
3.5
3.0
2.5
2.5
2.0
Census
Recidivism
Bias
Texicity
Consus
Recidivism
Bias
Toxicity
(a) Transparency
(b) Usefulness
Figure 4: Comparing the average human perceived transparency and usefulness of the benignly manipulated explanations, SHAP explanations, and LIME explanations. Error bars represent the 95% confidence intervals of the mean values.
whether these differences are statistically significant. The regression models incorporated set of covariates, including participants' demographic backgrounds (e.g., age, race, gender, education level), their knowledge of Al explanations, their trust in Al models, and the accuracy of the Al models. The regression results indicate that in the Census, Recidivism, and Bias tasks, substituting SHAP LIME explanations with manipulated explanations significantly improves the accuracy of human AI team. In contrast, for the Toxicity task, we observed no statistical difference, which could potentially be attributed to the high competence of humans in solving this task (e.g., when presented with SHAP or LIME explanations, the average decision accuracy of participants already exceeds 0.8, leaving limited room for further improvement..
6.2 How do humans perceive benignly manipulated AI explanations?
In Section 5.2 we observed that it is challenging for humans to detect abnormalities the adversari- ally manipulateds explanations, even though they unconsciously influenced by the manipulated explanations to make more unfair decisions. In this section, we revisit this question to investigate into whether humans' perceptions of the manipulated explanations change, when they are manipulated for benign purposes. Figures 4a and compare the average human perceived transparency and usefulness of the benignly manipulated explanations, SHAP explanations, and LIME explanations. Regression analyses reveal no statistically significant differences among the perceived transparency and usefulness of these three types of explanations. Similar trends were observed for other perceptual aspects of explanations, including perceived alignment, comprehensibility. satisfaction, and trust. For further results, please refer to Appendix
7 Conclusion and Limitations
In this paper, we explore whether we can quantitatively model how humans incorporate both AI recommendations and explanations into their decision making process, and whether we can utilize the quantitative understanding of human behavior obtained from these learned models manipulate Al explanations for both adversarial and benign purposes. Our extensive experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations toward targeted outcomes, regardless of the intent being benign or adversarial. Despite the significant influence of these falsified explanations on human decisions, individuals typically fail to detect or recognize any abnormalities. Our study has several limitations. For example, it focuses on modeling and manipulating core-based explanations. Further research is needed to explore how to model how humans incorporate other types of explanations, such as example-based and rule- based explanations, and how these can manipulated to influence human behavior as observed with score-based explanations in our study. Additionally, our study was limited to decision making tasks involving tabular and textual data, which are naturally suited to score-based explanations. Further explorations needed to extend these findings to decision tasks with other data types (c.g., images).

---

Ethical Consideration
This study was approved by the Institutional Review Board of the authors' institution. Through our findings, we aim to draw the community's 's attention to the ease with which third parties can manipulate Al explanations with the learned behavior models to influence human decision making. Users often lack the ability to accurately and appropriately interpret the Al explanations presented to them, yet their decision behavior is easily swayed by the manipulated Al explanations. Our findings highlight the critical importance of securing human-Al interaction data to prevent the misuse of human behavior models derived from it. Additionally, there is an urgent need to ensure that AI explanations provided to humans are more secure and inherently benign. Moreover, providing pre-education is essential to assist humans in establishing a proper understanding of Al explanations, which may potentially mitigate the risks of manipulation.
In addition, our experiments are based on datasets that are publicly available: "correct" decisions for the tasks in these datasets are generally considered as recording the real-world ground truth. While these datasets are not intentionally biased toward any specific groups, we acknowledge that there might be implicit biases introduced to these datasets during the curation process, which are beyond our control. Importantly, we note that we made no alterations to the datasets that would introduce additional bias in our experiment.
Acknowledgments
We thank the support of the National Science Foundation under grant 122-2229876 and IIS- IS--2302299 on this work. Any opinions, findings, conclusions, recommendations expressed here those of the authors alone.
References
[I] Gagan Bansal ct al. "Is the most accurate ai the best teammate? optimizing ai for teamwork". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 13. 2021, pp. 11405-11414.
[2] Aakriti Kumar et al. "Explaining algorithm aversion with metacognitive bandits". In: Proceedings of the annual meeting of the cognitive science sociery. Vol, 43, 43. 2021.
[3] Peter Hase and Mohit Bansal. "Evaluating explainable Al: Which algorithmic explanations help users predict model behavior?" In: arXiv preprint arXiv:2005. 501831 (2020).
[4] Heliodoro Tejeda et al. "Al-assisted decision-makinn A cognitive modeling approach to infer latent reliance strategies". In: Computational Brain & Behavior 5.4 (2022), pp. 491-508.
[5] Hussein Mozannar, Arvind Satyanarayan, and David Sontag. "Teaching humans when to defer to classifier via exemplars". In: Proceedings of the aaai conference on artificial intelligence. Vol. 36. 2022, pp. 5323-333333 5323-5331.
[6] Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. "Uncalibrated models can improve human-ai collaboration.. In: Advances in Neural Information Processing Systems 35 (2022), pp. 4004-4001
[7] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. .. Why should trust you?" Explaining the predictions of any classifier". In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135-1144.
[8] Scott M Lundherg and Su-In Lec. "A unified approach to interpreting model predictions". In: Advances in neural information processing systems 30 (2017).
[9] Himabindu Lakkaraju. Stephen H Bach, and Jure Leskovee, "Interpretable decision sets: A joint frame- work for description and prediction". In: Proceedings the 22nd ACM SIGKDD international conference knowledge discovery and data mining. 2016, pp. 1675-1684.
[10] Himabindu Lakkaraju et al. "Faithful and customizable explanations of black box models". In: Proceed- ings of the 2019 AAAI/ACM Conference AI, Ethics, and Society. 2019, pp. 131-138.
[1]] Chirag Agarwal al. "Openxai: Towards a transparent evaluation of model explanations", In: Advances in Neural Information Processing Systems 35 (2022), pp. 15784-15799.
[12] Gagan Bansal al. "Does the whole exceed its parts? the effect of ai explanations on complementary team performance". In: Proceedings of the 2021 CHI conference on human factors in computing systems. 2021, pp. 1-16.
[13] Zana Buçinca et al. "Proxy tasks and subjective measures can be misleading in evaluating explainable Al systems". " In: Proceedings of the 25th international conference on intelligent user interfaces. 2020, pp. 454-464.
10

---

[14] Yunfeng Zhang. Q Vera Liao, and Rachel KE Bellany. "Effect of confidence and explanation accuracy and trust calibration in Al-assisted decision making". In: Proceedings of the 2020 conference on fairness. accountability, and transparency. 2020, pp. 295-305.
[15] Jonathan Dodge al. "Explaining models: an empirical study of how explanations impact fairness judgment". In: Pruceedings of the 24th international conference on intelligent user interfaces. 2019.
pp. 275-285.
[16] Forough Muusabzi-Sangdeer et al. "Manipulating and measuring model interpretability.. In: Proceedings of the 2021 CHI conference on human I factors in computing systems. 2021, pp. 1-52.
[17] Arjun Chandrasekaran al. "Do explanations make VQA models more predictable to human?" In: arviv preprint arXiv:1810.1 12366 (2018).
[18] Siddhant Arora "Explain, edit, and understand: Rethinking user study design for evaluating model explanations". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. 5. 2022, Pp. 5277-5285.
[19] Hao-Fei Cheng et al. "Explaining decision- making algorithms through UI: Strategics to help non-expert stakeholders". In: Proceedings of the 2019 chi conference on human factors in computing systems. 2019,
pp. 1-12.
[20]
Xinru Wang and Ming Yin. "Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making" In: 26th international conference intelligent user interfaces. 2021, pp. 318-328.
[21] Nina Corvelo Benz and Manuel Rodriguez. "Human-aligned calibration for ai-assisted dccision making". In: Advances in Neural Information Processing Systems 36 (2023).
[22] Zhuoran Lu al. "Mix and Match: Characterizing Heterogencous Human Behavior in Al-assisted Decision Making". In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing.
Vol. 12. 2024, pp. 95-104.
[23] Zhuoyan Li, Zhuoran Lai, and Ming Yin. "Decoding Al's Nudge: A Unified Framework to Predict Human Behavior in Al-assisted Decision Making". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 9. 2024, pp. 10083-10091.
[24] Zhuoyan Li, Zhuoran Lui, and Ming Yin. "Modeling human trust and reliance in ai-assisted decision making: A markovian approach". In: Proceedings of the AAAI Conference on Artificial Intelligence.
Vol. 37. 5. 2023, pp. 6056-6064.
[25] Shuai Ma et al. "Who should trust: Ai or myself? leveraging human and ai correctness likelihood to promote appropriate trust in ai-assisted decision decision-making"" In: Proceedings of the 2023 CHI Conference Human Factors Computing Systems. 2023, pp. 1-19.
[26]
Zana Buçinca et al. "Towards Optimizing Human-Centric Objectives in Al-Assisted Decision-Making With Offline Reinforcement Learning.. In: arXiv preprint arXiv:2403.00 .0591 (2024).
[27] Joey Hong, Sergey Levine, and Anca Dragan. "Learning to influence human behavior with offline reinforcement learning". In: Advances in Neural Information Processing Systems 36 (2023).
[28] Hussein Mozannar et al. "Effective Human-Al Teams via Learned Natural Language Rules and Onboard- ing". In: Advances in Neural Information Processing Systems 36 (2022).
[29] Syed Hasan Amin Mahmood, Zhuoran Lu, and Ming Yin. "Designing Behavior-Awar Al to Improve the Human-Al Team Performance in Al-Assisted Decision Making". In: Proceedings of the Third International Joint Conference on Artificial Intelligence. 2024.
[30] Hussein Mozannar et al. "When show suggestion? Integrating human feedback in Al-assisted programming". : In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 9. 2024. pp. 10137-10144.
[31] Valerie Chen et al. less--cas-grounddee simulations for explanation evaluation". In: Advances in neural information processing systems 35 (2022), pp. 1764-1775.
[32] Julien Colin et al. "What cannot predict, not understand: cemanneenttred evaluation framework for explainability methods". In: Advances in neural information processing systems 35 (2022), pp. 2832- 2845.
[33] Chacha Chen et al. "Machine explanations and human understanding". In: arXiv preprint arXiv:2202.04000 04092 (2022).
[34] Zhuoran Lu et al. "Strategic adversarial attacks in Al-assisted decision making to reduce human trust and reliance". In: Proceedings of the -ecord International Joint Conference on Artificial Intelligence. 2023, pp.
[35] Himabindu Lakkaraju et al. "Rethinking explainability as dialogue: A practitioner's perspective". In: arXiv preprint 2rXiv:2202.01875 (2022).
[36] Dylan Slack et al. "Counterfactual explanations can be manipulated". In: Advances in neural information processing systems 34 (2021), pp. 62-75.
[37] Dylan Slack et al. "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods".
In: AAAUACM Conference on Artificial Intelligence, Ethics, and Society (AIES) (2020).
11

---

[38]
Rohan Paleja et al. "The utility of explainable ai in ad hoc human-machine teaming". In: Advunces in neural information processing systems 34 (2021), pp. 610-623.
[39]
Hua Shen et al. "Are shortest rationales the best explanations for human understanding?" In: arXiv preprint arXiv:2203.085 08788 (2022).
[40] Han Liu, Vivian Lai, and Chenhao Tan. "Understanding the effect of out-of out-of-distribution examples and interactive explanations on human decision making". In: Proceedings of the ACM on Human- Computer Interaction 5, 5.CSCW2 (2021), pp. 1-45.
[41] Vivian Lai, Han Liu, and Chenhao Tan. "" Why is Chicago deceptive?" Towards Building Dodel-Driven Tutorials for Humans". In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020, pp. 1-13.
[42] Himabindu Lakkaraju and Osbert Bastani. '" How do I fool you?" Manipulating User Trust via Misleading Black Box Explanations". In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2020, pp. 79 85.
[43]
Vivian Lai et al. "Selective explanations: Leveraging human input align explainable ai". In: Proceedings of the ACM Human-Comppunn Interuction 7. 7CCSC22 (2023), pp. 1-35. Ron Kohavi. Census Income. UCI Machine Learning Repository. DOI: https://doi..rr/1100 http://bb..or//00222433111CCPP 1996.
[44]
[45]
Julia Dressel and Hany Farid. "The accuracy, fairness, and limits of predicting recidivism". In: Science advances 4.1 (2018), eaao5580.
[46] Timo Spinde al. "Neural Media Bias Detection Using Distant Supervision With BABE Bias An- notations By Experts". In: Findings of the Association for Computational Linguisties: EMNLP 2021. Ed. by Marie-Francine Moens et al. Punta Cana, Dominican Republic: Association for Computational
Linguistics, Nov. 2021, pp. 1166-11 1177. DOI: 10. 18653/ v1 2021 findings- emnlp. 101 URL:
https://acl antholog org/2021. http:/221..inndnn findings--- emmlp. 101
[47] Jacob Devlin et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing". In: Proceedings the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). Ed. by Jill Burstein, Christy Doran, and Thamar Solorio. Minneapolis, Minnesota: Association for
Computational Linguistics, June 2019. pp. 4171 117764 DOL: 10. 1ww111666//vv// 1423 URL: https
//aclanthology- hwwphhhoooooyyor//199119 org/ 1423
[48] Chris J Kennedy et al. "Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application". In: arXiv preprint arXiv:2009, 10277 (2020).
[49] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization". In: arXiv preprint 141xiv:442..000 6980 (2014).
[50] Moritz Hardt, Eric Price. and Nati Srebro. "Equality of opportunity in supervised learning". In: Advances neural information processing systems 29 (2016).
[51] Lucas Dixon al. "Measuring and mitigating unintended bias in text classification". In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018, pp. 67-73.
[52] Mark Steyvers et al. "Bayesian modeling of human-Al complementarity". In: Proceedings of the National Academy of Sciences 119.11 (2022), e21 1547119.
[53] Gavin Kerrigan, Padhraic Smyth, and Mark Steyvers. "Combining human predictions with model probabilitics via confusion matrices and calibration". In: Advances in Neural Information Processing Systems 34 (2021), pp. 4421 14444.
[54] Yudian Zheng et al. "Truth inference in crowdsourcing: Is the problem solved?" In: Proceedings of the VLDB Endowment 10.5 (2017), pp. 541-552,
[55] Jacoh Whitchill et al. "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise". In: Advances in neural information processing systems 22 (2009).
[56]
Qi Li et al. "A confidence-aware approach for truth discovery on long-tail data". In: Proceedings of the VLDB Endowment 8.4 (2014), pp. 425-436.
[57]
Vikas C Raykar et al. "Learning from crowds." In; Journal of machine learning research 11.4 (2010).
12

---

Table A.I: The average hourly payment received by participants our study across four tasks. the row "Number of Participants" the number in parentheses indicates the number of invalid participants who did not pass the attention check questions.
Recidivism
Census
Bias
Toxicity
Number of Participants Average Working Time (Minute)
336 (16)
310 (25)
259 (20)
286 (17)
6.67
6.25
6.98
6.34
Hourly Payment (Base) Hourly Payment (Base Bonus)
$10.8
$11.7
$10.2
$11.3
$11.9
$11.8
$11.4
$14.4
dilw D0%
Cer
rold sean
(a) Census prediction
(b) Recidivism prediction
Figure A.I: The task interfaces for the census prediction and recidivism prediction.
A
The Design of Human Study (Additional Details)
A.1
Compensation Details
To determine the appropriate payment level for each type of task, we first conducted a preliminary study to estimate the time workers might spend on the tasks. Our pilot study indicated that a base payment of $1.2 per task translates to an approximate hourly rate of $10. To provide greater trans- parency about the compensation received by participants in our formal study, Table A.1 summarizes the average hourly payment and the average time spent on each task.
A.2 Task Interfaces
Figure A. A.Ib A.2a and A.2b show the interfaces participants used in the Census Prediction, Recidivism Prediction, Bias Detection, and Toxicity Detection tasks, respectively.
A.3
Exit Survey Questions
In the study, after the main tasks, the participants need to fill in an exit survey to report their perceptions presented AI explanations. The survey questions are detailed as:
Alignment: On a scale of 1-5, how well do you think the explanations align with your understanding the problem?
Usefulness: On scale of 1-5, how useful are the explanations helping you make decisions?
Transparency: On scale of 1-5, how well do you think the explanations reveal the AI model's decision making process?
Comprehensibility: On a scale of 1-5, how easy is it for you to understand the explanations?
Satisfaction: On a scale of 1-5, how satisfied are you with the explanations provided by the Al model?
Trust:
On scale of 1-5, would you trust the Al model's prediction or decision based on the explanations?
13

---

Prediction
Prediction Task
P Pascar rivine the senience from made raport Lekon and delermine whather this senteriaa
Please nertex the senience from socill madia below and destermine whather ths centence can aen su apench, as coummnnt
scdly retr pecule wtnrive wrennv the pull une
asd seter
myy Rorente Horrs Whyy Chock
Modal Prediction:
Modde Prodicton
nonvbleadd
ucdSs ana con-iiaaad urregor, and nec sords ame seaoc ated with
Bug wards abscituted willi sariging and work assosiated with 00/ng pioos
binend The
Ivo carage-
your
You. Falkk thi senterce blsadd.
Yex, mark is serience is tocie.
No. teek sentence non-blised.
(a) Bias detection
(b) Toxicity detection
Figure A.2: The task interfaces for the bias detection and the toxicity detection.
Table B.1: Agreement between the sum of feature importance in explanations and AI predictions,
measured in terms of the Pearson correlation coefficient.
Census
Recidivism
Bias
Toxicity
SHAP
0.88
0.97
0.88
0.91
LIME
0.41
0.40
0.76
0.78
Adversarially Manipulated
0.57
0.38
0.73
0.68
Benignly Manipulated
0.87
0.68
0.89
0.89
B Evaluation I: Manipulating AI Explanations for Adversarial Purposes (Additional Results)
B.1 Visual Consistency of Explanations
Table B.1 compares the agreement between the sum of feature importance in explanations and AI predictions, the Pearson correlation coefficients, measured in terms Pearson correlation coefficient, for adversarially manipulated, LIME, and SHAP explanations, respectively. We observed that the visual consistency of the manipulated explanations lower than that of SHAP but very close that of LIME.
B.2 Human Perceptions of Explanations
compare the average human perceived alignment, comprehensibility. satisfaction with the provided explanations ,and the trust in the AI models under the under the adversarially manipulated explanation, SHAP explanation, or LIME explanation. In general, our findings indicate that there are significant differences people's ' perceptions of the three explanations across four tasks, with the exceptions for the alignment and trust the Toxicity task. Specifically, for the Toxicity task, participants perceived LIME explanations as aligning more closely with their own rationales than the adversarially manipulated explanations, with a marginally significant difference (p 0.1). Furthermore, participants reported significantly greater trust in the AI models accompanied by LIME explanations compared to those with adversarially manipulated explanations (p < 0.01).
Figures B.la to
B.3 Examples of Manipulated Explanations
Figures B.2 to B.5 show the visual comparisons of adversarially manipulated, LIME, and SHAP explanations for Census Prediction task, Recidivism Prediction task, Bias Detection task, and Toxicity Detection task, respectively.
14

---

amprchensiiility
Masipalbied
5 S 5
4.5
(a) Alignment
(b) comprehensibility
(c) Satisfaction
(d) Trust
Figure B.I: Comparing the average human perceived alignment, comprehensibility, satisfaction with the provided explanations, and the trust in the Al models under the adversarially manipulated explanation, SHAP explanation, or LIME explanation. Error bars represent the 95% confidence intervals of the mean values. and denote significance levels of 0.1 and 0.01 respectively.
1. Agg::
2. Sac::
Nale
Hours pet wook
Al Prediction: Label: 0, Targeted Decision: 1
Number years education:
Occopation:
Carl
Work Closs:
Governent Bedtrr
(a) LIME
(b) SHAP
(c) Manipulated
Profll:
Famale
Hours week:
Al Prediction: 1, Label: 1, Targeted Decision:
Numbbr of years aducation:
Marial Gtouus
Marked
Occupation:
Profozsioal Spocialy
(a) LIME
(b) SHAP
(c) Manipulated
Figure B.2: The visual comparisons of adversarially manipulated, LIME, and SHAP explanations for the Census Prediction task.
C Evaluation II: Manipulating AI Explanations for Benign Purposes (Additional Results)
C.1 Combining Human Decisions and AI Predictions
In the main paper, we aim to benignly manipulate AT explanations to encourage human DMs rely more appropriately on AI models. Following previous research [22 533 we combined independent human decisions with AI model predictions to determine the targeted decision for each task instance. We evaluated the human-Al combination method (53] and several truth inference methods used in crowdsourcing for truth discovery. We detailed the process of evaluation below.
Table C.I: The average accuracy of the independent human behavior model through 5-fold validation for each task.
Census
Recidivism
Bias
Toxicity
Accuracy
0.81
0.84
0.62
0.79
15

---

Rack:
Black
Age
16:
Al Prediction: 0, Label: 0, Targeted Decision: 1
5. Felony Crime Count before Age
Age
7. Current Charge Deginel
Fakeyy
E: Current Charge Issua:
Appressted Asssutt WiDeod Weop
Rex
(a) LIME
(b) SHAP
(c) Manipulated
Defendant Profile:
Raca:
Wheie
Ses:
Male
Age:
27
Prior Crima Count atter Age
Fetony Count Age 18:
Al Prediction: 1, Label: 1, Targeted Decision: 0
Mademmaaer Count before
Charge Degres:
Felory
Charge
(Molo Venci))
10
(a) LIME
(b) SHAP
(c) Manipulated
Figure B.3: The visual comparisons of adversarially manipulated, LIME, and SHAP explanations for the Recidivism Prediction task.
Political leaning: right-leaning, Al Prediction: Label: 1, Targeted Decision: 0
Pecsts corossenss part House
FRosE Socninna Skx
cliwing portuer sppotiie snleet deine whtwire puttedd conraton staciinn athicialy deltect andees number azserase sakon from veset.
balet harvering lodesl rection aftecivey
Nosaall
eaciion mudeet
endicts namber of stseesse zafors from vonee.
andees
o M balion Fro wcters
(a) LIME
(b) SHAP
(c) Manipulated
Political leaning: left-leaning, Al Prediction: 1, Label: 0, Targeted Decision: 1
Kandley hat for years been acciting Puth and No amourage of rergant compttin.
Nenelery years beer inceenn Faln tei
werr
accuviny
and
entzurage
compption
(a) LIME
(b) SHAP
(c) Manipulated
Figure B.4: The visual comparisons of adversarially manipulated, LIME, and SHAP explanations for the Bias Detection task.
Simulating Human Independent Decision. To understand how humans independently make decisions on each task instance, we first conducted study again on the Prolific to collect independent human decision behavior data across four tasks. We recruited 40 participants for each task. Each recruited participant needed to complete 15 tasks. With the collection of human behavior data, we then fitted two-layer neural networks to simulate human independent decision behavior. For Toxicity Detection and Bias Detection textual tasks, we used the pretrained BERT encoder to initially extract features from the original sentences as the input to the independent behavior models. We optimized these independent behavior models using Adam
49 with initial learning rate of le 4 and
16

---

Victim: Black, Al Prediction: 0, Label: 0, Targeted Decision: 0
this Milen amoud be veceet Blick namen be nere incloves Net wrougn diverite they of Chinsse
abouid blick WETKE more hot enough sivernts ar
recuit smooilh diveritt
(a) LIME
(b) SHAP
(c) Manipulated
Victim: Black, AI Prediction: 1, Label: 1, Targeted Decision: 0
tnt ine
start
(a) LIME
(b) SHAP
(c) Manipulated
Figure B.5: The visual comparisons of adversarially manipulated, LIME, and SHAP explanations for the Toxicity Detection task.
Marippdated
Maniguritted
SHAP
Masipuibbie
amprcheeabbli
(a) Alignment
(b) comprehensibility
(c) Satisfaction
(d) Trust
Figure C.I: Comparing the average human perceived alignment, comprehensibility, satisfaction with the provided explanations, and the trust in the Al models under the benignly manipulated explanation, SHAP explanation, or LIME explanation. Error bars represent the 95% confidence intervals of the
mean values.
a batchsize of each training iteration of 128. The number of training epochs is set as 10. The average accuracy of 5-fold validation for each model is reported Table C.I which we found be satisfactory. We then utilized these fitted models to simulate independent human decisions Vindopendent in the human-Al combination process to determine the potentially better decisions.
Comparing Combination Performance. We consider the human Al combination method [53] and a few truth inference methods in crowdsourcing as baselines in the evaluation, including GLAD 55 CATD (56). LFC (57). EM (54), and MV (54). These methods combine the hu- Vindependent predicted by the fitted independent human behavior models and Al model recommendations y" to produce combined decisions Ycombine- The accuracy each method on holdout task pools for each task to be used in the subsequent evaluation is reported in Table C.2 In general, we found that human + Al combination method outperforms other baselines. By integrating human decisions with AI predictions, this method shows superior performance to either AI solo human solo across all four tasks. Consequently, we used the combined decisions Ycombine from the human AI combination method the targeted decision ÿ in subsequent experiments manipulate explanations.
man independent decisions l
C.2 Visual Consistency of Explanations
Table B.I compares the agreement between the sum of feature importance in explanations and AI predictions, the Pearson correlation coefficients, measured in terms Pearson correlation coefficient, for benignly manipulated, LIME, and SHAP explanations, respectively. We observed that the visual consistency of the manipulated explanations very close to that of SHAP and higher than that of LIME.
C,3 Human Perceptions of Explanations
Figures C.1a Taab to compare the average human perceived alignment, comprehensibility, satisfaction with the provided explanations and the trust in the AI models under the under the adversarially
17

---

Table C.2: The accuracy of each method on the holdout task pools, used in following experiments to
manipulate Al explanations. The best result in each row is highlighted in bold.
Humnn Solo
Al Solo
Hurman Al 53
GLAD 55
CATD 56
LPC
EM 54
MV [54
Census
0.61
0.73
0.76
0.62
0.74
0.61
0.60
0.69
Recidivism
0.54
0.58
0.61
0.59
0.61
0.58
0.59
0.62
0.66
0.67
0.69
Toxicity 0.76
0.86
0.86
0.78
0.79
0.81
0.77
0.82
Ago!
20
2. Bexs
Ferralo
Hour per weak:
36
Al Prediction: 1, Label: 1, Targeted Decision:
Number of years edacation:
Moriaal Btatus:
Nover Morred
Occupation:
Sperbelly
Class:
Privlee
(a) LIME
(b) SHAP
(c) Manipulated
Proffie:
Al Prediction: 0, Label: 0, Targeted Decision:
Number education:
Marial Blatus:
Nover Mariod
6. Occupation:
Professonal Specialy
Work Class:
Covprsment Secto
(a) LIME
(b) SHAP
(c) Manipulated
Figure C.2: The visual comparisons of benignly manipulated, LIME, and SHAP explanations for the Census Prediction task.
manipulated explanation, SHAP explanation, or LIME explanation. We found that there are no significant differences in people's perceptions of the three explanations across four tasks on these aspects.
C.4 Examples of Manipulated Explanations
Figures to C.5 show the visual comparisons of benignly manipulated, LIME, and SHAP explanations for Census Prediction task, Recidivism Prediction task, Bias Detection task, and Toxicity Detection task, respectively.
18

---

16:
Prediction: 0, Label: 1, Targeted Decision:
6. Felory Orime Count before Age
Wadermeane Courti
Current Charge
Current
Grand That (Matrr vildd)
Mendine
Sechood
(a) LIME
(b) SHAP
(c) Manipulated
Defendont Profile:
Roce:
White
Sex:
Male
Age:
87
Prior Crime Count ofter Age
AI Prediction: 0, Label: 0, Targeted Decision: 0
Age 16:
6. Mandemanno Count before
Degres:
Aw:th
(a) LIME
(b) SHAP
(c) Manipulated
Figure C.3: The visual comparisons of benignly manipulated, LIME, and SHAP explanations for the Recidivism Prediction task.
Political leaning: left-leaning, Al Prediction: 1, Label: 1, Targeted Decision: 1
Donald Truna, che-chllc okcant moid ferright
distant culture wor cruedde; has takon keen inerest atertion rights kowy.
for-ight
distrn::
risht lamel:
(a) LIME
(b) SHAP
(c) Manipulated
Political leaning: left-leaning, Al Prediction: 1, Label: 1, Targeted Decision: 1
froter and artivecche gracee are swoing the and gevemnane nentinett moked by comsrratee laniing proteeters adurce thai: cass.
Hatoc Boad macine government sentiment staked comserative
leaning protenters advence their cane.
(a) LIME
(b) SHAP
(c) Manipulated
Figure C.4: The visual comparisons of benignly manipulated, LIME, and SHAP explanations for the Bias Detection task.
19

---

Victim: Black, Al Prediction: 0, Label: 0, Targeted Decision:
Theres
merel Lean Feed
socual
64
senual hannormnn
UB
(a) LIME
(b) SHAP
(c) Manipulated
Victim: Black, AI Prediction: 1, Label:1, Targeted Decision:
KoOR the
cum
head
detier why fook Sheat
V91
EM
WTV
1991
me
2hat you
Selver
pou
(a) LIME
(b) SHAP
(c) Manipulated
Figure C.5: The visual comparisons of benignly manipulated, LIME, and SHAP explanations for the Toxicity Detection task.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes].
Justification: In this paper, we aim to explore whether we can model how humans incorporate both AI predictions and explanations into their decision making process, and whether we can utilize the quantitative understanding of human behavior from these models to manipulate explanations, thereby nudging human decisions in Al-assisted decision making. Through extensive human subject experiments, we showed the good, bad, and the ugly side of this. The claims in the abstract and the introduction accurately reflect the paper's contributions and scope.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We discuss the limitations of the work in the Conclusion section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: This paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes].
Justification: We report the settings of the hyper-parameters for training behavior models and manipulating Al explanations in the paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No].
Justification: will ensure that the collected human behavior data is used responsibly by implementing strict access controls minimize potential risks associated with unauthorized use or misuse. Individuals who wish to have access to the data or code must apply for permission, which will only be granted to those who meet the necessary authorization criteria.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We describe training and test details in the paper.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
21

---

Justification: The experimental results are accompanied by the 95% confidence intervals, and the statistical significance tests.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes].
Justification:
We used one RTX 4060 for behavior model training and Al explanation manipulation.
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics hattps://neurt httppc/ccbb.///iiicccciiie.e.e.e
Question:
Answer: [Yes].
Justification: We checked the ethics guidelines, and the research conducted in the paper conform with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We discuss the societal impacts of the work in the Ethical Consideration
section.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No].
Justification: We will ensure that the collected human behavior data is used responsibly by implementing strict access controls minimize potential risks associated with unauthorized use misuse.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We cited all datasets used in the paper.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes].
Justification: We included the task interfaces, the experimental procedure, and the compen- sation structure in the paper.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes].
Justification: The study was approved by the Institutional Review Board of the authors' institution. To minimize the risk of leaving our study participants with inappropriate impressions of manipulated AI explanations, upon the end of the study, provided debrief session directly through the Prolific chat system. Each participant who received manipulated AI explanations the study were individually contacted through the Prolific platform. In the chat- basdd session, we clarified that the Al explanations were manipulated and did not accurately represent the underlying rationales of the Al models. We emphasized that these explanations were intentionally biased for the purpose of the study, based on the specific task participants were involved in.
23


Reviews:
[{"review_id": "CvMeP5Fxah", "reviewer": "Reviewer_f7Sf", "summary": "The authors show that by modeling a human decision maker they can manipulate the provided information in ways that reliable influence their decisions towards even non-benign outcomes.", "questions": "Are the manipulations working possibly just due to the bar plots being being bigger, and numbers being clearer? So no behavioral model needed.\n\nHow should I read figure 1? It looks like it's saying the means 95% confidence intervals overlap, but that they are also below 5% to overlap (bias 1a)?\n\nMore generally I think this result is weak, if people could tell the models apart this would be more interesting. As it stands I'm not surprised the authors can manipulate the responses, but I'm not convinced their method is close to optimal.", "limitations": "This is an area where ethics should be given additional scrutiny, but I do not see anything alarming in the paper, and the authors take the correct tone in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "# originality\n\nManipulating Mturk works is a well studied area of research, but this is a unique approach and highlights the limitations of interpretability research using similar techniques.\n\n# quality\n\nThe results are statistically significant and done across a range of tasks which strengthens their claims. \n\n# clarity\n\nMost of the plots are readable and I think I was mostly able to understand the results after reading a few time.\n\n# significance\n\nThis is an interesting result, presenting an example of \"attacking\" these score based systems via manipulating the interpretably metrics is an important result for the XAI literature.", "weaknesses": "# originality\n\nManipulating workers on digital platforms is a well studied area.\n\n# quality\n\nReading the plots (fig 3) some of these results look to be supporting the null-hypothesis (all interventions have the same effect). Having a table giving the numbers and more details on the statistical tests would make the quality of the results much easier to judge. The results also are weak, in part this is due to a small sample size.\n\n# clarity\n\nIf found the plots difficult to read either both being too small and the lack of numbers makes analyzing them tricky. Could the authors include the results as a table in the appendix with the error ranges clearly laid out so we don't have to eyeball tiny error bars.\n\nI also had to refer to the appendix to understand the experimental procedure, making it clearer what exactly is happening in the experiments would greatly aid the readability of this paper.\n\n # significance\n\nLooking at figure B.2 it looks like the manipulation is mostly making the difference much more extreme and noticeable, this suggests these results could simply be due to the differences in the visual presentation between interventions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! Below, we address your questions.\n\n>  Are the manipulations working possibly just due to the bar plots being being bigger, and numbers being clearer? So no behavioral model needed.\n\n- Our behavior model-based manipulation demonstrated that changing the bar length in some task instances—indicating a change in the perceived importance of a feature—indeed influences human decision making. This shift in behavior, captured by our human behavior model, aligns with previous research which showed that overstating model confidence can similarly impact decisions [1,2,3]. \n\n- However, our manipulation strategy goes beyond merely increasing the length of bars to make them more salient. In some cases, the manipulation also involves reducing the bar length or even reversing the direction of the bars to downplay or misrepresent a feature's importance.  We observed that in many cases, especially when we manipulate explanations for adversarial purposes and AI models may actually be unfair, the model-based manipulation intentionally retains certain features as protective to mislead human decision making. For example, for the census prediction task, when predicting whether a person's income is below \\\\$50k, if the person is a female, the manipulated explanation might retain gender as a positive contributor to reaching the \\\\$50k threshold, whereas for males, gender might be depicted as a negative factor. By doing so,  humans are more likely to make unfair decisions compared to those who got unmanipulated explanations based on our evaluation data.\n\nTherefore, the manipulations based on behavior models are not solely about making bars longer but are tailored to strategically influence human perceptions in line with the model's understanding of decision making behavior.\n\n> How should I read figure 1? It looks like it's saying the means 95% confidence intervals overlap, but that they are also below 5% to overlap (bias 1a)?\n\nIn Figure 1(a), the circle (or square or rectangular) represents the mean values, and the error bars represent the 95% confidence intervals. This plot is intended to illustrate how the average False Positive Rate Difference (FPRD) or False Negative Rate Difference (FNRD) of human decisions might be distributed within each of the three treatments (i.e., Manipulated, LIME, SHAP). The statistical significance between a pair of treatments is directly determined by the linear regressions, rather than relying on the visual check of the overlap of confidence intervals. Significance levels are based on the p-values from t-tests on the coefficients derived from these regression models.\n\n> More generally I think this result is weak, if people could tell the models apart this would be more interesting. As it stands I'm not surprised the authors can manipulate the responses, but I'm not convinced their method is close to optimal.\n\nIf we understand your comment correctly, you are suggesting that the results would  be more interesting if participants could tell that the explanations had been manipulated. Additionally, it seems you think that the current results might simply be due to the fact that the bars in the explanations were made longer, as mentioned in your earlier point.\n\n- We would like to clarify that our manipulation is not simply about making the bars longer. Instead, as discussed in response to the first question, our approach is based on a human behavior model and is strategically tailored to influence human perceptions in alignment with the model's understanding of humans’ decision-making behavior (i.e., how humans will factor AI recommendations and explanations into their final decisions).\n\n- Regarding whether people can tell if the explanations are manipulated, we find it is also interesting—and more concerning—that human behavior can be changed without participants detecting these manipulations, which indicates significant implications for secure and reliable human-AI collaboration. If participants are unaware of the subtle influences on their decision making, it raises important questions about the transparency and trustworthiness of AI systems, particularly in contexts where security and reliability are paramount.\n\n> Paper clarity\n\nFinally, we will revise the manuscript to enhance the readability of the figures by increasing their size and adding the corresponding labels. Additionally, we will include the linear regression results as tables in the appendix.\n\n\n[1] Vodrahalli, Kailas, Tobias Gerstenberg, and James Y. Zou. \"Uncalibrated models can improve human-ai collaboration.\" Advances in Neural Information Processing Systems 35 (2022): 4004-4016.\n\n[2] Zhang, Yunfeng, Q. Vera Liao, and Rachel KE Bellamy. \"Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making.\" Proceedings of the 2020 conference on fairness, accountability, and transparency. 2020.\n\n[3] Rechkemmer, Amy, and Ming Yin. \"When confidence meets accuracy: Exploring the effects of multiple performance indicators on trust in machine learning models.\" Proceedings of the 2022 chi conference on human factors in computing systems. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRUbeX75Dk", "reviewer": "Reviewer_L1hd", "summary": "This paper proposes a novel method that manipulates human decision-making by manipulating AI explanations in human-AI interaction scenarios. By utilizing human behavior models and minimizing the cross-entropy function between human and AI agreement with constraint to generating the same AI recommendation outcomes, the authors investigate this method for adversarial and benign purposes. The results show that the manipulation in AI explanations significantly negatively affects human decisions in the four tasks with adversarial purpose and enhances human decision accuracy, over-reliances, and diminishes under-reliances. The paper also discusses how the human perception to the AI explanations varies with the manipulation. Overall, it provides a novel and practical way to intervene in human decision-making in human-AI interactions.", "questions": "- My question is humans themselves may have a lot of biases, for example, peak and anchoring effects; or in risky decision-making, humans can have probability distortion. This is to say, for humans, the change of 'numerical values' may change their perception of certain things. Probably the manipulations here are not directly related to the importance. So how would the authors know whether the manipulation takes effect on the importance but not simple number perception? (For example, changing from positive to negative, or changing small numbers to large numbers; changing the rank in overall dimensions). This may require a baseline test, whether about the numerical values (in an irrelevant context) or about other measures of importance (e.g., relative rank, or a Likert scale on each dimension separately).", "limitations": "- The limitation of this paper as discussed, is only certain features in each task are selected to manipulate. It would reveal more about the societal impact if comprehensive features are considered. Moreover, the consideration of interactions between features is also helpful, though this does require a larger dataset.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper formulates the AI explanation manipulation problems, as optimization problems to minimize human-AI decision disagreement, with constraint to keep the original AI outcomes. This can indeed be applied in both positive or negative ways (in respective to adversarial and benign purposes in the paper). The authors discuss both sides of the methods and provide an implication for society about such methods.\n\n- The paper selects multiple AI explanation baselines and collects human data on multiple tasks, enhancing the robustness of the manipulation method and broadening potential social impacts in various scenarios. The paper also collects human perception, providing an extra view from human participants in subjective feelings.", "weaknesses": "- Only particular features are selected in each task. This may weaken the effectiveness of the manipulation. For example, in the census task, the paper selects 'gender' as the dimension of manipulation; while 'degree of education' or 'ages' can possibly be biased and effect human decision-making. Thus, the authors may need to address such representativeness (or probably other dimensions do not exhibit significant effects in manipulations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! Below, we address your questions.\n\n> Only particular features are selected in each task. This may weaken the effectiveness of the manipulation.\n\n- Thank you for your insightful feedback. Firstly, we’d like to clarify that in Section 5 (Evaluation I, when AI explanations are manipulated for adversarial purposes), the chosen “targeted” feature is only used for defining the goal of manipulation (i.e., whether the human decision makers’ decisions are “fair” or not is defined with respect to the chosen feature); it does not mean that we only manipulate the explanation for that feature. In Section 6 (Evaluation II, when AI explanations are manipulated for benign purposes), no feature is “selected” because the goal of the manipulation is to improve the accuracy of the human decision makers’ final decisions. Again, on each task instance, our manipulation can occur on any subset of the features. \n\n- In addition, for Section 5, to validate the effectiveness of adversarial manipulation of AI explanations, conducting large-scale human experiments is essential; however, such experiments are both costly and resource-intensive. Therefore, for each type of decision making task, we focused on manipulating the most sensitive feature, which we hypothesized would have the most significant societal impact on decision making.\n\n- To ensure the generalizability of our results , we conducted evaluations across four different types of decision making tasks. While we only selected one feature per task to define our fairness goal, we believe that similar manipulative effects could be observed if fairness is defined by other features. This belief is based on our successful manipulation of human behavior for both benign and adversarial purposes, where participants were unable to perceive any differences in the presented manipulated explanations. \n\n> My question is humans themselves may have a lot of biases, for example, peak and anchoring effects; or in risky decision-making, humans can have probability distortion. This is to say, for humans, the change of 'numerical values' may change their perception of certain things. \n\n- Human bias in decision making: Our manipulation is based on a learned human behavior model, which inherently captures the biases present in human decision-making, such as anchoring effects, probability distortion, and other cognitive biases (if they exist). Since these biases are integrated into the behavior model, our manipulation leverages them to nudge human decision makers to make the desired decisions.\n\n-  Human Interpretation of Number: During the tutorial phase of our experiment, we carefully followed established HCI research practices to educate participants on how to interpret AI explanations. We specifically instructed them that the numerical values or bar lengths associated with each feature explanation represent the feature's importance. A positive value indicates a positive contribution to the decision, while a negative value indicates a negative contribution. This framing ensured that participants understood the numbers as a direct reflection of feature importance, rather than as arbitrary values. Thus, in our study, the change of numerical values was directly tied to perceived feature importance as intended.\n\n- Baseline Test Consideration: We appreciate your suggestion regarding a baseline test. To better understand your suggestion, could you please clarify whether you are suggesting that we conduct a study to determine if participants perceive features with larger numerical values as more important, irrespective of context? We are happy and prepared to conduct this test if it would help address any concerns about the study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELLy1oaGJB", "reviewer": "Reviewer_1Xcr", "summary": "This paper proposes to train a computational model to predict how humans would respond to model predictions and their explanations to make the final decision. Using this model, the authors then demonstrate that it could be used to manipulate explanations for both good and bad purposes, specifically to steer human predictions toward the label that is likely to be correct, or make human decisions intentionally biased. Furthermore, the humans have very little idea that the explanations have been manipulated in both cases.", "questions": "See weaknesses. In addition, the reliance metrics of Fig. 3 are not defined here, and the readers need to search the referenced papers for definitions.", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper focuses on an important topic: the role of explanations in human decision making. \n\nThe presentation is generally clear and the writing structure is good. \n\nExtensive experiments are conducted to demonstrate the main arguments of the paper.", "weaknesses": "1. I noticed that the user study payment is only \\\\$1.2 base pay with a potential bonus of \\\\$1.0 (Sec. 4.2). The study consists of a tutorial, 5 predictions without AI assistance, 15 predictions with AI prediction and explanations, and an exit survey. Given that this study is deployed to US-based participants, the compensation is extremely meager: even at the federal minimum wage of \\\\$7.25 per hour, the \\\\$1.2 base pay would be equivalent to 10 min of work, which, given the user interface of Fig. A.1 and A.2, is extremely unlikely to be enough for all the tasks. Thus, I have serious concern about the ethics of the study, despite its IRB approval, and thus decide to request additional ethics reviewers for this paper. \n\n2. The use of computational model for human behavior is not novel, and the applications to human decision manipulation seems quite straightforward. \n\n3. Furthermore, it would be helpful have some additional analysis on the learned computation model itself, maybe with the help of various interpretability tools. For example, when is the human prediction most likely influenced by the provided explanation, and in what way? These quantitative and qualitative insights could be helpful to understand human behaviors better. \n\n4. For the \"benign\" use case of improving human model prediction, the authors looked at cases \"when the AI model decision is likely incorrect\" (Line 285). How is this \"likely incorrect\" determined? And if we know when the AI prediction is likely incorrect, why can't we simply fix/patch the AI model directly? The authors demonstrated that the human performance is better after explanation manipulation, compared to the original explanation in Fig. 3, but is the human prediction better than the \"fixed AI model\" performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We noticed that you are particularly concerned with the ethics of this study. We hope the clarifications below satisfactorily address these issues, and we are open to discussing any further concerns you may have. We are fully committed to ensuring that our work meets all ethical requirements.\n\n>  I noticed that the user study payment is only \\\\$1.2 base pay with a potential bonus of \\\\$1.0 (Sec. 4.2). The study …..\n\nTable 1: The average hourly payment received by participants in our study across four tasks. In the row \"Number of Workers,\" the number in parentheses indicates the number of invalid participants who did not pass the attention check questions.\n\n|                               | Recidivism |   Census  |    Bias   | Toxicity |\n|:-----------------------------:|:----------:|:---------:|:---------:|:--------:|\n|       Number of Participants       |  336 (16)  | 310 (25)  | 259 (20)  | 286 (17) |\n|          Average Working Time (minute)         |    6.67    |    6.25  |    6.98   |   6.34   |\n|     Hourly Payment (Base)     |    $10.8   |   $11.7   |   $10.2   |  $11.36  |\n| Hourly Payment (Base + Bonus) |    $11.9   |   $11.8   |   $11.4   |   $14.4  |\n\n- To determine the appropriate payment level for each task, we first conducted a preliminary study to estimate the time workers might spend on the tasks. Our pilot study indicated that a base payment of \\\\$1.2 per task translates to an approximate hourly rate of \\\\$10. To provide greater transparency about the compensation received by participants in our formal study, Table 1 summarizes the average hourly payment and the average time spent on each task. As shown in Table 1, the average base hourly payment across the four tasks exceeds \\\\$10 per hour, and the average hourly payment including bonus is close to \\\\$12 per hour, both of which are well above the federal minimum wage. In addition, to minimize any potential negative effects of the manipulation, we provided a debrief session through the Prolific system after participants completed the experiment. This session clarified that the provided explanations were intentionally manipulated to influence their decisions.\n\n- In addition, to ensure the quality of participant responses, we included two attention check questions in our study, where participants were required to select a pre-specified answer. These attention checks were randomly inserted among 15 formal tasks. Only participants who passed both attention checks were considered valid for our analysis.\nLastly, we acknowledge that conducting an online study presents challenges in ensuring that participants maintain full attention throughout the tasks. We will include a discussion of this limitation in our manuscript.\n\n> The use of computational model for human behavior is not novel, and the applications to human decision manipulation seems quite straightforward.\n\n\nWhile much research has focused on modeling human behavior in AI-assisted decision making, our study is among the first to explore and answer whether we can model the impact of AI predictions with explanations on human decisions. Furthermore,  informed by the learned human behavior models, we further investigate the possibility to  manipulate AI explanations, aiming to advance our understanding of what roles AI explanations play in human decisions. Through our initial exploration of manipulating AI explanations to influence human decision making, we seek to lay a base for demonstrating the potential to strategically manipulate information presented to humans with behavior modeling, thereby impacting their decisions.\n\n> Furthermore, it would be helpful have some additional analysis on the learned computation model itself, maybe with the ….\n\nThanks for your suggestion! We used SHAP to provide explanations on how task features, AI model predictions, and explanations impact human decisions based on our behavior models. Through our SHAP analysis, we first observed that AI model predictions influence human decisions, often leading them to align with the AI's predictions. In addition, we found that, especially when explanations are manipulated for adversarial purposes, model-based manipulation often intentionally retains certain features as protective to mislead humans into making biased decisions. For example, when predicting whether a person's income is below \\\\$50k, if the person individual is a female, the manipulated explanation might present gender as a positive contributor to earning above \\\\$50k, and vice versa for males. If we intentionally change the manipulated contribution of gender to be negative for females, we observed that the predicted probability of this female's income being below $50k by our decision model decreases. We also observed similar trends in recidivism prediction when targeting the 'Black’ race. This phenomenon suggests that when humans make decisions, they may focus intentionally on sensitive features. When AI explanations indicate traces of unfairness in these features, people may naturally make decisions to counteract this perceived bias. And we will include the discussion in the manuscript.\n\n> Continued on the next comment\n.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NlLEC9LGMr", "reviewer": "Reviewer_9s1B", "summary": "This paper addresses the manipulation of explanations in AI-assisted decision-making, presenting a comprehensive study that explores how human behavior models can be used to adjust explanations provided by AI systems. The aim is to understand if these manipulations can nudge decision-makers towards specific outcomes, which might be either beneficial or malicious.", "questions": "I still have a positive view of this work, and have the following questions:\n1. Could the authors please provide more detail on which covariates were found to be significant in the regression analyses? It would be helpful to understand not only which variables significantly impacted the model outcomes but also how they were selected and their relative influence on the dependent variables.\n2. Regarding the use of $\\sum_i e_i$ in Eq. 2 for evaluating consistency, I wonder why this summation was chosen over other potential methods. Could the authors clarify the rationale behind this choice?", "limitations": "1. Score-based explanations may not be adequate. Incorporating Large Language Models could introduce more versatile and comprehensive explanations.\n2. Include more diverse datasets, possibly extending to more complex data types like images or videos, to test the robustness of the behavior models across different AI applications.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's focus on quantitatively modeling human behavior to manipulate AI explanations is highly novel and impactful. This approach not only extends the current understanding of AI-human interactions but also opens new avenues for both enhancing and securing AI-assisted decision-making systems.\n\n2.  The experiments conducted across various decision-making tasks provide a robust validation of the proposed models. The inclusion of both adversarial and benign manipulations allows for a balanced view of the potential impacts of this technology.", "weaknesses": "1. The scope of the experiments is restricted to tasks such as census and recidivism prediction, which may not adequately represent the complexities and stakes of decision-making environments in sectors like healthcare or finance. Expanding the range of tasks to include high-stakes decision-making could improve the generalizability of the results.\n2. The behavior model does not account for the inherent variability and noise in human decision-making, potentially oversimplifying the complexities of real-world human-AI interactions.\n2. The data and code are not immediately available for replication and further study, which could hinder the verification of the results and the advancement of the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review! Below we address your questions.\n\n>  Could the authors please provide more detail on which covariates were found to be significant in the regression analyses? It would be helpful to understand not only which variables significantly impacted the model outcomes but also how they were selected and their relative influence on the dependent variables.\n\n- The selection criteria for the covariates, including participants' demographic information, their knowledge of AI explanations, and their trust in AI models, were based on prior HCI research [1,2,3], which empirically examines how AI explanations impact human decisions in AI-assisted decision-making and how these factors might influence that impact.\n\n- In our regression analysis, we observed an interesting trend across all four types of decision tasks, particularly when AI explanations were manipulated for adversarial purposes. Participants with high trust in AI systems consistently rated the manipulated explanations higher in terms of alignment, satisfaction, usefulness, etc. Similarly, participants with greater knowledge of AI explanations exhibited the same trend as those with high trust, suggesting that individuals more familiar with AI systems might be more vulnerable to these manipulations.\n\nWe will include the full linear regression results as tables in the manuscript and add a discussion on these findings to provide further insight.\n\n> Regarding the use of  in Eq. 2 for evaluating consistency, I wonder why this summation was chosen over other potential methods. Could the authors clarify the rationale behind this choice?\n\nThe choice of the summation form for consistency constraints in Equation 2 is primarily motivated by the alignment with SHAP explanations, which emphasize local accuracy. This property ensures that the sign of the sum of all feature contributions from SHAP can match the sign of the model’s output. Additionally, in our setting, we assume that third parties do not have access to the AI model when manipulating AI explanations. Therefore, alternative metrics that would require probing the model’s outputs with perturbed inputs to evaluate the consistency of explanations are challenging to implement in this study.\n\n> No code available   \n\nWe will release the collected human behavior data, along with the code for training the human behavior model and manipulating AI explanations, upon the acceptance of our paper. \n\n\n[1] Lai, Vivian, Han Liu, and Chenhao Tan. \"\" Why is' Chicago'deceptive?\" Towards Building Model-Driven Tutorials for Humans.\" Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020.\n\n[2] Zhang, Yunfeng, Q. Vera Liao, and Rachel KE Bellamy. \"Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making.\" Proceedings of the 2020 conference on fairness, accountability, and transparency. 2020.\n\n[3] Wang, Xinru, and Ming Yin. \"Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making.\" Proceedings of the 26th International Conference on Intelligent User Interfaces. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach to manipulating human decision-making through AI-generated explanations, exploring both benign and adversarial uses. The method's originality and robust experimental validation across multiple tasks make it a valuable contribution to the field of explainable AI (XAI). 

However, several reviewers raised ethical concerns, particularly regarding participant compensation in the user study and potential biases in datasets. Addressing these issues in the final version would be necessary.

In addition, the paper would benefit from improvements in clarity and comprehensiveness. Reviewers noted that the presentation of results could be enhanced, particularly with more detailed statistical analyses and clearer visualizations. Expanding the scope of experiments to include more complex decision-making scenarios and providing access to the data and code for replication would further validate the findings.
Decision: Accept (poster)