Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary

Authors: zhuoyan  li
ming  yin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-28 16:14:12

Abstract: Recent advances in AI models have increased the integration of AI-based decision aids into the human decision making process. To fully unlock the potential of AI-assisted decision making, researchers have computationally modeled how humans incorporate AI recommendations into their final decisions, and utilized these models to improve human-AI team performance. Meanwhile, due to the ``black-box'' nature of AI models, providing AI explanations to human decision makers to help them rely on AI recommendations more appropriately has become a common practice.  In this paper, we explore whether we can quantitatively model how humans integrate both AI recommendations and explanations into their decision process, and whether this quantitative understanding of human behavior from the learned model can be utilized to manipulate AI explanations, thereby nudging individuals towards making targeted decisions. Our extensive human experiments across various tasks demonstrate that  human behavior can be easily influenced by these manipulated explanations towards targeted outcomes, regardless of the intent being adversarial or benign. Furthermore, individuals often fail to detect any anomalies in these  explanations, despite their decisions being affected by them.

Main paper: 
 introduction: 
 Recent advances in AI models have significantly increased the integration of AI-based decision aids into human decision making process. The widespread adoption of such AI-based decision aids has opened up a new paradigm of human-AI collaboration-the AI model provides recommendations for a given decision making task, while human decision makers are responsible for making the final decisions. To fully unlock the potential of AI-based decision aids in enhancing human decision making, a few studies [1] [2] [3] [4] [5] have developed computational models to capture how humans factor AI recommendations into their decision-making process, and explored how these behavioral models can be utilized to improve human-AI team performance. For example, Vodrahalli, Gerstenberg, and Zou [6] developed a human behavior model to characterize the impact of AI predictions and confidence levels on human final decisions. This model was then utilized to adjust the model confidence displayed to people, with the objective of calibrating human trust in AI assistance.Meanwhile, the black-box nature of prevalent AI models has driven a greater integration of model explanations, generated through various explainable AI (XAI) methods [7] [8] [9] [10] [11] , into AI-assisted decision making. These explanations seek to provide some insights into the the underlying decision rationales of AI models, assisting humans in evaluating the reliability of AI decisions and identifying the optimal strategies to rely on AI recommendations. However, many empirical studies [12] [13] [14] [15] [16] [17] [18] , which evaluate the effectiveness of current XAI methods for improving people's understanding of the AI model [16, 19, 20] and supporting their calibrated trust in the model [13, 14, 20] , have demonstrated that humans often struggle to use the explanations generated by these methods in the optimal way. Thus, despite designers' expectations that XAI methods will positively shape human interaction with AI models, they often fall short of their intended goals, such as fostering appropriate levels of trust and reliance in AI-assisted decision-making. This could be because existing XAI methods do not account for human reactions, making them unadaptive to human cognitive processes. If this is the case, one may naturally wonder if it is feasible to quantitatively model how humans incorporate both AI recommendations and explanations into their decision making process. If so, can the quantitative understanding of human behavior obtained from the learned behavior models be utilized to directly manipulate AI explanations, thereby nudging human decision-makers towards making targeted decisions?To answer these questions, in this paper, we begin by training behavior models that characterize how AI recommendations and explanations are factored into human decisions based on the collected behavior datasets for various decision making tasks. Utilizing these learned models, we then adjust the AI explanations for different purposes through gradient-based optimizations. Our extensive human experiments across various tasks demonstrated that such human behavior modeling can bring forth both benefits and risks -when used for benign purposes, the human behavior models can inform the adjustment of AI explanations such that the manipulated explanations could significantly enhance the decision-making performance of human-AI teams in most tasks; however, the same behavior models can also be exploited by adversarial parties for adversarial purposes, such as increasing human decision-makers' biases against a certain protected group and significantly decreasing the fairness level of humans' final decisions across different groups. Finally, examining human perceptions of these manipulated explanations reveals the "scary" truth -while human decisions are easily swayed by these altered explanations, individuals generally fail to detect any anomalies in the explanations, underscoring a significant vulnerability in human-AI interaction.2 Related Work 
computational modeling of ai-assisted decision making: 
 There has been a surge of interests among researchers recently in computationally modeling human behavior in AI-assisted decision making [1] [2] [3] [4] [5] [21] [22] [23] [24] . The goals of these studies for modeling human behavior are diverse, encompassing improving human-AI team performance through intelligent interventions or model recommendation adjustments [6, [25] [26] [27] [28] [29] , deciding when to present AI powered code suggestion in programming [30] , evaluating the utility of AI explanations to improve user understanding of AI model behavior [31] [32] [33] , and deploying adversarial attacks on AI models to reduce human trust [34] . In this paper, we take a more holistic view and explore whether we can model how humans integrate both AI recommendations and explanations into their decision making process, and what the implications of such behavior modeling are.With the increasing use of AI technologies as decision aids for humans, a variety of explainable AI techniques have been developed to increase the interpretability of AI models [7-11, 33, 35-37] . To understand the effectiveness of these explanation methods, a growing body of empirical humancentered evaluations have been conducted to examine how AI explanations would affect the ways that humans perceive and interact with AI models [12-18, 31, 32, 38, 39] . These evaluations look into various aspects of impacts of AI explanations, such as the influence on people's trust and reliance on the AI model [13, 14, 20] , understanding of AI model [16, 19, 20] , and the collaboration performance of the human-AI team [12, 40, 41] . Recently, some research has explored the modification of AI explanations to influence human behavior. For example, Lakkaraju and Bastani [42] demonstrated that handcrafting modifications in AI explanations-such as hiding sensitive features-can mislead human trust in AI models. Another study [43] found that aligning AI explanations with humans' own decision rationales can increase agreement between human decisions and the AI model's predictions. Different from previous work which required intensive handcrafting of AI explanations, in this paper, we explore whether it is possible to directly exploit the computational human behavior models to manipulate AI explanations, even without the access to AI models, with the goal of nudging human decisions towards targeted directions. 
problem formulation: 
 In this study, we explore the scenario of human-AI collaboration within the context of AI-assisted decision making, and we now formally describe it. Consider a decision making task represented by a n-dimension feature vector x ∈ R n , and y is the correct decision to make in this task. Specifically, in this study, we focus on decision making tasks with binary choices of decisions, i.e., y ∈ {-1, 1}. The AI model's recommendation on the decision task is represented as y m = M (x) , y m ∈ {-1, 1}. Following the explainable AI methods like LIME [7] or SHAP [8] , the AI model could also provide some "explanations" of its decision, e = E(M (x)), e ∈ R n , by showing the contributions of each feature to the decision. With all these information, the human decision maker (DM) needs to make the final decision y h ∈ {-1, 1} by either accepting or rejecting AI model's decision recommendation y m , which can be characterized by y h = H(x, y m , e). The goal of our study is to explore whether we can quantitatively model such decision making process-specifically, H(x, y m , e)-and whether this quantitative understanding of human behavior can be utilized to adjust AI explanations (i.e., change e to e ′ ) without accessing to the original AI model M (•), thereby nudging human DMs to make the targeted decision ŷh ∈ {-1, 1}, denoted as ŷh = H(x, y m , e ′ ).We first build computational models to characterize how humans integrate both AI recommendations and explanations into their decision process. Following previous works on modeling human behavior in different scenarios of AI-assisted decision making [23, 31] , we adopted a two-layer neural network as the structure for modeling the human decision in this study:EQUATIONThe inputs to the behavior model include the task features x, the AI model's prediction y m , the AI explanation e, and the interaction term between the task features and the AI explanation x ⊙ e that reflects how humans may redirect their attention to the corresponding features highlighted by the AI explanation. Given the human behavior dataset D = {x i , y m i , e i , y h i } N i=1 , we can employ the maximum log-likelihood estimation to learn the behavior model H w h .We next proceed to explore how the quantitative understanding of human behavior from H w h can be utilized to manipulate AI explanations. In particular, given the targeted decision ŷh for the task instance x, we want to identify a new AI explanation e ′ that maximizes the likelihood that human DMs make the targeted decision ŷh according to the learned behavior model H w h . In addition, to prevent the case where the manipulated explanations e ′ has a very low level of fidelity [10] , such as suggesting a recommendation that is inconsistent with the AI model's prediction y m , we also impose a constraint that the new explanation e ′ should still support the original AI recommendation y m . Since we assume no access to the original AI model, we define L consistency (e, y m ) as a measurement of agreement consistency between the manipulated AI explanations and the AI recommendation:EQUATION)Together, we use the following optimization problem to manipulate AI explanations:EQUATIONwhere L behavior is defined as the cross entropy function. Since exactly solving the above optimization problem is intractable, we used the gradient-based optimization to approximate it:EQUATIONwhere η is the step size, λ is the trade-off parameter, and e ′ θ represents the parameterized explanations in the optimization process. We can iteratively optimize manipulated explanations e ′ until L behavior is smaller than a threshold τ or reach the maximum number of rounds T . 
human behavior model learning: 
 To develop the human behavior model for manipulating AI explanations, we first conduct a human subject experiment to collect human behavior data.We consider four decision making tasks in this study:• Census Prediction (Tabular Data) [44] : This task was to determine a person's annual income level. In each task, the human DM was presented with a profile with 7 features, including the person's gender, age, education level, martial status, occupation, work type, and working hour per week. The subject was asked to decide whether this person's annual income is higher or lower than $50k for each task. We trained a random forest model to make the income prediction, and the accuracy of the AI model was 76%.• Recidivism Prediction (Tabular Data) [45] : This task was to determine a person's recidivism risk. In each task, the human DM was presented with a profile with 8 features, including their basic demographics (e.g., gender, age, race), criminal history (e.g., the count of prior non-juvenile crimes, juvenile misdemeanor crimes, juvenile felony crimes committed), and information related to their current charge (e.g., charge issue, charge degree). The subject was asked to decide whether this person would reoffend within two years. We trained a random forest model to make the prediction, and the accuracy of the AI model was 62%.• Bias Detection (Text Data) [46] : In this task, the human DM was presented with a text snippet and needed to decide whether it contained any bias. We fine-tuned a BERT [47] model to identify bias in the snippet, and the accuracy of the AI model is 79%.• Toxicity Detection (Text Data) [48] : In this task, the human DM was presented with a text snippet and needed to decide whether it contained any toxic content. We fine-tuned a BERT model to identify the toxic content, and the accuracy of the AI model is 86%.To understand how people respond to various AI explanations, we employed LIME and SHAP to explain the predictions made by the AI model. Additionally, we augment the LIME or SHAP explanations by either randomly masking out contributions from some features or amplifying contributions of some features (referred to as the "Augmented" explanations) to see how humans react to them. These explanations are provided with AI recommendations together to humans in decision making.We posted our data collection study on the Prolific1 to recruit human participants. Upon arrival, we randomly assigned each participant to one of the four decision making tasks and they needed to fill in an initial survey to report their demographic information and their knowledge of AI models and explanations. Participants started the study by completing a tutorial that described the decision making task that they needed to work on. To familiarize participants with the task, we initially asked them to complete five tasks independently without AI assistance. During these training tasks, we immediately provided the correct answer at the end of the task. After the completion of training tasks, participants moved on to the formal tasks. In the formal tasks, participants would receive one type of AI explanations among SHAP, LIME, or Augmented. Specifically, each participant was asked to complete a total of 15 tasks. In each task, participants were provided with the AI prediction and the explanations along with the task instance. They were then required to make their final decisions. Finally, participants were required to complete an exit survey to report their perceptions of the AI explanations they received during the study. They were asked to rate the alignment of AI explanations with their own rationale, as well as the usefulness, transparency, comprehensibility, satisfaction with the provided explanations, and their trust in the AI models, on a 5-point Likert scale. For the detailed survey questions, please refer to Appendix A.3. We offered a base payment of $1.2 and a potential bonus of $1 if the participant's accuracy is above 85%. The study was open to US-based workers only, and each worker can complete the study once. After collecting data on human behavior, we developed human behavior models for each type of task.For the human behavior models for two textual tasks-Toxicity Detection and Bias Detection-we employed the pretrained BERT encoder to extract features from the original sentences, which were then used as the task feature x in the human behavior model H w h . We optimized these behavior models using Adam [49] with an initial learning rate of 1e -4 and a batchsize of each training iteration of 128. The number of training epochs is set as 10. Table 1 shows the number of participants recruited, as well as the average accuracy of the human behavior model evaluated through 5-fold cross validation for each task. We observed that the average accuracy of all human behavior models exceeds 0.65, which is considered to be reasonable. Consequently, we utilized these learned human behavior models to manipulate AI explanations in the following evaluations. 
evaluation i: manipulating ai explanations for adversarial purposes: 
 In our first evaluation, we adopted the role of an adversarial party to explore whether they could utilize the learned human behavior model to manipulate AI explanations. The manipulation goal was to nudge human DMs to be biased against certain protected groups in the decision making process. We are particularly interested in comparing the fairness level of human decision outcomes between human DMs who receive original explanations, such as SHAP or LIME, and those who receive manipulated explanations. Notably, all human DMs are provided with the same AI predictions for the same decision making task. Additionally, we also explore differences in human perceptions of original AI explanations versus manipulated AI explanations.Evaluation Metrics and Manipulating AI Explanations. Following previous work [50, 51] , we used the false positive rate difference (i.e., FPRD) and the false negative rate difference (i.e., FNRD) to measure the fairness level of human decision outcomes-the closer these values are to zero, the more fair the decisions are. To manipulate AI explanations and nudge human DMs toward biasing against certain protected groups, we define the targeted human decision ŷh for each task as follows:• Census Prediction: In this task, we considered a person's sex as the protected attribute. The targeted human decision is defined as ŷh = 1 (indicating a person's annual income exceeds $50K) when x sex = male, and ŷh = -1 (indicating a person's annual income does not exceed $50K) when x sex = female. The fairness metrics can be computed as FPRD = FPR female -FPR male , and FNRD = FNR female -FNR male .• Recidivism Prediction: In this task, we considered the defendant's race as the protected attribute. The targeted human decision is defined as ŷh = 1 (indicating the defendant will reoffend) when x race = black, and ŷh = -1 (indicating the defendant will not reoffend) when x race = white. The two fairness metrics can be computed as FPRD = FPR white -FPR black , and FNRD = FNR white -FNR black .• Bias Detection: In this task, we divided text snippets into groups based on their political leaning. The targeted human decision is defined as ŷh = 1 (indicating the text is biased) when x leaning = democratic, and ŷh = -1 (indicating the text is not biased) when x leaning = republican. The fairness metrics can be computed as FPRD = FPR rep -FPR dem , and FNRD = FNR rep -FNR dem .• Toxicity Detection: In this task, we divided text snippets into groups based on the victim of the text. The targeted human decision is defined as ŷh = 1 (indicating the text is toxic) when x victim = white, and ŷh = -1 (indicating the text is non-toxic) when x victim = black. The two fairness metrics can be computed as FPRD = FPR black -FPR white , and FNRD = FNR black -FNR white . For the Bias Detection and Toxicity Detection tasks, the original datasets [46, 48] provide annotations for the political leanings of the sentences and the targeted victims of the text snippets, respectively. After determining the targeted decision ŷh for each task instance, we then followed the gradient-based optimization procedure (i.e., Equation 4) to identify the manipulated explanation. We set the step size η as 0.01, the trade-off λ as 0.01, the optimization threshold τ as 0.1, and the maximum optimization number T as 100. For the initial AI explanation e θ 0 at the start of the optimization process, we directly initialize e ′ θ 0 as e ′ θ 0 ∼ U (-1, 1). We then repeated this optimization process for 5 times and took the average to use in the following human evaluations. For the examples of manipulated explanations, please refer to Appendix B.3. Data Collection. We followed the experimental procedure described in Section 4.2 to collect data on human responses to and perceptions of different AI explanations. We randomly assigned either SHAP or LIME explanations, or the manipulated explanations, to participants. Participants were required to complete 15 tasks with AI model predictions and the assigned explanations (SHAP or LIME or manipulated). We offered a base payment of $1.2 and a potential bonus of $1 if the participant's accuracy is above 85%. Table 2 reports detailed statistics of the participants in each task. Below, we analyzed how the manipulated explanations affect fairness level of human decisions and how do humans perceive those explanations.The fairness levels of participants' decision outcomes under the manipulated explanation, SHAP explanation, and LIME explanation are presented in Figure 1 . Visually, it appears that when human DMs are provided with manipulated explanations, both FPRD and FNRD scores of their decision outcomes tend to deviate more from zero compared to when DMs receive SHAP or LIME explanations.To examine whether these differences are statistically significant, we conducted regression analyses. Specifically, the focal independent variable was the type of explanation received by participants, while the dependent variables were the participants' FPRD and FNRD scores. To minimize the impact of potential confounding variables, we included a set of covariates in our regression models, such as participants' demographic background (e.g., age, race, gender, education level), their knowledge of AI explanations, their trust in AI models, and the FPRD or FNRD scores of the AI model decisions they received in the study. These covariates were selected based on prior HCI research [14, 20, 41] which empirically reveal how characteristics of human DMs may moderate the impacts of AI explanations on human decisions in AI-assisted decision making.Our regression results indicate that the adversarial party can significantly increase the level of unfairness in human decision outcomes with manipulated explanations through human behavior modeling. Specifically, when examining FPRD, we found that participants who received manipulated AI explanations made more unfair decisions compared to those who received SHAP or LIME explanations (p < 0.05) in the Census and Recidivism tasks. The difference was marginally significant (p < 0.1) in the Toxicity task. When examining FNRD, results show that participants who received manipulated explanations made decisions that were significantly more unfair than those who received SHAP or LIME explanations (p < 0.01) in the Bias task.5.2 How do humans perceive the adversarially manipulated AI explanations? In Section 5.1, we found that the adversarial party can manipulate AI explanations to nudge human DMs toward making more unfair decisions compared to those who received the original AI explanations, aligning with the adversarial party's intentions. To determine whether DMs could detect any abnormalities in the manipulated explanations, we examined how their perceptions of AI explanations varied among manipulated, SHAP, and LIME explanations.Figures 2a and 2b compare the average perceived transparency and usefulness of three types of AI explanations. Visually, there are no significant differences in how the explanations are perceived by people who received different explanations. We also applied regression models to predict human perceptions of these explanations by accounting for their demographic background (e.g., age, race, gender, education level), their knowledge of AI explanations and their trust in AI models. The regression results indicate that there are no significant differences in the perceived transparency and usefulness of manipulated explanations compared to SHAP or LIME explanations. Similar patterns were observed for perceptions of alignment, comprehensibility, satisfaction, and trust between the manipulated and unmanipulated explanations. While adversarially manipulated explanations significantly influence human decision making behavior, individuals generally do not detect abnormalities in the manipulated AI explanations across most tasks. For further details, please refer to Appendix B.2. 
evaluation ii: manipulating ai explanations for benign purposes: 
 In the previous section, we found that the adversarial party could use the behavior model to manipulate AI explanations, thereby misleading humans into making unfavorable decisions against specific groups. Naturally, one might wonder could a third party also use behavior models to manipulate AI explanations for benign purposes, such as promoting more appropriate human reliance on AI models? For instance, can manipulated AI explanations lead humans to reject AI recommendations when the AI model decision is likely incorrect, and encourage acceptance when the decision is likely correct?We aim to explore the answers to this question in this section.Evaluation Metrics and Manipulating AI Explanations Following previous work [14, 41, 43] , we used the accuracy, underreliance, and overreliance to measure human DMs' appropriate reliance level on AI models. Underreliance refers to the fraction of tasks where the participant's decision was different from the AI model's decision when the AI model's decision was correct. Overreliance refers to the fraction of tasks where the participant's decision was the same as the AI model's decision when the AI model's decision was incorrect. To manipulate AI explanations for the promotion of more appropriate reliance on AI models, it is necessary to determine the reliability of AI model prediction on each task instance. Recent work [52, 53] has proposed methods to leverage the complementary strengths of humans and AI by combining human independent decisions and AI model, which is often shown to result in more accurate decisions than those made by either humans or AI models alone. Specifically, given the human independent decision y h independent , the AI model recommendation y m , and the task instance x, these methods learn models to combine y h independent and y m to produce a combined result:EQUATIONTo see whether y combine can yield better decisions compared to AI alone or human alone, we evaluated various combination models including the human-AI combination method [53] and several truth inference methods [54] [55] [56] [57] used in crowdsourcing. Our results showed that the human-AI combination method [53] generally outperformed AI solo and independent human decision, as well as other combination methods. Thus, y combine produced by the human-AI combination method [53] is defined as the targeted decision ŷh for manipulating AI explanations. For detailed information on the evaluations of each combination method, please refer to the Appendix C.1. We again followed Equation 4 to manipulate the AI explanations. We set the step size η as 0.01, the trade-off λ as 0.01, the optimization threshold τ as 0.1, and the maximum optimization number T as 100, and the initial AI explanation e ′ θ 0 at the start of the optimization process is initialized as e ′ θ 0 ∼ U (-1, 1). We repeated this optimization process for 5 times and took the average to use in the following experiments. For the examples of manipulated explanations, please refer to Appendix C.4. Data Collection. We recruited participants from Prolific once again to collect behavioral data under the benignly manipulated explanations, following the experimental procedure described in Section 4.2. We offered a base payment of $1.2 and a potential bonus of $1 if the participant's accuracy is above 85%. Table 2 shows the detailed statistics of the participants we recruited for each task. Subsequently, we analyzed whether the benignly manipulated explanations can promote appropriate reliance of human DMs on AI models, as well as their perceptions of these AI explanations.Figures 3a, 3b, and 3c compare the average accuracy, overreliance, and underreliance of human decision outcomes under manipulated, SHAP, and LIME explanations, respectively. It is clear that providing human DMs with manipulated AI explanations leads to an increase in the accuracy of their decision outcomes for most of tasks. We subsequently conducted regression analyses to determine whether these differences are statistically significant. The regression models incorporated a set of covariates, including participants' demographic backgrounds (e.g., age, race, gender, education level), their knowledge of AI explanations, their trust in AI models, and the accuracy of the AI models. The regression results indicate that in the Census, Recidivism, and Bias tasks, substituting SHAP or LIME explanations with manipulated explanations significantly improves the accuracy of human-AI team.In contrast, for the Toxicity task, we observed no statistical difference, which could potentially be attributed to the high competence of humans in solving this task (e.g., when presented with SHAP or LIME explanations, the average decision accuracy of participants already exceeds 0.8, leaving limited room for further improvement).In Section 5.2, we observed that it is challenging for humans to detect abnormalities in the adversarially manipulated explanations, even though they are unconsciously influenced by the manipulated explanations to make more unfair decisions. In this section, we revisit this question to investigate into whether humans' perceptions of the manipulated explanations change, when they are manipulated for benign purposes. Figures 4a and 4b compare the average human perceived transparency and usefulness of the benignly manipulated explanations, SHAP explanations, and LIME explanations. Regression analyses reveal no statistically significant differences among the perceived transparency and usefulness of these three types of explanations. Similar trends were observed for other perceptual aspects of explanations, including perceived alignment, comprehensibility, satisfaction, and trust. For further results, please refer to Appendix C.3. 
conclusion and limitations: 
 In this paper, we explore whether we can quantitatively model how humans incorporate both AI recommendations and explanations into their decision making process, and whether we can utilize the quantitative understanding of human behavior obtained from these learned models to manipulate AI explanations for both adversarial and benign purposes. Our extensive experiments across various tasks demonstrate that human behavior can be easily influenced by these manipulated explanations toward targeted outcomes, regardless of the intent being benign or adversarial. Despite the significant influence of these falsified explanations on human decisions, individuals typically fail to detect or recognize any abnormalities. Our study has several limitations. For example, it focuses on modeling and manipulating score-based explanations. Further research is needed to explore how to model how humans incorporate other types of explanations, such as example-based and rule-based explanations, and how these can be manipulated to influence human behavior as observed with score-based explanations in our study. Additionally, our study was limited to decision making tasks involving tabular and textual data, which are naturally suited to score-based explanations. Further explorations are needed to extend these findings to decision tasks with other data types (e.g., images). 
 
 Appendix: This study was approved by the Institutional Review Board of the authors' institution. Through our findings, we aim to draw the community's attention to the ease with which third parties can manipulate AI explanations with the learned behavior models to influence human decision making. Users often lack the ability to accurately and appropriately interpret the AI explanations presented to them, yet their decision behavior is easily swayed by the manipulated AI explanations. Our findings highlight the critical importance of securing human-AI interaction data to prevent the misuse of human behavior models derived from it. Additionally, there is an urgent need to ensure that AI explanations provided to humans are more secure and inherently benign. Moreover, providing pre-education is essential to assist humans in establishing a proper understanding of AI explanations, which may potentially mitigate the risks of manipulation.In addition, our experiments are based on datasets that are publicly available; "correct" decisions for the tasks in these datasets are generally considered as recording the real-world ground truth. While these datasets are not intentionally biased toward any specific groups, we acknowledge that there might be implicit biases introduced to these datasets during the curation process, which are beyond our control. Importantly, we note that we made no alterations to the datasets that would introduce additional bias in our experiment. A The Design of Human Study (Additional Details)ATo determine the appropriate payment level for each type of task, we first conducted a preliminary study to estimate the time workers might spend on the tasks. Our pilot study indicated that a base payment of $1.2 per task translates to an approximate hourly rate of $10. To provide greater transparency about the compensation received by participants in our formal study, Table A .1 summarizes the average hourly payment and the average time spent on each task. In the study, after the main tasks, the participants need to fill in an exit survey to report their perceptions of presented AI explanations. The survey questions are detailed as:• Alignment: On a scale of 1-5, how well do you think the explanations align with your understanding of the problem?• Usefulness: On a scale of 1-5, how useful are the explanations in helping you make decisions?• Transparency: On a scale of 1-5, how well do you think the explanations reveal the AI model's decision making process?• Comprehensibility: On a scale of 1-5, how easy is it for you to understand the explanations?• Satisfaction: On a scale of 1-5, how satisfied are you with the explanations provided by the AI model?• Trust: On a scale of 1-5, would you trust the AI model's prediction or decision based on the explanations? with the provided explanations ,and the trust in the AI models under the under the adversarially manipulated explanation, SHAP explanation, or LIME explanation. In general, our findings indicate that there are no significant differences in people's perceptions of the three explanations across four tasks, with the exceptions for the alignment and trust in the Toxicity task. Specifically, for the Toxicity task, participants perceived LIME explanations as aligning more closely with their own rationales than the adversarially manipulated explanations, with a marginally significant difference (p < 0.1). Furthermore, participants reported significantly greater trust in the AI models accompanied by LIME explanations compared to those with adversarially manipulated explanations (p < 0.01). In the main paper, we aim to benignly manipulate AI explanations to encourage human DMs to rely more appropriately on AI models. Following previous research [52, 53] , we combined independent human decisions with AI model predictions to determine the targeted decision for each task instance. We evaluated the human-AI combination method [53] and several truth inference methods used in crowdsourcing for truth discovery. We detailed the process of evaluation below. Simulating Human Independent Decision. To understand how humans independently make decisions on each task instance, we first conducted a study again on the Prolific to collect independent human decision behavior data across four tasks. We recruited 40 participants for each task. Each recruited participant needed to complete 15 tasks. With the collection of human behavior data, we then fitted two-layer neural networks to simulate human independent decision behavior. For Toxicity Detection and Bias Detection textual tasks, we used the pretrained BERT encoder to initially extract features from the original sentences as the input to the independent behavior models. We optimized these independent behavior models using Adam [49] with an initial learning rate of 1e -4 and C .1, which we found to be satisfactory. We then utilized these fitted models to simulate independent human decisions y h independent in the human-AI combination process to determine the potentially better decisions.Comparing Combination Performance. We consider the human + AI combination method [53] and a few truth inference methods in crowdsourcing as baselines in the evaluation, including GLAD [55] , CATD [56] , LFC [57] , EM [54] , and MV [54] . These methods combine the human independent decisions y h independent predicted by the fitted independent human behavior models and AI model recommendations y m to produce combined decisions y combine . The accuracy of each method on holdout task pools for each task to be used in the subsequent evaluation is reported in Table C .2. In general, we found that human + AI combination method outperforms other baselines. By integrating human decisions with AI predictions, this method shows superior performance to either AI solo or human solo across all four tasks. Consequently, we used the combined decisions y combine from the human + AI combination method as the targeted decision ŷh in subsequent experiments to manipulate explanations.Table B .1 compares the agreement between the sum of feature importance in explanations and AI predictions, the Pearson correlation coefficients, measured in terms of Pearson correlation coefficient, for benignly manipulated, LIME, and SHAP explanations, respectively. We observed that the visual consistency of the manipulated explanations is very close to that of SHAP and higher than that of LIME. manipulated explanation, SHAP explanation, or LIME explanation. We found that there are no significant differences in people's perceptions of the three explanations across four tasks on these aspects. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] . Justification: The study was approved by the Institutional Review Board of the authors' institution. To minimize the risk of leaving our study participants with inappropriate impressions of manipulated AI explanations, upon the end of the study, we provided a debrief session directly through the Prolific chat system. Each participant who received manipulated AI explanations in the study were individually contacted through the Prolific platform. In the chat-based session, we clarified that the AI explanations were manipulated and did not accurately represent the underlying rationales of the AI models. We emphasized that these explanations were intentionally biased for the purpose of the study, based on the specific task participants were involved in.https://www.prolific.com/ 


Decision: The paper presents a novel approach to manipulating human decision-making through AI-generated explanations, exploring both benign and adversarial uses. The method's originality and robust experimental validation across multiple tasks make it a valuable contribution to the field of explainable AI (XAI). 

However, several reviewers raised ethical concerns, particularly regarding participant compensation in the user study and potential biases in datasets. Addressing these issues in the final version would be necessary.

In addition, the paper would benefit from improvements in clarity and comprehensiveness. Reviewers noted that the presentation of results could be enhanced, particularly with more detailed statistical analyses and clearer visualizations. Expanding the scope of experiments to include more complex decision-making scenarios and providing access to the data and code for replication would further validate the findings.

Reviews: review: paper_summary: paper_summary: The authors show that by modeling a human decision maker they can manipulate the provided information in ways that reliable influence their decisions towards even non-benign outcomes.
strength_weakness: strength_weakness: # originality

Manipulating workers on digital platforms is a well studied area.

# quality

Reading the plots (fig 3) some of these results look to be supporting the null-hypothesis (all interventions have the same effect). Having a table giving the numbers and more details on the statistical tests would make the quality of the results much easier to judge. The results also are weak, in part this is due to a small sample size.

# clarity

If found the plots difficult to read either both being too small and the lack of numbers makes analyzing them tricky. Could the authors include the results as a table in the appendix with the error ranges clearly laid out so we don't have to eyeball tiny error bars.

I also had to refer to the appendix to understand the experimental procedure, making it clearer what exactly is happening in the experiments would greatly aid the readability of this paper.

 # significance

Looking at figure B.2 it looks like the manipulation is mostly making the difference much more extreme and noticeable, this suggests these results could simply be due to the differences in the visual presentation between interventions.
questions: questions: Are the manipulations working possibly just due to the bar plots being being bigger, and numbers being clearer? So no behavioral model needed.

How should I read figure 1? It looks like it's saying the means 95% confidence intervals overlap, but that they are also below 5% to overlap (bias 1a)?

More generally I think this result is weak, if people could tell the models apart this would be more interesting. As it stands I'm not surprised the authors can manipulate the responses, but I'm not convinced their method is close to optimal.
limitations: limitations: This is an area where ethics should be given additional scrutiny, but I do not see anything alarming in the paper, and the authors take the correct tone in the paper.
paper_summary: paper_summary: This paper proposes a novel method that manipulates human decision-making by manipulating AI explanations in human-AI interaction scenarios. By utilizing human behavior models and minimizing the cross-entropy function between human and AI agreement with constraint to generating the same AI recommendation outcomes, the authors investigate this method for adversarial and benign purposes. The results show that the manipulation in AI explanations significantly negatively affects human decisions in the four tasks with adversarial purpose and enhances human decision accuracy, over-reliances, and diminishes under-reliances. The paper also discusses how the human perception to the AI explanations varies with the manipulation. Overall, it provides a novel and practical way to intervene in human decision-making in human-AI interactions.
strength_weakness: strength_weakness: - Only particular features are selected in each task. This may weaken the effectiveness of the manipulation. For example, in the census task, the paper selects 'gender' as the dimension of manipulation; while 'degree of education' or 'ages' can possibly be biased and effect human decision-making. Thus, the authors may need to address such representativeness (or probably other dimensions do not exhibit significant effects in manipulations).
questions: questions: - My question is humans themselves may have a lot of biases, for example, peak and anchoring effects; or in risky decision-making, humans can have probability distortion. This is to say, for humans, the change of 'numerical values' may change their perception of certain things. Probably the manipulations here are not directly related to the importance. So how would the authors know whether the manipulation takes effect on the importance but not simple number perception? (For example, changing from positive to negative, or changing small numbers to large numbers; changing the rank in overall dimensions). This may require a baseline test, whether about the numerical values (in an irrelevant context) or about other measures of importance (e.g., relative rank, or a Likert scale on each dimension separately).
limitations: limitations: - The limitation of this paper as discussed, is only certain features in each task are selected to manipulate. It would reveal more about the societal impact if comprehensive features are considered. Moreover, the consideration of interactions between features is also helpful, though this does require a larger dataset.
paper_summary: paper_summary: This paper proposes to train a computational model to predict how humans would respond to model predictions and their explanations to make the final decision. Using this model, the authors then demonstrate that it could be used to manipulate explanations for both good and bad purposes, specifically to steer human predictions toward the label that is likely to be correct, or make human decisions intentionally biased. Furthermore, the humans have very little idea that the explanations have been manipulated in both cases.
strength_weakness: strength_weakness: 1. I noticed that the user study payment is only \\$1.2 base pay with a potential bonus of \\$1.0 (Sec. 4.2). The study consists of a tutorial, 5 predictions without AI assistance, 15 predictions with AI prediction and explanations, and an exit survey. Given that this study is deployed to US-based participants, the compensation is extremely meager: even at the federal minimum wage of \\$7.25 per hour, the \\$1.2 base pay would be equivalent to 10 min of work, which, given the user interface of Fig. A.1 and A.2, is extremely unlikely to be enough for all the tasks. Thus, I have serious concern about the ethics of the study, despite its IRB approval, and thus decide to request additional ethics reviewers for this paper. 

2. The use of computational model for human behavior is not novel, and the applications to human decision manipulation seems quite straightforward. 

3. Furthermore, it would be helpful have some additional analysis on the learned computation model itself, maybe with the help of various interpretability tools. For example, when is the human prediction most likely influenced by the provided explanation, and in what way? These quantitative and qualitative insights could be helpful to understand human behaviors better. 

4. For the "benign" use case of improving human model prediction, the authors looked at cases "when the AI model decision is likely incorrect" (Line 285). How is this "likely incorrect" determined? And if we know when the AI prediction is likely incorrect, why can't we simply fix/patch the AI model directly? The authors demonstrated that the human performance is better after explanation manipulation, compared to the original explanation in Fig. 3, but is the human prediction better than the "fixed AI model" performance?
questions: questions: See weaknesses. In addition, the reliance metrics of Fig. 3 are not defined here, and the readers need to search the referenced papers for definitions.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper addresses the manipulation of explanations in AI-assisted decision-making, presenting a comprehensive study that explores how human behavior models can be used to adjust explanations provided by AI systems. The aim is to understand if these manipulations can nudge decision-makers towards specific outcomes, which might be either beneficial or malicious.
strength_weakness: strength_weakness: 1. The scope of the experiments is restricted to tasks such as census and recidivism prediction, which may not adequately represent the complexities and stakes of decision-making environments in sectors like healthcare or finance. Expanding the range of tasks to include high-stakes decision-making could improve the generalizability of the results.
2. The behavior model does not account for the inherent variability and noise in human decision-making, potentially oversimplifying the complexities of real-world human-AI interactions.
2. The data and code are not immediately available for replication and further study, which could hinder the verification of the results and the advancement of the research.
questions: questions: I still have a positive view of this work, and have the following questions:
1. Could the authors please provide more detail on which covariates were found to be significant in the regression analyses? It would be helpful to understand not only which variables significantly impacted the model outcomes but also how they were selected and their relative influence on the dependent variables.
2. Regarding the use of $\sum_i e_i$ in Eq. 2 for evaluating consistency, I wonder why this summation was chosen over other potential methods. Could the authors clarify the rationale behind this choice?
limitations: limitations: 1. Score-based explanations may not be adequate. Incorporating Large Language Models could introduce more versatile and comprehensive explanations.
2. Include more diverse datasets, possibly extending to more complex data types like images or videos, to test the robustness of the behavior models across different AI applications.

Comments: title: Thank you for the responses
Rebuttal (continued)
comment: Thanks for the clarification by the authors.

Regarding the last question, yes. The authors could possibly consider conducting a baseline test to probe how the selected participants generally perceive the numbers at the scale in the conducted formal experiment. This may help to control individual biases and provide a more accurate prediction.

I think the authors well addressed my concerns. But given the constraint of the overall scope, I will keep my current evaluation.
Thank you for the review and suggestions. Below, we provide clarifications regarding the potential concerns. We are open to discussing any further concerns you may have. We are fully committed to ensuring that our work meets all ethical requirements.



- [Potential Misuse and Harm; Deception and Fraud]: In response to the concern raised, we want to clarify that in Section 5, where we manipulate AI explanations for adversarial purposes, our aim is exactly to discuss and demonstrate to the wider community the potential misuse and harm associated with this practice. Additionally, in the Introduction section (lines 49-60), we also discuss the potential misuse of human behavior modeling to manipulate AI explanations as the "bad" side of this technology. Finally, in the Ethical Considerations section, we briefly discuss the potential impact of such technology and suggest possible interventions to prevent its misuse. 

- [Debrief session]: In addition, for participants in Section 5 who received adversarially manipulated explanations, we conducted an immediate debrief session after they completed the study. During this session, we explained that the explanations they received were intentionally manipulated and do not reflect real-world decision-making or the actual rationale of the AI model. This debriefing was designed to mitigate any potential negative impact of the adversarial manipulations on the participants.  


- [bias in data]: The datasets we used in our study are widely publicly available and are considered to reflect real-world ground truth. These datasets are not intentionally biased toward any specific groups. However, we acknowledge that there might be implicit biases introduced during the curation process, which are beyond our control. In our experiments, we made no alterations to the datasets that would introduce additional bias. We relied on the original data as provided to in our study to control for other effect.

We will add more discussions in the maniscurpt to highlight the parts you mentioned.
Thanks for the review and suggestions. Below, we provide more details regarding to the payment. We are open to discussing any further concerns you may have. We are fully committed to ensuring that our work meets all ethical requirements.

Table 1: The average hourly payment received by participants in our study across four tasks. In the row "Number of Workers," the number in parentheses indicates the number of invalid participants who did not pass the attention check questions.

|                               | Recidivism |   Census  |    Bias   | Toxicity |
|:-----------------------------:|:----------:|:---------:|:---------:|:--------:|
|       Number of Workers       |  336 (16)  | 310 (25)  | 259 (20)  | 286 (17) |
|          Average Time (minute)         |    6.67    |    6.25  |    6.98   |   6.34   |
|     Hourly Payment (Base)     |    $10.8   |   $11.7   |   $10.2   |  $11.36  |
| Hourly Payment (Base + Bonus) |    $11.9   |   $11.8   |   $11.4   |   $14.4  |

-  As shown in Table 1, the average base hourly payment across the four tasks exceeds \\$10 per hour, and the average hourly payment including bonus is close to \\$12 per hour, both of which are well above the federal minimum wage.  Additionally, the average hourly payment on Prolific, the platform we used for recruiting participants, is approximately \\$8 per hour. Therefore, our hourly payment is also well above the platform’s requirements.

-  the payment covers the entire study, including both the tutorial and the tasks.

We will include this information in the manuscript to ensure transparency regarding the payment structure of our study.
Thank you to the authors for their responses. Most of my questions have been addressed. After considering your responses and the feedback from other reviewers, I will maintain my evaluation.
Thank you for the clarification. I've read your response and the other reviews/responses. I still think this work is a marginal accept and maintain my score, but will check back if there is more discussion.
> For the "benign" use case of improving human model prediction, the authors looked at cases ...


- To determine when the AI model's decision is likely correct, we followed an established approach [1], which combines AI predictions with independent human decisions to leverage the complementary strengths of both AI and human judgment. Our evaluation revealed that the combined decision's accuracy on the test set was higher than the accuracy of either the AI model or human judgment alone. Therefore, we used this combined prediction to evaluate when the AI model's decision is likely correct. For more details on this evaluation, please refer to Appendix C.1.

- In AI-assisted decision making, particularly in critical contexts, it is crucial that humans remain responsible for the final decision and its consequences. In addition, simply making AI predictions more accurate may not be enough, because humans could still rely on a highly accurate AI very inappropriately, resulting in low team performance.  Our study, therefore,  is not to enhance AI predictions but to improve the performance of human decision makers as they interact with AI models. In fact, during our evaluation, we found that presenting manipulated explanations can significantly reduce human underreliance—where humans fail to rely on the AI model when it is correct—and thus improve human-AI team performance.


- We examined the performance of humans after manipulating explanations and compared it to the performance of a "fixed AI model." In the recidivism prediction task, we found that humans with manipulated explanations achieved an average accuracy of 0.66, which was higher than the fixed AI model's accuracy of 0.62. However, in other tasks, human performance was lower than that of the fixed AI model.

> In addition, the reliance metrics of Fig. 3 are not defined here, and the readers need to search the referenced papers for definitions.

We will revise the manuscript as suggested to include the definition of the reliance metric.


[1]  Kerrigan, Gavin, Padhraic Smyth, and Mark Steyvers. "Combining human predictions with model probabilities via confusion matrices and calibration." Advances in Neural Information Processing Systems 34 (2021): 4421-4434.