Abstract:
Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decision-making tasks. Further, the subjective measures on evaluations with actual decision-making tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing human+AI teams that can reliably perform better than humans or AIs alone.
 CCS CONCEPTS • Human-centered computing → Interaction design; Empirical studies in interaction design.

Zana Buçinca
Phoebe Lin
Krzysztof Z Gajos
Elena L 2020 Glassman

Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

INTRODUCTION Because people and AI-powered systems have complementary strengths, many expected that human+AI teams would perform better on decision-making tasks than either people or AIs alone  [1, 21, 22] . However, there is mounting evidence that human+AI teams often perform worse than AIs alone  [16, 17, 28, 34] . We hypothesize that this mismatch between our field's aspirations and the current reality can be attributed, in part, to several pragmatic decisions we frequently make in our research practice. Specifically, although our aspiration is formulated at the level of sociotechnical systems , i.e., human+AI teams working together to make complex decisions, we often make one of two possible critical mistakes: (1) Rather than evaluating how well the human+AI team performs together on a decision-making task, we evaluate by using proxy tasks, how accurately a human can predict the decision or decision boundaries of the AI  [13, 27, 29, 34] .  (2)  We rely on subjective measures of trust and preference, e.g.,  [35, 36, 44] , instead of objective measures of performance. We consider each of these two concerns in turn. First, evaluations that use proxy tasks force study participants to pay attention to the AI and the accompanying explanationssomething that they are unlikely to do when performing a realistic decision-making task. Cognitive science provides compelling evidence that people treat cognition like any other form of labor  [24]  and favor less demanding forms of cognition, i.e., heuristics over analytical thinking, even in high stakes contexts like medical diagnosis  [31] . Therefore, we hypothesize that user performance and preference on proxy tasks may not accurately predict their performance and preference on the actual decision-making tasks where their cognitive focus is elsewhere and they can choose whether and how much to attend to the AI. Second, subjective measures such as trust and preference have been embraced as the focal point for the evaluation of explainable systems  [35, 36, 44] , but we hypothesize that subjective measures may also be poor predictors of the ultimate performance of people performing realistic decision-making tasks while supported by explainable AI-powered systems. Preference and trust are important facets of explainable AI systems: they may predict users' intent to attend to the AI and its explanations in realistic tasks settings and adhere to the system's recommendations. However, the goal of explainable interfaces should be instilling in users the right amount of trust  [10, 32, 33] . This remains a remarkable challenge, as on one end of the trust spectrum users might over-rely on the system and remain oblivious of its errors, whereas on the other end they might exhibit self-reliance and ignore the system's correct recommendations. Furthermore, evaluating an AI's decision, its explanation of that decision, and incorporating that information into the decisionmaking process requires cognitive effort and the existing evidence suggests that preference does not predict performance on cognitive tasks  [8, 12, 37] . To evaluate these two hypotheses, we conducted two online experiments and one in-person study of an AI-powered decision support system for a nutrition-related decision-making task. In one online study we used a proxy task, in which participants were asked to predict the AI's recommendations given the explanations produced by the explainable AI system. In the second online study, participants completed an actual decision-making task: actually making decisions assisted by the same explainable AI system as in the first study. In both studies, we measured participants' objective performance and collected subjective measures of trust, preference, mental demand, and understanding. In the in-person study, we used a think-aloud method to gain insights into how people reason while making decisions assisted by an explainable AI system. In each study, we presented participants with two substantially distinct explanation types eliciting either deductive or inductive reasoning. The results of these studies indicate that (1) subjective measures from the proxy task do not generalize to the actual decision-making task, and (2) when using actual decision-making tasks, subjective results do not predict objective performance results. Specifically, participants trusted and preferred inductive explanations in the proxy task, whereas they trusted and preferred the deductive explanations in the actual task. Second, in the actual decision-making task, participants recognized AI errors better with inductive explanations, yet they preferred and trusted the deductive explanations more. The in-person think-aloud study revealed insights about why participants preferred and trusted one explanation type over another, but we found that by thinking aloud during an actual decision-making task, participants may be induced to exert additional cognitive effort, and behave differently than they would during an actual decision-making task when they are, more realistically, not thinking aloud. In summary, we show that the results of evaluating explainable AI systems using proxy tasks may not predict the results of evaluations using actual decision-making tasks. Users also do not necessarily perform better with systems that they prefer and trust more. To draw correct conclusions from empirical studies, explainable AI researchers should be wary of evaluation pitfalls, such as proxy tasks and subjective measures. Thus, as we recognize that explainable AI technology forms part of sociotechnical systems, and as we increasingly use these technologies in high-stakes scenarios, our evaluation methodologies need to reliably demonstrate how the entire sociotechnical systems (i.e., human+AI teams) will perform on real tasks.
 RELATED WORK 2.1 Decision-making and Decision Support Systems Decision-making is a fundamental cognitive process that allows humans to choose one option or course of action from among a set of alternatives  [42, 43, 45] . Since it is an undertaking that requires cognitive effort, people often employ mental shortcuts, or heuristics, when making decisions  [40] . These heuristics save time and effort, and frequently lead to good outcomes, but in some situations they result in cognitive biases that systematically lead to poor decisions (see, e.g.,  [4] ). To help people make good decision reliably, computer-based Decision Support Systems (DSS) have been used across numerous disciplines (e.g., management  [15] , medicine  [20] , justice  [47] ). While DSS have been around for a long time, they are now increasingly being deployed because the recent advancements in AI enabled these systems to achieve high accuracy. But since humans are the final arbiters in decisions made with DSS, the overall sociotechincal system's accuracy depends both on the system's accuracy and on the humans and their underlying cognitive processes. Research shows that even when supported by a DSS, people are prone to insert bias into the decision-making process  [16] . One approach for mitigating cognitive biases in decision-making is to use cognitive forcing strategies, which introduce self-awareness and self-monitoring of decision-making  [7] . Although not universally effective  [38] , these strategies have shown promising results as they improve decision-making performance, both if the human is assisted  [17, 34]  or is not assisted by a DSS  [31] . To illustrate, Green & Chen  [17]  showed that across different AI-assisted decision-making treatments, humans performed best when they had to make the preliminary decision on their own first before being shown the system recommendation (which forced them to engage analytically with the system's recommendation and explanation if their own preliminary decision differed from that offered by the system). Even though conceptual frameworks that consider cognitive processes in decision-making with DSS have been proposed recently  [41] , further research is needed to thoroughly investigate how to incorporate DSS into human decision-making and the effect of cognitive processes while making system-assisted decisions.
 Evaluating AI-Powered Decision Support Systems Motivated by the growing number of studies in interpretable and explainable AI-powered decision support systems, researchers have called for more rigorous evaluation of explainable systems  [9, 14, 19] . Notably, Doshi-Velez & Kim  [9]  proposed a taxonomy for evaluation of explainable AI systems, composed of the following categories: application grounded evaluation (i.e., domain experts evaluated on actual tasks), human grounded evaluation (i.e., lay humans evaluated on simplified tasks) and functionally grounded evaluation (i.e., no humans, proxy tasks). To put our work into context, our definition of the actual task falls into application grounded evaluation, where people for whom the system is intended (i.e., not necessarily experts) are evaluated on the intended task. Whereas, the the proxy task is closer to human grounded evaluation but addresses both domain experts and lay people evaluated on simplified tasks, such as the simulation of model's prediction given an input and an explanation. Studies using actual tasks evaluate the performance of human and the system, as a whole, on the decision-making task  [3, 17, 23, 46] . In these studies, participants are told to focus on making good decisions and it is up to them to decide whether and how to use the AI's assistance to accomplish the task. In contrast, studies that use proxy tasks evaluate how well users are able to simulate the model's decisions  [6, 13, 27, 34]  or decision boundaries  [29] . In such studies, participants are specifically instructed to pay attention to the AI. These studies evaluate the human's mental model of the system when the human is actively attending to the system's predictions and explanations, but do not necessarily evaluate how well the human is able to perform real decision-making tasks with the system. For example, to identify which factors make a model more interpretable, Lage et al. ask participants to simulate the interpretable model's predictions  [27] . In addition to the evaluation task, the choice of evaluation metrics is a critical one for the correct evaluation of intelligent systems  [2] . In explainable AI literature, subjective measures, such as user trust and experience, have been largely embraced as the focal point for the evaluation of explainable systems  [35, 36, 44, 48] . Hoffman et al.  [19]  proposed metrics for explainable systems that are grounded in the subjective evaluation of a system (e.g., user satisfaction, trust, and understanding). These may take the form of questionnaires on attitude and confidence in the system  [18]  and helpfulness of the system  [5, 26] . However, while these measures are informative, evidence suggests they do not necessarily predict user's performance with the system. For example, Green & Chen  [16]  discovered that self-reported measures could be misleading, since participant's confidence in their performance was negatively associated with their actual performance. Similarly, Lai & Tan  [28]  found that humans cannot accurately estimate their own performance. More closely related to our findings, Poursabzi-Sangdeh et al.  [34]  observed that even though participants were significantly more confident on the predictions of one model over the other, their decisions did not reflect the stated confidence. Furthermore, Lakkaraju & Bastani  [30]  demonstrated that participants trusted the same underlying biased model almost 10 times more when they were presented with misleading explanations compared to the truthful explanations that revealed the model's bias. These findings indicate that not only are subjective measures poor predictors of performance, but they can easily be manipulated and lead users to adhere to biased or malicious systems.
 EXPERIMENTS We conducted experiments with two different evaluation tasks and explanation designs to test the following hypotheses: H1: Results of widely accepted proxy tasks, where the user is asked to explicitly engage with the explanations, may not predict the results of realistic settings where the user's focus is on the actual decision-making task. H2: Subjective measures, such as self-reported trust and preference with respect to different explanation designs, may not predict the ultimate human+AI performance.
 Proxy Task 3.1.1 Task Description. We designed the task around nutrition because it is generally accessible and plausibly useful in explainable AI applications for a general audience. Participants were shown a series of 24 images of different plates of food. The ground truth of the percent fat content was also shown to them as a fact. Participants were then asked: "What will the AI decide?" given that the AI must decide "Is X% or more of the nutrients on this plate fat?". As illustrated in Figure  1 , each image was accompanied by explanations generated by the simulated AI. The participants chose which decision they thought the AI would make given the explanations and the ground truth. We designed two types of explanations, eliciting either inductive or deductive reasoning. In inductive reasoning, one infers general patters from specific observations. Thus, for the inductive explanations, we created example-based explanations that required participants to recognize the ingredients that contributed to fat content and draw their own conclusion about the given image. As shown in Figures 1a, the inductive explanations began with "Here are examples of plates that the AI knows the fat content of and categorizes as similar to the one above. " Participants then saw four additional images of plates of food. In deductive reasoning, in contrast, one starts with general rules and reaches a conclusion with respect to a specific situation. Thus, for the deductive explanations, we provided the general rules that the simulated AI applied to generate its recommendations. For example, in Figure  1b , the deductive explanation begins with "Here are ingredients the AI knows the fat content of and recognized as main nutrients:" followed by a list of ingredients. We chose a within-subjects study design, where for one half of the study session, participants saw inductive explanations and, for the other half of the study session, they saw deductive explanations. The order in which the two types of explanations were seen was counterbalanced. Each AI had an overall accuracy of 75%, which meant that in 25% of the cases the simulated AI misclassified the image or misrecognized ingredients (e.g., Figure  1b ). The order of the specific food images was randomized, but all participants encountered the AI errors at the same positions. We fixed the errors at questions 4, 7, 11, 16, 22 and 23, though which food the error was associated to was randomized. We included the ground truth of the fat content of plates of food, because the main aim of the proxy task was to measure whether the user builds correct mental models of the AI and not to assess the actual nutrition expertise of the participant. 3.1.2 Procedure. This study was conducted online, using Amazon Mechanical Turk. Participants were first presented with brief information about the study and an informed consent form. Next, participants completed the main part of the study, in which they answered 24 nutrition-related questions, divided into two blocks of 12 questions. They saw inductive explanations in one block and the deductive explanations in the other. The order of explanations was randomized across participants. Participants completed mid-study and end-study questionnaires so that they would provide a separate assessment for each of the two explanation types. They were also asked to directly compare their experiences with the two simulated AIs in a questionnaire at the end of the study. 3.1.3 Participants. We recruited 200 participants via Amazon Mechanical Turk (AMT). Participation was limited to adults in the US. Of the total 200 participants, 183 were retained for final analyses, while 17 were excluded based on their answers to two commonsense questions included in the questionnaires (i.e., What color is the sky?). The study lasted 7 minutes on average. Each worker was paid 2 USD. 3.1.4 Design and Analysis. This was a within-subjects design. The within-subjects factor was explanation type -inductive or deductive. We collected the following measures: • Performance: Percentage of correct predictions of AI's decisions • Appropriateness: Participants responded to the statement "The AI based its decision on appropriate examples/ingredients. " with either 0=No or 1=Yes (after every question) • Trust: Participants responded to the statement "I trust this AI to assess the fat content of food. " on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (at the end of each block) • Mental demand: Participants answered the question "How mentally demanding was understanding how this AI makes decisions?" on a 5-point Likert scale from 1=Very low to 5=Very high (every four questions) • Comparison between the two explanation types: Participants were asked at the end of the study to choose one AI over another on trust, preference, and mental demand. We used repeated measures ANOVA for within-subjects analyses and the binomial test for the comparison questions.
 Actual Decision-making Task 3.2.1 Task description. The actual decision-making task had a similar set up to the proxy task. Participants were shown the same series of 24 images of different plates of food, but were asked their own decision whether the percent fat content of nutrients on the plate is higher than a certain percentage. As illustrated in Figure  2 , each image was accompanied by an answer recommended by a simulated AI, and an explanation provided by that AI. We introduced two more conditions to serve as baselines in the actual decision-making task depicted in Figure  3 . There were three between-subjects conditions in this study: 1. the no-AI baseline (where no recommendations or explanations were provided), 2. the no-explanation baseline (where a recommendation was provided by a simulated AI, but no explanation was given), and 3. the main condition in which both recommendations and explanations were provided. In this last condition, two withinsubjects sub-conditions were present: for one half of the study participants saw inductive explanations and for the other they saw deductive explanations. The order in which the two types of explanations were seen was counterbalanced. In the no-AI baseline, participants were not asked any of the questions relating to the performance of the AI. The explanations in this task differed only slightly from the explanations in the proxy task, because they indicated the AI's recommendation. Inductive explanations started with: "Here are examples of plates that the AI categorizes as similar to the one above and do (not) have X% or more fat. " followed by four examples of images. Similarly, deductive explanations stated: "Here are ingredients the AI recognized as main nutrients which do (not) make up X% or more fat on this plate:" followed by a list of ingredients. 3.2.2 Procedure. The procedure was the same as for the proxy task. The study was conducted online, using the Amazon Mechanical Turk. Participants were first presented with a brief information about the study and an informed consent form. Next, participants completed the main part of the study, in which they answered 24 nutrition-related questions, divided into two blocks of 12 questions. All participants also completed a questionnaire at the end of the study, providing subjective assessments of the system they interacted with. Participants who were presented with AI-generated recommendations accompanied by explanations also completed a mid-study questionnaire (so that they would provide separate assessment for each of the two explanation types) and they were also asked to directly compare their experiences with the two simulated AIs at the end of the study.
 Participants. We recruited 113 participants via Amazon Mechanical Turk (AMT). Participation was limited to adults in the US. Of the total 113 participants, 102 were retained for final analyses, while 11 were excluded based on their answers to two commonsense questions included in the pre-activity and post-activity questionnaires (i.e., "What color is the sky?"). The task lasted 10 minutes on average. Each worker was paid 5 USD per task.
 Design and Analysis . This was a mixed between-and withinsubjects design. As stated before, the three between-subjects conditions were: 1. the no-AI baseline; 2. the no-explanation baseline, in which the AI-generated recommendations were provided but no explanations; 3. the main condition, in which both the AI-generated recommendations and explanations were provided. The withinsubjects factor was explanation type (inductive or deductive) and it was applied only for participants who were presented with AIgenerated recommendations with explanations. We collected the following measures: • Performance: Percentage of correct answers (overall for each AI, and specifically for questions when AI presented incorrect explanations) • Understanding: Participants responded to the statement "I understand how the AI made this recommendation." on a 5point Likert scale from 1=Strongly disagree to 5=Strongly agree (after every question) • Trust: Participants responded to the statement "I trust this AI to assess the fat content of food. " on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (every four questions) • Helpfulness: Participants responded to the statement "This AI helped me assess the percent fat content." on a 5-point Likert scale from 1=Strongly disagree to 5=Strongly agree (at the end of each block) • Comparison between the two explanation types: Participants were asked at the end of the study to choose one AI over another on trust, preference, understanding and helpfulness. We used analysis of variance (ANOVA) for between-subjects analyses and repeated measures ANOVA for within-subjects analyses. We used the binomial test for the comparison questions.
 RESULTS
 Proxy Task Results The explanation type had a significant effect on participants' trust and preference in the system. Participants trusted the AI more when presented with inductive explanations (M = 3.55), rather than deductive explanations (M = 3.40, F 1,182 = 5.37, p = .02). Asked to compare the two AIs, most of the participants stated they trusted more the inductive AI (58%, p = .04). When asked the hypothetical question: "If you were asked to evaluate fat content of plates of food, which AI would you prefer to interact with more?", again most of the participants (62%) chose the inductive AI over the deductive AI (p = .001). The inductive AI was also rated significantly higher (M = 0.83) than the deductive AI (M = 0.79) in terms of the appropriateness of examples (ingredients for the deductive condition) on which the AI based its decision (F (1, 182) = 13.68, p = 0.0003). When the AI presented incorrect examples/ingredients, there was no significant difference among the inductive (M = 0.47) and deductive (M = 0.50) conditions (F (1, 182) = 1.02, p = .31, n.s.). We observed no significant difference in overall performance when participants were presented with inductive (M = 0.64) or deductive explanations (M = 0.64, F (1, 182) = 0.0009, n.s.). When either AI presented incorrect explanations, although the average performance dropped for both inductive (M = 0.40) and deductive (M = 0.41) conditions, there was also no significant difference among them (F (1, 182) = .03, n.s.). In terms of mental demand, there was a significant effect of the explanation type. Participants rated the deductive AI (M = 2.94) as more mentally demanding than the inductive AI (M = 2.79, F (1, 182) = 7.75, p = .0006). The effect was noticed also when they were asked: "Which AI required more thinking while choosing which decision it would make?", with 61% of participants choosing deductive over inductive (p = .005).
 Actual Decision-making Task Results 18 participants were randomized into the no-AI condition, 19 into the AI with no explanation condition, and 65 were presented with AI recommendations supported by explanations. We observed a significant main effect of the presence of explanations on participants' trust in the AI's ability to assess the fat content of food. Participants who saw either kind of explanation, trusted the AI more (M = 3.56) than those who received AI recommendations, but no explanations (M = 3.17, F 1,483 = 11.28, p = .0008). Further, there was a significant main effect of the explanation type on participants' trust: participants trusted the AI when they received deductive explanations more (M = 3.68) than when they received inductive explanations (M = 3.44, F 1,64 = 5.96, p = .01). When asked which of the two AIs they trusted more, most participants (65%) said that they trusted the AI that provided deductive explanations more than the one that provided inductive explanations (p = .02). Participants also found the AI significantly more helpful when explanations were present (M = 3.78) than when no explanations were offered (M = 3.26, F 1,147 = 4.88, p = .03). Further, participants reported that they found deductive explanations more helpful (M = 3.92) than inductive ones (M = 3.65) and this difference was marginally significant (F 1,64 = 3.66, p = .06). When asked which of the two AIs they found more helpful, most participants (68%) chose the AI that provided deductive explanations (p = .006). Participants also reported that they understood how the AI made its recommendations better when explanations were present (M = 3.84) than when no explanations were provided (M = 3.67, F 1,2014 = 6.89p = .009). There was no difference in the perceived level of understanding between the two explanation types (F 1,64 = 0.44, p = .51). Asked about their overall preference, most participants (63%) preferred the AI that provided deductive explanations over the AI that provided inductive explanations (p = .05). In terms of actual performance on the task, participants who received AI recommendations (with or without explanations) provided a significantly larger fraction of accurate answers (M = 0.72) than those who did not receive AI recommendations (M = 0.46, F 1,2446 = 118.07, p < .0001). Explanations further improved overall performance: participants who saw explanations of AI recommendations had a significantly higher proportion of correct answers (M = 0.74) than participants who did not receive explanations of AI recommendations (M = 0.68, F 1,2014 = 5.10, p = .02) (depicted in Figure  4a ). There was no significant difference between the two explanation types in terms of overall performance (F 1,64 = 0.44, n.s.). However, we observed a significant interaction between explanation type and the correctness of AI recommendations (F 2,2013 = 15.03p < .0001). When the AI made correct recommendations, participants performed similarly when they saw inductive (M = .78) and deductive (M = .81) explanations (F 1,64 = 1.13, n.s.). When the AI made incorrect recommendations, however, participants were significantly more accurate when they saw inductive (M = 0.63) than deductive (M = 0.48) explanation (F 1,64 = 7.02, p = .01) (depicted in Figure  4b ). To ensure the results of our studies were not random, we replicated both experiments with almost identical setup and obtained the same main results (in terms of significance) reported in this section.
 QUALITATIVE STUDY Through the qualitative study, we explored the user reasoning and sought to gain insight into the discrepancy between subjective measures and performance. We asked participants to think aloud during an in-person study in order to understand how and why people perceive AI the way they do, in addition to what factors go into making decisions when assisted by an AI.
 Task The same task design was used in this study as in the actual decisionmaking task, except that all participants were presented with the main condition (where both recommendations and explanations were provided). As in the actual decision-making task, each participant saw both inductive and deductive explanations.
 Procedure Upon arriving to the lab, participants were presented with an informed consent form, including agreeing to being screen-and audiorecorded, and instructions on the task. Afterwards, the steps in this study were similar to those in the actual decision-making task, except that we added the think-aloud method  [11] : as participants completed the task, they were asked to verbalize their thought process as they made each decision. At the end of the task, there was a semi-structured interview, during which participants briefly discussed how they believed the two AIs were making their recommendations and why they did or did not trust them. Participants also discussed if and why they preferred one AI over the other.
 Participants We recruited 11 participants via community-wide emailing lists (8 female, 3 male, age range 23-29, M = 24.86, SD = 2.11). Participants were primarily graduate students with backgrounds from design, biomedical engineering, and education. Participants had varying levels of experience with AI and machine learning, ranging from 0-5 years of experience.
 Design and Analysis We transcribed the think-aloud comments and the post-task interviews. Transcripts were coded and analyzed for patterns using an inductive approach  [39] . We focused on comments about (1) how the AI made its recommendations; (2) trust in the AI; (3) erroneous recommendations; (4) why people preferred one explanation type over the other. From a careful reading of the transcripts, we discuss some of the themes and trends that emerged from the data.
 Results Preference of one explanation type over another. Eight out of the 11 participants preferred the inductive explanations. Participants who preferred inductive explanations perceived the four images as data. One participant stated that "Because [the AI] showed similar pictures, I knew that it had data backing it up" (P3). On the other hand, participants who preferred deductive explanations perceived the listing of ingredients to be reliable, and that "if the AI recognized that it's steak, then I would think, Oh the AI knows more about steak fat than I do, so I'm going to trust that since it identified the object correctly. " (P6). In our observations, we found that the way participants used the explanations was different depending on the explanation type. With inductive explanations, one participant often first made their own judgement before looking at the recommendation, and then used the recommendation to confirm their own judgement. In a cake example, one participant said, "So I feel it probably does have more than 30% because it's cake, and that's cream cheese. But these are all similar to that, and the AI also says that it does have more than 30% fat, so I agree" (P2). With deductive explanations, participants evaluated the explanations and recommendation more before making any decision. In the same cake example, a different participant said, "There are [the AI recognizes] nuts, cream cheese, and cake. That seems to make sense. Nuts are high in fat, so is dairy, so I agree with that. " (P6). Cognitive Demand. At the end of the study, participants were asked which AI was easier to understand. Ten out of 11 participants felt the inductive explanations were easier to understand than the deductive explanations. Several participants stated that the deductive explanations forced them to think more, and that generally they spent more time making a decision with deductive explanations. One participant said, for example, "I feel like with this one I have to think a bit more and and rely on my own experiences with food to see or understand to gauge what's fatty. " (P2). Errors and Over-reliance. Nine out of 11 participants claimed to trust the inductive explanations more. We intentionally introduced erroneous recommendations because we expected participants to utilize them to calibrate their mental model of the AI. When participants understood the error and believed the error was reasonable for an AI to make, they expressed less distrust in subsequent questions. However, when participants perceived the error to be inconsistent with other errors, their trust in subsequent recommendations was hurt much more. For example, one participant stated, "I think the AI makes the recommendation based on shape and color. But in some other dessert examples, it was able to identify the dessert as a dessert. So I wasn't sure why it was so difficult to understand this particular item" (P5). We found that there was also some observable correlation between explanation type and trust. Many participants claimed it was easier to identify errors from the inductive explanations, yet agreed with erroneous recommendations from inductive explanations more. In some of those instances, participants either did not realize the main food image was different from the other four or felt the main food image was similar enough though not exact. Lastly, one participant stated the inductive explanations were easier to understand because "you can visually see exactly why it would come to its decision,", but for deductive explanations "you can see what it's detecting but not why" (P8), and yet this participant also stated that the deductive explanations seemed more trustworthy. Impact of the Think-Aloud method on participant behavior. In this study, we asked participants to perform the actual decision-making task and we expected to observe similar results to those obtained in the previous experiment when using the actual tasks. Yet, in this study, 8 out of the 11 participants preferred the inductive explanations and 10 out of 11 participants felt the inductive explanations were easier to understand than the deductive explanations. These results are comparable to the results we obtained in the previous experiment when we used the proxy task rather than the actual task. We believe that the use of the think-aloud method may have impacted participants' behavior in this study. Specifically, because participants were instructed to verbalize their thoughts, they were more likely to engage in analytical thinking when considering the AI recommendations and explanations than they were in the previous experiment with the actual tasks, where their focus was primarily on making decisions. It is possible that while the think-aloud method is part of standard research practice for evaluating interfaces, it is itself a form of cognitive forcing intervention  [7] , which impacts how people perform on cognitively-demanding tasks such as interacting with an explainable AI system on decision-making tasks. The act of talking about the explanations led participants to devote more of their attention and cognition to the explanations, and thus made them behave more similarly to participants in working with the proxy task rather than those working with the actual task.
 DISCUSSION In this study, we investigated two hypotheses regarding the evaluation of AI-powered explainable systems: • H1: Results of widely accepted proxy tasks, where the user is asked to explicitly engage with the explanations, may not predict the results of realistic settings where the user's focus is on the actual decision-making task. • H2: Subjective measures, such as self-reported trust and preference with respect to different explanation designs, may not predict the ultimate human+AI performance. We examined these hypotheses in the context of a nutrition-related decision-making task, by designing two distinct evaluation tasks and two distinct explanation designs. The first task was a proxy task, where the users had to simulate the AI's decision by examining the explanations. The second task was the more realistic, actual decision-making task, where the user had to make their own decisions about the nutritional content of meals assisted by AIgenerated recommendations and explanations. Each of the tasks had two parts, where users interacted with substantially different explanation styles-inductive and deductive. In the experiment with the proxy task, participants preferred and trusted the AI that used inductive explanations significantly more. They also reported that the AI that used inductive explanations based its decision on more accurate examples on average than the AI that used deductive explanations. When asked "If you were asked to evaluate fat content of plates of food, which AI would you prefer to interact with more?"), the majority of participants chose the AI that provided inductive explanations. In contrast with the proxy task experiment, in the experiment with the actual decision-making task, participants rated the AI with deductive explanations as their preferred AI, and viewed it as more trustworthy and more helpful compared to the AI that used inductive explanations. The contrast in terms of performance measures was less pronounced. When attempting proxy tasks, participants demonstrated nearly identical accuracy regardless of explanation type. However, when attempting actual decision-making tasks and the AI provided an incorrect recommendation, participants ignored that incorrect recommendation and provided the correct answer significantly more often when they had access to inductive, not deductive, explanations for the AI's recommendation. These contradictory results produced by the two experiments indicate that results of evaluations that use proxy tasks may not correspond to results on actual tasks, thus supporting H1. This may be because in the proxy task the users cannot complete the task without engaging analytically with the explanations. Whereas, in the actual decision-making task, the user's primary goal is to make the most accurate decisions about the nutritional content of meals; she chooses whether and how deeply she engages with the AI's recommendations and explanations. This finding has implications for the explainable AI community, as there is a current trend to use proxy tasks to evaluate user mental models of the AI-powered systems, with the implicit assumption that the results will translate to the realistic settings where users make decisions about an actual task while assisted by an AI. We tested H2 on the actual decision-making task. The results show that participants preferred, trusted and found the AI with deductive explanations more helpful than the AI that used inductive explanations. Yet, they performed significantly better with the AI that used inductive explanations when the AI made erroneous recommendations. Therefore, H2 is also supported. This finding suggests that the design decisions for explainable interfaces should not be made by relying solely on user experience and subjective measures. Subjective measures of trust and preference are, of course, valuable and informative, but they should be used to complement rather than replace performance measures. Our research demonstrated that results from studies that use proxy tasks may not predict results from studies that use realistic tasks. Our results also demonstrated that user preference may not predict their performance. However, we recognize that evaluating novel AI advances through human subjects experiments that involve realistic tasks is expensive in terms of time and resources, and may negatively impact the pace of innovation in the field. Therefore, future research needs to uncover why these differences exist so that we can develop low burden evaluation techniques that correctly predict the outcomes of deploying a system in a realistic setting. We believe that the reason why explainable AI systems are sensitive to the difference between proxy task and actual task evaluation designs is that different AI explanation strategies require different kinds and amounts of cognition from the users (like our inductive and deductive explanations). However, people are reluctant to exert cognitive effort  [24, 25]  unless they are motivated or forced to do so. They also make substantially different decisions depending on whether they choose to exert cognitive effort or not  [12, 37] . In actual decision-making situations, people often choose not to engage in effortful analytical thinking, even in high-stakes situations like medical diagnosis  [31] . Meanwhile, proxy tasks force participants to explicitly pay attention to the behavior of the AI and the explanations produced. Thus, results observed when participants interact with proxy tasks do not accurately predict people's behavior in many realistic settings. In our study, participants who interacted with the proxy task felt that the deductive explanations required significantly more thinking than the inductive explanations. Therefore, in the proxy task where the participants were obliged to exert cognitive effort to evaluate the explanations, they said they preferred and trusted the less cognitively demanding explanations more, the inductive explanations. In contrast, in the actual task the participants could complete the task even without engaging with the explanations. Thus, we suspect that in the deductive condition participants perceived the explanations as too mentally demanding, and chose to over-rely on the AI's recommendation, just to avoid cognitive effort of examining those explanations. They also might have perceived the AI that provided deductive explanations as more competent because it required more thinking. One implication of our analysis is that the effectiveness of explainable AI systems can be substantially impacted by the design of the interaction (rather than just the algorithms or explanations). For example, a recent study showed that a simple cognitive forcing strategy (having participants make their own preliminary decision before being shown the AI's decision) resulted in much higher accuracy of the final decisions made by human+AI teams than any strategy that did not involve cognitive forcing  [17] . Inadvertently, we uncovered an additional potential pitfall for evaluating explainable AI systems. As the results of our qualitative study demonstrated, the use of the think-aloud method-a standard technique for evaluating interactive systems-can also substantially impact how participants allocate their mental effort. Because participants were asked to think aloud, we suspect that they exerted additional cognitive effort to engage with the explanations and analyze their reasoning behind their decisions. Together, these results indicate that cognitive effort is an important aspect of explanation design and its evaluation. Explanations high in cognitive demand might be ignored by the users while simple explanations might not convey the appropriate amount of evidence that is needed to make informed decisions. At the same time, traditional methods of probing users' minds while using explainable interfaces should also be re-evaluated. By taking into account the cognitive effort and cognitive processes that are employed during the evaluation of the explanations, we might generate explainable interfaces that optimize the performance of the sociotechnical (hu-man+AI) system as a whole. Such interfaces would instill trust, and make the user aware of the system's errors.
 CONCLUSION To achieve the aspiration of human+AI teams that complement oneanother and perform better than either the human or the AI alone, researchers need to be cautious about their pragmatic decisions. In this study, through online experiments and an in-person study, we showed how several assumptions researchers make about the evaluation of the explainable AI systems for decision-making tasks could lead to misleading results. First, choosing proxy tasks for the evaluation of explainable AI systems shifts the user's focus toward the AI, so the obtained results might not correspond to results of the user completing the actual decision-making task while assisted by the AI. In fact, our results indicate that users trust and prefer one explanation design (i.e. inductive) more in the proxy task, while they trust and prefer the other explanation design (i.e. deductive) more in the actual decision-making task. Second, the subjective evaluation of explainable systems with measures such as trust and preference may not correspond to the ultimate user performance with the system. We found that people trusted and preferred the AI with deductive explanations more, but recognized AI errors better with the inductive explanations. Lastly, our results suggest that think-aloud studies may not convey how people make decisions with explainable systems in realistic settings. The results from the think-aloud in-person study, which used the actual task design, aligned more with the results we obtained in the proxy task. These findings suggest that to draw correct conclusions about their experiments, explainable AI researchers should be wary of the explainable systems' evaluation pitfalls and design their evaluation accordingly. Particularly, the correct and holistic evaluation of explainable AI interfaces as sociotechnical systems is of paramount importance, as they are increasingly being deployed in critical decision-making domains with grave repercussions. Figure 1 : 1 Figure 1: The proxy task. Illustration of the simulated AI system participants interacted with: (a) is an example of an inductive explanation with appropriate examples. (b) is an example of a deductive explanation with misrecognized ingredients, where the simulated AI misrecognized apples and beets as avocados and bacon.
 Figure 2 :Figure 3 : 23 Figure 2: The actual task. Illustration of the simulated AI system participants interacted with. (a) is an example of incorrect recommendations with inductive explanations. Contrasting the query image with the explanations reveals that the simulated AI misrecognized churros with chocolate as sweet potato fries with BBQ sauce. (b) is an example of correct recommendation with deductive explanations.
 Figure 4 : 4 Figure 4: Performance in the actual decision-making task. (a) depicts the mean of performance among no-AI, no-Explanations and with-Explanations (overall) conditions. (b) depicts the mean of performance among inductive and deductive conditions, when the AI recommendation is correct and erroneous. Error bars indicate one standard error.
 Figure 5 : 5 Figure 5: Subjective evaluations in terms of trust and preference of the two AIs. Red and blue depict the percent of participants that chose inductive and deductive AI, respectively. (a) proxy task (b) actual decision-making task.


Abstract: Inverse Reinforcement Learning (IRL) is the problem of learning the reward function underlying a Markov Decision Process given the dynamics of the system and the behaviour of an expert. IRL is motivated by situations where knowledge of the rewards is a goal by itself (as in preference elicitation) and by the task of apprenticeship learning (learning policies from an expert). In this paper we show how to combine prior knowledge and evidence from the expert's actions to derive a probability distribution over the space of reward functions. We present efficient algorithms that find solutions for the reward learning and apprenticeship learning tasks that generalize well over these distributions. Experimental results show strong improvement for our methods over previous heuristic-based approaches.

Bayesian Inverse Reinforcement Learning

Introduction The Inverse Reinforcement Learning (IRL) problem is defined in  [Russell, 1998]  as follows: Determine The reward function that an agent is optimizing. Given 1) Measurement of the agent's behaviour over time, in a variety of circumstances 2) Measurements of the sensory inputs to that agent; 3) a model of the environment. In the context of Markov Decision Processes, this translates into determining the reward function of the agent from knowledge of the policy it executes and the dynamics of the state-space. There are two tasks that IRL accomplishes. The first, reward learning, is estimating the unknown reward function as accurately as possible. It is useful in situations where the reward function is of interest by itself, for example when constructing models of animal and human learning or modelling opponent in competitive games. Pokerbots can improve performance against suboptimal human opponents by learning reward functions that account for the utility of money, preferences for certain hands or situations and other idiosyncrasies  [Billings et al., 1998] . There are also connections to various preference elicitation problems in economics  [Sargent, 1994] . The second task is apprenticeship learning -using observations of an expert's actions to decide one's own behaviour. It is possible in this situation to directly learn the policy from the expert  [Atkeson and Schaal, 1997] . However the reward function is generally the most succint, robust and transferable representation of the task, and completely determines the optimal policy (or set of policies). In addition, knowledge of the reward function allows the agent to generalize better i.e. a new policy can be computed when the environment changes. IRL is thus likely to be the most effective method here. In this paper we model the IRL problem from a Bayesian perspective. We consider the actions of the expert as evidence that we use to update a prior on reward functions. We solve reward learning and apprenticeship learning using this posterior. We perform inference for these tasks using a modified Markov Chain Monte Carlo (MCMC) algorithm. We show that the Markov Chain for our distribution with a uniform prior mixes rapidly, and that the algorithm converges to the correct answer in polynomial time. We also show that the original IRL is a special case of Bayesian IRL (BIRL) with a Laplacian prior. There are a number of advantages of our technique over previous work: We do not need a completely specified optimal policy as input to the IRL agent, nor do we need to assume that the expert is infallible. Also, we can incorporate external information about specific IRL problems into the prior of the model, or use evidence from multiple experts. IRL was first studied in the machine learning setting by  [Ng and Russell, 2000]  who described algorithms that found optimal rewards for MDPs having both finite and infinite states. Experimental results show improved performance by our techniques in the finite case. The rest of this paper is organised as follows: In section 2 we define our terms and notation. Section 3 presents our Bayesian model of the IRL process. Section 4 discusses how to use this model to do reward learning and apprenticeship learning while section 5 discusses the sampling procedure. Sections 6, 7 and 8 then present experimental results, related work and our conclusions respectively.
 Preliminaries We recall some basic definitions and theorems relating to Markov Decision Processes and Reinforcement Learning. A (finite) Markov Decision Problem is a tuple (S, A, T, γ, R) where • S is a finite set of N states. • A = {a 1 , . . . , a k } is a set of k actions. • T : S × A × S → [0, 1] is a transition probability func- tion. • γ ∈ [0, 1) is the discount factor. • R : S → R is a reward function, with absolute value bounded by R max . The rewards are functions of state alone because IRL problems typically have limited information about the value of action and we want to avoid overfitting. A Markov Decision Process (MDP) is a tuple (S, A, T, γ), with the terms defined as before but without a reward function. To avoid confusion we will use the abbreviation MDP only for Markov Decision Processes and not Problems. We adopt the following compact notation from  [Ng and Russell, 2000]  for finite MDPs : Fix an enumeration s 1 . . . s N of the finite state space S. The reward function (or any other function on the state-space) can then be represented as an Ndimensional vector R, whose ith element is R(s i ). A (stationary) policy is a map π : S → A and the (discounted, infinite-horizon) value of a policy π for reward function R at state s ∈ S,denoted V π (s, R) is given by: V π (s t1 , R) = R(s t1 )+E st 1 ,st 2 ,... [γR(s t2 )+γ 2 R(s t3 )+. . . |π] where P r(s ti+1 |s ti , π) = T (s ti , π(s ti ), s ti+1 ). The goal of standard Reinforcement Learning is to find an optimal policy π * such that V π (s, R) is maximized for all s ∈ S by π = π * . Indeed, it can be shown (see for example  [Sutton and Barto, 1998] ) that at least one such policy always exists for ergodic MDPs. For the solution of Markov Decision Problems, it is useful to define the following auxilliary Q-function: Q π (s, a, R) = R(s) + γE s ∼T (s,a,•) [V π (s , R)] We also define the optimal Q-function Q * (•, •, R) as the Qfunction of the optimal policy π * for reward function R. Finally, we state the following result concerning Markov Decision Problems (see  [Sutton and Barto, 1998] ) : Theorem 1 (Bellman Equations). Let a Markov Decision Problem M = (S, A, T, γ, R) and a policy π : S → A be given. Then, 1. For all s ∈ S, a ∈ A, V π and Q π satisfy V π (s) = R(s) + γ s T (s, π(s), s )V π (s ) (1) Q π (s, a) = R(s) + γ s T (s, a, s )V π (s ) 2. π is an optimal policy for M iff, for all s ∈ S, π(s) ∈ argmax a∈A Q π (s, a) (2) 3 Bayesian IRL IRL is currently viewed as a problem of infering a single reward function that explains an agent's behaviour. However, there is too little information in a typical IRL problem to get only one answer. For example, consider the MDP shown in Figure  1 . There are at least three reasonable kinds of reward functions: R 1 (•) has high positive value at s 1 (and low values elsewhere) which explains why the policy tries to return to this state, while R 2 (•) and R 3 (•) have high values at s 2 and s 3 respectively. Thus, a probability distribution is needed to represent the uncertainty. S 0 S 1 S 3 S 2 a 1 a 1 a 1 a 1 0.4 0.3 0.3 a 2 a 2 a 2 a 2 Figure 1: An example IRL problem. Bold lines represent the optimal action a 1 for each state and broken lines represent some other action a 2 . Action a 1 in s 1 has probabilities 0.4,0.3 and 0.3 of going to states s 1 , s 2 , s 3 respectively, and all other actions are deterministic.
 Evidence from the Expert Now we present the details of our Bayesian IRL model (Fig.  2 ). We derive a posterior distribution for the rewards from a prior distribution and a probabilistic model of the expert's actions given the reward function. Consider an MDP M = (S, A, T, γ) and an agent X (the expert) operating in this MDP. We assume that a reward function R for X is chosen from a (known) prior distribution P R . The IRL agent receives a series of observations of the expert's behaviour O X = {(s 1 , a 1 ), (s 2 , a 2 ) . . . (s k , a k )} which means that X was in state s i and took action a i at time step i. For generality, we will not specify the algorithm that X uses to determine his (possibly stochastic) policy, but we make the following assumptions about his behaviour: 1. X is attempting to maximize the total accumulated reward according to R. For example, X is not using an epsilon greedy policy to explore his environment. 2. X executes a stationary policy, i.e. it is invariant w.r.t. time and does not change depending on the actions and observations made in previous time steps. For example, X could be an agent that learned a policy for (M, R) using a reinforcement learning algorithm. Because the expert's policy is stationary, we can make the following independence assumption: P r X (O X |R) = P r X ((s 1 , a 1 )|R)P r X ((s 2 , a 2 )|R) . . . P r X ((s k , a k )|R) The expert's goal of maximizing accumulated reward is equivalent to finding the action for which the Q * value at each state is maximum. Therefore the larger Q * (s, a) is, the more likely it is that X would choose action a at state s. This likelihood increases the more confident we are in X 's ability to select a good action. We model this by an exponential distribution for the likelihood of (s i , a i ), with Q * as a potential function: P r X ((s i , a i )|R) = 1 Z i e αX Q * (si,ai,R) where α X is a parameter 1 representing the degree of confidence we have in X 's ability to choose actions with high X R α X (s1, a1) (s2, a2) (s k , a k ) Figure  2 : The BIRL model value. This distribution satisfies our assumptions and is easy to reason with. The likelihood of the entire evidence is : P r X (O X |R) = 1 Z e αX E(OX ,R) where E(OX , R) = P i Q * (si, ai, R) and Z is the appropriate normalizing constant. We can think of this likelihood function as a Boltzmann-type distribution with energy E(O X , R) and temperature 1 αX . Now, we compute the posterior probability of reward function R by applying Bayes theorem, P r X (R|O X ) = P r X (O X |R)P R (R) P r(O X ) = 1 Z e αX E(OX ,R) P R (R) (3) Computing the normalizing constant Z is hard. However the sampling algorithms we will use for inference only need the ratios of the densities at two points, so this is not a problem.
 Priors When no other information is given, we may assume that the rewards are independently identically distributed (i.i.d.) by the principle of maximum entropy. Most of the prior functions considered in this paper will be of this form. The exact prior to use however, depends on the characteristics of the problem: 1. If we are completely agnostic about the prior, we can use the uniform distribution over the space −R max ≤ R(s) ≤ R max for each s ∈ S. If we do not want to specify any R max we can try the improper prior P (R) = 1 for all R ∈ R n . 2. Many real world Markov decision problems have parsimonious reward structures, with most states having negligible rewards. In such situations, it would be better to assume a Gaussian or Laplacian prior:  P Gaussian (R(s) = r) = 1 √ 2πσ e − r P Beta (R(s) = r) = 1 ( r Rmax ) 1 2 (1 − r Rmax ) 1 2 , ∀s ∈ S In section 6.1, we give an example of how more informative priors can be constructed for particular IRL problems.
 Inference We now use the model of section 3 to carry out the two tasks described in the introduction: reward learning and apprenticeship learning. Our general procedure is to derive minimal solutions for appropriate loss functions over the posterior (Eq. 3). Some proofs are omitted for lack of space.
 Reward Learning Reward learning is an estimation task. The most common loss functions for estimation problems are the linear and squared error loss functions: L linear (R, R) = R − R 1 L SE (R, R) = R − R 2 where R and R are the actual and estimated rewards, respectively. If R is drawn from the posterior distribution (3), it can be shown that the expected value of L SE (R, R) is minimized by setting R to the mean of the posterior (see  [Berger, 1993] ). Similarily, the expected linear loss is minimized by setting R to the median of the distribution. We discuss how to compute these statistics for our posterior in section 5. It is also common in Bayesian estimation problems to use the maximum a posteriori (MAP) value as the estimator. In fact we have the following result: Theorem 2. When the expert's policy is optimal and fully specified, the IRL algorithm of  [Ng and Russell, 2000 ] is equivalent to returning the MAP estimator for the model of (3) with a Laplacian prior. However in IRL problems where the posterior distribution is typically multimodal, a MAP estimator will not be as representative as measures of central tendency like the mean.
 Apprenticeship Learning For the apprenticeship learning task, the situation is more interesting. Since we are attempting to learn a policy π, we can formally define the following class of policy loss functions: L p policy (R, π) = V * (R) − V π (R) p where V * (R) is the vector of optimal values for each state acheived by the optimal policy for R and p is some norm. We wish to find the π that minimizes the expected policy loss over the posterior distribution for R. The following theorem accomplishes this: Theorem 3. Given a distribution P (R) over reward functions R for an MDP (S, A, T, γ), the loss function L p policy (R, π) is minimized for all p by π * M , the optimal policy for the Markov Decision Problem M = (S, A, T, γ, E P [R]). Proof. From the Bellman equations (1) we can derive the following: V π (R) = (I − γT π ) −1 R (4) where T π is the |S|×|S| transition matrix for policy π. Thus, for a state s ∈ S and fixed π, the value function is a linear function of the rewards: V π (s, R) = w(s, π) • R where w(s, π) is the s'th row of the coefficient matrix (I − γT π ) −1 in (4). Suppose we wish to maximize E[V π (s, R)] alone. Then, max π E[V π (s, R)] = max π E[w(s, π)•R] = max π w(s, π)•E[R] By definition this is equal to V * M (s), the optimum value function for M , and the maximizing policy π is π * M , the optimal policy for M . Thus for all states s ∈ S, E[V π (s, R)] is maximum at π = π * M . But V * (s, R) ≥ V π (s, R) for all s ∈ S, reward functions R, and policies π. Therefore E[L p policy (π)] = E( V * (R) − V π (R) p ) is minimized for all p by π = π * M . So, instead of trying a difficult direct minimization of the expected policy loss, we can find the optimal policy for the mean reward function, which gives the same answer.
 Sampling and Rapid Convergence We have seen that both reward learning and apprenticeship learning require the mean of the posterior distribution. However the posterior is complex and analytical derivation of the mean is hard, even for the simplest case of the uniform prior. Instead, we generate samples from these distributions and then return the sample mean as our estimate of the true mean of the distribution. The sampling technique we use is an MCMC algorithm GridWalk (see  [Vempala, 2005] ) that generates a Markov chain on the intersection points of a grid of length δ in the region R |S| (denoted R |S| /δ). However, computing the posterior distribution at a particular point R requires calculation of the optimal Q-function for R, an expensive operation. Therefore, we use a modified version of GridWalk called PolicyWalk (Figure  3 ) that is more efficient: While moving along a Markov chain, the sampler also keeps track of the optimal policy π for the current reward vector R. Observe that when π is known, the Q function can be reduced to a linear function of the reward variables, similar to equation 4. Thus step 3b can be performed efficiently. A change in the optimal policy can easily be detected when moving to the next reward vector in the chain R, because then for some (s, a) ∈ (S, A), Q π (s, π(s), R) < Q π (s, a, R) by Theorem 1. When this happens, the new optimal policy is usually only slightly different from the old one and can be computed by just a few  of R in R |S| /δ. (b) Compute Q π (s, a, R) for all (s, a) ∈ S, A. (c) If ∃(s, a) ∈ (S, A), Q π (s, π(s), R) < Q π (s, a, R) i. π := PolicyIteration(M, R, π) ii. Set R := R and π := π with probability min{1, P ( R,π) P (R,π) } Else i. Set R := R with probability min{1, P ( R,π) P (R,π) } 4. Return R Figure  3 : PolicyWalk Sampling Algorithm steps of policy iteration (see  [Sutton and Barto, 1998] ) starting from the old policy π. Hence, PolicyWalk is a correct and efficient sampling procedure. Note that the asymptotic memory complexity is the same as for GridWalk. The second concern for the MCMC algorithm is the speed of convergence of the Markov chain to the equilibrium distribution. The ideal Markov chain is rapidly mixing (i.e. the number of steps taken to reach equilibrium is polynomially bounded), but theoretical proofs of rapid mixing are rare. We will show that in the special case of the uniform prior, the Markov chain for our posterior (3) is rapidly mixing using the following result from [Applegate and  Kannan, 1993]  that bounds the mixing time of Markov chains for pseudo-logconcave functions. Lemma 1. Let F (•) be a positive real valued function defined on the cube {x ∈ R n | − d ≤ x i ≤ d} for some positive d, satisfying for all λ ∈ [0, 1] and some α, β |f (x) − f (y)| ≤ α x − y ∞ and f (λx + (1 − λ)y) ≥ λf (x) + (1 − λ)f (y) − β where f (x) = log F (x). Then the Markov chain induced by GridWalk (and hence PolicyWalk) on F rapidly mixes to within of F in O(n 2 d 2 α 2 e 2β log 1 ) steps. Proof. See [Applegate and  Kannan, 1993] . Theorem 4. Given an MDP M = (S, A, T, γ) with |S| = N , and a distribution over rewards P (R) = P r X (R|O X ) defined by (3) with uniform prior P R over C = {R ∈ R n | − R max ≤ R i ≤ R max }. If R max = O(1/N ) then P can be efficiently sampled (within error ) in O(N 2 log 1/ ) steps by algorithm PolicyWalk. Proof. Since the uniform prior is the same for all points R, we can ignore it for sampling purposes along with the normalizing constant. Therefore, let f (R) = α X E(O X , R). Now choose some arbitrary policy π and let Note that f π is a linear function of R and f f π (R) = α X i Q π (s, a i , R) IJCAI-07 (R) ≥ f π (R), for all R ∈ C. Also we have, max s,a Q * (s, a) = max s,a,π Q π (s, a) = max s,π V π max (s) ≤ R max 1 − γ Similarly, min s,a Q * (s, a) ≥ − Rmax 1−γ . Therefore, f (R) ≤ αX NRmax 1−γ and f π (R) ≥ − αX NRmax 1−γ and hence f π (R) ≥ f (R) − 2α X N R max 1 − γ So for all R 1 , R 2 ∈ C and λ ∈ [0, 1], f (λR 1 + (1 − λ)R 2 ) ≥ f π (λR 1 + (1 − λ)R 2 ) ≥ λf π (R 1 ) + (1 − λ)f π (R 2 ) ≥ λf (R 1 ) + (1 − λ)f (R 2 ) − 2α X N R max 1 − γ Therefore, f satisfies the conditions of Lemma 1 with β = 2αX NRmax 1−γ = 2N • O( 1 N ) 1−γ = O(1) and α = |f (R 1 ) − f (R 2 )| R 1 − R 2 ∞ ≤ 2α X N R max (1 − γ)O( 1 N ) = O(N ) Hence the Markov chain induced by the GridWalk algorithm (and the PolicyWalk algorithm) on P mixes rapidly to within of P in a number of steps equal to O(N 2 1 N 2 N 2 e O(1) log 1/ ) = O(N 2 log 1/ ). Note that having R max = O(1/N ) is not really a restriction because we can rescale the rewards by a constant factor k after computing the mean without changing the optimal policy and all the value functions and Q functions get scaled by k as well.
 Experiments We compared the performance of our BIRL approach to the IRL algorithm of  [Ng and Russell, 2000]  experimentally. First, we generated random MDPs with N states (with N varying from 10 to 1000) and rewards drawn from i.i.d. Gaussian priors. Then, we simulated two kinds of agents on these MDPs and used their trajectories as input: The first learned a policy by Q-learning on the MDP + reward function. The learning rate was controlled so that the agent was not allowed to converge to the optimal policy but came reasonably close. The second agent executed a policy that maximized the expected total reward over the next k steps (k was chosen to be slightly below the horizon time). For BIRL, we used PolicyWalk to sample the posterior distribution (3) with a uniform prior. We compared the results of the two methods by their average 2 distance from the true reward function (Figure  4 ) and the policy loss with 1 norm (Figure  5 ) of the learned policy under the true reward. Both measures show substantial improvement. Note that we have used a logarithmic scale on the x-axis. We also measured the accuracy of our posterior distribution for small N by comparing it with the true distribution of rewards i.e. the set of generated rewards that gave rise to the same trajectory by the expert. In Figure  6 , we show scatter plots of some rewards sampled from the posterior and the true distribution for a 16-state MDP. These figures show that the posterior is very close to the true distribution.
 From Domain Knowledge to Prior To show how domain knowledge about a problem can be incorporated into the IRL formulation as an informative prior, IJCAI-07 we applied our methods to learning reward functions in adventure games. There, an agent explores a dungeon, seeking to collect various items of treasure and avoid obstacles such as guards or traps. The state space is represented by an m-dimensional binary feature vector indicating the position of the agent and the value of various fluents such as hasKey and doorLocked. If we view the state-space as an m-dimensional lattice L S , we see that neighbouring states in L S are likely to have correlated rewards (e.g. the value of doorLocked does not matter when the treasure chest is picked up). To model this, we use an Ising prior (see  [Cipra, 1987] ): P R (R) = 1 Z exp(−J (s ,s)∈N R(s)R(s ) − H s R(s)) where N is the set of neighbouring pairs of states in L S and J and H are the coupling and magnetization parameters. We tested our hypothesis by generating some adventure games (by populating dungeons with objects from a common sense knowledge base) and testing the performance of BIRL with the Ising prior versus the baseline uninformed priors. The results are in figure  7  and show that the Ising prior does significantly better.
 Related Work The initial work on IRL was done by  [Ng and Russell, 2000]  while  [Abbeel and Ng, 2004]  studied the special case where rewards can be represented as linear combinations of features of the state space and gave a max-margin characterization of the optimal reward function.  [Price and Boutilier, 2003]  discusses a Bayesian approach to imitating the actions of a mentor during reinforcement learning whereas the traditional literature on apprenticeship learning tries to mimic the behaviour of the expert directly  [Atkeson and Schaal, 1997] . Outside of computer science, IRL-related problems have been studied in various guises. In the physical sciences, there is a body of work on inverse problem theory, i.e. infering values of model parameters from observations of a physical system  [Tarantola, 2005] . In control theory,  [Boyd et al., 1994]  solved the problem, posed by Kalman, of recovering the objective function for a deterministic linear system with quadratic costs.
 Conclusions and Future Work Our work shows that improved solutions can be found for IRL by posing the problem as a Bayesian learning task. We provided a theoretical framework and tractable algorithms for Bayesian IRL and our solutions contain more information about the reward structure than other methods. Our experiments verify that our solutions are close to the true reward functions and yield good policies for apprenticeship learning. There are a few open questions remaining: 1. Are there more informative priors that we can construct for specific IRL problems using background knowledge? 2. How well does IRL generalize? Suppose the transition function of the actor and the learner differed, how robust would the reward function or policy learned from the actor be, w.r.t the learner's state space? Figure 4: Reward Loss.
 Figure 6 : 6 Figure 5: Policy Loss.
 Figure 7 : 7 Figure 7: Ising versus Uninformed Priors for Adventure Games
 If the underlying MDP represented a planning-type problem, we expect most states to have low (or negative) rewards but a few states to have high rewards (corresponding to the goal); this can be modeled by a Beta distribution for the reward at each state, which has modes at high and low ends of the reward space: P Laplace (R(s) = r) = 1 2σ e − |r| 2σ , ∀s ∈ S 3. 2 2σ 2 , ∀s ∈ S on αX as well(Fig 2). But it will be simpler to treat αX as just a parameter of the distribution.
 Algorithm PolicyWalk(Distribution P , MDP M, Step Size δ )1. Pick a random reward vector R ∈ R |S| /δ. 2. π := PolicyIteration(M, R) 3. Repeat (a) Pick a reward vector R uniformly at random from the neighbours
			 Note that the probabilities of the evidence should be conditioned IJCAI-07