The Role of Explanation in Algorithmic Trust ∗ Finale Doshi-Velez Ryan Budish Mason Kortz

Our research focuses on the role of explanations in preventing errors and increasing trust in algorithmic decision-making. The question of when and what kind of explanation might be required of algorithmic decision-making systems is urgent: details about a potential “right to explanation” were debated in the most recent revision of the European Union’s General Data Protection Regulation (GDPR) [1, 2]. While the ultimate version of the GDPR only requires explanation in very limited contexts, we expect questions about explanations to be important in future regulation of algorithmic decision-making systems. In particular, there exist concerns that explainable algorithmic decision-making systems would be so difficult to engineer that legally requiring explanation would stifle innovation; that explanation would come at the price of some other performance objective, such as decreased system accuracy; and that explanation would force trade secrets being revealed [3, 4]. For the purposes of this whitepaper, when we discuss explanations, we shall mean ex post humaninterpretable descriptions of the process by which a decision-maker took a particular set of inputs and reached a particular conclusion [2]. When it comes to human decision-makers, we often want an explanation when someone makes a decision we do not understand or believe to be suboptimal [5]. For example, was the conclusion accidental or intentional? Was it caused by incorrect information or faulty reasoning? The answers to these questions permit us to weigh our trust in the decision-maker and to assign blame in case of a dispute. But although there are many circumstances in which we might want an explanation, or even feel we are owed one, the circumstances in which a decision-maker is legally obligated to provide an explanation are more limited. Accordingly, we examine why explanations for some human decisions, but not others, are required by law and whether the same criteria can be applied to explanations for algorithmic decisions. There are two primary reasons there is no general legal right to an explanation from a human decisionmaker. First, generating explanations takes effort that could be spent on other productive endeavors. Second, requiring an explanation for a decision may negatively affect the decision-making process. For example, in the U.S. legal system, juries are not required to explain their decisions, in part because public accountability could bias jurors in favor of making popular but legally incorrect decisions. [6]. Accordingly, legal requirements to generate an explanation are limited to situations where there is social utility to knowing how a decision was made. Specifically, there must be some value to knowing if the decision was made erroneously, for example, to provide compensation for a past harm or to improve future decision-making [7]. When, then, is explanation currently required? In most cases, before any legal right to an explanation can be invoked, there must be some evidence that an error occurred in the decision-making process. The amount and type of evidence required to trigger a right to an explanation–what in the legal system is called meeting the burden of production–is based on a number of factors. One is the significance of the decision. In most racial or gender discrimination litigation, the plaintiff must plausibly allege that the decision-making process was intentionally biased before the defendant is required to present a competing explanation [8]. But in certain circumstances, such as criminal jury selection, employment, or access to housing, showing that the results of the decision disproportionately exclude a particular race or gender is enough to shift the burden of explanation on the decision-maker [9, 10]. Criminal sentences, commonly regarded as one of the most important decisions a court can make, must be accompanied by an explanation even where there is no evidence of impropriety—although the level of detail required varies between jurisdictions [11]. There are other reasons an explanation may or may not be legally required in a given case. If a person