Explaining Explanation For “Explainable Ai”

What makes for an explanation of "black box" AI systems such as Deep Nets? We reviewed the pertinent literatures on explanation and derived key ideas. This set the stage for our empirical inquiries, which include conceptual cognitive modeling, the analysis of a corpus of cases of "naturalistic explanation" of computational systems, computational cognitive modeling, and the development of measures for performance evaluation. The purpose of our work is to contribute to the program of research on “Explainable AI.” In this report we focus on our initial synthetic modeling activities and the development of measures for the evaluation of explainability in human-machine work systems.  The importance of explanation in AI has been emphasized in the popular press, with considerable discussion of the explainability of Deep Nets and Machine Learning systems (e.g., Kuang, 2017). For such “black box” systems, there is a need to explain how they work so that users and decision makers can develop appropriate trust and reliance. As an example, referencing Figure 1, a Deep Net that we created was trained to recognize types of tools.  Some examples of Deep Net classification. Outlining the axe and overlaying bird silhouettes on it resulted in a confident misclassification. While a fuzzy hammer is correctly classified, an embossed rendering is classified as a saw. Deep Nets can classify with high hit rates for images that fall within the variation of their training sets, but are nonetheless easily spoofed using instances that humans find easy to classify. Furthermore, Deep Nets have to provide some classification for an input. Thus, a Volkswagen might be classified as a tulip by a Deep Net trained to recognize types of flowers. So, if Deep Nets do not actually possess human-semantic concepts (e.g., that axes have things that humans call "blades"), what do the Deep Nets actually "see"? And more directly, how can users be enabled to develop appropriate trust and reliance on these AI systems? Articles in the popular press highlight the successes of Deep Nets (e.g., the discovery of planetary systems in Hubble Telescope data; Temming 2018), and promise diverse applications "... the recognition of faces, handwriting, speech... navigation and control of autonomous vehicles... it seems that neural networks are being used everywhere" (Lucky, 2018, p. 24). And yet "models are more complex and less interpretable than ever... Justifying [their] decisions will only become more crucial" (Biran and Cotton, 2017, p. 4). Indeed, a proposed regulation before the European Union (Goodman and Flaxman, 2016) asserts that users have the "right to an explanation.” What form must an explanation for Deep Nets take? This is a challenge in the DARPA "Explainable AI" (XAI) Program: To develop AI systems that can engage users in a process in which the mechanisms and "decisions" of the AI are explained. Our tasks on the Program are to: (1). Integrate philosophical studies and psychological research in order to identify consensus points, key concepts and key variables of explanatory reasoning, (2). Develop and validate measures of explanation goodness, explanation satisfaction, mental models and human-XAI performance, (3) Develop and evaluate a computational model of how people understand computational devices, and C op yr ig ht 2 01 8 by H um an F ac to rs a nd E rg on om ic s So ci et y. D O I 1 0. 11 77 /1 54 19 31 21 86 21 04 7 Proceedings of the Human Factors and Ergonomics Society 2018 Annual Meeting 197