Is It Time to Get Rid of Black Boxes and Cultivate Trust in AI?

W a short time, artificial intelligence (AI) has taken center stage in radiology, with a large number of publications that is growing at a swift pace. Radiologists routinely assess medical images and report findings to detect, characterize, and monitor diseases. Such assessment is based both on education and experience and can be, at times, subjective. To augment such qualitative interpretation, AI excels at recognizing complex patterns in imaging data and can offer novel tools for quantitative medical image interpretation in an automated fashion. Deep learning, in particular, is a subset of AI that utilizes neural network architectures loosely inspired by the human brain. Although most of the earlier AI methods led to applications with subhuman performance, deep learning has recently demonstrated performance that matches and often surpasses humans (1–3). Recognizing its power, medical researchers and scientists have been exploring the role of deep learning methods in a variety of radiologic applications (4), including early detection, risk prediction, prognosis assessment, treatment planning, as well as lesion and organ segmentation, for various diseases. However, AI models are often considered “black boxes”; although we may be able to get accurate decisions and predictions, we cannot easily or clearly explain or interpret the logic behind an AI model’s outputs. But how can we gain useful insights? Which aspects of an AI model should we strive to elucidate, and what are the computational tools we need to achieve that? These questions are important to address when the issue of AI model interpretability or explainability is raised. Although interpretability can be a broad, poorly defined concept, in AI it can be thought of as the ability to explain or to present in understandable terms to a human the cause of a model’s decision. The more interpretable or explainable an AI model is, the easier it is for someone to comprehend why certain decisions or predictions have been made. The concept of interpretability in AI is not new. Inherently interpretable AI models, such as decision trees and decision rules, have received a great deal of attention from the research community. Moreover, several methods for feature importance analysis have been developed to offer insights into what the model is learning and what factors might be relevant. However, in the case of deep learning, the situation is more complicated. A key difference is that traditional AI systems primarily rely on predefined, engineered (ie, handcrafted) features based on expert knowledge. In contrast, deep learning allows the algorithm to learn discriminative features from data automatically, offering the ability to approximate very complex nonlinear relationships through several layers and neurons. Therefore, deep learning interpretability becomes even more challenging, as we need not only to explain the relationship between features and decisions or predictions, which usually involves a vast number of parameters, but also to understand the learned features themselves. In this issue of Radiology: Artificial Intelligence, Reyes et al address this important topic by reviewing computational approaches to interpret deep learning models, and discuss gaps and challenges that need to be addressed, focusing primarily on radiologic applications and their translation into clinical practice (6). The authors are to be commended for the comprehensive summary of existing interpretability technologies, as well as for the intuitive, yet detailed, descriptions of such sophisticated computational tools. Reyes and colleagues have categorized various interpretability methods, provided explanatory figures, demonstrated related applications, and offered a thorough discussion on the advantages and limitations. If a deep learning model performs well, why shouldn’t we simply trust the model and ignore why it made a specific decision? A lot of this has to do with the impact that Is It Time to Get Rid of Black Boxes and Cultivate Trust in AI?