Title: The Myth of the Box: Causation and Comprehensibility in Neural Network Behavior
Primary area: interpretability and explainable AI
Keywords: black box, opacity, explanable AI, causation, correlation, feature extraction

Abstract: Because neural network architectures defy attempts to trace the individual processes responsible for any given, particular output, deep learning systems that rely on these architectures are often characterized as “black boxes”. The supposition that there are inaccessible features of the internal process that are responsible for individual behaviors deeply informs our understanding of, our trust in, and our experimentation with these systems. This supposition, however, rests on an assumption about the nature of causation in systems: namely, that if some particular past feature of a system is causally responsible for a present feature, an intermediary correlate must in principle be able to be individuated in the system. This assumption is false. The consequent model of deep learning systems, wherein explanations of behavior are intrinsically partial, yet complete and without remainder, has conceptual repercussions for various discussions surrounding explainable artificial intelligence.

TL;DR: Characterizing neural network systems as "black boxes" is based on an unwarranted assumption about causation and individuation.