# Explanations for Neural Networks by Neural Networks
###### Abstract  
Understanding the function learned by a neural network is crucial in many domains, e.g., to detect a model's adaption to concept drift in online learning. Existing global surrogate model approaches generate explanations by maximizing the fidelity between the neural network and a surrogate model on a sample-basis, which can be very time-consuming. Therefore, these approaches are not applicable in scenarios where timely or frequent explanations are required. In this paper, we introduce a real-time approach for generating a symbolic representation of the function learned by a neural network. Our idea is to generate explanations via another neural network (called the _Interpretation Network_, or \(\mathcal{I}\)-Net), which maps network parameters to a symbolic representation of the network function. We show that the training of an \(\mathcal{I}\)-Net for a family of functions can be performed up-front and subsequent generation of an explanation only requires querying the \(\mathcal{I}\)-Net once, which is computationally very efficient and does not require training data. We empirically evaluate our approach for the case of low-order polynomials as explanations, and show that it achieves competitive results for various data and function complexities. To the best of our knowledge, this is the first approach that attempts to _learn_ mapping from neural networks to symbolic representations.  
Explainable AI (XAI); interpretability; explainability; neural networks; machine learning; symbolic representations +
Footnote †: journal: Applied  
underlying our approach is to use a neural network (called _Interpretation Network_, or \(\mathcal{I}\)-Net) which maps the parameters of the network \(\lambda\) to a corresponding symbolic representation (e.g., a low-order polynomial). Accordingly, we transfer the task of interpreting neural networks to a machine learning problem, which we solve using neural networks themselves. The \(\mathcal{I}\)-Net can be trained up-front for a family of functions, so that generating a symbolic expression for a given network \(\lambda\) only requires querying the \(\mathcal{I}\)-Net once. This way, our approach is applicable when real-time interpretations are required (e.g., in an online learning scenario) or when the training data are either not available anymore or cannot be shared (e.g., personal data). We demonstrate our approach on the example of low-order polynomials, which can approximate a wide range of functions and can efficiently be learned with neural networks, as shown by [PERSON] et al. [8]. Using low-order polynomials as explanations allows reading of the influence of variables (and interactions of variables) directly from the corresponding coefficients. This makes polynomials interpretable comprehensible functions, allowing interpretability on a modular level. Even if we focus on polynomials in this paper, the approach is general by design and can be applied to arbitrary functions with symbolic representations, e.g., Boolean functions or decision trees.  
This paper has four contributions:  
1. We introduce the general framework of _Interpretation Networks_ (\(\mathcal{I}\)-Nets) as means of learning the mapping from a neural network to a symbolic representation of the network function;
2. We show how \(\mathcal{I}\)-Nets can be trained up-front, without requiring access to training data or querying the target network (Section 2);
3. We propose a specific instance of this framework, where the symbolic representations are low-order polynomials (Section 3). Therefore, this instance allows explanations for models where sparse polynomials offer a reasonable explanation;
4. We empirically evaluate our approach against symbolic regression [6], showing that it achieves competitive results without time-consuming optimizations for each specific generated explanation (Section 4).