# Explanations for Neural Networks by Neural Networks
## 2 The \(\mathcal{I}\)-Net Approach  
### Explanations for Neural Networks  
We start by formalizing the task of generating symbolic representations (explanations) from black-box models, i.e., neural networks. A neural network represents a function \(\lambda:X\to Y\). Our goal is to obtain a function \(g:X\to Y\), such that \(g\) approximates \(\lambda\), i.e., \(\forall\mathbf{x}\in X:\lambda(\mathbf{x})\approx g(\mathbf{x})\). More specifically, we are interested in functions \(g\) that have a simple, _interpretable_ representation. There is no universal agreement about what constitutes an interpretable function [9]. A usual requirement is that \(g\) should have substantially fewer parameters and interdependencies than \(\lambda\)[2]. In Section 3, we propose to use sparse polynomials as function family of \(g\).  
For this paper, it is useful to explicitly distinguish between the functions \(\lambda\) and \(g\), and their _representations_\(\theta_{\lambda}\in\Theta_{\lambda}\) and \(\theta_{g}\in\Theta_{g}\). We can think of \(\theta_{\lambda}\) and \(\theta_{g}\) as parameters of \(\lambda\) and \(g\), respectively. For example, when \(\lambda\) is a neural network, \(\theta_{\lambda}\) is the vector of all weights and biases of \(\lambda\), in some systematic order that allows reconstructing the function \(\lambda\) from \(\theta_{\lambda}\). Similarly, \(\theta_{g}\) is the parameter vector for the symbolic model, e.g., coefficients of sparse polynomials, as we will introduce in Section 3.  
The process of generating explanations can be formalized as a function \(\mathcal{I}:\Theta_{\lambda}\to\Theta_{g}\) that maps representations of \(\lambda\) to representations of \(g\). We are interested in functions \(\mathcal{I}\) with a high fidelity to the neural network, i.e., where \(\forall\mathbf{x}:\lambda(\mathbf{x})\approx g(\mathbf{x})\). Existing approaches for generating symbolic explanations implement \(\mathcal{I}\) via a sample-based procedure. They generate a set of samples \(\{\mathbf{x}_{i},\lambda(\mathbf{x}_{i})\}_{i=1}^{M}\) from \(\lambda\) (where \(X\) usually constitutes the data used for the training of \(\lambda\)), and then fit a function \(g\) to those samples. This procedure must be repeated for each \(\theta_{\lambda}\in\Theta_{\lambda}\) independently.  
Symbolic regression [6] uses genetic programming to find a symbolic expression that maximizes the fidelity of \(g\) based on a set of data points. Accordingly, for interpretability purposes, we can apply the algorithm to the predictions of \(\lambda\), assuming we have query access. Therefore, the functional form of the expressions has to be selected in terms of the allowed operations prior to application. In contrast, symbolic metamodeling [5] does not require predefining the functional form of the expressions. Instead, it uses Meijer G-functions [10] and their parameterization, which allows uncovering infinitely many functional forms (for instance arithmetic, polynomial, algebraic, and analytic expressions) based on a fixed-dimensional parameter space [5]. Furthermore, the corresponding metamodels can be optimized using gradient descent.  
While these approaches can lead to concise and accurate explanations in \(g\), they can be computationally very expensive due to the large number of required samples and difficult, time-consuming optimization of the parameters of \(g\). Furthermore, they require the network function \(\lambda\) to start the optimization process which precludes real-time explanations. Specifically, in an online learning situation, it can be necessary to generate explanations in real time, making existing methods become infeasible.