## 2 The \(\mathcal{I}\)-Net Approach
### Explanations for Neural Networks by Neural Networks  
To overcome this problem, we propose to implement \(\mathcal{I}\) as a neural network. This concept is visualized in Figure 1. Thus, we can train \(\mathcal{I}\) up-front, so that generating an explanation \(g\) only requires querying the \(\mathcal{I}\)-Net once, which is possible in (close to) real-time. As described before, our goal is to obtain an \(\mathcal{I}\)-Net that computes \(\theta_{g}\) with \(\forall\mathbf{x}:\lambda(\mathbf{x})\approx g(\mathbf{x})\). More precisely, we can use a distance measure, such as the mean absolute error (MAE) over a set of sample points \(\{\mathbf{x}_{i}\}_{i=1}^{M}\) to quantify the agreement (the fidelity) between \(\lambda\) and \(g\):  
\[\text{MAE}(\theta_{\lambda},\theta_{g})=\frac{1}{M}\sum_{i=1}^{M}|\lambda( \mathbf{x}^{(i)})-g(\mathbf{x}^{(i)})|\]  
Given a set of network parameters \(\Theta_{\lambda}=\{\theta_{\lambda}^{(i)}\}_{i=1}^{N}\), the loss function of the \(\mathcal{I}\)-Net can be computed as  
\[\mathcal{L}_{\mathcal{I}}=\frac{1}{|\Theta_{\lambda}|}\sum_{\theta_{\lambda} \in\Theta_{\lambda}}\text{MAE}(\theta_{\lambda},\mathcal{I}(\theta_{\lambda})).\]  
Note that training an \(\mathcal{I}\)-Net does not require any supervision of the desired network output \(\theta_{g}\), but instead we only measure the fidelity between network input \(\lambda\) and network output \(g\). Since both the polynomial function \(g\) and the neural network function \(\lambda\), which are required in the loss, are differentiable, we can ensure an efficient computation. Here, we focus on generating explanations for fixed function families. That is, each \(\mathcal{I}\)-Net is learned based on networks \(\lambda\), which were trained using functions belonging to  
Figure 1: Overview of the \(\mathcal{I}\)-Net translation approach. The neural network parameters as input are translated into a symbolic representation of a mathematical function. the same family, e.g., polynomials of fixed dimensionality. For training such an \(\mathcal{I}\)-Net, we require a set \(\Theta_{\lambda}\), where each \(\theta_{\lambda}\in\Theta_{\lambda}\) represents a neural network trained on that function family. Fortunately, it is not necessary to use real data for training. Instead, it is sufficient to sample a set of functions from the family of interest and train networks \(\lambda\) to approximate these functions. Interestingly, as we will see later, \(\mathcal{I}\)-Nets trained on such samples can still generate accurate explanations for networks that were _not_ trained explicitly on the same function family (see Section 4.2.3). More specifically, training data for the \(\mathcal{I}\)-Net can be created in three steps:  
1. Sample a set of parameter vectors \(\theta_{\mathcal{S}^{*}}\);
2. For each \(\theta_{\mathcal{S}^{*}}\), generate a set of input-output pairs \(\{(x,g^{*}(x))\}_{i=1}^{M}\);
3. For each \(\theta_{\mathcal{S}^{*}}\), train a network \(\lambda\) on \(\{(x,g^{*}(x))\}_{i=1}^{M}\)  
Here, \(g^{*}\) represents the _target function_ of a network \(\lambda\). We explicitly distinguish between \(g^{*}\) and \(g\), since our goal is finding a function \(g\) that approximates what the neural network has actually learned (i.e., the network function) and not the target function \(g^{*}\) (i.e., what the neural network should learn). A specific example of this procedure for the case of sparse polynomials is shown in Algorithm 1. Note that sampling is only necessary during generation of \(\Theta_{\lambda}\) and in the loss function of the \(\mathcal{I}\)-Net, but not when generating explanations using the \(\mathcal{I}\)-Net. This has a number of advantages compared to conventional, sample-based algorithms. First, we can generate explanations in (close to) real-time, because generating explanations only requires querying the \(\mathcal{I}\)-Net once instead of performing a costly optimization, and the \(\mathcal{I}\)-Net can be trained up-front, even without knowing the network function \(\lambda\).