# Explanations for Neural Networks by Neural Networks
## 4 Evaluation
### Experimental Results
#### 4.2.2 Runtime and Performance Evaluation  
The goal of this experiment was to compare our approach with symbolic regression based on the runtime, as well as the fidelity between the predicted symbolic expression and the corresponding \(\lambda\)-Net.  
Figure 5 shows the time required for generating an explanation for a single network. At the smallest setting (\(n=5\)), symbolic regression needs approximately 160 s for the interpretation. There is no notable increase in the runtime with an increasing number of variables. This can again be explained by the parsimony_coefficient, which restricts the complexity resulting in functions with similar complexity even when the dataset complexity and the number of variables increases (see Section 4.2.1 for a more in-depth evaluation of the function complexity). However, if more complex functions are considered within symbolic regression, the runtime increases significantly, especially for high-dimensional learning problems. Using \(\mathcal{I}\)-Nets, the training process can be performed up-front, and interpretations can be generated in close to real-time (\(\sim\)0.003 s), regardless of the number of variables and function complexity.  
Figure 6 shows the error of the explanation in terms of MAE between function values of the \(\lambda\)-Net and the corresponding explanation \(g\) computed by the \(\mathcal{I}\)-Net.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Number of Variables** & \(\mathcal{I}\)**-Net** & **Symbolic Regression** \\ \hline \multirow{7}{*}{Function **(Fidelity)**} & \multirow{7}{*}{\(n=10\)} & \(3.1812x^{4}-3.1752x^{3}-0.0694\) & \(-0.5060x\) \\  & & **(MAE: 0.0220)** & **(MAE: 0.0566)** \\  & & \(-0.0749x_{0}x_{1}x_{4}+0.5084x_{1}x_{2}-0.2185x_{1}x_{4}-\) & \\  & & \(0.432x_{2}x_{4}^{2}+0.0939x_{3}^{2}x_{4}\) & \(0.2180x_{2}^{2}+0.4360x_{1}x_{2}-0.6540x_{4}x_{2}\) \\  & & **(MAE: 0.0068)** & **(MAE: 0.0373)** \\  & & \(0.1837x_{0}^{2}x_{3}-0.2915x_{0}x_{3}x_{4}-0.4966x_{0}x_{5}x_{6}-\) & \\  & & \(0.3588x_{1}^{2}+0.0340x_{1}x_{3}x_{8}-0.1769x_{2}x_{3}+\) & \\  & & \(0.1238x_{3}^{2}-0.0746x_{3}x_{6}x_{7}-0.3337x_{6}^{2}-\) & \\  & & \(0.3666x_{6}x_{8}\) & **(MAE: 0.1371)** \\  & & **(MAE: 0.1436)** & \\  & & \(0.0722x_{0}^{3}+0.6574x_{0}x_{2}x_{8}+0.0555x_{0}x_{7}x_{13}+\) & \\  & & \(0.0764x_{10}x_{10}+0.1436x_{1}^{2}-0.0359x_{2}x_{3}-\) & \\  & & \(0.1764x_{2}x_{3}x_{13}+0.2182x_{2}x_{13}x_{14}+\) & \(1.0000x_{3}x_{8}+1.0000x_{1}x_{5}x_{7}x_{11}+\) \\ \(n=15\) & & \(0.5176x_{3}x_{11}+0.0369x_{4}+0.165x_{5}x_{6}x_{7}x_{12}-\)When comparing the mean errors of both approaches, we can observe that symbolic regression achieves a higher fidelity to the neural network, especially for \(n=5\) and \(n=10\). For \(n=15\), however, the error difference diminishes. Additionally, we can see that the standard deviation of the error for symbolic regression is low, while \(\mathcal{I}\)-Nets have a comparatively higher standard deviation. While there are cases in which the performance of \(\mathcal{I}\)-Nets is superior to symbolic regression, there are also networks where the \(\mathcal{I}\)-Net is not able to make reasonable predictions, resulting in high errors that lower the mean performance of \(\mathcal{I}\)-Nets. We suspect that a more sophisticated architecture, a more in-depth hyperparameter optimization, and a larger training data set with higher variety would allow increased the performance of \(\mathcal{I}\)-Nets, subject to further work.  
Furthermore, optimization for symbolic regression was conducted on only 5000 samples, which can be considered to be very low compared to most real-world tasks. However, there was no significant improvement in performance for a larger sample size in our evaluation, which would justify the use of a larger set of samples. This can be traced back to the fact that the underlying network function is close to a polynomial function, which can already be approximated well with a small number of samples. However, this assumption may not hold in other scenarios.