# _Applied Sciences_
## 4 Discussion
### Reliability  
Good performance on the reliability metric indicates that, for the target distribution, a high proportion of instances covered by the explanation will receive the same classification from the black box model as was given to the explanandum. At the same time, the end user can be certain that the rule does not cover a trivially small region of the input space.  
A cursory visual inspection of the precision (Figure 5) demonstrates the trade-off between precision and coverage. The BRL, inTrees and defragTrees methods that had scored relatively well for coverage do not deliver state-of-the-art precision on any data set. Both precision and reliability (Figure 6) score distributions show that Anchors and gbt-HIPS vie for first position over almost all of the data sets. Anchors appears to have a slight advantage for precision while gbt-HIPS appears to do better for reliability. The placement is often so close that it requires recourse to the tabulated results (supplementary materials) and the significance tests to be certain of the leading method.  
The results of hypothesis tests of the pairwise comparisons for the top two ranking methods are shown in Table 6. The tests seem to show that Anchors is leading for reliability on three out of the nine data sets, joint first (no significant difference between first and second place) on a further two data sets, and second on a further two data sets. gbt-HIPS appears to be the second place method, leading on two data sets, and joint first on a further three data sets. These results, it seems, are inconsistent with the tabulated and visualised mean scores for reliability.  
These inconsistencies are, unfortunately, an artefact of the choice of significance test, which is non-parametric and, therefore, insensitive to outliers. Specifically, the long tail of under-fitting instances visible as colour saturated dots in the lower parts of each facet of Figure 6. For gbt-HIPS, instead almost the entire set of scores occupies a narrow band near to the upper bound for reliability. It is for this reason that the reliability floor statistic (Figure 7) is so enlightening. The reliability floor quantifies the propensity to over-fit by measuring the proportion of explanations in the test set that scored above the threshold. Over-fitting explanations are too granular and cover too few instances. Furthermore, a significant number of explanations score zero, demonstrating a critical over-fit. That is, an explanation that covers only the explanandum but not a single instance in the held out  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Data** & **1 st** & **Mrnk** & **2 nd** & **Mrnk** & **N** & \(z\) & \(p\)**Value** \\ \hline adult & **Anchors** & 2.19 & defragTrees & 2.62 & 1000 & 19.4 & \(\approx\)0 \\ bank & **Anchors** & 2.23 & LORE & 2.83 & 1000 & 26.3 & \(\approx\)0 \\ car & Anchors & 2.60 & gbt-HIPS & 2.61 & 517 & 0.1 & 0.4266 \\ cardio & **gbt-HIPS** & 1.95 & defragTrees & 2.58 & 637 & 7.04 & \(<\)0.00417 \\ credit & Anchors & 1.69 & gbt-HIPS & 1.94 & 206 & 1.4 & 0.00865 \\ german & gbt-HIPS & 2.18 & Anchors & 2.56 & 299 & 2.5 & 0.00671 \\ lending & **gbt-HIPS** & 1.23 & Anchors & 2.18 & 631 & 13.1 & \(\approx\)0 \\ nursery & **BRL** & 2.28 & Anchors & 2.61 & 1000 & 9.21 & \(\approx\)0 \\ rcdv & **Anchors** & 1.74 & LORE & 1.89 & 1000 & 4.3 & \(<\)0.00417 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Reliability significance tests: top two methods by mean rank (mrnk) for the stochastic GBT model.  
Figure 6: Distribution of reliability for stochastic GBT model explanations.  
set. gbt-HIPS leads on all nine data sets for reliability floor. The reliability floor scores are presented visually in Figure 7 and tabulated in the supplementary materials.