# _Applied Sciences_
## 3 Materials and Methods
### Quantitative Study  
There is a very strong case that coverage and precision are not appropriate quality metrics for explanations-based research [31; 32]. Coverage is trivially maximised by critically under-fitting solutions. For example, the _null_ rule \(\mathcal{X}\implies g(\mathbf{x})\) (all inputs result in the given output) is critically under-fitting, yet scores 1.0 for coverage. Precision, conversely, is trivially maximised by critically over-fitting solutions. For example, the \"tautological\" rule \(\{\textit{feature}_{1}=x_{1},\dots,\textit{feature}_{p}=x_{p}\}\implies g( \mathbf{x})\) (the unique attributes of the explanandum result in the given output) is critically over-fitting yet scores 1.0 for precision. These metrics are absolutely ubiquitous throughout the ML, statistical and data mining literature, which might explain their continued application in XAI experimental research. This research prefers _reliability_ and _exclusive coverage_, first proposed in [31], because they penalise any explanations that approach the ill-fitting situations described above. However, to assist the user in understanding the utility of these novel metrics, both sets of results (novel and traditional) are presented. The rule length (cardinality of the rule antecedent) is also measured. These metrics are supplemented by the reliability floor and the rule length floor. The reliability floor is the proportion of evaluated explanations that clear the threshold of 0.75 reliability, and the rule length floor is the proportion of explanations with a length greater than zero. Both of these supplementary statistics are useful for quantifying the prevalence of over- and under-fitting. These pathological behaviours can easily be masked when only looking at aggregate scores (means, mean ranks, etc.). We will also present the fidelity scores that reveal when methods/proxy models do not agree with the black box reference model. The computational complexity will be compared using the mean time (sec) to generate a single explanation. The authors of [19; 22] state that their methods take a few seconds to a few minutes to generate an explanation. We conjecture that something less than thirty seconds would be considered acceptable for many Human-in-the-Loop processes because each explanation requires further consideration prior to completion of a downstream task. Consideration and completion steps would likely be much longer than this simple time threshold. Significance (where required) shall be evaluated with the modified Friedman test, given in [40]. The Friedman test [41] is a non-parametric equivalent to ANOVA and an extension of the rank sum test for multiple comparisons. The null hypothesis of this test is that the mean ranks for all groups are approximately equal. In these experiments, the groups are the competing algorithms. The alternative hypothesis is that at least two mean ranks are different. On finding a significant result, the pairwise, post-hoc test can be used to determine which of the methods perform significantly better or worse than the others. It is sufficient for this study to demonstrate whether the top scoring method was significantly greater than the second place method. Note, however, that the critical value is applied as if all the pairwise comparisons were made. The critical value for a two-tailed test with the \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Data Set** & **Target** & **Classes** & **Class Balance** & **Features** & **Categorical** & **N** \\ \hline adult & income & 2 & 0.77:0.23 & 14 & 9 & 48,842 \\ bank & y & 2 & 0.89:0.11 & 20 & 11 & 45,307 \\ car & acceptability & 2 & 0.71:0.29 & 7 & 7 & 1728 \\ cardio & NSP & 3 & 0.78:0.14:0.08 & 22 & 1 & 2126 \\ credit & A16 & 2 & 0.57:0.43 & 16 & 10 & 690 \\ german & rating & 2 & 0.70:0.30 & 21 & 14 & 1000 \\ lending & loan\_status & 2 & 0.79:0.21 & 75 & 9 & 2105 \\ nursery & decision & 4 & 0.33:0.33:0.31:0.02 & 9 & 9 & 12,958 \\ rcdv & recid & 2 & 0.38:0.62 & 19 & 19 & 18,876 \\ \hline \hline \end{tabular}
\end{table}