# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 4 Experimental Evaluation
### Metrics  
One of the main are where the XAIB contributes to the evaluation of XAI is implementing the clear evaluation system proposed in the work of [PERSON] et al. (2018) in practice. The Co-12 framework is a set of desirable properties for an explainer that can be used to evaluate and compare different methods. Since the focus of this work is on the benchmark itself, the details of metric implementations will be left outside the scope of this work, but they are still available in the documentation; their importance is well understood. The results of the experiments still require explanations for the meaning of the metrics that were computed, and this is what this section is devoted to. Not every one of the 12 properties for every explainer type was covered during the implementation of metrics; some of them may be inapplicable, and some were not covered yet. In the next section, only the properties currently addressed are listed. **Correctness** for example, was measured for both types of explainers in different ways with a metric called model randomization check (MRC). A Correctness metric should \begin{table}
\begin{tabular}{l l} \hline \hline
**Entity Type** & **Names** \\ \hline Datasets & breast cancer, digits, wine, iris, synthetic, synthetic noisy \\ Models & SVC, MLPClassifier, KNeighborsClassifier \\ Feature importance explainers & Constant, LIME, Random, Shap \\ Example selection explainers & Constant, KNN, Random \\ Feature importance metrics & MRC, SNC, LD, DMA, SP, CVR \\ Example selection metrics & MRC, SNC, TGD, SCC, CVR \\ Cases & Correctness, Continuity, Contrastivity, Covariate Complexity, Compactness, Coherence \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of entities present in XAIB. The set of entities is not fixed by design and was created to support proof-of-concept with the intention to expand the number of available options. For the abbreviations of metrics, see Section 4.2.  
indicate that the explanations describe the model correctly and truthfully. The explanations may not always be reasonable to the user, but they should be true to the model to satisfy this criterion. **Continuity** is measured for both types with a small noise check (SNC), which measures the stability of the explanation function. Continuous functions are desirable because they are considered more predictable and comprehensive. **Contrastivity** measure should show how discriminative the explanation is in relation to different targets. The contrast between different concepts is very important, and the explanation method should explain instances of different classes in different ways. For this benchmark, we implemented label difference (LD) and target discriminativeness (TGD) for feature importance and example selection types, respectively. **Coherence** metric should show to what extent the explanation is consistent with the relevant background knowledge, beliefs, and general consensus. The agreement with domain-specific knowledge can be measured, but this can be difficult to define and it can be very task-dependent. The measure can be proxied using the agreement between different methods. It is represented by a metric called different methods agreement (DMA) for feature importance and same class check (SCC) for example selection. **Compactness** measures the size of the explanations. The explanations should be sparse, short, and not redundant. The size can be measured directly in some cases. In the case of feature importance, the size is always the same, but sparsity is different. This is why Compactness was measured through sparsity (SP) for feature importance. **Covariate Complexity** means that the features used in the explanation should be comprehensible. Furthermore, non-complex interactions between features are desired. It is measured as covariate regularity (CVR) for both types. For the precise formulations and implementations of all the metrics and up-to-date information, the reader is encouraged to visit the documentation.