# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 4 Experimental Evaluation
### Method Comparison  
In terms of evaluation, one of the most important aspects that sets the XAIB apart from existing benchmarks is its ability to compare explainers in different contexts. In most existing XAI evaluation solutions, this is not the case. Datasets and models are treated as constants in evaluation experiments and are usually embedded or hardcoded. This is acceptable for research and the theoretical study of explanation quality. However, when it comes to the evaluation of explainers for specific applications, when models and datasets are different from the evaluation scenarios, this scheme is not flexible enough to allow it. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **(MRC \(\uparrow\))** & **(SNC \(\downarrow\))** & **(LD \(\uparrow\))** & **(DMA \(\downarrow\))** & **(SP \(\uparrow\))** & **(CVR \(\uparrow\))** \\ \hline Const & 0.00 & 0.00 & 0.00 & 1.80 & 0.00 & 0.00 \\ Random & 1.78 & 1.79 & 1.77 & 1.81 & 0.31 & 56.38 \\ shap & 0.88 & **0.23** & **1.27** & **1.16** & 0.11 & **71.06** \\ LIME & **0.99** & 0.48 & 0.86 & 1.28 & **0.16** & 69.51 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results for the **feature importance** methods. The score for each method is averaged across datasets and models. Arrows represent the direction of the metric as follows: the greater the better (\(\uparrow\)) or the lower the better (\(\downarrow\)). The best performing method is shown in bold. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **(MRC \(\uparrow\))** & **(SNC \(\downarrow\))** & **(TGD \(\uparrow\))** & **(SCC \(\downarrow\))** & **(CVR \(\uparrow\))** \\ \hline Const & 1.00 & 1.00 & 0.18 & 0.25 & 12.98 \\ Random & 0.00 & 0.00 & 0.28 & 0.37 & 16.38 \\ KNN (l2) & 0.98 & **0.62** & **0.63** & **0.65** & 16.80 \\ KNN (cos) & **1.00** & 0.63 & **0.63** & **0.65** & **16.84** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Evaluation results for the **example selection** methods. The score for each method is averaged across datasets and models. Arrows represent the direction of the metric as follows: the greater the better (\(\uparrow\)) or the lower the better (\(\downarrow\)). The best performing method is shown in bold. In the XAIB, the nature of most metrics makes data and models pluggable into a measurement process. Since a dataset and a model are variables, it easily enables, for example, the kind of experiments where one of them is varied while the other stays constant. This and other experiments, including evaluations on a specific scenario, also become available. Using the terms established earlier, previous works on benchmarking XAI methods focused on the needs of Researchers, but not those of the Engineers. With the benchmark proposed, Engineers are now able to use their own models and data to evaluate several explainers on their own setup and figure out what works best for the application. The following example demonstrates how different metric values can be derived when the dataset-model pair changes while the explainers stay the same. In Figure 3, the results of shap and LIME on the SVM classifier trained on the breast_cancer dataset are shown against all metrics. Metric values were not taken as this is only transformed for visualization purposes. They were normalized to the range [0, 1], and the ones with a downward direction (the lower the better) were multiplied by \(-1\). This transformation makes all metrics comparable in one graph and gives them an upward direction. Taking into account these results, shap achieves better metric values than LIME on almost all metrics. In this case, if there are no preferences for some set of metrics, one could choose shap over LIME. However, if the pair dataset-model changes, (assuming that another task is solved), the results change as shown in Figure 4. Methods change places, and LIME becomes better on a majority of metrics, so if there are no preferences among metrics, one could choose LIME for this application.