# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 5 Discussion
### Comparison
Users can also install the XAIB with a single command, making it much more accessible to users from different scientific backgrounds, who may find manual installation more difficult. Table 5 shows which of the Co-12 properties are measured with the metrics of existing benchmarks. The benchmarks feature numerous metrics; however, in terms of completeness in property coverage, they lack variability. The coverage of the Co-12 properties was analyzed by attributing each of the metrics presented in the respective papers to some of the properties whose definition best fits the given metric. The existing benchmarks feature multitudes of metrics that should facilitate a comprehensive evaluation. However, the particular set of metrics that is chosen each time is rarely discussed or justified. The completeness of the chosen metrics is also a topic that is usually not brought up in papers. After analyzing the proposed metrics through the lens of one of the most complete systems of properties available, their completeness becomes clearer. For example, the saliency eval benchmark provides the _Human Agreement_ metric, which is considered to belong to the _Context_ property; _Faithfullness_ measures _Correctness_; _Confidence Indication_ is conceptually very similar to _Contrastivity_; and the _Rationale/Dataset Consistency_ metrics could belong to the _Consistency_ property. The XAI-Bench is very narrow in terms of properties; however, it goes deep into the measures of _Completeness_. _Faithfullness_, _ROAR_, _Monotonicity_, and _Infidelity_ are all different measures of the same property and can reflect different aspects of it. _GT-Shapley_ is likely to measure _Coherence_ since this ground truth can reflect some form of \"common sense\". \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Benchmark** & **Saliency Eval [17]** & **XAI-Bench [18]** & **OpenXAI [19]** & **Compare-AI [20]** & **AI [21]** & **XAI [60 nm]** \\ \hline Code publication date & Sep 2020 & Jun 2021 & Jun 2022 & Mar 2022 & Dec 2022 & Oct 2022 \\ \hline Use of ground truth & Human annotations & Synthetic & Logistic regression & No & No pseudo, synthetic & No \\ \hline Documentation & No & Reproduce & Use & Reproduce, add explainer, add & Reproduce, see & Reproduce, see add dataset, add metric \\ \hline Explanation types & Feature importance & Feature importance & Feature importance & Feature importance & Feature importance, example-based \\ \hline Data types & Test & Tabular & Tabular & Tabular & Text, image & Tabular, more well implemented \\ \hline ML tasks & Classification & Classification & Classification & Sentiment, classification & Classification, more will be implemented \\ \hline Results & In paper & In paper & Online & Online & In paper & Online \\ \hline Applicability & Specific datasets & Synthetic tests & Specific datasets/models & Hardcoded tests & Datasets and models & Any compatible \\  & & & & & & \\  & & & & & & \\  & & & & & & \\ \hline Versioning & No & No & SemVer & No & SemVer (as a part of \\  & & & & & InteroperID) & SemVer \\ \hline Distribution & No & No & No & No & PyPI (as a part of \\  & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison table of existing XAI benchmarks. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Benchmark** & **Saliency Eval [17]** & **XAI-Bench [18]** & **OpenXAI [19]** & **Compare-AI [20]** & **AI [21]** & **XAIB (Own)** \\ \hline Correctness & Yes & Yes & Yes & Yes & Yes \\ Continuity & Yes & Yes & Yes & Yes & Yes \\ Coherence & Yes & Yes & Yes & Yes & Yes \\ Completeness & Yes & Yes & & & \\ Covariate Complexity & & & & & & \\ Computers & Yes & & & & & \\  
OpenXAI, although featuring 22 metrics, does not cover as many properties. They feature _GT Faithfullness_, similarl to the previous case, is a measure of _coherence_. Aside from that, metrics are divided into two groups as follows: _Faithfullness_ as a measure of _Correctness_ and _Stability_ as a measure of _Continuity_.