# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 5 Discussion
### Comparison  
In order to highlight the contribution of this work, this section provides a comparative analysis of the benchmark proposed against the currently existing solutions. Flexibility, extensibility, and usability are the main focus of this analysis, although other criteria such as interpretability abd property coverage were also considered. Table 4 summarizes the overview of different aspects of the existing XAI benchmarks and also includes the one proposed in this work. The flexibility of an XAI benchmark can be achieved in many ways. Given the diversity of the ML landscape with different data types, tasks, models, and ways to explain them, a flexible benchmark should be able to cover as much as possible. However, current solutions do not provide such flexibility in terms of tasks, models, and especially explanation types. Most of the works do not go beyond tabular data classification. All of them solely focus on feature importance methods. The XAIB, in turn, provides an example of a flexible benchmark, offering the ability to evaluate not only feature importance explainers but also those that use examples as explanations. It can be argued that the only pair of a data type and a task implemented in the XAIB is also the classification over tabular data, but there is one crucial difference. Compared to other solutions, the XAIB was built with different tasks in mind. Although tabular data classification was the first choice for obvious reasons, the addition of other data types and tasks can occur without rebuilding everything from scratch. The applicability row in Table 4 illustrates this very clearly. Most of the other solutions can only use predefined sets of datasets, models, explainers, or tests. Figure 4: Results on the second setup —NN—on synthetic noisy dataset. Metric values are normalized for visualization. Each line represents a single explanation method. In this setup, **LIME outperforms shap** on the majority of metrics. The extensibility is closely related to the previous property. The extensible benchmark should not only allow the evaluation of different explainers (this would be the borderline), but it should also provide means of experimentation with different data, models, and metrics. The applicability and documentation rows in Table 4 suggest that some solutions offer the ability to evaluate custom explainers. For example, OpenXAI, Compare-xAI, and \(\mathcal{M}^{4}\) have this extensibility option. Alternative directions are not currently allowed. For example, users who want to evaluate an explainer on their own dataset and model cannot use OpenXAI or Compare-xAI. They can leverage \(\mathcal{M}^{4}\) if their data are images or text and their model is a neural network. In other cases, those users have no options. In this regard, at present, only the XAIB provides a full set of extension directions. It provides the user with the ability to experiment with their own implementations of datasets, models, explainers, and metrics. As long as interface compatibility is ensured, users can combine existing entities with custom ones, which is supported not only with the code but also with the detailed documentation. Usability is an important criterion in the open source community. The benchmark should be open, providing the documentation and the results table. It should not be difficult to setup and run experiments. Almost every existing benchmark provides some documentation that highlights different aspects of its usage. Mostly, this covers the basics, for example, how to reproduce results. Usually, there is no information on how to submit the new method or contribute to the development in any other way. Providing up-to-date results is also an important and seemingly underrepresented feature. The ability to easily install and use the package is also excluded from most of the methods. The proposed benchmark provides detailed documentation, not only on reproducing the results but also on usage with custom entities and separate sections on how to contribute.