# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 1 Introduction  
Approaches based on machine learning (ML) models are incorporated into an increasing number of fields, replacing or augmenting traditional approaches. As the size of the models increases over time to tackle more complex tasks, it becomes harder to understand the outcomes. In addition, there is an increasing interest in systems that satisfy not only the accuracy criterion but also a set of additional criteria such as fairness, safety, or providing the right to explanation [1]. For some areas, satisfying those criteria is a dealbreaker when it comes to the adoption of an ML system [2, 3, 4].  
To address these issues, eXplainable Artificial Intelligence (XAI) has emerged [5]. It encompasses a variety of approaches, from highlighting relevant parts of the input data to showing similar examples or even providing verbal explanations.  
While actively expanding the variety of explanation algorithms, the field of XAI is often criticized for its lack of rigor and evaluation standards [5, 6]. Given the uncertainty of explainability definitions noted in many recent works [7], it seems necessary to explicitly state which one is used in this work. In this work, the definition is the same as in the work of [PERSON]. Interpretability is the ability to explain or present information inunderstandable terms to a human. To further clarify, for inclusion purposes, no distinction is made between interpretability and explainability, similar to the work of [PERSON] et al. (2018).  
The growing number of explanation solutions with different approaches demands a way to compare them. Isolated evaluation-centered papers exist, but they are very limited compared to benchmarks.  
In recent years, several attempts have been made to build a benchmark for XAI methods. However, the existing benchmarks have a number of drawbacks. They, for example, focus only on feature importance, limited or no openness and extensibility, a limited set of measured interpretability properties, which are usually not justified explicitly, focusing on ground-truth-based metrics alone, and missing important documentation, versioning, and software distribution aspects.  
In this work, a new XAI Benchmark (XAIB) is proposed ([[https://oxid15.github.io/xai-benchmark/index.html](https://oxid15.github.io/xai-benchmark/index.html)]([https://oxid15.github.io/xai-benchmark/index.html](https://oxid15.github.io/xai-benchmark/index.html)) (accessed on 29 October 2024)). It was designed keeping in mind the diversity of the XAI landscape. It was important for the generality to be able to include different explanation types, models, and datasets. At the start, it already features two different explanation types, as well as various datasets, models, and explainers that can be used in many combinations. Special efforts were made to make it easy to use and extend. It features thin, high-level interfaces for easier setup while not obfuscating the internals, opening itself for customization and extension. Finally, its XAI evaluation system is based on a comprehensive framework of 12 properties, ensuring the completeness of the evaluation for every explainer type.