# Open and Extensible Benchmark for Explainable Artificial Intelligence Methods
## 2 Related Work
### Benchmarks  
For this overview, the projects were selected on the basis of several criteria. For one, each of them explicitly articulates that it constitutes a benchmark for explainable AI methods. The second important criterion was open source code. Openness is considered an important aspect of a benchmark. Being open source should help reach end users and become widely employed. The ensuing overview will follow a chronological order, as in previous sections, despite the fact that benchmarks cannot claim a long history. All projects in this overview were published between 2020 and 2022. The authors of \"A Diagnostic Study of Explainability Techniques for Text Classification\" propose several diagnostic properties that measure different aspects of interpretability [17]. Mainly, they focus on text data and the saliency methods used for it. Their metrics are partially application-grounded, using the agreement with human annotations of token importance; however, four out of five of them are functionally grounded. The methods used are not model-agnostic because they explicitly extract and compare the weights of a neural network model. As metrics or diagnostic properties, they use agreement with human rationales in importance scores for the sentiment classification task. This work lies somewhere on the boundary between benchmark papers and regular XAI evaluation papers. Regarding explainer types, this work is focused exclusively on feature importance in text classification. Due to the highly specific nature of experiments, they were not intended to be extended or customized, which impedes further development. Although the proposed metrics are valid, the particular set and its completeness are not justified in the paper. It features a table with the evaluation results and visualizations, but since it all resides inside a paper, this information cannot be updated. [PERSON] et al. propose an approach that is based on synthetic datasets. Real-world data require human annotation if ground-truth importance is required [18]. When creatingsynthetic datasets, researchers can manipulate them in such a way so as to know which features are important for the outcome and to what extent. In their work, they mainly use the order of the features when they are sorted by their importance scores, since it is difficult to compare different scores and set the exact ground-truth value. The authors use their generated datasets to compute several popular metrics such as correctness, faithfulness, monotonicity, and remove-and-retrain (ROAR). XAI-Bench is the name of the project that is published along with the paper. It contains the source code and instructions on how to reproduce the results of the paper. The project is focused on classification and feature importance methods and seemingly does not show any intention of growing beyond that. The metrics employed there are known, although the choice of metrics was not explicitly discussed. This project is very similar to the evaluation papers in the aspect of further development; documentation on how to contribute seems missing, as do the instructions on running the user's own setup. It also seems to lack versioning, making it difficult to track how changes in the implementation of evaluation procedures influence the results in the final table. [PERSON] et al. propose OpenXAI, a benchmark for functionally grounded evaluation of XAI methods with a leaderboard that is available and updatable online [19]. The authors propose 22 metrics measuring the different properties of explainers and argue that their solution can be applied to several modalities and extended by adding new models, metrics, and datasets. The main focus of the project is classification and feature importance, which are built into the benchmark architecture. The results of the evaluation are presented and available online, along with the documentation. The documentation itself is clear, with instructions on how to use dataloaders, models, and explainers.