SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As noticed, ABPC, INFD, and PScore are potential alternatives to one another, each requiring a good amount of computation from different perspectives. Besides one trial of explanation algorithm, ABPC needs tens of forward passes of the original model. INFD requires generating random masks, which
(a) Modality of images.
(b) Modality of texts.
Figure 3: Averaged metric scores. A higher value indicates better faithfulness. The blanks indicate that the algorithm in a vanilla style is not suitable for the model.
is sampled from a very large space, especially for images. Meanwhile, PScore necessitates trained models' availability, with each model passing the explanation algorithm once. Practitioners should choose the most appropriate method based on the availability of models and computational resources.
There are two other key observations worth discussing. First, MoRF is weakly correlated with ABPC, with a correlation coefficient of 0.29, indicating that they are not measuring exactly the same characteristic. As discussed by Samek et al. [42], MoRF focuses only on the most important features, while ABPC also considers the ranking of the least important features. Therefore, if the goal is to filter out irrelevant features, ABPC scores should be more heavily weighted. Second, we found that ABPC, PScore, and INFD are strongly correlated with each other. In contrast, MoRF and SynScore evaluate faithfulness from different perspectives than the former three metrics.
3.2 Which Explanation Algorithm Demonstrates the Best Faithfulness?
Another interesting question is determining which explanation algorithm is the most faithful. Depending on the models and faithfulness metrics used, the optimal algorithm may differ. However, we still can draw several useful and instructive conclusions.
To simplify the comparison of faithfulness across explanation algorithms, we aggregate the multidimensional metrics into a single one. First, we make the assumption that the metrics in our benchmark measure the faithfulness from different aspects since no pairs are perfectly correlated. Then we can propose a single score by averaging all of the metrics while negating the scores of INFD which is the only one ranking the faithfulness in a descent order. Moreover, we do the standardization for scores of all metrics within each model to balance the contributions of metrics. Specifically, the averaged metric is defined as:
$$
\operatorname {A v g S c o r e} = z (\text {M o R F}) + z (\text {A B P C}) + z (\text {P S c o r e}) - z (\text {I N F D}) + z (\text {S y n S c o r e}), \tag {7}
$$
where  $z(s) = (s - \bar{s}) / \sigma(s)$  is the standardization within each model. Following the formulas, we compute the averaged faithfulness score for each model-algorithm pair and show the results in Figure 3. We provide the results of each metric in the supplementary materials.
We summarize the observations as follows:
- Overall, IG generally outperforms SG except for Vision Transformers. For Transformers in NLP, SG is not an optimal choice. One possible reason is that SG adds noise in the embedding layer, and the noise scale is difficult to tune. Though further investigation is needed, we believe that the theoretical guarantee (IG satisfies the Completeness axiom, i.e., the attributions add up to the difference between the output of the model at the input and a chosen baseline) may be one of the reasons to support wide applications.
Figure 4: Sensibility measured by standard deviations across attribution methods. Note that the std for VGG16-INFD is very high (86.5) whereas we set the limit to 0.2 for better visualization.
- LIME steadily gets a high averaged score for NLP Transformers<sup>5</sup>. One reason is that the algorithmic computation of LIME overlaps in some degree with the evaluation of MoRF and ABPC. This is more obvious if we see the results of MoRF and ABPC, which are in the supplementary materials. Nevertheless, LIME is among the best algorithms measured by other metrics as well.
- For attention-based networks, including both ViTs and NLP Transformers, BT generally demonstrates higher faithfulness than others. This may stem from its accurate approximation of Transformer computations. This observation coincides with the implication of the first observation, motivating future work to design explanation algorithms through mathematical analysis of network structure, e.g., attention modules in Transformers.
- Even within the same network structure (here we have three sets of model families: ResNet{50,101,152}, ViT-{Small,Base,Large} and Bert-{Distil,Base,Large})), no algorithm consistently achieved high faithfulness. For example, IG performed well on R101 and R152 but not R50. Although BT demonstrated the highest faithfulness in most cases, it did not do so for Bert/L. Developing a faithful algorithmic technique that works across different metrics, modalities and models remains an open challenge.
3.3 Which Model is the Most (In)sensitive to Explanation Algorithms?
The model's complexity and interpretability is also a key factor influencing the benchmarking performance. From the extensive evaluation results, we further investigate the sensitivity of the models to attribution methods. Given a model and a metric, we calculate the standard deviation of all metric scores across possible attribution methods. Take ResNet-50 as an example. For each evaluation metric, a standard deviation is computed among GradCAM, IG, and SG. The results are shown in Figure 4 for image classification models and Figure 5 for NLP models.
A model that is insensitive to attribution methods may indicate that the model can be easily explained, or all attribution methods fail to explain the model. Fortunately, the latter case does not occur in our experiments, because the attribution
Figure 5: Sensibility measured by standard deviations across attribution methods. Since the vocabulary of Roberta/B is different from other models, thus PScore is excluded.
methods we selected for the models are relatively suitable. So, with the sensitivity, we can roughly estimate the difficulty of explaining a model. This may be helpful when designing novel network structures.
For fair comparison, we did not involve the results of LIME in Figure 3 because the implementation of LIME involves a procedure of superpixel segmentation for reducing the computation complexity.
Although the most insensitive model is not easy to identify, the most sensitive one is VGG16, which gets the highest sensitivity in almost all metrics. For network families, we can find that ResNets get higher sensitivity than ViTs in MoRF and INFD but lower in PScore and SynScore. In either network family, the sensitivity does not vary much. As for NLP models, they have similar sensitivity as well, except Bert/B in the metrics of MoRF and ABPC. A similar observation is found in ViT/B by the ABPC metric. Existing attribution methods often use ViT/B or BERT as the primary model and achieve good evaluation results. However, some methods may work especially well for ViT/B or BERT/B but not as well for other models. Therefore, we encourage the research community to evaluate attribution methods on a variety of models with different network architectures.
Existing work has developed benchmarks for evaluating and comparing explainability approaches. For example, Rathee et al. proposed BAGEL, a benchmark for evaluating explanation methods on graphical neural networks [39]. OpenXAI provides an open-source framework for evaluating post hoc explanation methods in tabular data [3]. Similarly, other benchmarks focus on NLP models, such as [14, 52]. The XAI-Bench library benchmarks feature attribution methods on synthetic datasets [33]. Chou et al. proposed a benchmark for counterfactual explanation methods on tabular data [11]. However, these benchmarks are limited to specific data modalities and explanation methods. A benchmark that considers multiple data modalities and explanation paradigms is still lacking. Our work addresses this gap by proposing a unified benchmark for explainable AI across different modalities, with the goal of facilitating holistic progress in the field of XAI.
In this section, we provide discussions and limitations on the proposed benchmark  $\mathcal{M}^4$ . We also present our plans of future work for addressing the limitations.
In addition to including a wide range of feature attribution methods, faithfulness evaluation metrics, data modalities, and deep models, our benchmark  $\mathcal{M}^4$  has two other properties. The first one is efficiency and facility. The benchmark  $\mathcal{M}^4$  utilizes subsets of public datasets and evaluates the same datasets for each modality, i.e. ImageNet and MovieReview. Moreover, the models used during evaluation consist of publicly available pre-trained model weights, avoiding training new models from scratch, except for fine-tuning required in sentiment analysis. The second one is objectiveness. The  $\mathcal{M}^4$  pipeline is objective and performed by evaluation algorithms because the faithfulness evaluation depends only on attribution methods and deep models.
We would also like to distinguish between faithfulness and interpretability. Interpretability refers to the alignment between the explanations of a model and human understanding [25]. Faithfulness is a prerequisite for interpretability and refers to how well an explanation reflects the model's functioning. This paper focuses on evaluating the faithfulness, specifically of feature attribution methods, across metrics, models, and modalities. We do not focus much on the interpretability of deep models in this work but our benchmark can be easily extended to its evaluations, e.g., with the help of ground-truth labels of image segmentation [19] and language reasoning [14].
We present the limitations and plans in the future work.
(1) The evaluation metrics in XAI contain several others beyond faithfulness, e.g., interpretability, sparsity, stability etc, while the current version of  $\mathcal{M}^4$  only focuses on the faithfulness. For the comprehensive applicability of the XAI benchmark, we will progressively integrate other evaluation metrics for feature attributions. Some are easy to be plugged in. For example, the sparsity can be directly computed via the entropy of normalized attributions, but for the reason that the sparsity is not one of the faithfulness metrics, it is thus not reported in the current version of  $\mathcal{M}^4$  benchmark. Another reason that we do not involve other aspects in the benchmark and focus on the faithfulness evaluations of feature attributions, is that we believe that based on faithful explanation results, we can more easily and accurately analyze other aspects of XAI. In the future, various metrics will be included in  $\mathcal{M}^4$ , contributing  $\mathcal{M}^4$  for more comprehensive applicability in XAI.
(2) Our benchmark  $\mathcal{M}^4$  did not include many good attribution methods, such as Shapley values based methods [35, 10], CAM variants [51, 36, 23], LIME variants [58, 30], LRPs [5, 49] attention-
based [1] and many others. However, explanations are of great variety. Although many advanced explanations beyond feature attributions have been proposed to facilitate deeper understanding of deep neural networks, their faithfulness is difficult to evaluate a posteriori and would be evaluated ad hoc. As an initial stride toward establishing a benchmark for evaluating XAI methods, concentrating on feature attribution methods would be an attainable endeavor, albeit one that still presents its own set of challenges.
(3) Language models and their explanations are only evaluated by the task of sentiment analysis. Though we are interested in the explanation faithfulness instead of language models' capacities and the sentiment analysis is one of the accessible tasks for faithfulness evaluations, it would be comprehensive to evaluate on other NLP tasks, e.g., those from the GLUE benchmark [50, 14, 57].
(4) Our benchmark  $\mathcal{M}^4$  contains currently the image and text modalities. One of the future directions is to enhance the benchmark by integrating more data modalities, such as graphs, audio clips, tabular data, and multi-modality, whereas several pioneering studies [6, 9] have explored multi-modal explanations using some of the evaluation metrics in our benchmark pipeline.
(5) Social impacts and ethics. Our framework can assess bias and fairness issues of DNNs in high-stake applications involving sensitive attributes like gender, race, and age. This is a challenging topic and would be investigated in our future research.
Although existing benchmarks have advanced XAI in specific domains, a universal benchmark is lacking to compare explanation methods between models and modalities. Our work aims to address this gap by evaluating feature attribution methods on computer vision and NLP tasks using a variety of metrics. In our benchmark, we evaluated nine models using six of the most common explanation methods (LIME, SG, IG, GradCAM, GA and BT) on two modalities (image and texts) based on five evaluation metrics. We gain several observations that can inform the future design and applications of the XAI method. For future work, we plan to expand the benchmark to other modalities, including but not limited to graphs and audio. We also plan to incorporate additional evaluation perspectives, such as interpretability, stability, sparsity, etc.
Xuhong Li and Haoyi Xiong were supported in part by the National Key R&D Program of China under the gradnt No. 2021ZD0110303.
[1] Samira Abnar and Willem H. Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Proceedings of the Advances in Neural Information Processing Systems, 2018.
[3] Chirag Agarwal, Eshika Saxena, Satyapriya Krishna, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. Advances in Neural Information Processing Systems, 2022.
[4] Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pages 435-451. PMLR, 2022.
[5] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks. Springer, 2016.
[57] Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. arXiv preprint arXiv:2309.01029, 2023.
[58] Zhengze Zhou, Giles Hooker, and Fei Wang. S-lime: Stabilized-lime for model explanation. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2429-2438, 2021.