Benchmarking XAI Explanations with Human-Aligned Evaluations

Authors: Rémi Kazmierczak, Steve Azzolin, Eloïse Berthier, Anna Hedström, DELHOMME, Nicolas Bousquet, Goran Frehse, Massimiliano Mancini, Baptiste Caramiaux, Andrea Passerini, Gianni Franchi
Primary area: interpretability and explainable AI
Keywords: Explainable artificial intelligence (XAI), Dataset, Benchmark

Abstract:
In this paper, we introduce PASTA (Perceptual Assessment System for explanaTion of Artificial intelligence), a novel framework for a human-centric evaluation of XAI techniques in computer vision.
Our first key contribution is a human evaluation of XAI explanations on four diverse datasets—COCO, Pascal Parts, Cats Dogs Cars, and MonumAI—which constitutes the first large-scale benchmark dataset for XAI, with annotations at both the image and concept levels. This dataset allows for robust evaluation and comparison across various XAI methods. Our second major contribution is a data-based metric for assessing the interpretability of explanations. It mimics human preferences, based on a database of human evaluations of explanations in the PASTA-dataset. With its dataset and metric, the PASTA framework provides consistent and reliable comparisons between XAI techniques, in a way that is scalable but still aligned with human evaluations. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. Our findings indicate that humans tend to prefer saliency maps over other explanation types. Moreover, we provide evidence that human assessments show a low correlation with existing XAI metrics that are numerically simulated by probing the model.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730287490795

thread_type:
official_review

<thread_object>
creator: reviewer_1rmp

<content>
summary:
This paper proposes backbone for XAI, introducing a human evaluation protocol. The authors construct a benchmark dataset consisting of triplets of images, explanations, and labels, allowing for quantitative evaluation of XAI methods. Additionally, the paper consolidates different evaluation criteria and presents a question-based protocol for data annotation.

strengths:
1. The paper conducts a comprehensive investigation of XAI methods, providing a convincing and promising set of evaluation criteria and protocols that offer a strong foundation for future research.

2. Experiments demonstrate that PASTA achieves human-level or better performance under the proposed new protocol.

weaknesses:
1. The process of generating computed explanations and annotating concepts is unclear. The PASTA-metric’s applicability to unseen datasets or new XAI methods appears limited, as the generalization to other contexts or modalities is not fully validated. The four datasets used may not represent the full range of XAI applications, potentially limiting the framework’s relevance for domains beyond standard object recognition. From Table 5 in the supplementary material, it seems that concepts are defined by class names in the datasets. Would these pre-defined concepts limit the model’s generalization? If concepts are simply classes, it may keep the model as a “black-box”. So how to define the explainability?

2. Including more figures and equations to illustrate the metrics would be beneficial. For instance, how are scores for faithfulness calculated? This additional detail would enhance clarity.

3. Participant Background: The study included 15 participants—are they experts in this field? Is this number sufficient for robust annotation?

4. The main sections are somewhat unclear, and parts seem to serve as an overview of the supplementary material, which makes the paper challenging to follow. I recommend the authors provide a clearer narrative or at least one example before referring to supplementary sections.

5. There are minor typos, such as “tree losses” in Line 1563, which should be “three losses,” and issues with brackets in Fig. 8.

6. I also recommend comparing PASTA with traditional protocols or providing a visualization to demonstrate the effectiveness of the backbones.

questions:
Please see the points under Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730321977985

thread_type:
official_review

<thread_object>
creator: reviewer_2wcf

<content>
summary:
This paper conducts a user study on the comparative effectiveness of different explainable AI (XAI) methods, and proposes an automatic XAI evaluation model. In particular, it asks human participants to rate different aspects of the explanations given carefully designed questions, and train a model that takes in the explanations (either text or saliency maps) to predict the ratings. Experimental results reveal the advantages of visual explanations over concept ones, and show the feasibility of predicting human ratings based on model explanations.

strengths:
(1) This paper tackles an important challenge in XAI: the alignment between model explanations and human evaluations.

(2) It carries out human study with a broad range of explanation methods and multiple datasets, which can facilitate the development of future XAI methods.

(3) With an automatic model for estimating human ratings, the study can be extended to help improve the trustworthiness of deep networks.

(4) Extensive experiments are provided in the supplementary materials, providing the opportunity for more in-depth analyses.

weaknesses:
(1) The paper emphasizes the limitations of existing studies in scaling to broader domains, due to difficulties of data collection and the subjectivity of human evaluation. Nevertheless, these problems are also well addressed in the proposed method. Specifically, the paper centers around a user study for rating model explanations, which also requires significant amounts of manual labor and does not alleviate annotator biases. While a quantitative evaluation model is proposed, with only the results in Table 3 (comparison to the baseline without explanations and inter-subject agreement), it is unclear how it can help evaluate XAI approaches in different domains. I would suggest involving the PASTA model for training other models, and exploring its effectiveness in enhancing the trustworthiness of deep networks.

(2) Despite the wide coverage of the user study, the analyses in the main paper are relatively shallow and fall short of providing important insights. In particular, while the paper highlights comparing over 10 classifiers and XAI methods, its conclusions focus on only two categories of XAI methods and three backbones. Table 1 indicates close-to-zero correlations between human ratings and the XAI methods, but without detailed explanations. It is reasonable to move certain results from the supplementary materials and perform more in-depth analyses following previous studies (e.g., [ref1, ref2]).

(3) The proposed PASTA model essentially feeds the visual or textual explanations to CLIP encoders to derive the human ratings. Such a design can have several limitations: First, CLIP is heavily tuned toward semantics and can have its own biases. It would be good to test with various designs, e.g., with recent approaches of projecting multi-modal data to a pretrained LLM space (e.g., [ref3]). Second, the model seems to only utilize explanations (or explanations on top of images) as inputs. While this is okay with simple visual stimuli that have a dominant object for classification, it may not generalize to other applications that demand complex reasoning with a rich set of visual elements (e.g., visual reasoning). It can be useful to consider at least including the ground truth answers as inputs. Third, relating to Table 3, I would expect comparisons with various methods, as the problem itself is just a regression. 

References:

[ref1] Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations. IEEE TPAMI, 2024.

[ref2] What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. NeurIPS, 2022.

[ref3] Visual Instruction Tuning. NeurIPS, 2023.

questions:
(1) Please justify the difference between the proposed study and previous user studies on XAI.

(2) How can the PASTA model scale to broader domains, and help model development?

(3) Please consider reorganizing the paper, and including more in-depth analyses of the main paper

(4) It would be reasonable to perform a more comprehensive study on the automatic rating part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731129519716

thread_type:
official_review

<thread_object>
creator: reviewer_cta3

<content>
summary:
This paper introduces a framework for human centric method of the explainable AI techniques (XAI). This paper performs evaluation on four datasets: COCO, Pascal Parts, Cats Dogs Cars and Monum AI.

strengths:
1. The paper tries to answer an important question about the evaluation of the XAI methods. 
2. The overall evaluation protocol and the research questions designed based on the fidelity, complexity, objectivity and robustness are presented in a right manner.

weaknesses:
1. First and foremost to perform any sort of evaluation the authors should consider the methodology used and the architecture used specially in gradient based methods. eg: GradCAM does not work well with transformers, rather it is a methodology that works significantly better on CNNs due to the architectural composition. Chefer et. al. Transformer interpretability beyond attention visualization.
2. One of the major limitation of this work is use of another network suck as CLIP as explanations evaluator, this limits the model to the limitations of the CLIP. eg: CLIP embedding space limits it for the fine grained understanding in the joint embedding space.
3. The paper is more suitable for a user-based study conference rather than ICLR.

questions:
Please check the limitations and kindly answer the correlation of the design of XAI methods and a deep learning architecture for the evaluation of the XAI methods. Another question that needs to be highlighted are the use of CLIP in the pipeline as it will be dependent on the embedded representations on the CLIP space.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731157257902

thread_type:
official_review

<thread_object>
creator: reviewer_cj8g

<content>
summary:
The paper introduces PASTA, a perceptual assessment system designed to benchmark explainable AI (XAI) techniques in a human-centric manner. The authors first integrate annotated images from the COCO, Pascal Parts, Cats Dogs Cars, and Monumai datasets to create a benchmark dataset for XAI, which is also used to train classifier models for generating explanations. For the final assessment of XAI methods, the authors curate a PASTA evaluation benchmark comprising 100 images. They apply 46 distinct combinations of 21 different XAI techniques on these 100 images, creating a dataset of 4,600 instances. Each instance includes an image, its ground truth label, a classifier’s predicted label, and the explanation generated by a specific XAI technique.

Additionally, to compare various XAI methods, including saliency-based and explanation-based approaches, the authors develop an automated evaluation metric that models human preferences based on a database of human evaluations. This metric enables robust evaluations across modalities. The experiments, conducted on 21 different XAI techniques and datasets, demonstrate that saliency-based explanation techniques such as LIME and SHAP align more closely with human intuition than explanation-based methods.

strengths:
1. The paper addresses an important problem of benchmarking XAI methods in a comprehensive and efficient way, aligning evaluations with human assessments, which I believe is an important gap in the literature. 

2. The proposed benchmark allows for the comparison of XAI methods across different modalities, facilitating evaluations of both explanation-based and saliency-based methods.

3. The paper introduce a data-driven metric to mimic human assessments in evaluating the interpretability of explanations.

4. The authors created six carefully designed questions and a comprehensive set of criteria to assess various desired properties while collecting human labels.

5. Different XAI methods are evaluated against human assessments to benchmark the quality of their explanations. Additionally, human scores are compared with different XAI metrics. 

6. The authors have indicated their willingness to open-source their code, annotations, and models.

weaknesses:
1. How many human subjects were involved in the benchmark creation? Is it five? This information is not explicitly stated in the main paper. Since the benchmark aims to align with human evaluations, it would be valuable to provide details about the annotators, including their ethnicity, gender, age, and other relevant demographics. Ideally, involving annotators from diverse backgrounds—such as varying ages, genders, and ethnicities—would help reduce potential biases in the benchmark.

2. The evaluation benchmark includes only 100 images, which may not be statistically sufficient for conclusive insights into model behavior across different XAI techniques.

3. There appears to be a possibility that some images could overlap between training and test sets. While the authors demonstrated generalization by separating these sets explicitly in Section 4.4, it is not clear to me  why they did not consistently apply such strict separation across all experiments. 

4. The method utilizes existing object grounding techniques, such as Grounding DINO, to generate bounding boxes for the “Cats Dogs Cars” dataset, which might lead to suboptimal performance. It is also unclear if the authors conducted any ablation studies on different grounding or object detection methods before selecting Grounding DINO. Was there a particular reason for choosing this specific method? 

5. Many essential information related to the proposed approach are not included in the main paper. For example, the complete list of perturbations need to be included in the main paper, not on the appendix.

questions:
1. On page 18, in Appendix A1.1, it’s stated that "the task we focus on is indoor scene labeling," but this was not mentioned in the main paper. Could you please clarify this?

2. The authors mention a low correlation between widely used XAI methods and human assessments. Does this imply that existing metrics do not accurately align with human evaluations? If so, how do the authors conclude that human and metric-based assessments cover complementary aspects? What complementary information is provided by the existing metric-based approaches that human-subject-based evaluation does not capture?

Also, See the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731938265516

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer

comment:
Dear AC and reviewers,

We would like to express our sincere thanks for your constructive comments and questions regarding our work. We greatly appreciate the reviewers’ feedback and are grateful for the opportunity to address the concerns raised.

We believe that PASTA and the dataset could serve as a valuable benchmark for the community to assess the quality of XAI techniques. In response to the specific questions raised by the reviewers, we have made the following updates and clarifications (in purple in the revised manuscript):

- **In the appendix (Section B.1 Comparison with Existing Benchmarks):** We have added a comparison between the PASTA dataset and existing human-aligned evaluation benchmarks.
- **In Section A.4 (PERTURBATIONS) of the appendix:** We have included the list of perturbations and additional details on their usage.
- **In Section 3.6 (CORRELATION WITH OTHER METRICS):** We have clarified that human evaluations and computational metrics should be viewed as complementary aspects of the evaluation process.
- **In Section 4.3 (CLASSIFIER RESULTS):** We have tested variants of the feature extraction process, including replacing CLIP with the LLaVa encoder and using handcrafted feature extraction. Details on how we compute these handcrafted features are provided in Section D (VARIANT WITH HANDCRAFTED FEATURES).
- **In Section C.4 (SCORING FUNCTIONS):** We tested alternative scoring networks, including Ridge Regression, Lasso Regression, Support Vector Machines, and a Multi-Layer Perceptron with a single hidden layer of 100 units.
- **In Section B.2 (COMPARISON WITH EXISTING METRICS):** We have added additional formulas and figures to better illustrate the process of computing our scores.
- **In Section B.5 (EVALUATION QUESTIONS):** We provide insights into annotators' feedback and their opinions on XAI, as well as the questions they were asked to answer.
- **In Section A.1.1 (DATASET):** We have clarified the classes and concepts used in the datasets for training our inference models, and we have included a table of labels.
- **In Section C.3:** We have added an experiment measuring the impact of label information on the embedding in the PASTA metric.
- **We have expanded our analyses** in Sections 3.5 and 3.6.
- **In the conclusion:** We have highlighted interesting avenues for future work, including the potential for PASTA-metric as a perceptual loss.
- **In Section 3.2 (XAI METHODS):** We provide further clarification on our testing across multiple backbones.
- **We have re-run all our experiments** related to the PASTA-metric, considering the restrictions on images as indicated by $img_{id}$ in Table 4. As such, all values have been updated, though the observations remain consistent.
- **We have corrected some typos.**
- **In Section B.2 (COMPARISON WITH EXISTING METRICS):** We have added more detailed information on how the computational metrics were obtained.

By Wednesday, we plan to make the following revisions to the paper:
- Correct the typo in Figure 8.
- Include examples of both high- and low-rated explanations, showcasing human annotations alongside PASTA-metric ratings.
- Add more detailed examples in the main text before referencing supplementary sections.

We sincerely hope the reviewers will appreciate the efforts we've made to address their feedback. Once again, we deeply appreciate the reviewers’ thorough feedback. We look forward to engaging in further discussions with the reviewers. We are happy to address any further questions and look forward to your continued input.

Best regards,
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission initially had mixed reviews, and the major concerns raised are:

1. limited benchmark size (5 humans, 100 images) [cj8g, 2wcf]
2. share images in training/test sets? [cj8g]
3. missing ablation study on grounding method [cj8g]
4. missing explanation/analysis about why there is low correlation between XAI methods and human assessments. [cj8g, 2wcf]
5. need to use other gradient-based method for transformers [cta3]
6. limited usefulness since new human annotations would be needed for new XAI methods [cta3]
7. uses CLIP to evaluate explanations, which limits the model to the limitations/biases of CLIP (or whatever VLM is used) [cj8g, 2wcf]
8. perhaps more suitable for user-based study conference rather than ICLR [cj8g]
9. how to use the method to evaluate XAI approaches in different domains? limited generalization? [2wcf, 1rMP]
10. shallow analysis, lacks insight [2wcf]
11. missing comparison with other regression methods [2wcf, 1rMP]

The authors wrote a response, and after the discussion period, the reviewers were still split on the paper. One major concern (Point 9) raised by a majority of reviewers was about the generalizability of the PASTA-metric to other XAI approaches and other domains. Another concern (Point 7) was the use of CLIP as a feature extractor for regressing the evaluation scores for other XAI approaches, and the potential bias introduced by this. (Although authors provide results with LLAVA, it should be noted that LLAVA is based on CLIP). Finally, the suitability of the user-study-based work (Point 8) was also mentioned.

Regarding Point 7, the AC also agrees that using CLIP or other VLMs as feature extractors makes the generated metric for new XAI methods potentially biased towards high-level semantic content (and the limitations of CLIP), especially as the image contents are visible in the input heat map. It is unclear why the training data couldn't be used to train / fine-tune a basic CNN regressor.

Regarding Point 8, the paper consists of a user-study of XAI methods (PASTA dataset), and a method to predict the user evaluations on new XAI methods (PASTA metric). In this sense, the paper seems to be more suitable for a user-study based conference -- the main goal is to evaluate XAI in terms of user preferences. The authors argue that "the objective is not to interpret human annotations but rather to assess the performance of XAI techniques."  However, this is not entirely convincing since only the human-interpretability performance is evaluated here, whereas the primary focus of XAI methods should be faithfulness to the explained model. 

Overall, the paper still has a few outstanding problems that need to be carefully addressed. 

As a side note, there is additional related work that evaluates plausibility of XAI methods using human eye gaze:
https://ojs.aaai.org/index.php/HCOMP/article/view/22002