Explain Any Concept: Segment Anything Meets Concept-Based Explanation
Authors: Ao Sun
Pingchuan Ma
Yuanyuan Yuan
Shuai Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: EXplainable AI
Machine Learning
Computer Vision

Abstract: EXplainable AI (XAI) is an essential topic to improve human understanding of deep neural networks (DNNs) given their black-box internals. For computer vision tasks, mainstream pixel-based XAI methods explain DNN decisions by identifying important pixels, and emerging concept-based XAI explore forming explanations with concepts (e.g., a head in an image). However, pixels are generally hard to interpret and sensitive to the imprecision of XAI methods, whereas “concepts” in prior works require human annotation or are limited to pre-defined concept sets. On the other hand, driven by large-scale pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promotable framework for performing precise and comprehensive instance segmentation, enabling automatic preparation of concept sets from a given image. This paper for the first time explores using SAM to augment concept-based XAI. We offer an effective and flexible concept-based explanation method, namely Explain Any Concept (EAC), which explains DNN decisions with any concept. While SAM is highly effective and offers an “out-of-the-box” instance segmentation, it is costly when being integrated into defacto XAI pipelines. We thus propose a lightweight per-input equivalent (PIE) scheme, enabling efficient explanation with a surrogate model. Our evaluation  over two popular datasets (ImageNet and COCO) illustrate the highly encouraging performance of EAC over commonly-used XAI methods.

Explain Any Concept: Segment Anything Meets Concept-Based Explanation
Ao Sun, Pingchuan Ma, Yuanyuan Yuan, and Shuai Wang The Hong Kong University of Science and Technology fasunac, pmaab, yyuanaq, Scuaiw)Ocse. ust. .hk
Abstract
EXplainable AI (XAI) is essential topic improve human understanding of deep neural networks (DNNs) given their black-box internals. For computer vision tasks, mainstream pixel-based XAI methods explain DNN decisions by identifying important pixels, and emerging concept-based XAI explore forming explanations with concepts (e.g., head an image). However, pixels are generally hard interpret and sensitive to the imprecision of XAI methods, whereas "concepts" in prior works require human annotation are limited to pre-defined concept sets. On the other hand, driven by large-scale pre- rre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promotable framework for performing precise and comprehensive instance segmentation, enabling automatic preparation of concept sets from given image. This paper for the first time explores using SAM to augment concept-based XAI. We offer an effective and flexible concept- based explanation method, namely Explain Any Concept (EAC), which explains DNN decisions with any concept. While SAM is highly effective and offers an "out-of-the-box" instance segmentation, it is costly when being integrated into de facto XAI pipelines. We thus propose lightweight per-input equivalent (PIE) scheme, enabling efficient explanation with a surrogate model. Our evaluation over two popular datasets (ImageNet and COCO) illustrate the highly encouraging performance EAC over commonly-used XAI methods.
Introduction
In recent years, Deep Neural Networks (DNNs) have exhibited exceptional performance a variety of computer vision (CV) tasks such image classification [I], object detection (2), and semantic segmentation 3 However, due to the "black-box" nature of these complex models, their use in security-sensitive applications where interpretability is critical is still limited. As result, there is growing demand for increased transparency and comprehensibility the decision decision-making processes of DNNs. To address this issue, Explainable AI (XAI) [4] has emerged with the purpose of providing explanations for DNNs' predictions.
For CV tasks, conventional XAI works primarily focus proposing and enhancing pixel-level interpretation, which explains the model prediction by identifying important pixels. Despite the strides made in XAI, these techniques often involve trade-offs between three key desiderata among bunch of criteria: faithfulness, understandabiliry, and efficiency [5] 6 7 g. Faithfulness ensures that the generated explanation aligns with the internal decision-making process of the DNN, understand- ability ensures that the explanations are comman-coppeehnniblle, and efficiency guarantees that the explanations generated with reasonable computational cost.
From the perspective of explanation forms, existing methods often provide pixel or superpixel-level explanations [9] 10 12 [ which are constantly hard to interpret (i.e., low understandability) and sensitive to the potential imprecision of XAI techniques (low faithfulness). Some recent works
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

aim to explain DNN predictions with concepts (e.g., a head rather than pixels in the head) in images. However, they either require human annotation or are limited specific concept discovery methods.
From the XAI methodology perspective, Shapley value [14 14Jaeseed explanation has become the mainstream, given its well-established guarantee in theory. However, the inherent complexity of Shapley value calculations and the target model makes it highly costly and time-consuming (low efficiency). To reduce the high overhead, existing methods rely on Monte Carlo sampling (15), model- specific approximations 16 17.. (smaller) surrogate models [8] 19 or combination of the above to approximate the Shapley value. With the growth of model size and complexity, surrogate models are widely employed, even though it may suffer from low faithfulness owing to the discrepancy between the surrogate model and the target model under consideration.
Our Solution, Driven by large-scale pre-training, Segment Anything Model (SAM) 20 has been demonstrated as a powerful and promotable framework for performing precise and comprehensive instance segmentation, enabling automatic extraction of a concept set from an given image. Hence, this paper for the first time explores using SAM as a concept discovery method to augment concept- based XAL. We advocate to line up SAM and XAI, such that SAM's instance segmentation delivers high accurate and numan-understandable concept set from arbitrary images, which in turn facilitates the XAI task with high faithfulness and understandability.
Nevertheless, while SAM is effective and offers an -out-of-theebox' solution, computing the Shapley value is still expensive. Thus, to achieve high efficiency (our third desiderata), besides standard Monte Carlo sampling method to reduce overheads, we propose a lightweight per-inpul equivalent (PIE) scheme to approximate the target model with a low-cost surrogate model. Our PIE scheme allows the surrogate model to share some carefully-chosen parameters with the target model, which effectively close the discrepancy between the two models.
Our evaluation over two popular datasets (ImageNet and COCO) illustrate the highly encouraging and superior accuracy of EAC over popular pixel-level and superpixel-basee XAI methods. Moreover, we also demonstrate the high interptertability of EAC with durefully-desiieee user study. As confirmed by human experts, EAC offers high interpretability, largely outperforming de facto XAI methods. We also justify the effectiveness of our technical pipeline with ablation studies, and discuss potential extension and future work. In summary, this paper makes the following contributions:
We for the first time advocate the usage of SAM concept discovery method to facilitate concept-based XAI with high faithfulness and understandability.
We propose general and flexible concept- based explanation pipeline, namely Explain Any Concept (EAC), which can explain the model prediction with any concept. We introduce a set of design considerations and optimizations to make EAC practically efficient while maintaining high faithfulness and understandability. In particular, to reduce the generally expensive computational burden of Shapley Value, we propose the a novel scheme named Per-Input Equivalence (PIE).
We conduct extensive experiments and human studics to demonstrate the effectiveness of EAC on diverse settings. We also illustrate the generalizability of EAC and its security benefits.
Open Source. We publicly release and maintain EAC under the following github page: http:///giibbbbooo/////001/////s
Background and Related Works
EAC belongs to local XAI which offers model-agnostic explanations for DNN decisions over each image input. Below, we introduce prior works from both XAI methodology and input data perspective.
XAI Methodology, From the methodology perspective, XAI can be classified into two categories: backpropagation-baate based and perturbation-based -asedd. The former case, also known as gradient- gradient-based, leverages the backward pass of a neural network to assess the influence and relevance of an input feature on the model decision. Representative works include saliency maps [9) Gradient class activation mapping (Grad-CAM) [1]. Salient Relevance (SR) maps (21), Attributes Maps (AM) (22. DeepI.IFT (23), and GradSHAP [2]. For the latter case, it primarily perturbs the input data into variants, and then measures the change in model output to generate explanations. Representative

---

works include LIME [33. SHAP 18 and DeepSHAP [8 Given that EAC employs Shapley value, we detail it in the following.
Shapley Value for XAI. The Shapley value [4], initially introduced in the cooperative game theory, quantifies each player's contribution to the total payoff of coalition. This concept has been adapted to machine learning to measure each feature's contribution to model's prediction. The Shapley value for player is determined by their average marginal contribution across all possible coalitions, taking into account all potential player orderings. Formally, it can be expressed as: (s(v)) = x Di= (T)) Eses. (1) (u(S (i)) - u()) where N represents the set of all players,
(i) the collection of all coalitions of size that include player i, is the utility function, and S is a coalition of players. In short, the Shapley value enumerates all possible coalitions and calculates the marginal contribution of the player each coalition. The Shapley value possesses several desirable attributes, including efficiency, symmetry. and additivity, and uniquely satisfies the property of locality. According to this property, a player's contribution depends solely on the members involved in the respective coalition.
In the context of XAI, N often represents the size of feature space, S is subset of features, and is the model's prediction. For instance, consider the pixel-level Shapley value for image classification. In this case, N is the number of pixels in the image, S is subset of pixels, and uSS) is the probability of the image belonging to a particular class when all pixels excluding S are masked. Likewise, in superpixel-basse
and concept-based methods, N the number of superpixels or concepts, 8 is a subset of super- apper-piells or concepts, and uSS) is the prediction of the model when the remaining superpixels or concepts are masked.
Different Forms of Explanations. Given an input image, existing XAI techniques may explain the DNN decision at the level of individual pixels, superpixels, concepts.
Pixel-Based XA/ explains DNN decision at the level of individual image pixels. Prior works often use saliency maps 9 10]. which highlight the most important pixels for given decision. Activation maximization [24] modifies an input image to maximize the activation of a particular neuron or layer in DNN to explain its decision. Attention 225 26 is also employed in XAI, offering relatively
low-cost explanation in comparison to standard gradient-baa based methods. However, these techniques have limitations, such as being highly dependent the model architecture and not providing complete understanding of the decision-making process.
Superpixel-Based XAI explains DNN decision at the level of super-pixels, which are often perceptual groups of pixels. SLIC (27 is a common superpixel algorithm that clusters pixels together based on their spatial proximity and color similarity. SLIC forms the basis of various mainstream XAI methods, such as Class Activation Mapping (CAM) I and Grad-CAM [1]. LIME offers a model agnostic approach to XAI that explains model decisions by training local explainable model over superpixels. RISE (28) randomly masks parts of the input image and observes the model prediction change. Superpixels that have the most significant impact the output are then used create an importance map for explanation. Superpixel based methods group pixels together to simplify the image, but this may result in loss of resolution and detail, particularly when attempting to identify small or intricate details in an image. The accuracy of superpixel- based methods is highly dependent on the quality of the image being analyzed, and more importantly, objects that recognized in the image. Ill-conducted superpixel segmentation can notably undermine the accuracy of the explanation.
Concept-Based XAI leverages concept activation vectors (CAVs) 29 extracted from neuron activa- tions to distinguish between images containing user-specified concepts and random images. It offers quantitative way of depicting how a concept influences the DNN decision. Furthermore, the CAV method is extended in recent works with more flexibility and utility (e.g., with causal analysis or Shapley values) [0 31 32). However, these techniques generally require human annotation limited to a pre-defined set of concepts, and this paper advocates a general and flexible concept-basee XAI technique.
3

---

Method
3.1 Desiderata of XAI Techniques
XAI aims to address the opaque nature of DNNs and provide explanations for their predictions. Aligned with prior research [5 7] we outline three essential desiderata for XAI techniques applied in computer vision tasks.
Faithfulness. The explanation should be highly faithful with respect to the internal mechanism the target DNN. Let E' be the explanation and f be the target DNN model under analysis, the faithfulness can be defined as the correlation between E and the actual decision-
process of f. A higher correlation indicates more effective explanation. To date, consensus is yet to be reached on how to quantitatively measure faithfulness [28 33
35 this paper, we employ insertion/deletion experiments 28 frequently-used approach to assessing faithfulness.
necessary element, it alone does suffice for desirable XAI process. It is equally crucial that the explanation delivered by the XAI process is comprehensible humans. In other words, the XAI process must provide lucid and human- understandable explanation that allows users to easily grasp the reasoning behind the model's decision-making process.
Understandabiliy. Although faithfulness
Efficiency. The explanation generation process should have relatively low computational cost so as cause unacceptable delays. This ensures that the XAI technique can be practically implemented in real-world applications without compromising system performance.
Trade-off. Note that the three desiderata are frequently in conflict with each other. For example, highly faithful explanation, strictly reflecting the internal mechanism of the target DNN, may be too obscure understand. Namely, too large superpixel size can yield inaccurate yet more "complete" (and thus more readable) outputs, and vice versa for a too small superpixel size. fact, a proper superpixel size is often hard decide. Moreover, faithfulness may also impose significant challenge to efficiency. In fact, generating a explanation that faithfully explains the target model is usually costly, especially for models with popular large backbones. Hence, many existing XAI techniques trade faithfulness for efficiency by using surrogate models to approximate the target model. sum, this paper advocates strike balance between the three desiderata obtain effective XAI technique, as will be illustrated in our technical design (Sec. 33 and evaluation (Sec.
3.2 EAC Design
H
"Dog"
Original Input
Target Model
Festuree Map
Decision
|
MC
Sampling
0.05
Binary Feature
Surrogate Model (PIF)
Shared FC
Shapley Value
Concept-based Explasation
Sogmented Image
Phase One
Phase Two
Phase Three
Figure 1: The technical pipeline of EAC in three-
three-phase phase form.
Technical Pipeline. As depicted in Fig. I EAC features three-phase, pipeline to explain DNN's prediction for an input image. the phase, we employ the de facto instance segmentation model, SAM, to partition an input image into a set of visual concepts. In the second phase, we train a per-input equivalent (PIE) surrogate model to approximate the behavior of the target DNN. In the third phase, we use the surrogate model to efficiently explain the model prediction with the concepts obtained in the first phase. In sum, EAC lines up SAM with XAI, such that the facto instance segmentation model forms the basis of XAI XAI faithfulness and understandability, whereas

---

novel PIE scheme offers high efficiency while maintaining high faithfulneess. Below, we present the technical pipeline of EAC in detail.
Phase One: Concept Discovery. Concepts are defined as prototypes that are understandable for humans [3 37. Traditionally, methods such as ACE [11 and CONE-SHAP (5) leverage superpixel method on the validation set. Then, these superpixels are clustered into set of concepts using its feature vectors. However, this does not necessarily render the concepts that are understandable for humans. In this regard, we advocate that the concepts should be semantically meaningful and buman-understandable. To this end, we employ the de facto instance segmentation model, SAM, to obtain the concepts. Given an input image x, SAM outputs a set of instances in the image and these instances constitute the concept for x. We present a sample case in "Phase One" of Fig. [ Here, we
denote the set of concepts as C = {c1, C2, ),,), where n is the number of concepts.
Phase Two: Per-Input Equivalence (PIE). Aligned with recent research in XAI, we use the Shapley value [1] to identify key concepts that contribute to the target model's prediction; the design detail presented below. However, despite the general effectiveness of Shapley value, it highly costly to compute due to the exponential complexity. Overall, the Shapley value needs to enumerate all possible coalitions and calculates the marginal contribution for each concept. While the exponential complexity is usually avoided by Monte Carlo sampling, it is still costly due to the inherent complexity of the target model (e.g., models with large backbones).
To alleviate this hurdle, we propose the scheme of Per-Input Equivalence (PIE) to reduce the complexity of the target model. Intuitively, we would expect to substitute the target model with a surrogate model I' that is computationally efficient while maintaining the same functionality of the target model. However, it is challenging to obtain a simple model f' that is fully equivalent to the target DNN model f. Since each time we only explain the target model for one certain input, we can employ a surrogate model f' that is only equivalent to f over the given certain input (i.e., PIE). Therefore, we have (b( == fie(h(()), where b is the one-hot encoding of the concepts in C, h mimics the feature extractor of f, and fie is the fully-connected (FC) layer in
Formally, we have h (0. 1)10 -> R.m where m is the size of features in the last layer of f- Considering the "Phase Two" in Fig- [ where to constitute the PIE scheme, f' takes the one-hot encoding of the concepts as input, leverages h to extract the feature vector with the same semantics as f. and predicts the expected output of ff using the same FC layer (referred to as "Shared FC Layer" in Fig. D of f. The training data for f' is obtained by sampling the concepts in C and the corresponding probability distribution of f by masking the concepts in C. Then, we can train f' by plugging the FC layer of f into f' as freezed parameters and optimize h with the cross-entropy loss.
Hence, we only use the target model f to obtain few training samples for our surrogate model fr and then use I' to explain J. Given that f' is much smaller than f' the PIE scheme can significantly reduce the cost of computing the Shapley value, enabling efficient Shapley value calculations for f.
Phase Three: Concept-based Explanation. Given a set of visual concepts C = {CI, ... .ca) identified in the first phase, we aim to explain the model prediction for a given input image x. The explanation can be expressed as a subset of C, i.e., E C C. Recall the definition of Shapley value in Sec. 2 we consider each concept as a player and the model prediction on the original class of x as the utility function. Then, we can define the marginal contribution of concept as the difference between the model prediction on SU {ct} and S where S S C) {ct]. That is, the marginal contribution of c; is defined as
D (5)
(5) = ulS (c)) u(S)
Here, u(S)= == I (mask( C \. ()) is the prediction of the target model f on the image with only concepts in 5 (remaining concepts are masked). With the aforementioned PIE scheme, we can use the surrogate model f' to approximate I (i.e., a(S) = f'(mask(rr, ())). Then, the Shapley value of c; is defined as
n n
(c((x)
Σ Ae, (S)
(E)
where (i) is the collection of all coalitions of size k that does not contain q. Since the size of all coalitions is prohibitively large, we approximate the Shapley value using Monte Carlo (referred to as MMC Sampling" in Fig. [ sampling. In particular, sample K coalitions for each concept
5

---

and approximate the Shapley value as (x) & Ek= (S,) where Sş is the k-th sampled coalition. The optimal explanation is defined as the subset of concepts that maximizes the Shapley value, i.e.,
E arggce max (x)
where (g(x)= = =C,EEE (2) is the Shapley value of E for z. Finally, we mask the concepts in C. E and provide the masked image as the visual explanation to the user.
Comparison with Existing Surrogate Models. We aware of existing methods such LIME [B] that also use a low cost surrogate model to mimic the target model. We highlight the key differences between method and LIME. First, LIME uses linear model the surrogate model, which is not expressive enough to approximate the target model. Second, LIME learns the relation between the input and output of the target model. In contrast, our method effectively reuses the FC layer of the target model f to form the surrogate model f'. This shall generally enable more accurate approximation of the target model the FC layer retained.
Evaluation
In this section, we evaluate EAC from three aspects: (I) the faithfulness of EAC in explaining the model prediction, (2) the understandability of EAC from the human perspective, and (3) the effectiveness of PIE scheme compared with the standard Monte Carlo sampling and surrogate model-based methods.
4.1 EAC Faithfulness
Setup. We evaluate EAC on two popular datasets, ImageNet [38] and COCO 33 and use the standard training/validation split for both datasets. We use the ResNet-50 [1] pre-trained on ImageNet/COCO as the target DNN model for EAC. We present discussion on generalizability of EAC in See. 5 and corresponding empirical evaluations in Supplementary Materials. We clarify that EAC does not involve many Scheme, i.c. simple linear neural network learning scheme and the Monte Carlo (MC) sampling: we set = 0.008 and the number of MC sampling as 50000 throughout all experiments.
The only syper-parameter considered in EAC is when fitting the PIE
We use the insertion and deletion schemes to form our evaluation metrics; these two schemes are commonly used in the literature for evaluation of XAI techniques [28]. To clarify, these metrics involve generating predictions by gradually inserting and deleting concept features from the most important to the least important ones, and then measuring the Area Under the Curve (AUC) of prediction probabilities. Particularly, for insertion, it starts with fully masked image and gradually reveals the concepts, while for deletion, it starts with fully unmasked image and gradually masks the concepts. Intuitively, the AUC reflects the impact of the inserted/deleted concepts on the model prediction. For insertion, higher AUC indicates better faithfulness, while for deletion, lower AUC indicates better faithfulness. For both settings, report the average results and standard deviation of three random runs.
Baselines. We compare EAC with nine baseline methods: (1) DeepLIFT 23 and (2) GradSHAP [2] are two representative backpropagation-ben based methods, which have been noted in Sec. 2 (3) Int- Grad [4)] a gradient-based method, yields explanation by computing the path integral of all the gradients in the straight line between an input and the corresponding reference input; (4) Ker-
neISHAP [18] approximates Shapley values by solving linear regression problem; (5) FeatAbl (Feature Ablation) is a frequently-used perturbation based approach for feature attribution [41]; (6) CRAFT [42] is an attribution-bast based method allowing heatmaps to identify the most influential regions of an image. (7) LIME 13,. a popular perturbation based method introduced in Sec. 2 uses surrogate model to approximate the target model. To apply these methods, we first employ superpixel 27 to generate concept patches from the input image, and then calculate their concept importance. The details of the superpixel hyper-param setting and the corresponding experiment provided in the Supplementary Material.
Besides, we also compare EAC with three variants of DeepLIFT, GradSHAP, and IntGrad, which leverages morphological operations 43 (e.g., image erosion and dilation 44 as postprocessing
step to cluster pixels into concepts. We denote these variants as DeepLIFT* GradSHAP*, and

---

IntGrad respectively, Overall, we compare EAC with six superpixel-based XAI methods and three postprocessing- based concept XAI.
Table 1:
Comparison with baseline methods across four settings. For each setting, the first and the second row report the mean and std. dev. of the results of three runs, respectively. 1 and + indicate higher and lower is better, respectively.
||
EAC
Deepl.IFT
GndSHAP
IntGrd ||
KemelSFHAP
FoutAl
LIME
CRAFT
Deepl.FF
GradSHAP
IniCledd
ImsageNetinserrin"
K3,000
0.033
75.255
0,000
64,55
0.035
68.772
0.000
64,544
0.002
70,187
0.000
76,638
0.027
0.000
04,40
14,707
0.000
14,794
0.067
15.120
0,000
CoCallussstioa f
83,004
0.12
78.199
0,000
61,109
0,212
65.087
54,570
72,260
79,028
51.49
0.000
0.000
8,500
21.643
19.755
ImageNevDeeletion
23,799
25.262
40.996
%6314
26.583
37.352
54,66
40600
44,800
46.015
0.000
0.034
0.000
0.000
0.246
CoCw/elttion
16,640
0.041
17,036
0.000
34/138
0.144
0.000
20,064
0.040
0.000
17.387
0.089
0.000
44.93
49/077
0.000
35.902
0.173
0.000
We report the effectiveness of EAC and compare it with the baseline methods in Table 1 In particular, we observe that EAC consistently outperforms the baseline methods across all settings. For example, in the ImageNet dataset, EAC achieves 83.400% AUC for insertion, which is 8.165 higher than the second-best method DeepLIFT. Similarly, in the COCO dataset, EAC achieves 83.404% AUC for insertion, which is 5. 5.205% % higher than the second-best method DIePLIFT. We observe similarly promising results for the deletion evaluations. We also observe that the standard deviation of EAC is comparable to these baseline methods, which indicates that EAC is as stable as the majority of the baseline methods. Moreover, it seen that the variants of DeepLIFT, GradSHAP, and IntGrad perform much worse than the original methods, which indicates the incapability of those standard morphological operations in clustering pixels, and the superiority of SAM in concept discovery.
4.2 EAC Understandability
To further explore the understandability of outputs from different XAI methods, we conduct a human evaluation to assess whether EAC can generate more human-understandaan baseline methods. To do so, we randomly select 100 images from the ImageNet and COCO datasets, respectively. We then generate eight explanations for each image using EAC and the seven baseline methods (for those three morphological operation- based variants in Table [ we pick the best one in this evaluation). Then, for each image, shuffle the explanations generated by EAC and the baseline methods, and ask our human participants (see below) to pick an explanation that they think is most understandable.
explanations than the
At this step, we recruit six participants to evaluate the explanations. We clarify that all participants are graduate students in relevant fields. We spent about 15 minutes to train the participants to understand our task. Before launching our study, we also provide a few sample cases to check whether the participants understand the task. To reduce the workload, for each image and its associated eight explanations, we randomly select three participants for the evaluation. Thus, each participant needs to evaluate 100 explanations. We report that each participant spends about 45 minutes to finish the evaluation.
Table 2: Human evaluation results.
EAC|
DeepLIFT
GradSHAP
IntGrad
|| KerneSHHAP
FeatAbl
Lime
GradSHAP
Discord
ImageNet|
70
6
3
4
5
COCO
67
7
3
5
11 3
5
We report the results in Table 2 Among 200 images in total, participants reach a consensus (i.c., at least two out of three participants favors the same explanation) on 184 images (92.0%) with 93 images from ImageNet and 91 images from COCO. Among these 184 images, EAC is favored on 137 images (74.5%) while the second best baseline method is only favored seven images (7.7%). From Table 2
it is seen that among the baseline methods, no one is significantly better than the others.
Overall, we conclude that EAC is significantly better than the baseline methods in terms of under- standability. This is an encouraging finding that is in line with our study in Table [ Below, we present several cases to demonstrate the EAC's superiority in understandability.
7

---

Original Image
FAC
Deepl IFT
GradSFAPP
IntGind
KemelSHAP
FearAbl
LIME
GradHAAP*
Figure 2: Sample explanations generated by EAC and the baseline methods.
Case Study. We present several cases in Fig. to demonstrate the effectiveness of EAC in explaining the model prediction. Besides these four cases, observe that EAC consistently demonstrates its superiority in explaining the model prediction across other test cases. Overall, is clear that EAC generates more "well-formed," concept-level explanations that human understandable across all four cases in Fig. 2 For example, in the first case, EAC correctly highlights the "train" as the concept- level explanation, whereas the bascline methods yield some negligible pixels (the first three baselines), a fragment of the image (the 4th, 5th, and 6th baselines), or the entire image (the 7th baseline). This clearly illustrates the superiority of EAC in terms of the faithfulness and understandability.
4.3 Ablation Study of the PIE Scheme
In this experiment, we explore the effectiveness of our proposed PIE scheme. We compre the PIE scheme with three baselines that compute the Shapley value using 1) the original model under explanation, 2) standard linear surrogate model and, 3) the same model our PIE scheme while parameter sharing is disabled. In all settings, approximate the Shapley value while using different schemes to represent the target DNN model.
the same Monte Carlo sampling method to
First, for the setting of directly using the original model, we observe significantly longer processing time than others. Indeed, we report that it takes more than 24 hours to process one image when using the same Monte Carlo sampling method. When processing time still much longer than the other three methods (two baselines find that the AUC is also much lower than that of the other three methods. As result, deem that the original model is impractical for computing the Shapley value and a surrogate model is necessary. Accordingly, omit reporting the results of the original model and mark its results as "N/A" in Table 3
slightly reduce the number of samples, the EAC). Similarly,
Table Ablation study of the PIE scheme. For each setting, report the the mean of the results of ten runs. Aligned with Table [
t and 1 indicate higher and lower is better, respectively.
Processing (sec.)
Model
Processing
PiBs
81.78
245
PiE
12,47
244
ImageNevInsertion
Original Model
NIA
N/A
ImageNe/DDletton.
Original Model
N/A
N/A
PIE
Lincar Model
78.11
Lincar Model
14.08
31
PIE
87.08
252
PIE
13.71
203
COCOOinssetion
Original Model
PIE 74.86
NIA
N/A
cocorDeettio
Original Model
N/A
N/A
222
Linear Model
Linear Model
14.36
[3]
We report the results of our PIE scheme and the baselines in Table 3 Overall, we interpret the evaluation results as highly encouraging: the PIE scheme is notably better than all three baselines, In particular, the ablated PIE scheme without parameter sharing (the "PIE w/o PS rows in Table is

---

significantly worse than the PIE scheme in terms of both AUC and the processing time. This indicates that parameter sharing effective in reducing the processing time while preserving high accuracy. Moreover, when comparing with the linear surrogate model (the "Linear Model" rows in Table ) the PIE scheme is consistently better terms of AUC. This indicates that the PIE scheme is more accurate, because the linear surrogate model over-simplifies the target model. Overall, we interpret that this ablation study illustrates the necessity of our PIE scheme, which empowers high faithfulness and efficiency to our technical pipeline.
5 Discussion
In this section, we first analyze the generalizability of EAC from the following three aspects. We then present statements on the security impact of EAC. Evaluations noted in this section are presented in the Supplementary Material.
Different Target Models. In this paper, we primarily use ResNet-50 as the target DNN model to evaluate EAC. Nevertheless,
evident that the technical pipeline of EAC is independent of particular target DNN models. Further evaluation of EAC's performance using different target DNN models is presented in the Supplementary Material. In short, our findings demonstrate the persistent superiority of EAC over baselines when explaining different target DNN models.
Different Visual Domains. The concept extraction process for EAC is executed using the SAM framework, which means that it functions optimally on identical visual domains to SAM. It is advocated that SAM can be effective wide range of visual domains, including medical images, simulation images, and painting images [20]. However, our preliminary study over some other visual domains illustrate potentially suboptimal performance of SAM. Overall, it is clear that the accuracy of the concept set identified by SAM significantly impacts the performance of EAC, and ill-identified concepts may lead suboptimal results of EAC. We remain exploring further applications of EAC on other visual domains for future research. Also, with the recent development of knowledge-specific area SAMs. such as medical image 45), remote sensing [46), and UVA [47), we believe that EAC has the potential to improve DNN decisions in other targeted areas in the future. Since this is the first attempt deliver humanly understandable and computationally feasible concept extractors to the field XAI, we believe our work can shed light on their explainability.
Different Visual Tasks. This paper illustrates the effectiveness of EAC on interpreting image classification, a common and core task widely studied in previous XAI research. However, EAC is limited to image classification. In fact, the technical pipeline of EAC can be applied and extended to other common visual tasks that can be explained using Shapley value-based explanations, such as object detection. We leave exploring EAC's potential applications other visual tasks future work.
Security Impact. Analyzing and detecting backdoors in DNNs is an emerging downstream applica- tion of XAI techniques. While the main paper primarily focuses on the core functionality EAC, also consider the security impact of EAC. In particular, we present evaluations of using EAC to analyze and detect backdoors in DNNs in the Supplementary Material, We report promising results and present discussions on potential future improvement accordingly.
6 Conclusion and Impact Statement
In this paper, we propose EAC, a novel method for explaining the decision of a DNN with any concept. EAC is based on SAM, a powerful and promotable framework for performing precise and comprehensive instance segmentation over a given image. We propose a highly efficient neural net- work pipeline to integrate the SAM and shapley value techniques. We conduct extensive experiments to demonstrate the effectiveness and interpretability of EAC.
Broader Impact. EAC is general framework for explaining the prediction of a DNN with any concept. can be used in many applications, such as medical image analysis, autonomous driving, and robotics. For example, EAC can be used to explain the prediction of a DNN for medical image with any concept, such as tumor. This can help doctors to understand the prediction of the DNN and improve the trustworthiness of the DNN. Overall, we believe that EAC can be used to improve

---

the trustworthiness of DNNs in many applications, and we publicly release and maintain EAC to facilitate its adoption the real world.
However, terms of Negative Societal Impacts, EAC may also be used to explain the prediction of DNN for medical image with concept that is not related the medical image, such as a cat, train, or some subtle emror. This may mislead doctors and cause serious consequences, Therefore, it may cause harm when using EAC in real-world sensitive domains without sufficient safety checks on its outputs, We will continue to improve EAC to reduce its negative societal impacts.
Acknowledgement
We thank the anonymous reviewers for their insightful comments. This research is supported in part by the HKUST 30 for 30 research initiative scheme under the the contract Z1283.
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[2] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pages 91-99, 2015.
[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[4] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52160, 2018.
[5] Jiahui Li, Kun Kuang, Lin Li, Long Chen, Songyang Zhang, Jian Shao, and Jun Xiao. Instance- wise or class-wise? tale of neighbor shapley for concept-based explanation. Proceedings of the 29th ACM International Conference on Multimedia, pages 3664-3672 2021.
[6] Janet Hui-wen Hsiao, Hilary Hei Ting Ngai, Luyu Qiu, Yi Yang. and Caleb Chen Cao. Roadmap of designing cognitive metrics for explainable artificial intelligence (xai). arXiv preprint arXiv:2108.01732 2021.
[7] Lukas-Valentio Herm. Impact of explainable ai on cognitive load: Insights from an empirical study, arXiv preprini trXiv:2304.0886 2023.
[8] Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang, Xinsight: explainable data analysis through the lens of causality. Proceedings of the ACM on Management of Data, 1(2):1277, 2023.
[9] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[10] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3437, 2017.
[11] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[12] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626 2017.
10

---

[13] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[14] Lloyd S Shapley et al. A value for n-person games. 1953.
[15] Eunhye Song, Barry L Nelson, and Jeremy Staum. Shapley effects for global sensitivity analysis: Theory and computation. SIAM/ASA Journal on Uncertainty Quantification, 4(1):1060-1083, 2016.
[16] Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence, 2(1):56-67, 2020.
[17] Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pages 272-281. PMLR, 2019.
[18] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[19] Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference Artificial Intelligence and Statistics, pages 3457-3465. PMLR, 2021.
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Ye Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[21] Heyi Li, Yunke Tian, Klaus Mueller, and Xin Chen. Beyond saliency: understanding convolu- tional neural networks from saliency prediction layer-wise relevance propagation. Image and Vision Computing, 83:70-86, 2019.
[22] Marco Ancona, Enea Ccolini, Cengiz Öztireli, and Markus Gross. Towards better under- standing of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06100 2017.
[23] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages 3145-3153. PMLR, 2017.
[24] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):333-255, 2016.
[25] Ioanna Gkartzonika, Nikolaos Gkalelis, and Vasileios Mezaris. Learning visual explanations for denn- based image classifiers using attention mechanism. In Computer ECvV 2022 Workshops: Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 491- Springer, 2023.
[26] Mariano Ntrougkas, Nikolaos Gkalelis, and Vasileios Mezaris. Tame: Attention mechanism based feature fusion for generating explanation maps of convolutional neural networks. In 2022 IEEE International Symposium on Multimedia (ISM), pages 58-65. IEEE, 2022.
[27] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(1 1):274-2228, 2012.
[28] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint 1rXivv/806..7440 2018.
[29] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668-26688 1668-2677. PMLR, 2018.

---

[30] Yash Goyal, Amir Feder, Uni Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint vrXiv:1907.07/1 07165, 2019.
[31] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-
based explanations. Advances in Neural Information Processing Systems, 32, 2019.
[32]
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Pradeep Ravikumar, and Tomas Pfister. On concept-based explanations in deep neural networks. 2019.
[33] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE Iransactions neural networks and learning systems, 28( (6611))666-2677, 2016.
[34] Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature- based model explanations. arXiv preprint arXiv:2005.0065 2020.
[35] Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems, 35:2832-2845, 2022.
[36] Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R Lyu, and tu-Wing Tai. Towards global explanations of convolutional neural networks with concept attribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1652-8661, 2020.
[37] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. completeness-aver awarr concept-base explanations in deep neural networks. Advances in Neural Information Processing Systems,
2020.
[38] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[39] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. European conference on computer vision, pages 740-755 Springer, 2014.
[40] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In International conference on machine learning, pages 3319-3328. PMLR, 2017.
[41] Feature Ablation. https: btttps//captunn
http://efaauueaaabaaaaeeeee htmmllnn.mmm html 2023.
[42] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 271 1-221,,,, 2023.
[43] Yuanyuan Yuan, Qi Pang, and Shuai Wang. Unveiling hidden dnn defects with decision-basee metamorphic testing. In 37th IEEF/ACM International Conference on Automated Software Engineering, pages 1-13, 2022.
[44] G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000.
[45] Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, and Kang Li. Sam on medical images: A comprehensive study three prompt modes. arXiv preprint arXiv:2305.0000 00035, 2023.
[46] Lucas Prado Osco, Qiusheng Wu, Eduardo Lopes de Lemos, Wesley Nunes Gonçalves, Ana Paula Marques Ramos, Jonathan Li, and José Marcato Junior. The segment anything model (sam) for remote sensing applications: From zero to one shot. arXiv preprint ArXiv:2306.166) 2023.
[47] Liangliang Yao, Haobo Zuo, Guangze Zheng, Changhong Fu, and Jia Pan. Sam-da: Uav tracks anything at night with sam-powered domain adaptation. arXiv preprint MrXiiv2307.01022 2023.
12

---

[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[49] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1rXiv:1704.0461,, 2017.
[50] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks deep neural networks. IEEE Access, 7:44230-44244,, 2019.
[51] Yingqi Liu, Shiqing Ma. Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018.
[52] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[53] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack deep neural networks. In Computer Vision-ECCO 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 182-199. Springer, 2020.
13

---

Different Target Models
Table 4: ViT-bl6, 50 runs
EAC
DIT
GradSHAP
IntGrad
KernelSHAP
FeatAbl
LIME
ImageNet/Insertion
89.594
54,455
68. 185
69.480
75.152
65.656
76.161
CoCo/Insertion f
76.759
37.659
48.889
50.323
63,503
59.072
64.244
ImageNet/Delltion. 4
17.298
40.784
30.948
29.903
21.825
34.191
19.254
CoCo/Deletion
8.318
28.762
18.422
17.440
9.950
15.946
8,426
Table 5: MobileNNt-v2., 50 runs
EAC
DIe
GradSHAP
IntGrad
KernelSHAP
FeatAbl
LIME
ImageNe//Inserton
74.651
34.197
47.848
48.662
60.837
59.197
61.282
CoCo/Insertion f
68.556
28.951
37,393
37.719
48.658
44.420
50.387
ImageNet/Delltion |
6.002
26.381
14.679
13.382
7.766
8.866
7.344
CaCo/Deletion
6.684
21.467
14.237
14.936
9.308
11.706
7.106
Table 6: ResNet-18 18, 50 runs
EAC
DeepLIFT
GradSHAP
IntGrad
KemneSSHAP
FeatAbl
LIME
ImageNet/Insertion
73.558
47.799
38.877
36.806
50.547
43.448
50.592
CuCo/Insertio f
65.669
50.689
42.937
45.252
54.046
53.835
53.837
ImageNet/Delltion J
6.596
8.588
11.273
11.555
6.638
8.352
6.776
CoCo/Deletion 5.015
9.097
11.758
11.483
7.007
9.325
6.495
We explore the performance of EAC on different target models. We choose three representative visual models, including VIT [48), MobileNet [49]. and ResNet-18 M. and use the same experimental setup as in the main text. We run each method for 50 times to report the average performance of each method. Overall, observe a similar performance as shown in the main text. particular, EAC consistently outperforms other methods on all target models.
Backdoor Defense
Table Backdoor-Defense on CIFAR-10
ASR
Victim Model
EAC
DeepLIFT
GradSHAP
IntGrad
KemeeSHHA
FeatAbl
LIME
BadNet
50
0.99
0.042
0.542
0.622
0.618
0.91
0.47
0.574
TrojanNN [1
0.99
0.038
0.094
0.122
0.122
0.65
0.098
0.11
To evaluate the security impact of EAC, this section conducts backdoor removal experiments on CIFAR-I0 [52]. We compare EAC and other XAI methods. Specifically, we perform two representa- tive backdoor attacks, BadNet [50] and TrojanNN [5T). on ResNet-18 18 as the target model. During the evaluation process, aligned with relevant works in this field [53). we remove the top three patches among every poisoned image for each XAI tool, and record the corresponding Attack Success Rate
(ASR) after the removal. Overall, we randomly generate 250 poisoned images, and report their average ASR in Table 7 The evaluation results are highly encouraging: EAC has the lowest ASR under both attack settings. We interpret that this evaluation shows the high generalizability of EAC over different target models.
14

---

80
80
70
60
Inceronn Aur
90
Inser Inserton AUI
20-
20
20
KerneSShop Reacureiblstion
FeotureAbletion
20
100
100
Supprr Rouel
SuperRioel
(a) COCO, 100 runs
(b) ImageNet, 100 runs
Figure 3: The effect of the superpixel size on AUC. A trade-off can be observed.
Analysis of the Trade-off between SuperPixel Size and AUC
Overall, superpixel-base XAI tools are sensitive to the size of the superpixel. To obtain a fair comparison between EAC and de facto superpixel-based XAI tools, we carefully studied how the size of superpixel influence the performance of LIME, KernelShap, and FeatureAblation. The evaluation results using both ImageNet and COCO shown in Fig. 3 We observed that there exists a trade-off between AUC and the superpixel size for both datasets. Empirical observation shows that a proper range of the superpixel size ranges from 40 to 80.
To unleash the full capability of superpixel-basee based methods, we set the superpixel size as 75 for ImageNet evaluations, and 58 for COCO evaluations, respectively, when conducting the experiments the main paper. In contrast, EAC does not require such a hyperparameter tuning step, and is able to achieve superior performance over those superpixel-based methods.
15


Reviews:
[{"review_id": "65Be7sanAR", "reviewer": "Reviewer_xxGh", "summary": "The paper introduces Explain Any Concept (EAC), a concept-based explanation approach that enhances the interpretability of deep neural networks in computer vision. EAC utilizes the Segment Any Model (SAM) as a concept discovery technique, and the authors propose a lightweight Per-Input Equivalence (PIE) scheme that employs a proxy model to approximate the target model. This scheme reduces computational costs while maintaining reasonable fidelity, effectively addressing the challenges related to fidelity, comprehensibility, and efficiency in Explainable Artificial Intelligence (XAI). The effectiveness of EAC is demonstrated through experiments conducted on the ImageNet and COCO datasets, where it outperforms pixel-level and superpixel-based methods in terms of accuracy and interpretability. The authors emphasize the potential applications of EAC in various domains, including medical image analysis, autonomous driving, and robotics.", "questions": "May need proofreading:\nAll references in Supplementary Material are shown incorrectly.", "limitations": "Lack of real-world deployment evaluations: While this paper discusses the potential impact and applications of EAC in various domains, it does not provide real-world deployment evaluations or case studies to demonstrate the practical effectiveness of EAC in real-world scenarios.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The EAC is a novel concept-based interpretable method that integrates the recently released SAM. It leverages the zero-shot/few-shot capabilities of the segment anything model, addressing the limitations of pixel-based methods and the need for human annotation in concept-based approaches.\nThe proposed lightweight per-input equivalent (PIE) scheme improves the efficiency of the explanation process while maintaining high faithfulness and understandability.\nThe paper conducts extensive experiments and evaluations on popular datasets such as ImageNet and COCO to demonstrate the effectiveness of EAC. Comparisons with traditional pixel-level and superpixel-based XAI methods showcase the superior performance of EAC.", "weaknesses": "All datasets used in the paper are natural images in the general domain. However, it is important to include some knowledge-specific domains, such as medical images, for evaluation purposes.\nThe performance of the proposed methods can be affected by the SAM segmentation performance. It has already been found that SAM struggles to segment medical images or tiny-scale objects.\nSome failure cases should be included to provide a comprehensive overview of the model's faithfulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: why not include some knowledge-specific domain dataset to eval?**\n\nA1: Thank you for your comments, and we agree with you. Yes, our observation and exploration also show that the Meta SAM model was only trained on the general image domain, and it may struggle to segment images in knowledge-specific domains. \n\nAnd from a more general perspective, a common challenge for neural models is their poor generalization in domains that require specific knowledge. Besides SAM, even the latest and most advanced Large Language Models (LLMs) like ChatGPT/LLAMA suffer from this issue as well [1,2,3].\n\nHaving that said, with the recent development of knowledge-specific area SAM such as medical image [4], remote sensing [5], and UVA [6], we believe EAC has the potential to improve DNN decisions on other targeted areas. We leave it as one future work, as discussed in Section 6. Since this is the first attempt to bring humanly understandable and computationally feasible concept extractors to the field XAI, we believe our work can shed light on their explainability.\n\nIn revision, we would like to follow your suggestions to clarify this point (SAM’s limitation) in the paper; this shall better provide a fair and comprehensive overview of the model’s faithfulness without potential overclaims.\n\n\n[1]: Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond.\n\n[2]: Is ChatGPT a good translator? A preliminary study.\n\n[3]: The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain.\n\n[4]; SAM on Medical Images: A Comprehensive Study on Three Prompt Modes.\n\n[5]: The Segment Anything Model (SAM) for Remote Sensing Applications: From Zero to One Shot.\n\n[6]: SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation."}, {"review_id": "6YJdXwRGww", "reviewer": "Reviewer_6va6", "summary": "This article proposes EAC, which aims to use the Segment Anything Model to generate some prior concepts. By constructing a surrogate model, the concept combination area most relevant to the decision category is calculated by Shapley Value. The author evaluates the proposed model from three perspectives: faithfulness, understandability and effectiveness. Experimental results demonstrate the superiority of the proposed method.", "questions": "- Are the deletion and insertion indicators used by the author deleted according to the score of shapley value?\n- Since the results generated by SAM have no labels, can this segmentation result be called a concept?", "limitations": "The authors have discussed potential social implications.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- SAM-based prior methods will provide more accurate localization on conventional data than superpixel-based methods.\n- This approach can explain both CNN and ViT models (in supplementary material).\n- The authors provided the code to ensure reproducibility.", "weaknesses": "- The authors compare DeepLIFT, GradSHAP, and IntGrad methods, which use superpixel-based methods. Can the authors replace the superpixel method with the input result of Segment Anything to make the method comparison fairer?\n- Why is it necessary to train a surrogate model? Is this to improve inference efficiency?\n- Since this paper only uses SAM to generate prior knowledge, and then uses Shapley Value to estimate the importance score of prior concepts. I would like to see a concrete comparison with other Shapley Value based methods such as GradShap+SAM or FastShap+SAM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: compare our results with  GradShap+SAM or FastShap+SAM.**\n\nA1:\n\nThank you for this insightful question. We indeed explored this direction before, and our preliminary observation shows that this is unpromising; please see our response to **Q2** below on the conceptual-level clarification of “training a surrogate model.”\n\nFollowing your suggestion, below we launch more experiments to compare with baselines. Due to the large baselines for evaluation and the short timing window during rebuttal, we focus on evaluating the \"Adding\" operation: adding and removing operations share conceptually similar difficulty, as reflected in our results in the paper (tools behave consistently under \"Adding\" and \"Removing\" operations; as shown in Table 1, tools with better score in adding also preserve approximately the same performance ranking in deleting, such as EAC, LIME, DeepLift).\n\n| SOTA         | Imagenet adding | COCO adding |\n|--------------|-----------------|-------------|\n| KernelSHAP+sam  | 81.76           | 75.65       |\n| DeepLIFT+sam | 52.82           | 49.27       |\n| LIME+sam     | 79.85           | 77.50       |\n| EAC (Ours)     | 83.40           | 83.404      |\n| FeatAbl+sam  | 72.74           | 71.24       |\n| GradSHAP+sam | 44.47           | 41.37       |\n\nFrom the table above, we interpret that our EAC method consistently achieves the best result and outperforms others. In the COCO dataset, it is even six percentages higher than that of the second best (lime+sam). This empirically justifies the advantage of employing our customized lightweight surrogate model. \n\nWe’ll add the above results (with further evaluations on “Removing”) in revision. \n\n**Q2: Why is it necessary to train a surrogate model? Is this to improve inference efficiency?**\n\nA2: First, using a surrogate model appears to be a common practice in this field of research [1,2,3]; for instance, the seminal work, LIME, also uses surrogate models to approximate a neural network’s decision and ease XAI. \n\nNevertheless, Unlike LIME, the surrogate model in our work mainly serves to efficiently compute the Shapley value of each concept’s contribution (thus to answer your question: yes, it’s mainly for “improving inference efficiency”).\n\nOverall, computing the exact Shapley Value is computationally expensive O(2^(N)), so we take a common practice to employ a surrogate model that can estimate it [1,2,3]. To ensure that our surrogate model can produce the same output as the original model, we use Monte Carlo (MC) sampling; [2] has theoretically shown that MC makes the surrogate-provided Shapley Value closer to the true value of the target model, when more samples are made. Accordingly, we use a large number of MC samples (50000 iterations) for each experiment. Our empirical observation shows that this setting offers sufficiently good accuracy. We’ll note this in revision.\n\n[1] GraphSVX: Shapley Value Explanations for Graph Neural Network. ECML PKDD 2021.\n\n[2] Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms. VLDB 2019.\n\n[3] Shapley Computations Using Surrogate Model-Based Trees. IISA 2019.\n"}, {"review_id": "6HFaUp8dG7", "reviewer": "Reviewer_bcLx", "summary": "This work proposes EAC to study the interpretability of models. Instead of making element-wise explanations, EAC segments an input into sub-parts, then uses Shapley value to characterize important features for a model decision. User studies were conducted to show the explainability ability of this method.", "questions": "This paper is well presented with an interesting idea based on Shapley value. Since it’s mainly based on Shapley value technique, this may limit the originality contribution.", "limitations": "The authors discussed broader societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "In this work, the idea that splits an input into performing sub-parts and subsequence-level explanation are reasonable. The proposed  method is also easy to understand. The user studies show the effectiveness of EAC.", "weaknesses": "This method is built upon LIME with certain modifications specific to approximate the target model, which compromises the technical novelty of this work. And the selection of parameters is not clear to me, which should influence the final performance. For example, how to choose an optimal surrogate model suitable for datasets across different domains. Besides, the comparison methods in Tables 1 & 2 are vanilla so that they not quite convincing.  Maybe more stronger methods need to be considered. In addition, the baseline methods, i.e., LIME and KernelSHAP, used in Tables 1 & 2 are proposed in 2016 and 2017 respectively. More recent methods should be considered as baselines, for example, “On locality of local explanation models, NeurIPS 2021”, “Craft: Concept recursive activation factorization for explainability, CVPR 2023”, “RKHS-SHAP: Shapley values for kernel methods, NeurIPS 2022”. Please address the weakness parts in the rebuttal.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: EAC built upon LIME, compromise technical novelty**\n\nA1:  Thanks for the comment. Indeed, we found that the binary feature expression by LIME is very inspiring, such that we adopted those expressions in our pipeline. Nevertheless, there are two main major differences between our work EAC and LIME: 1) Concept-wise SAM+Shapley and 2) Per-Input Equivalence (PIE).\n\n1）Unlike LIME, we choose to use Shapley Value as our importance score measurement. This is because Shapley Value accounts for the average expected marginal contribution of a concept after all possible combinations fairly. In contrast, LIME only uses Super-Pixel for concept discovery, which is often less human-friendly and understandable than SAM extractor, as shown in Fig.2.\n\n2）To reduce the heavy computational burden of Shapley Value, we propose the scheme of Per-Input Equivalence (PIE). Moreover, while LIME only uses a set of linear weights as the surrogate model, ours not only mimics the feature extractor but also retains its original fully-connected layer.\n\nAlso, besides the above *technical novelty* comparison, we wish to clarify that our paper has *conceptual-level novelty* (i.e., we for the first time advocate using SAM as a concept discovery method to facilitate concept-based XAI and achieve high faithfulness and understandability) and also *highly encouraging empirical results*. We’ll better clarify our contributions in the revision.\n\n**Q2: unclear selection of hyperparam for EAC?**\n\nA2: Thanks for the question. We clarify that EAC does not require much “hyper-param.” The only hyper-param involved is when fitting the PIE Scheme, i.e. a simple linear neural network learning scheme and the Monte Carlo (MC) sampling: we set `lr=0.008` and the number of MC sampling as `50000` throughout all experiments. We’ll clarify this in the revision, and we also explain why using a MC sampling threshold of `50000` when answering the **Q2** of reviewer 6va6 above.\n\n**Q3: more and stronger baselines**\n\nA3:\n| SOTA  | Imagenet adding (higher the better) | Imagenet removing (lower the better) | COCO adding(higher the better) | Coco removing(lower the better) |\n|-------|-------------------------------------|--------------------------------------|--------------------------------|---------------------------------|\n| EAC   | 83.40                               | 23.799                               | 83.404                         | 16.640                          |\n| CRAFT | 60.40                               | 54.66                                | 51.49                          | 44.93                           |\n\n\nFollowing your suggestion, we launch more experiments to compare with baselines during the rebuttal phase. In particular, we report the performance of Craft (CVPR’23) compared with EAC under the same experiment setting in table 1. For Craft, we report its best performance after carefully tuning its hyper-param: patch_size=32, num_super-pixel=75/58 imagenet/coco. EAC outperforms CRAFT by 20 AUC percentage points in almost all cases.\n\nOn the other hand, RKHS-SHAP and Local Explanation are originally designed for numerical data with two dimensions (rows and columns). To adapt them for image tensors with four or three dimensions would take too much time and effort during the rebuttal timing window. We attempted to use the Local Explanation Models pipeline by choosing only the last fully-connected layer of the nn model, which produces a two-dimensional feature vector of size `n x 2048`, where `n` is the number of superpixel concepts. However, this did not work because the Local Explanation Models pipeline needs `n` to be larger than 2048, which is not feasible.\n\n"}, {"review_id": "9fF0kflGvD", "reviewer": "Reviewer_58qS", "summary": "This paper proposed EAC, which lines up SAM with XAI. The technique includes there phases: (1) generates concepts with SAM; (2) trains a surrogate model to represent the target model, using the same FC layer; (3) regards results of SAM in the first phases as players and calculates Shapley values with the surrogate model. Finally, obtain the masked image based on Shapley values as the visual explanation. ", "questions": "see the \"Weaknesses\"", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The first one to use SAM as a concept discovery method in the concept-based XAI.\n2. Good performance in the quantitative evaluation of faithfulness and user study.\n3. The paper is well-organized and easy to follow. ", "weaknesses": "1. Better to give some examples to show the trade-off. It’s hard to understand the trade-off mentioned in Sec.3.1. The highly faithful visual explanation map can show the importance of image pixels for the model to make a prediction accurately, so we use deletion/insertion to measure faithfulness. This is not conflict with human’s understanding. For example, through the explanation map from Grad-CAM or RISE, human can get where the model was looking at when did the prediction, so then human can make analysis of the model’s error mode.  \nAnd in [A, B], the user studies show that the faithfulness of the visual XAI map is consistent with human’s confidence (higher faithfulness map gets a higher rank in the user study), demonstrating that faithfulness and understandability are not conflict.\n\n[A] Chenyang Z, Chan A B. ODAM: Gradient-based Instance-Specific Visual Explanations for Object Detection[C]//The Eleventh International Conference on Learning Representations. 2023.\n\n[B] Petsiuk V, Jain R, Manjunatha V, et al. Black-box explanation of object detectors via saliency maps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 11443-11452.\n\n2. My major concern is the significance of the work, although the paper gives good quantitative evaluation results and discussions about the potential usage of EAC. \n- I’m not sure what the explanations are supposed to explain. From the samples in Figure 2, the explanation seems to be a correct segmentation of the salient object on the image, which corresponds to the classification result. The method is also tested on COCO, and I’m curious about the explanation in the situation of several same-class objects. In the fourth row, there are two zebras, and the explanation only marks out one of them; what does this mean? The model classifies the image as “zebra” because it has seen the right one, but not because of the other one?\n- The explanation map is supposed to help developers to understand the predictions and analyze the model. The paper should also provide some cases of applying EAC on wrong-classified samples. \n- I know that the baseline LIME also uses a surrogate model, but is that reasonable to use a surrogate model to replace the target model to be explained? How can we ensure that the same input (any coalition S of concepts) can always generate the same output between the two models? So why can use the explanation for the surrogate model to represent the explanation for the target model?\n\n3. Line 87, the reference of GradSHAP is actually for Grad-CAM. Reference for Grad-SHAP is missing.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: explain the trade-off in sec 3.1**\n\nA1: Sorry for the confusion. Our responses follow:\n\nFirst, the “trade-off” primarily refers to previous super-pixel based XAI methods (the mainstream), because it’s generally hard to decide an “one-size-fit-all’ superpixel size for previous works.\n\nWe present the following figure, showing that superpixel size mainly decides the faithfulness and understandability. The ResNet model predicts the original image as “ballplayer,” and we use EAC with two baselines from the main paper (Superpixel-based IntGrad and cluster pixel-based GradShap) to explain this prediction.\n\n[A figure has been submitted via the \"Official Comment\" button to the AC]\n\nAs shown in this figure, GradShap correctly identifies the player, but the cluster area is too small to be clear. Superpixel-based IntGrad is more readable than GradShap, but it is however imprecise. EAC strikes a balance by producing a well-shaped player that is easy to understand and also focuses on the ballplayer rather than irrelevant objects.\n\nOverall, a too large superpixel size can yield inaccurate yet more “complete” (and thus more readable) outputs, and vice versa for a too small superpixel size. Accordingly, the Figure 1 in our submitted Supplementary Material quantitatively reports that a too large/small superpixel size undermines the faithfulness. On the other hand, our EAC by design avoids this tradeoff problem, given that the size of the concept is intelligently extracted by SAM’s object detection.\n\nAgain, thank you for your advice, and we will extend the “trade-off” paragraph to avoid confusion.\n\n\n> \"And in [A, B], the user studies show that the faithfulness of the visual XAI map is consistent with human's confidence (higher faithfulness map gets a higher rank in the user study), demonstrating that faithfulness and understandability are not conflict.\"\n\nWe wish to clarify a confusion here. Those two works empirically justify that their methods have better faithfulness & understandability than previous works. In this regard, it is aligned with our paper, because our evaluation shows that EAC achieves better faithfulness and understandability. Those two papers are however not studying the inherent tradeoff across superpixel-based methods.\n\n**Q2: Explain the “correctness” in Fig 2 of what EAC should deliver to the audiences**\n\nA2: We answer this from three aspects:\n\nShapley Value by design measures the average expected marginal contribution of a concept after all possible combinations fairly. Namely, Shapley computes the exact importance of whether the model prefers the zebra on the left or on the right. Thus, it is technically correct to flag the right zebra as it contributes more.\n\nIn fact, as we can see in Fig.2, for any baseline using Shapley Value (GradShap, KernelShap, and EAC itself), they consistently prefer the zebra on the right. \n\nCompared with other Shapley baselines using super-pixel, output by our EAC method renders a clear and well-formed zebra to the audiences, which enhances human readability a step further.\n\n**Q3: What is our rationality to use a surrogate model replacing the target model? How can we ensure that the same input (any coalition S of concepts) can always generate the same output between the two models? So why can use the explanation for the surrogate model to represent the explanation for the target model?**\n\nA3: Unlike LIME, our aim is to calculate the Shapley Value of each concept’s contribution. Computing the exact Shapley Value is computationally expensive O(2^(N)), so we take a common practice to employ a surrogate model that can estimate it [1,2,3]. To ensure that our surrogate model can produce the same output as the original model, we use Monte Carlo (MC) sampling; [2] has theoretically shown that MC makes the surrogate-provided Shapley Value closer to the true value of the target model, when more samples are made. Accordingly, we use a large number of MC samples (50000 iterations) for each experiment. Our empirical observation shows that this setting offers sufficiently good accuracy. We’ll note this in revision.\n\n[1] Graphsvx: Shapley value explanations for graph neural networks. ECML PKDD 2021.\n\n[2] Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms. VLDB 2019.\n\n[3] Shapley Computations Using Surrogate Model-Based Trees. IISA 2019.\n\n\n**Q4: Line 87, the reference of GradSHAP is actually for Grad-CAM. Reference for Grad-SHAP is missing.**\n\nA4: Sorry about this and thank you for pointing it out; we will carefully proofread the paper and fix those errors.\n"}]

Author rebuttal:

Meta-review: This paper introduces Explain Any Concept (EAC), a method that leverages Segment Any Model (SAM) to enhance Explainable AI (XAI) for deep neural networks. SAM is used to provide instance segmentation for any concept and is incorporated into the Shapley method. A per-input equivalent (PIE) scheme is introduced to approximate SAM computation with a lightweight surrogate model. Experiments were conducted on ImageNet and COCO.

The paper initially received mixed reviews, with two reviewers recommending acceptance and three reviewers leaning towards rejection. The main concerns raised by reviewers related to the application domain of EAC, its positioning with respect to LIME, the discussion on faithfulness vs. understandability, and the need for clarification regarding experiments (such as stronger baselines, using SAM with other XAI baselines, and setting hyper-parameters). The rebuttal effectively addressed these concerns, and the three most negative reviewers increased their ratings to 5. As a result of the authors' feedback, a consensus among reviewers was reached for a (weak) paper acceptance.

The AC has thoroughly reviewed the submission and the discussions. Although the proposed method may appear incremental, the AC finds the combination of SAM and Shapley for XAI timely and well-conducted and described. The AC is convinced that this solution may be of great interest to the community and inspire follow-up works in explaining any concept. The AC thus recommends paper acceptance and encourages the authors to include the rebuttal in the final version of the paper.
Decision: Accept (poster)