SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines. We compare EAC with nine baseline methods: (1) DeepLIFT [23] and (2) GradSHAP [12] are two representative backpropagation-based methods, which have been noted in Sec. 2. (3) Int-Grad [40], a gradient-based method, yields explanation by computing the path integral of all the gradients in the straight line between an input and the corresponding reference input; (4) KernelSHAP [18] approximates Shapley values by solving a linear regression problem; (5) FeatAbl (Feature Ablation) is a frequently-used perturbation based approach for feature attribution [41]; (6) CRAFT [42] is an attribution-based method allowing heatmaps to identify the most influential regions of an image. (7) LIME [13], a popular perturbation-based method introduced in Sec. 2, uses a surrogate model to approximate the target model. To apply these methods, we first employ superpixel [27] to generate concept patches from the input image, and then calculate their concept importance. The details of the superpixel hyper-param setting and the corresponding experiment are provided in the Supplementary Material.
Besides, we also compare EAC with three variants of DeepLIFT, GradSHAP, and IntGrad, which leverages morphological operations [43] (e.g., image erosion and dilation [44]) as a postprocessing step to cluster pixels into concepts. We denote these variants as DeepLIFT*, GradSHAP*, and
IntGrad*, respectively. Overall, we compare EAC with six superpixel-based XAI methods and three postprocessing-based concept XAI.
Table 1: Comparison with baseline methods across four settings. For each setting, the first and the second row report the mean and std. dev. of the results of three runs, respectively.  $\uparrow$  and  $\downarrow$  indicate higher and lower is better, respectively.
EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAbl	LIME	CRAFT	DeepLIFT*	GradSHAP*	IntGrad*
ImageNet/Insertion ↑	83.400	75.235	64.658	68.772	64.544	70.187	76.638	60.40	14.707	14.794	15.120
0.023	0.000	0.035	0.000	0.002	0.000	0.027	0.000	0.000	0.067	0.000
CoCo/Insertion ↑	83.404	78.199	61.109	65.037	54.570	72.260	79.028	51.49	8.580	21.643	19.755
0.012	0.000	0.212	0.000	0.004	0.000	0.061	0.000	0.000	0.094	0.000
ImageNet/Deletion ↓	23.799	25.262	40.996	36.214	26.583	37.332	25.307	54.66	40.620	44.830	46.015
0.005	0.000	0.061	0.000	0.034	0.000	0.064	0.000	0.000	0.246	0.000
CoCo/Deletion ↓	16.640	17.026	34.038	30.074	20.054	26.535	17.337	44.93	49.697	35.302	38.148
0.041	0.000	0.144	0.000	0.040	0.000	0.049	0.000	0.000	0.173	0.000
We report the effectiveness of EAC and compare it with the baseline methods in Table 1. In particular, we observe that EAC consistently outperforms the baseline methods across all settings. For example, in the ImageNet dataset, EAC achieves  $83.400\%$  AUC for insertion, which is  $8.165\%$  higher than the second-best method DeepLIFT. Similarly, in the COCO dataset, EAC achieves  $83.404\%$  AUC for insertion, which is  $5.205\%$  higher than the second-best method DeepLIFT. We observe similarly promising results for the deletion evaluations. We also observe that the standard deviation of EAC is comparable to these baseline methods, which indicates that EAC is as stable as the majority of the baseline methods. Moreover, it is seen that the variants of DeepLIFT, GradSHAP, and IntGrad perform much worse than the original methods, which indicates the incapability of those standard morphological operations in clustering pixels, and the superiority of SAM in concept discovery.
4.2 EAC Understandability
To further explore the understandability of outputs from different XAI methods, we conduct a human evaluation to assess whether EAC can generate more human-understandable explanations than the baseline methods. To do so, we randomly select 100 images from the ImageNet and COCO datasets, respectively. We then generate eight explanations for each image using EAC and the seven baseline methods (for those three morphological operation-based variants in Table 1, we pick the best one in this evaluation). Then, for each image, we shuffle the explanations generated by EAC and the baseline methods, and ask our human participants (see below) to pick an explanation that they think is most understandable.
At this step, we recruit six participants to evaluate the explanations. We clarify that all participants are graduate students in relevant fields. We spent about 15 minutes to train the participants to understand our task. Before launching our study, we also provide a few sample cases to check whether the participants understand the task. To reduce the workload, for each image and its associated eight explanations, we randomly select three participants for the evaluation. Thus, each participant needs to evaluate 100 explanations. We report that each participant spends about 45 minutes to finish the evaluation.
Table 2: Human evaluation results.
EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAb1	Lime	GradSHAP*	Discord
ImageNet	70	6	3	4	5	1	2	2	7
COCO	67	7	3	5	3	5	1	0	9
We report the results in Table 2. Among 200 images in total, participants reach a consensus (i.e., at least two out of three participants favors the same explanation) on 184 images  $(92.0\%)$  with 93 images from ImageNet and 91 images from COCO. Among these 184 images, EAC is favored on 137 images  $(74.5\%)$  while the second best baseline method is only favored on seven images  $(7.7\%)$ . From Table 2, it is seen that among the baseline methods, no one is significantly better than the others.
Overall, we conclude that EAC is significantly better than the baseline methods in terms of understandability. This is an encouraging finding that is in line with our study in Table 1. Below, we present several cases to demonstrate the EAC's superiority in understandability.
DeepLIFT
SHAP
IntGrad
KernelSHAP
FeatAbl
GradSHAP*
Figure 2: Sample explanations generated by EAC and the baseline methods.
Case Study. We present several cases in Fig. 2 to demonstrate the effectiveness of EAC in explaining the model prediction. Besides these four cases, we observe that EAC consistently demonstrates its superiority in explaining the model prediction across other test cases. Overall, it is clear that EAC generates more "well-formed," concept-level explanations that are human understandable across all four cases in Fig. 2. For example, in the first case, EAC correctly highlights the "train" as the concept-level explanation, whereas the baseline methods yield some negligible pixels (the first three baselines), a fragment of the image (the 4th, 5th, and 6th baselines), or the entire image (the 7th baseline). This clearly illustrates the superiority of EAC in terms of the faithfulness and understandability.
4.3 Ablation Study of the PIE Scheme
In this experiment, we explore the effectiveness of our proposed PIE scheme. We compare the PIE scheme with three baselines that compute the Shapley value using 1) the original model under explanation, 2) a standard linear surrogate model and, 3) the same model in our PIE scheme while parameter sharing is disabled. In all settings, we use the same Monte Carlo sampling method to approximate the Shapley value while using different schemes to represent the target DNN model.
First, for the setting of directly using the original model, we observe a significantly longer processing time than the others. Indeed, we report that it takes more than 24 hours to process one image when using the same Monte Carlo sampling method. When we slightly reduce the number of samples, the processing time is still much longer than the other three methods (two baselines and EAC). Similarly, we find that the AUC is also much lower than that of the other three methods. As a result, we deem that the original model is impractical for computing the Shapley value and a surrogate model is necessary. Accordingly, we omit reporting the results of the original model and mark its results as "N/A" in Table 3.
Table 3: Ablation study of the PIE scheme. For each setting, we report the mean of the results of ten runs. Aligned with Table 1,  $\uparrow$  and  $\downarrow$  indicate higher and lower is better, respectively.
Model	AUC	Processing Time (sec.)	Model	AUC	Processing Time (sec.)
ImageNet/Insertion ↑	PIE	81.78	245	ImageNet/Deletion ↓	PIE	12.47	244
Original Model	N/A	N/A	Original Model	N/A	N/A
PIE w/o PS	50.40	288	PIE w/o PS	32.87	289
Linear Model	78.11	36	Linear Model	14.08	31
COCO/Insertion ↑	PIE	87.08	252	COCO/Deletion ↓	PIE	13.71	203
Original Model	N/A	N/A	Original Model	N/A	N/A
PIE w/o PS	42.86	250	PIE w/o PS	38.36	222
Linear Model	74.86	67	Linear Model	14.36	131
We report the results of our PIE scheme and the baselines in Table 3. Overall, we interpret the evaluation results as highly encouraging: the PIE scheme is notably better than all three baselines. In particular, the ablated PIE scheme without parameter sharing (the "PIE w/o PS" rows in Table 3) is
significantly worse than the PIE scheme in terms of both AUC and the processing time. This indicates that parameter sharing is effective in reducing the processing time while preserving high accuracy. Moreover, when comparing with the linear surrogate model (the "Linear Model" rows in Table 3), the PIE scheme is consistently better in terms of AUC. This indicates that the PIE scheme is more accurate, because the linear surrogate model over-simplifies the target model. Overall, we interpret that this ablation study illustrates the necessity of our PIE scheme, which empowers high faithfulness and efficiency to our technical pipeline.
In this section, we first analyze the generalizability of EAC from the following three aspects. We then present statements on the security impact of EAC. Evaluations noted in this section are presented in the Supplementary Material.
Different Target Models. In this paper, we primarily use ResNet-50 as the target DNN model to evaluate EAC. Nevertheless, it is evident that the technical pipeline of EAC is independent of particular target DNN models. Further evaluation of EAC's performance using different target DNN models is presented in the Supplementary Material. In short, our findings demonstrate the persistent superiority of EAC over baselines when explaining different target DNN models.
Different Visual Domains. The concept extraction process for EAC is executed using the SAM framework, which means that it functions optimally on identical visual domains to SAM. It is advocated that SAM can be effective on a wide range of visual domains, including medical images, simulation images, and painting images [20]. However, our preliminary study over some other visual domains illustrate potentially suboptimal performance of SAM. Overall, it is clear that the accuracy of the concept set identified by SAM significantly impacts the performance of EAC, and ill-identified concepts may lead to suboptimal results of EAC. We remain exploring further applications of EAC on other visual domains for future research. Also, with the recent development of knowledge-specific area SAMs, such as medical image [45], remote sensing [46], and UVA [47], we believe that EAC has the potential to improve DNN decisions in other targeted areas in the future. Since this is the first attempt to deliver humanly understandable and computationally feasible concept extractors to the field XAI, we believe our work can shed light on their explainability.
Different Visual Tasks. This paper illustrates the effectiveness of EAC on interpreting image classification, a common and core task widely studied in previous XAI research. However, EAC is not limited to image classification. In fact, the technical pipeline of EAC can be applied and extended to other common visual tasks that can be explained using Shapley value-based explanations, such as object detection. We leave exploring EAC's potential applications on other visual tasks as future work.
Security Impact. Analyzing and detecting backdoors in DNNs is an emerging downstream application of XAI techniques. While the main paper primarily focuses on the core functionality of EAC, we also consider the security impact of EAC. In particular, we present evaluations of using EAC to analyze and detect backdoors in DNNs in the Supplementary Material. We report promising results and present discussions on potential future improvement accordingly.
6 Conclusion and Impact Statement
In this paper, we propose EAC, a novel method for explaining the decision of a DNN with any concept. EAC is based on SAM, a powerful and promotable framework for performing precise and comprehensive instance segmentation over a given image. We propose a highly efficient neural network pipeline to integrate the SAM and shapley value techniques. We conduct extensive experiments to demonstrate the effectiveness and interpretability of EAC.
Broader Impact. EAC is a general framework for explaining the prediction of a DNN with any concept. It can be used in many applications, such as medical image analysis, autonomous driving, and robotics. For example, EAC can be used to explain the prediction of a DNN for a medical image with any concept, such as a tumor. This can help doctors to understand the prediction of the DNN and improve the trustworthiness of the DNN. Overall, we believe that EAC can be used to improve
the trustworthiness of DNNs in many applications, and we publicly release and maintain EAC to facilitate its adoption in the real world.
However, in terms of Negative Societal Impacts, EAC may also be used to explain the prediction of a DNN for a medical image with a concept that is not related to the medical image, such as a cat, a train, or some subtle errors. This may mislead doctors and cause serious consequences. Therefore, it may cause harm when using EAC in real-world sensitive domains without sufficient safety checks on its outputs. We will continue to improve EAC to reduce its negative societal impacts.
We thank the anonymous reviewers for their insightful comments. This research is supported in part by the HKUST 30 for 30 research initiative scheme under the contract Z1283.
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[2] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[4] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access, 6:52138-52160, 2018.
[5] Jiahui Li, Kun Kuang, Lin Li, Long Chen, Songyang Zhang, Jian Shao, and Jun Xiao. Instance-wise or class-wise? a tale of neighbor shapley for concept-based explanation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3664-3672, 2021.
[6] Janet Hui-wen Hsiao, Hilary Hei Ting Ngai, Luyu Qiu, Yi Yang, and Caleb Chen Cao. Roadmap of designing cognitive metrics for explainable artificial intelligence (xai). arXiv preprint arXiv:2108.01737, 2021.
[7] Lukas-Valentin Herm. Impact of explainable ai on cognitive load: Insights from an empirical study. arXiv preprint arXiv:2304.08861, 2023.
[8] Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. Xinsight: explainable data analysis through the lens of causality. Proceedings of the ACM on Management of Data, 1(2):1-27, 2023.
[9] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[10] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3437, 2017.
[11] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[12] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[49] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[50] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
[51] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018.
[52] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[53] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pages 182-199. Springer, 2020.
Different Target Models
Table 4:ViT-b16,50 runs
EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAbl	LIME
ImageNet/Insertion ↑	89.594	54.455	68.125	69.480	75.152	65.656	76.161
CoCo/Insertion ↑	76.759	37.659	48.888	50.323	63.503	59.072	64.244
ImageNet/Deletion ↓	17.298	40.784	30.948	29.903	21.825	34.191	19.254
CoCo/Deletion ↓	8.318	28.762	18.422	17.440	9.950	15.946	8.426
Table 5:MobileNet-v2,50 runs
EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAbl	LIME
ImageNet/Insertion ↑	74.651	34.197	47.848	48.662	60.837	59.197	61.282
CoCo/Insertion ↑	68.556	28.951	37.393	37.719	48.658	44.420	50.387
ImageNet/Deletion ↓	6.002	26.381	14.679	13.382	7.766	8.866	7.344
CoCo/Deletion ↓	6.684	21.467	14.237	14.936	9.308	11.706	7.106
Table 6: ResNet-18, 50 runs
EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAbl	LIME
ImageNet/Insertion ↑	73.558	47.799	38.877	36.806	50.547	43.448	50.592
CoCo/Insertion ↑	65.669	50.689	42.937	45.252	54.046	53.835	53.837
ImageNet/Deletion ↓	6.596	8.588	11.273	11.555	6.638	8.352	6.776
CoCo/Deletion ↓	5.015	9.097	11.758	11.483	7.007	9.325	6.495
We explore the performance of EAC on different target models. We choose three representative visual models, including ViT [48], MobileNet [49], and ResNet-18 [1], and use the same experimental setup as in the main text. We run each method for 50 times to report the average performance of each method. Overall, we observe a similar performance as shown in the main text. In particular, EAC consistently outperforms other methods on all target models.
Backdoor Defense
Table 7: Backdoor-Defense on CIFAR-10
ASR	Victim Model	EAC	DeepLIFT	GradSHAP	IntGrad	KernelSHAP	FeatAbl	LIME
BadNet [50]↓	0.99	0.042	0.542	0.622	0.618	0.91	0.47	0.574
TrojanNN [51]↓	0.99	0.038	0.094	0.122	0.122	0.65	0.098	0.11
To evaluate the security impact of EAC, this section conducts backdoor removal experiments on CIFAR-10 [52]. We compare EAC and other XAI methods. Specifically, we perform two representative backdoor attacks, BadNet [50] and TrojanNN [51], on ResNet-18 as the target model. During the evaluation process, aligned with relevant works in this field [53], we remove the top three patches among every poisoned image for each XAI tool, and record the corresponding Attack Success Rate (ASR) after the removal. Overall, we randomly generate 250 poisoned images, and report their average ASR in Table 7. The evaluation results are highly encouraging; EAC has the lowest ASR under both attack settings. We interpret that this evaluation shows the high generalizability of EAC over different target models.
(a) COCO, 100 runs
(b) ImageNet, 100 runs
Figure 3: The effect of the superpixel size on AUC. A trade-off can be observed.
Analysis of the Trade-off between SuperPixel Size and AUC
Overall, superpixel-based XAI tools are sensitive to the size of the superpixel. To obtain a fair comparison between EAC and de facto superpixel-based XAI tools, we carefully studied how the size of superpixel influence the performance of LIME, KernelShap, and FeatureAblation. The evaluation results using both ImageNet and COCO are shown in Fig. 3. We observed that there exists a trade-off between AUC and the superpixel size for both datasets. Empirical observation shows that a proper range of the superpixel size ranges from 40 to 80.
To unleash the full capability of superpixel-based methods, we set the superpixel size as 75 for ImageNet evaluations, and 58 for COCO evaluations, respectively, when conducting the experiments in the main paper. In contrast, EAC does not require such a hyperparameter tuning step, and is able to achieve superior performance over those superpixel-based methods.