Less is More: Fewer Interpretable Region via Submodular Subset Selection
Authors: Ruoyu Chen
Hua Zhang
Siyuan Liang
Jingzhi Li
Xiaochun Cao
Venue: ICLR 2024 oral (2024)
Keywords: Interpretable AI
Submodular subset selection
Explainable AI
Image Attribution
Primary area: visualization or interpretation of learned representations
TL;DR: This paper re-models the image attribution problem as a submodular subset selection problem, aiming to enhance model interpretability using fewer regions.

Abstract: Image attribution algorithms aim to identify important regions that are highly relevant to model decisions. Although existing attribution solutions can effectively assign importance to target elements, they still face the following challenges: 1) existing attribution methods generate inaccurate small regions thus misleading the direction of correct attribution, and 2) the model cannot produce good attribution results for samples with wrong predictions. To address the above challenges, this paper re-models the above image attribution problem as a submodular subset selection problem, aiming to enhance model interpretability using fewer regions. To address the lack of attention to local regions, we construct a novel submodular function to discover more accurate small interpretation regions. To enhance the attribution effect for all samples, we also impose four different constraints on the selection of sub-regions, i.e., confidence, effectiveness, consistency, and collaboration scores, to assess the importance of various subsets. Moreover, our theoretical analysis substantiates that the proposed function is in fact submodular. Extensive experiments show that the proposed method outperforms SOTA methods on two face datasets (Celeb-A and VGG-Face2) and one fine-grained dataset (CUB-200-2011). For correctly predicted samples, the proposed method improves the Deletion and Insertion scores with an average of 4.9\% and 2.5\% gain relative to HSIC-Attribution. For incorrectly predicted samples, our method achieves gains of 81.0\% and 18.4\% compared to the HSIC-Attribution algorithm in the average highest confidence and Insertion score respectively. The code is released at https://github.com/RuoyuChen10/SMDL-Attribution.

Published as a conference paper at ICLR 2024
LESS IS MORE: FEWER INTERPRETABLE REGION VIA SUBMODULAR SUBSET SELECTION
Ruoyu Chen Hua Z Siyuan Liang', Jingzhi , Xiaochun Cao Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China Schooo of Cyber Security, University Chinese Academy of Sciences, Beijing 100049, China [chenruoyu, zhanghua, 11 1 jinghi e@i aegie.ase en "Scoool of Computing, National University of Singapore, 119077, Singapore
pandal i ang521 gmail com
*School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China
ca ox i cocoochunmmm @ma il sysu. www..edu.uc.o. edu.cr.
ABSTRACT
Image attribution algorithms aim identify important regions that are highly rel- evant to model decisions. Although existing attribution solutions can effectively assign importance to target elements, they still face the following challenges: existing attribution methods generate inaccurate small regions thus misleading the direction of correct attribution, and 2) the model cannot produce good attribution results for samples with wrong predictions. To address the above challenges, this paper re-models the above image attribution problem as a submodular subset se- lection problem, aiming to enhance model interpretability using fewer regions. To address the lack of attention to local regions, we construct a novel submodu- lar function discover more accurate small interpretation regions. To enhance the attribution effect for all samples, we also impose four different constraints the selection of sub-regions, i.e., confidence, effectiveness, consistency, and collaboration scores, to assess the importance of various subsets. Moreover, our theoretical analysis substantiates that the proposed function is in fact submodu- lar. Extensive experiments show that the proposed method outperforms SOTA methods on two face datasets (Celeb-A and VGG-Face2) and one fine-gra ane-grained dataset (CUB-200-201 1). For correctly predicted samples, the proposed method improves the Deletion and Insertion scores with an average of 4.9% and 2.5% gain relative to HSIC-Attribution For incorrectly predicted samples, our method achieves gains of 81.0% and 18.4% compared to the Attr-butnibttion algorithm in the average highest confidence and Insertion score respectively. The code is WOS.L/CCCCCOOSASSAAAAAAAAA....
released at
INTRODUCTION
Building transparent and explainable artificial intelligence (XAI) models is crucial for humans to reasonably and effectively exploit artificial intelligence (Dwivedi al., 2023; Ya al., 2024; Li et al., b0 Tu et al., 2023; Liang et al., 2022a;b; 2023b). Solving the interpretability and security problems of AI has become an urgent challenge in the current field of AI research (Arrieta al., 2020; Chen et al., 2023; Zhao et al., 2023; Liang et al., 2023a; 2024; Li et al., 2023). Image attribu- tion algorithms are typical interpretable methods, which produce saliency maps that explain which image regions are more important to model decisions. It provides a deeper understanding of the op- erating mechanisms of deep models, aids in identifying sources of prediction errors, and contributes to model improvement. Image attribution algorithms encompass two primary categories: white-box methods (Chattopadhay et al., 2018; Wang et al., 2020) chiefly attribute on the basis of the model's internal characteristics decision gradients, and black-box methods (Petsiuk et al., 2018; Novello et al., 2022), which primarily analyze the significance of input regions via external perturbations.
Although attribution algorithms have been well studied, they still face two following limitations: 1) Some attribution regions are not small and dense enough which may interfere with the optimization
"Corresooniigg Author

---

Published as a conference paper at ICLR 2024
Sample correctlyy predieted by the model
Sample incorrectly predieted by the model
Ansetioon
Origial proditkon oofilane
Confdence the regias viceoonddd: 008543
6.75%
Predicion: Grouse Trath: Landi Baving
Pooeriing of langgh worerca
For correctly predicted samples, our method can find fewer regions that make the model predictions more confident. For incorrectly predicted samples, our method can find the reasons that caused the model to predict incorrectly.
Figure 1:
orientation. Most advanced attribution algorithms focus on understanding how the global image contributes to the prediction results of the deep model while ignoring the impact of local regions the results. 2) It difficult to effectively attribute the latent cause of the error to samples with incor- rect predictions. For incorrectly predicted samples, although the attribution algorithm can assign the correct class, it does not limit the response to the incorrect class, so that the attributed region may still have high response to the incorrect class.
To solve the above issues, we propose a novel explainable method that reformulates the attribution problem submodular subset selection problem. We hypothesize that local regions can achieve better interpretability, and hope to achieve higher interpretability with fewer regions. We first di- vide the image into multiple sub-regions and achieve attribution by selecting a fixed number of sub-regions to achieve the best interpretability. To alleviate the insufficient dense of the attribution region, we employ regional search to continuously expand the sub-region set. Furthermore, we introduce new attribution mechanism to excavate what regions promote interpretability from four aspects, i... the prediction confidence of the model, the effectiveness of regional semantics, the con- sistency of semanties, and the collective effect of the region. Based on these useful clues, design a submodular function to evaluate the significance of various subsets, which can limit the search for regions with wrong class responses. As shown in Fig. we find that for correctly predicted samples, the proposed method can obtain higher prediction confidence with only a few regions as input than with all regions as input, and for incorrectly predicted samples, our method has the ability to find the reasons (e.g., the dark background region) that caused its prediction error.
We validate the effectiveness of our method VGG-Face2 (Cao et al., 2018), and fine-grained dataset CUB-200-200 (Welinder et al., 2010). For correctly predicted samples, compared with the SOTA method HSIC-Attribution (Novello al., 2022), our method improves the Deletion scores by 8.4%, 1.0%, and 5.3%, and the Insertion scores by 1.1% 0.2%, and 6. 1%, respectively. For incorrectly predicted samples, our method excels in identifying the reasons behind the model's decision- decision-making errors. In particular, compared with the SOTA method HSIC-Attribution, the average highest confidence is increased by 81.0%, and the Insertion score is increased by 18.4% on the CUB-200-201 I dataset. Furthermore, our analysis at the theoretical level confirms that the proposed function is indeed submodular. Please see Appendix for list of important notations employed in this article.
face datasets, i.e., Celeb-A (Liu et al., 2015),
Our contributions can be summarized as follows:
We reformulate the attribution problem as a submodular subset selection problem that achieves higher interpretability with fewer dense regions.
A novel submodular mechanism is constructed to evaluate the significance various sub- from four different aspects, i.e., the confidence, effectiveness, consistency, and collab- oration scores. It not only improves the dense of attribution regions for existing attribution algorithms but also better discovers the causes of image prediction errors.
The proposed method has good versatility which can improve the interpretability of both on the face and fine-grained datasets. Experimental results show that it can achieve better attribution results for the correctly predicted samples and effectively discover the causes of model prediction errors for incorrectly predicted samples.
2 RELATED WORK
White-Box Attribution Method Image attribution algorithms designed to ascertain the im- portance of different input regions within an image with respect to the the model. White-box attribution algorithms primarily rely computing gradients with respect
process of
2

---

Published as a conference paper at ICLR 2024
the model's decisions. Early research focused assessing the importance of individual input pixels in mmage-based model decision decision-making (Bachrens et al., 2010; Simonyan et al., 2014). Some meth- ods propose gradient integration to address the vanishing gradient problem encountered in carlier approaches (Sundararajan et al., 2017; Smilkov et al., 2017). Recent popular methods often focus on feature-level activations in neural networks, such as CAM (Zhou al., 2016), Grad-CAM (Sel- varaju et al., 2020), Grad-CAM++ (Chattopadhay et al., 2018), and Score-CAM (Wang et al., 2020). However, these methods all rely on the selection of network layers in the model, which has a greater impact on the quality of interpretation (Novello et al., 2022).
Black-Box Attribution Method Black-box attribution methods assume that the internal structure of the model is agnostic. LIME (Ribeiro et al., 2016) locally approximates the predictions of a black- box model with a linear model, by only slightly perturbing the input. RISE (Petsiuk et al., 2018) perturbs the model by inputting multiple random masks and weighting the masks to get the final saliency map. HSIC-Atribution method (Novello et al., 2022) measures the dependence between the input image and the model output through Hilbert Schmidt Independence Criterion (HSIC) based on RISE. Petsiuk al. (2021) propose D-RISE to explain the object detection model's s decision.
Submodular Optimization Submodular optimization (Fujishige, 2005) has been successfully studied in multiple application scenarios (Wei et al., 2014; Yang et al., 2019; Kothawade et al., 2022; Joseph et al., 2019), small amount of work also uses its theory to do research related to model interpretability. Elenberg et al. (2017) frame the interpretability of black-box classifiers as combinatorial maximization problem, it achieves similar results to LIME and is more efficient. Chen et al. (2018) introduce instance- wwie wise feature selection to explain the deep model. Pervez et al. (2023) proposed a simple subset sampling alternative based on conditional Poisson sampling, which they applied to interpret both image and text recognition tasks. However, these methods only re- tained the selected important pixels and observed the recognition accuracy (Chen et al., 2018). In this article, we elucidate our model based on submodular subset selection theory and attain state-of- the-art results, as assessed using standard metrics for attribution algorithm evaluation. Our method also effectively highlights factors leading to incorrect model decisions.
3 PRELIMINARIES
In this section, we first establish some definitions. Considering finite set V, given set function F : 2V - R that maps any subset 5 C V to a real value. When F is a submodular function, its definition is as follows:
Definition 3.1 (Submodular function (Edmonds, 1970)). For any set S. S S, C V. Given an element a where a V \ Sb. The set function F submodular function when it satisfies monotonically non-decreasing (F (S, U {a]] - F (Sa) 2 0) and:
F FS. (a)) - F (S.) 2 (S, U (a)) (S.)).
Problem formulation We divide image into finite number of sub-regions, denoted as V = (IM . I where M indicates a sub-region I' formed by masking part of image I. Giving a monotonically non-decreasing submodular function F 2 + R, the image recognition attribution problem
be viewed as maximizing the value F (S) with limited regions. Mathemat- ically, the goal is to select a set S consisting of a limited number k of sub-regions in the set that maximize the submodular function F::
max SC ScV.skk
F(S),
(2)
we can transform the image attribution problem into a subset selection problem, where the submod- ular function F relates design to interpretability.
4 PROPOSED METHOD
In this section, we propose a novel method for image attribution based on submodular subset selec- tion theory. In Sec. 4.1 we introduce how to perform sub-region division, in Sec. 4.2 introduce our designed submodular function, and in Sec. 4.3 we introduce attribution algorithm based on greedy search. Fig. 2 shows the overall framework of our approach.
3

---

Published as a conference paper at ICLR 2024
Evaluation Metric (Instrtion AUC score)
Input Image
Artibution Map
Submedular Function
0.8
Oug, Altibutiin Methed Ouns
Amtiiaation Mosian Medinn
Confidenss
14%
Ellectiennes Soare (Bq
Senter Selection
Consisency Scoce
20%
40%
100%
Collaboration Score (Eq.
Elanera Devision
Combinstio
Elensent S
Selection Set.
Searched Region
Regina Importacce
Figure 2: The framework of the proposed method.
4.1 SUB-REGION DIVISION
To obtain the interpretable region in an image, we partition the image I E R2xxxx3 into m sub- regions IM where M indicates sub-region 1 formed by masking part of image Traditional methods (Noroozi & Favaro, 2016; Redmon & Farhadi, 2018) typically divide images into regu- lar patch areas, neglecting the semantic information inherent in different regions. contrast, our method employs sub-region division strategy that is informed and guided by an priori saliency map. detail, we initially divide the image into N x patch regions. Subsequently, an existing image attribution algorithm is applied to compute the saliency map A E RRxx for correspond- ing class I. Following this, we resize A N x N dimension, where its values denote the importance of each patch. Based on the determined importance of each patch, d patches are sequen- tially allocated to each sub-region IM, while the remaining patch regions are masked with 0, where d = N N/m. This process finally forms the element set V (I I IMM satisfying the condition I Σ I. The detailed calculation process outlined in Algorithm
4.2 SUBMODULAR FUNCTION DESIGN
Confidence Score We adopt a model trained by evidential deep learning (EDL. to quantify the uncertainty of sample (Sensoy et al., 2018), which is developed to overcome the limitation of softmax-basee networks the open-set environment (Bao et al., 2021). We denote the network with prediction uncertainty by F.(-). For K-class classification, assume that the predicted class probability values follow a Dirichlet distribution. When given sample x, the network is optimized by the following loss function:
K
LEDL = Σ Yk. (log Spir log (Ck, + 1)),
(3)
N==1
where y is the one-hot label of the sample x, and e E R is the output and the learned evidence of the network, denoted as e exp (F,, (x)). Spi Σ (ek, + 1) is referred to as the Dirichlet strength. In the inference, the predictive uncertainty can be calculated as K/Spir. where [0, 1]. Thus, the confidence score of the sample x predicted by the network can be expressed as:
K
Sconf. (x) = u
(4)
Σk_
(ek. +
By incorporating the Sconf,, we can ensure that the selected regions align closely with the In- Distribution (InD). This score acts reliable metric to distinguish regions from out-of-distribution, ensuring alignment with the InD.
Effectiveness Score We expect to maximize the response of valuable information with fewer re- gions since some image regions have the same semantic representation. Given element a, and
4

---

Published as a conference paper at ICLR 2024
sub-set S, we measure the distance between the element and all elements in the set, and calculate the smallest distance, as the effectiveness score of the judgment element a for the sub-set S:
Se (ax () min dist (F (a), F (s:)), S.
(5)
where .)st(:;.) denotes the equation to calculate the distance between two elements. Traditional distance measurement methods (Wang et al., 2018; Deng et al., 2019) are tailored to maximize the decision margins between classes during model training, involving operations like feature scaling and increasing angle margins. In contrast, our method focuses solely on calculating the relative distance between features, for which we utilize the general cosine distance. F(-) denotes a pre- trained feature extractor. To calculate the element effectiveness score of set, can compute the sum of the effectiveness scores for each element:
Seff. (S)
Σ min dist (F:), $IES 85E8,
By incorporating the Seff., we aim to limit the selection of regions with similar semantic representa- tions, thereby increasing the diversity and improving the overall quality of region selection.
Consistency Score As the scores mentioned previously may be maximized by regions that aren't 't target-dependent, we aim to make the representation of the identified image region consistent with the original semantics. Given a target semantic feature. fx. we make the semantic features of the searched image region close to the target semantic features. We introduce the consistency score:
((EMES ES 1 fz Scons. (5) = | (EEMES IM) l[f
where F(-) denotes pre-trained feature extractor. The target semantic feature fs. can either adopt the features computed from the original image using the pre-trained feature extractor, expressed as
f= F (I), or directly implement the fully connected layer of the classifier for a specified class. By incorporating the scons., our method targets regions that reinforce the desired semantic response. This approach ensures precise selection that aligns closely with our specific semantic goals.
Collaboration Score Some individual elements may lack significant individual effects in isola- tion, but when placed within the context of group system, they exhibit an indispensable collec- tive effect. Therefore, we introduce the collaboration score, which is defined as:
F (I- (I-EIMES EIMES IM) fx Scolla. (S, = - | (I- EiMeS IM]
(8)
II
where F-)) denotes a pre-trained feature extractor, f. is the target semantic feature. By introduc- ing the collaboration score, we can judge the collective effect of the element. By incorporating the Scolla., our method pinpoints regions whose exclusion markedly affects the model's predictive confi- dence. This effect underscores the pivotal role of these regions, indicative significant collective impact. Such a metric is particularly valuable in the initial stages of the search, highlighting regions essential for sustaining the model's s accuracy and reliability.
We construct our objective function for selecting elements through com- bination of the above scores, F(S), as follows:
Submodular Function
F(S)= X1 Scnnf.
Σ M 1 ES
A2Seft. (S) AgSeons. fs) + A48colla. (S
where \1, A2, Ag. and da represent the weighting factors used to balance each score. To simplify parameter adjustment, all weighting coefficients are set to by default.
Lemma 1. Consider two sub-sets Sa and S₁ in set V, where S. S C. Given an element a, where a C VV \ Sb. Assuming that a is contributing to model interpretation, then, the function F(.) in Eq. 9 a submodular function and satisfies Eq. I.
Proof. Please see Appendix B.1 for the proof.
5

---

TM T: Published as a conference paper at ICLR 2024
Lemma 2. Consider subset S, given element a, assuming that is contributing to model interpretation. The function F() of Eq. 9 is monotonically non-decreasing.
Proof. Please see Appendix B.2 for the proof.
4.3 GREEDY SEARCH ALGORRIHM
Algorithm 1: A greedy search based algorithm for interpretable region discovery
Input: Tinage I E Ruxaxa, mumher of divided patches N x N, prior saliency map A E R x4x3 of I, mumber of image division sub-regions maximum number of constituent regions k. Output: A set 9 C V. where [S] S k.
/x initiate the operation of sub-regicn division
A resize (A,, newRows N, newCols N):
d x N/m: for
for to
for;
to N
rank (A)
1x Index of Assr ordered descendingly v/
1) and > >d then
1 [1, x +1: b/N, x) +1: w/N non-relevant patch region, fill it with x7
Kask
11
end
12
13
1
14
end
15
Ø
/+ Initiate the operaticn of submedular subset selection +/
16
for
to do
Sa V\S:
18
- arg max.es, (S {a]);
/+ uptimize the submodular value +/
19
S + Su {a}
20
end
Given a set V = [IM, II
IM) we can follow Eq. 2 to search the interpretable region by selecting l elements that maximize the value the submodular function F(.). The above prob- lem be effectively addressed by implementing greedy search algorithm. Referring related works (Mirzasoleiman et al., 2015; Wei et al., 2015), we use Algorithm to optimize the value of the submodular function. Based Lemma and Lemma we have proved that the function F(-) is a submodular function. According to the theory of Nemhauser et al. (1978), we have:
Theorem ((Nemhauser al., 1978)). Let S denotes the solution obtained by the greedy search approach, and S* denotes the optimal solution. F()) is submodular function, then the solution has the following opproximation guarantee:
- F(S*),
F(S) 2
(10)
where is the base of natural logarithm.
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Datasets We evaluate the proposed method on two face datasets Celeb-A (Liu et al., 2015) and VGG-Face2 (Cao et al., 2018). and fine-gratd fine-grained dataset CUB-200-2011 (Welinder et al., 2010). Celeb-A dataset includes 10, 177 IDs, we randomly select 2, 000 identities from Celeb-A's validation set, and one test face image for each identity is used to evaluate our method; the VGG-Face2 dataset includes 8, 631 IDs, we randomly select 2, 000 identities from VGG-Face2's validation set, and one test face image for each identity is used to evaluate our method: CUB-200-201 dataset includes 200 bird species, we select 3 samples for each class that is correctly predicted by the model from the validation set for 200 classes, total of 600 images are used to evaluate the image attribution effect. Additionally, 2 incorrectly predicted samples from each class are selected, totaling 400 images, to evaluate method for identifying the cause of model prediction emros..

---

Published as a conference paper at ICLR 2024
Table 1: Deletion and Insertion AUC scores on the Celeb-A, VGG-Face2, and CUB-200-2011 vali-
dation sets.
Celeb-A
VGGFace2
CUB-200-2011
Method
Deletion (U)
Insertion (1)
Deletion (4)
Insertion (f)
Deletion (L)
Insertion (f)
Saliency (Simonyan et al., 2014)
0.1453
0.4632
0.1907
0,5612
0.0682
0.6585
Saliency ours)
0.1254
0.5465
0.1589
0.6287
0.0675
0.6927
Gmd-CAM (Selvariuu et al,, 2020)
0.2865
0.3721
0.3103
0,4733
0.0810
0.7224
Grad-CAM (w/ ours)
0.1549
0.4927
0.1982
0.5867
0.0726
0.7231
LIME (Ribeiro et 2016)
0.1484
0.5246
0.2034
0.6185
0.1070
0.6812
LIME (w/ ours)
0.1366
0,5496
0.1683
0.6314
0.0941
0.6994
Kemee Shap (Lundberg & Loc, 2017)
0,1409
0.5246
0.21199
0.6132
0.1016
0.6763
Kernel Shap (w/ ours)
0.1352
0,5504
0.1669
0.6314
0.0951
0.6920
RISE (Petsiuk al., 2018)
0.1444
0.5703
0.1375
0.6530
0.0665
0.7193
RISE (w/ ours)
0,1264
0.5719
0,1346
0.6548
0.0630
0.7245
HSSIC-Attribatinn (Novello et al., 2022)
0.1151
0.5692
0.1317
0.6694
0.0647
0.6843
HSIC-Artribution (w/ ours)
0.1054
0,5752
0.1304
0.6705
0.0613
0.7262
Evaluation Metric We use Deletion and Insertion AUC scores (Petsiuk et al., 2018) to evaluate the faithfulness of our method in explaining model predictions. To evaluate the ability of method to search for causes of model prediction errors, we use the model's highest confidence in correct class predictions over different search ranges the evaluation metric.
Baselines We select the following popular attribution algorithm for calculating the prior saliency map in Sec. 4.1, and also for comparison methods, including the white-box- boxeeasedd methods: Saliency (Simonyan et al., 2014), Grad-CAM (Selvaraju et al., 2020), Grad-CAM++ (Chattopadhay al.. 2018), Score-CAM (Wang et al., 2020), and the black-box-based methods: LIME (Ribeiro et al., 2016), Kernel Shap (Lundberg & Lee, 2017), RISE (Petsiuk et al., 2018) and HSIC- Attribution (Novello et al., 2022).
Implementation Details
Please see Appendix C for details.
5.2 FAITHFULNESS ANALYSIS
To highlight the superiority of our method, we evaluate its faithfulness, which gauges the consis- tency of the generated explanations with the deep model's decision-making process (Li et al., 2021a). We use Deletion and Insertion AUC scores to form the evaluation metric, which measures the de- crease (or increase) in class probability as important pixels (given by the saliency map) are gradually removed (or increased) from the image. Since our method based on greedy search, we can de- termine the importance of the divided regions according to the order in which they are searched. We can evaluate the faithfulness of the method by comparing the Deletion or Insertion scores under different image perturbation scales (i.e. different & sizes) with other attribution methods. We set the search subset size k to be the same as the set size m, and adjust the size the divided regions according the order of the returned elements. The image is perturbed to calculate the value of Deletion or Insertion of our method.
Table I shows the results on the Celeb-A, VGG-Face2, and CUB-200-2011 validation sets. We find that no matter which attribution method's saliency map is used as the baseline, our method can im- prove its Deletion and Insertion scores. The performance of our approach is also directly influenced by the sophistication of the underlying attribution algorithm, with more advanced algorithms lead- ing superior results. The state-of-the-arr method, HSIC-Attribttinn (Novello al., 2022), shows improvements when combined with our method. On the Celeb-A dataset, our method reduces its Deletion score by 8.4% and improves its Insertion score by 1.1%. Similarly, on the CUB-200- 2011 dataset, our method enhances the HSIC- Attribittitttion method by reducing its Deletion score by 5.3% and improving its Insertion score by 6.1 %. On the VGG-Face22 dataset, our method enhances the HSIC-Atribution method by reducing its Deletion score by 1.0% and improving its Insertion score by 0.2%, this slight improvement may be caused by the large number of noisy images in the VGG-Face2 dataset. It is precise because we reformulate the image attribution problem into sub- set selection problem that we can alleviate the insufficient fine- sreiggaiinness of the attribution region, thus improving the faithfulness of existing attribution algorithms. Our method exhibits significant improvements across a spectrum of vision tasks and attribution algorithms, thereby highlighting its extensive generalizability. Please Appendix H for visualizations of our method these datasets.

---

Published as a conference paper at ICLR 2024
Table 2: Evaluation of discovering the cause of incorrect predictions.
Average highest confidence (1)
Method
Insertion (1)
(0-25%)
(0-50%)
(0-75%)
(0-100%)
Grad-CAM++ (Chattopadhay et al., 2018)
0.1988
0.2447
0.2544
0.2647
0.1094
Grad-CAM++ (w/ ours)
0.2424
0.3575
0.3934
0.4193
0.1672
Score-CAM (Wang et al., 2020)
0.1896
0.2323
0.2449
0.2510
0.1073
Score-CAM (w/ ours)
0.2491
0.3395
0.3796
0,4082
0.1622
HSIC-Attribution (Novello et al., 2022)
0.1709
0.2091
0.22550
0.2493
0.1446
(w/ ours)
0.2430
0.3519
0.3984
0.4513
0.1772
Patch 10x
0.2020
0.4065
0.4908
0.5237
0.1519
HSIC-Atiibaatoo
Chrry
HSIC--Arribniin
Oura
Mcciomn SIA Score
Prediction: Sparnow
Peronsaag laage roweled
Prediction: Spurtow
Porronagge corealed
Ground Truth: Lazali Bunting
Ground Truth: Chipping Sparrow
DSSC-trriibttion
Churs
10
Insertion
JISIC-Arributee
1.
frvertion
Sooggaiae Scate
doNt-tee Our
Incorrect Prodiction: White Crowned Spariow Ground Truii:: Great Gizyy Shrike
Perensage Canggo
06
Incorrect Prediction: Hoodad Oriole
980
Pertestoge Camage
Ground Truth: Orchard Onole
Visualization of the method for discovering what causes model prediction errors. The Insertion curve shows the correlation between the searched region and ground truth class prediction confidence. The highlighted region matches the searched region indicated by the red line in the curve, and the dark region is the error cause identified by the method.
Figure 3:
5.3 DISCOVER THE CAUSES INCORRECT PREDICTIONS In this section, we analyze the images that are misclassified in the CUB-200-2011 validation set and try to discover the causes for their misclassification. We use Grad-CAM++, Score-CAM, and HSIC-Attribution as baseline. We specify its ground truth class and use our attribution algorithm to discover regions where the model's response improves on that class much as possible. We evaluate the performance of each method using the highest confidence score found within different search ranges. For instance, we determine the highest category prediction confidence that the algorithm can discover by searching up to 25% of the image region (i.e., set k = 0.25m). We also introduce the Insertion AUC score as an evaluation metric.
Table 2 shows the results for samples that were misclassified in the CUB-200-2011 validation set, we find that our method achieves significant improvements. The average highest confidence score searched by our method in any search interval is higher than the baseline. In the global search interval (0-100%). the average highest confidence level searched by our method improved by 58.4% Grad-CAM++, 62.6% on Score-CAM and 81.0% on HSIC-Attribution. Its Insertion score has also been significantly improved, increasing by 52.8% on Grad-CAM++, 51.2% on Score- CAM, and 18.4% on map of the attribution algorithm as priori. Instead, divides the image into N x N patches, where each element in the contains only one patch. In this case, our method achieves higher average highest confidence score, in the global search interval (0-100%), our method (divide the image into 10 10 patches) 97.8% better than CCMd+ 108.6% better than Score- CAM, and 110.1 % better than HSIC-Attribution. Fig. 3 shows some visualization results, we compare our method with the SOTA method HSIC-Attribution. The Insertion curve represents the relationship between the region searched by the methods and the ground truth class prediction confidence. We find that our method can search for regions with higher confidence scores predicted by the model than the HSIC- Attribution method with small percentage of the searched image region. The highlighted region shown in the figure can be considered as the cause of the correct prediction of the model, while the dark region is the cause of the incorrect prediction of the model. This also demonstrates that our method can achieve higher interpretability with fewer fine- aee-grained regions. For further experiments and analysis of this method across various network backhones, please refer to Appendix D)
In another scenario, method does not rely on the saliency
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study on components of different score functions of submodular function on the Celeb-A, and CUB-200-2000 validation sets.
Submodular Function
Colob-A
CUB-200-2011
(Equation 6)
Score
Bff. Score (Equation 6)
(Equation 7)
[Equation
Score
Deletion ())
Insertion (f)
Deletion ())
Insertion (f)
0.3161
0.1795
0.3850
0,3455
XxX
0.1211
0.5615
0.0835
0,6383
X
0,2849
0.2291
0.1019
0.6905
x
/
0.1591
0.3053
0.0771
0.5409
X
0.0865
X
X
V
0.1558
0.3617
0.0641
V
0.1074
0.5735
0.0632
0,7169
0.1993
0.2616
0.0623
0.7227
V
0.1067
0,5712
0,0651
0.6753
x
0,1088
0,5750
0.0811
0.9090
0.1054
0,5752
0.0613
0.7262
Table 4: Impact on whether to use a priori attribution map.
Coleb-A
CUB-200-201 200-2011
Method
Divided set size m
Deletion (U)
Insertion (1)
Deletion (L)
Insertion (f)
Putch 7x7
49
0.1493
0,5642
0.1061
0.6903
Patch 10x
100
0.1365
0.5459
0.1024
0.6159
Patch 14x14
196
0.1284
0.5562
0.0853
0.5805
+HSIC-Attibution
25
0.1054
0,5752
0.0613
0.7262
5.4 ABLATION STUDY
Components of Submodular Function We analyze the impact of various submodular function- based score functions on the results for both the Celeb-A and CUB-200-2011 datasets, the saliency map generated by the HSIC-Altribution method is used as a prior. As shown in Table 3, we ob- served that using a single score function within the submodular function limits attribution faithful- ness. When these score functions are combined in pairs, the faithfulness of our method will be further improved. We demonstrate that removing any of the four score functions leads to deterio- rated Deletion and Insertion scores, confirming the validity of these score functions. We find that the effectiveness score is key for the Celeb-A dataset, while the collaboration score more important for the CUB -UBB-00-200 200-2011 dataset.
Whether to Use a Priori Attribution Map Our method uses existing attribution algorithms as priors when dividing the image into sub-regions. Table 4 shows the results, we directly divide the image into N x patches without using priori attribution map. Fach element the set only retains the pixel value of one patch area, and other areas filled with values O. We can observe that, whether on the Celeb-A dataset or the CUB-200-201 2011 dataset, the more patches are divided, the Deletion AUC score increases, and the fewer patches divided, the Insertion AUC score increases. However, no matter how the number of patches is changed, we find that introducing prior saliency map generated by the HSIC attribution method works best, in terms of Insertion Deletion scores. Therefore, we believe that introducing prior saliency map to assist in dividing image areas is effective and can improve the computational efficiency of the algorithm.
CONCLUSION
In this paper, we propose a new method that reformulates the attribution problem submodular subset selection problem. To address the lack of attention to local regions, we construct novel submodular function to discover more accurate fine-grained interpretation regions. Specifically, four different constraints implemented on sub-regions are formulated together to evaluate the sig- nificance various subsets, i.c., confidence, effectiveness, consistency, and collaboration scores. The proposed method has outperformed state-of-the-art methods on two face datasets (Celeb-A and VGG-Face2) and fine-gras gr-igriindd dataset posed method can improve the Deletion and Insertion scores for the correctly predicted samples. While for incorrectly predicted samples, our method excels in identifying the reasons behind the model's decision-makin error..
Experimental results show that the pro-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Key R&D Program of China (Grant No. 2022ZD01 18102) in part by the National Natural Science Foundation of China (No. 62372448, 62025604, 62306308), and part by Beijing Natural Science Foundation (No. L212004).
REFERENCES
Alejandro Barredo Arrieta, Natalia Díaz- Boaz-Rodrig,ez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alherto Barbado, Salvador García, Sergio Gil-ILópez, Daniel Molina, Richard Benjamins, et al. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58:82-115, 2020.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 111800-18311 2010.
Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In International Conference Computer Vision (ICCV), pp. 13349-13358, 2021. 4
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp. 67-74, 2018. 2, 6
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian, Grad- CAM++ Generalized gradient-based visual explanations for deep convolutional networks. In Winter Conference on Applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018. 1,
7, 8. 17
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An cnformaaion-theoreeti perspective model interpretation. In International Conference Ma- chine Learning (ICML), pp. 883-892. PMLR, 2018. 3
Ruoyu Chen, Jingzhi Li, Hua Zhang, Changchong Sheng, Li Liu, and Xiaochun Cao. Sim2Word: Explaining similarity with representative attribute words via counterfactual explanations. ACM Transactions on Multimedia Computing, Communications and Applications, 19(6):1-22 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690-4699, 2019. 5, 16
Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, al. Explainable Al (XAI): Core ideas, techniques, and solutions. ACM Computing Surveys, 55(9):1-33, 2023. 1
Jack Edmonds. Submodular functions, matroids, and certain polyhedra. Combinatorial Structures and Their Applications, pp. 69-87, 1970.
Ethan Elenberg, Alexandros G Dimakis, Moran Feldman, and Amin Karbasi. Streaming weak sub- modularity: Interpreting neural networks on the fly. In Annual Conference on Neural Information Processing Systems (NeurIPS). pp. 4044 4054, 2017.
Satoru Fujishige. Submodular functions and optimization. Elsevier, 2005. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778,
2016. 16, 17
KJ Joseph, Krishnakant Singh, and Vinceth N Balasubramanian. Submodular batch selection for training deep neural networks. In International Joint Conference on Artificial Intelligence (IJ- CAI), pp. 2677-2683, 2019.
10

---

Published as a conference paper at ICLR 2024
Suraj Kothawade, Saikat Ghosh, Sumit Shekhar, Yu Xiang, and Rishabh Iyer. Talisman: targeted active learning for object detection with rare classes and slices using submodular mutual informa- tion. In European Conference Computer Vision (ECCV), pp. 1-16. Springer, 2022. 3
Jiahui Li, Kun Kuang, Lin Li, Long Chen, Songyang Zhang, Jian Shao, and Jun Xiao. Instance- wise or class wise? tale of neighbor shapley for concept-based explanation. In ACM International Conference on Multimedia (ACM MM), pp. 3664-3672, 2021a.
Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. IntentQA: Context-aware video intent reason- ing. In International Conference on Computer Vision (ICCV), pp- 11963-11974, 2023.
Jingzhi Li, Lutong Han, Ruoyu Chen, Hua Zhang. Bing Han, Lili Wang. and Xiaochun Cao. Seentttt-preserving face anonymization via adaptively facial attributes obfuscation. In ACM In- ternational Conference on Multimedia (ACM MM), pp. 3891-3899, 2021b.
Jiawei Liang, Siyuan Liang, Aishan Liu, Ke Ma, Jingzhi Li, and Xiaochun Cao. Exploring incon- sistent knowledge distillation for object detection with data augmentation. In ACM International Conference on Multimedia (ACM MM), pp. 768-778, 2023a.
Jiawei Liang, Siyuan Liang Aishan Liu, Xiaojun Jia, Junhao Kuang, and Xiaochun Cao. Poi- soned forgery face: Towards backdoor attacks on face forgery detection. arXiv preprint arXiv:2402.11473, 2024. 1
Siyuan Liang, Longkang Li, Yanbo Fan, Xiaojun Jia, Jingzhi Li, Baoyuan Wu, and Xiaochun Cao. A large-scale multiple-objective method for black-box attack against object detection. European Conference on Computer Vision (ECCV), pp. 619-636. 2022a.
Siyuan Liang, Aishan Liu, Jiawei Liang, Longkang Li, Yang Bai, and Xiaochun Cao. Imitated detectors: Stealing knowledge of black-box object detectors. In ACM International Conference on Multimedia (ACM MM), pp. 4839 4847, 2022b.
Siyuan Liang, Mingli Zha, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ee-Chien Chang. Bad- CLIP: Dual-embedding guided backdoor attack on multimodal contrastive learning. arXiv preprint arXiv:2311. 2023b. 1
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild.
In International Conference on Computer Vision (ICCV), pp. 3730-3738, 2015. 2, 6
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 4765-4774, 2017. 7
Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrák, and Andreas Krause. Lazier than lazy greedy. In AAAI Conference Artificial Intelligence (AAAI), pp. 1812- 1818, 2015. 6
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65:211-222 2017. 14
George L Nemhauser, Laurence A Wolsey, and Marshall Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming. 14:265-22 2944, 1978. 6
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European Conference on Computer Vision (ECCV), pp. 69-84. Springer, 2016. 4
Paul Novello, Thomas Fel, and David Vigouroux. Making sense of dependence: Efficient black-box explanations using dependence measure. In Annual Conference Neural Information Processing Systems (NeurIPS), pp. 1344-4357, 2022. 1, 2, 3, 7, 8, 17, 18, 19
Adeel Pervez, Phillip Lippe, and Efstratios Gavves. Scalable subset sampling with neural condi- tional poisson networks. In International Conference on Learning Representations (ICLR), pp- 1-21, 2023. 3
11

---

Published as a conference paper at ICLR 2024
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. In British Machine Vision Conference (BMVC), pp. 151, 2018. 1,3,7
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko. Black-box explanation of object detectors via saliency maps. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 11443-11452, 2021. 3
Joseph Redmon and Ali Farhadi. YOLOv3: An incremental improvement. arXiv preprint arXiv: 120.00070 2018. 4
Marco Tulio Ríbeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In ACM Knowledge Discovery and Data Mining (SIGKDD), pp-
1135-1144, 2016. 3.7
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen, Mo- bileNetV2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018. 17
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Frad-CAM, Visual explanations from deep networks via gradient-based local- ization. International Journal of Computer Vision, 128(2):336 (569),333-359 2020. 3,
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifi- cation uncertainty. In Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 1193, 2018. 4
K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. In ICLR (Workshop Poster), 2014. 3,7
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015. 17
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenherg. SmoothGrad: Removing noise by adding noise. arXiv preprint arXiv: 03825, 2017. 3
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), pp. 3328.3328. PMLR, 2017.
Mingxing Tan and Quoc Le. EfficientNetV2:: Smaller models and faster training. In International Conference on Machine Learning (ICML), pp. 10096-10106. PMLR, 2021. 17
Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? a safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311.16101, 2023. 1
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. CosFace: Large margin cosine loss for deep face recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5265-5274, 2018. 5
Haofan Wang. Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-CAM: Seirh-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 24-25, 2020. 1, 8, 17
Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. Unsupervised submodular subset selection for speech data. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4107-4111. IEEE, 2014.
Kai Wei, Rishabh lyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning (ICML), pp. 1954-1963. PMLR, 2015. 6
P Welinder, S Branson, T Mita, C Wah, F Schroff, S Be-longie, and P Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR ONS-TR-2010-000 Technical report, Technical Report CNS-TR-2010 001, California Institute of Technology, 2010. 2010. 2,
12

---

Published as a conference paper at ICLR 2024
Mengxi Ya, Yiming Li, Tao Dai, Bin Wang. Yong Jiang, and Shu-Tao Xia. Towards faithful xai eval- uation via generalization-linet backdoor watermark. In International Conference on Learning Representations (ICLR), 2024. 1
Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3323-3332, 2019. 3
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning LayerNorm in Attention:
Towards efficient multi-modal Ilm finetuning. arXiv preprint urXiv:2312.114200 2023. 1
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929 2016. 3
13

---

Published as a conference paper at ICLR 2024
A
NOTATIONS
The notations used throughout this article are summarized in Table 5.
Table 5: Some important notations used in this paper.
Notation
Description
an input image
sub-region into which I is divided
V
finite set of divided sub-regions subset of an element from V S the size of the S function that maps set to value saliency map calculated by attribution algorithms
S
F()
A
N N
the number of divided patches of I the number sub-regions into which the image I divided the number of patches in I M an input sample
m
the one-hot label
the predictive uncertainty ranges from 0 to 1 deep evidential network for calculating the u of x traditional network encoder
F.()
F())
K
the number classes
dist(:,-)
function to calculate the distance between two feature vectors the target semantic feature vector
f.
B THEORY PROOF
B.1 PROOF OF LEMMA
Proof. Consider two sub-sets Sa and S₁ in set V, where Sa S. Sb S V. Given an element a, where V.S. The necessary and sufficient conditions for the function F(-) satisfy the submodular property are:
F - - -
(11)
For Eq. 4, assuming that the individual element a of the collection division is relatively small, according to the Taylor decomposition (Montavon et al., 2017), can locally approximate Fg (S,++ a) = (Sa) + (S.( a. Thus:
K
Sconf. (Sa + a) Scouf. (Sa)
exp (Fu, (Sa)) + K
exp (Fu (S. + a)) + K
K
K
exp (Fu, (Sa)) + K
exp (Fu (Sa)) exp (V (S..)) a) + K"
(12)
since S, a = Ø, Sa and a do not overlap in the image space, and a is small. Therefore, we can regard F., -a O. Follow up:
K
K
Scont. (Sa a Sconf. (Sa) exp (Fa (Sa)) K exp (F, (Sa)) exp (0) K
(13)
=0+
and in the same way, Scouf. (S6 at) Scouf. (Sb) 04. We have:
Sconf. (Sa + a:) Sconf. (Sa) (Sconf. (Sh ai) Scont. (Sb)) 0.
(14)
14

---

Published as a conference paper at ICLR 2024
For Eq. 6, when element is added to the set Sa. the minimum distance between ele- ments Sa and other elements may be further reduced, ie., for any element 8; Sa, we have mms.),,,(o))sstts dist (F (s.) F (sz)) I mins, E8a.,, dist (F (s;) F (sy)). Thus:
Seft. (Sa U {a})
min dist (F (a), F. s,E8.
E syES,
min
dist (F (s:) F (s;))
ey ts
(15)
min dist (F (a) F (8:))
Σ
min dist (s:). F (s;)) Ear 5y SyE
s.ES.
where Ea is constant, which the sum of the minimum distance reductions of the elements in the original Sa after a is added. Then, we have:
Seft. (Sa U {a))
Seft. (Sa) min dist (F (a), (F) (s:)) Ea,
(16)
s,
and in the same way.
Seff. (S. (a)) Seft. (S,) min dist (a), (8:)) Eb: eE
(17)
since Sn C Sh. the minimum distance between alpha and elements in Ss. \ Sn may be smaller than the minimum distance between alpha and elements in Sa, thus,
min dist (F (a). F (s:)) > min dist F s,E.,
s,ES,
since there are more elements in Sy than in Sa, more elements in S₁ have the shortest distance from a, that, Eb 2 Ea. Therefore, we have:
Seff. (S. U (a}) Seff. (S.) 2 Seff. (So U (a))) Seff. (Sp)
(18)
For Eq. 7, let G (S.) (Sa) fs. Assuming that the individual element of the collection division is relatively small, according to the Taylor decomposition, can locally approximate G (Sa aa) = G (S.) + VG (Sa) a. Assuming that the searched a is valid, i.e., VG (Sa) > 0. Thus:
G(S.) + (S.)
G(S.)
Scons. (Sa + a, f.) Scons. (So, fa)
[F((.) + (S.) ) a alllfall
[[F(S))]
(19)
VG (Sa) a
[F((s)]||JA)
since S₂ aa = S, So and do not overlap in the image space, and a is small, (G -a small.
Then, we have:
Scous. (Sa + O. fs) Scons. (Sa, fs) (Scons. (Sp + a, fs) Scons. (S., fa)) 0.
(20)
For Eq. 8, let Get Sa) = F F -I- Sa)- f.. Assuming that the individual element a of the collection division is relatively small, according to the Taylor decomposition,
locally approximate G (I - S. a) = (I- S.) VG [I Sa) a. Assuming that the searched alpha valid, i.e.,
VG [I S.) Thus:
G - -a)
G ((I-Sa) - VG S.) a
Scolla. (S. + a. I,) Scolla. (Sa,
F (I S.) [[]
[ S. a) f
VG [I S.) -a).a
(I- S.) ||]
(21)
since is small, (G (S) a is small. Then, we have: Scolla. (Sa + ac f.) - Scolla. (Sa. I, f.) (Scolla. (Sp + a, I, fe) - Scolls. I, I.
= 0. (22)
Combining Eq. 14, 18, 20, and 22 we can get:
F F(S U (a]) - F (
(23)
hence, prove that Eq. 9 is submodular function.
15

---

Published as a conference paper at ICLR 2024
B.2 PROOF OF LEMMA 2
Proof. Consider a subset S, given an element assuming that a is contributing to interpretation. The necessary and sufficient conditions for the function F(-) satisfy the property of monotonically non-decreasing is:
-
- F() 0,
(24)
where, for Eq. 4:
K exp (Pu (5)) K exp (Pu (5)) exp (V) .)) K
K
Sconf. (S + ai Sconf. (S)
(25)
since a is contributing to interpretation, VF.(S) > 0, and exp (VFu(S) a) > 1, thus:
Sconf. (S + aa) - Sconf. (5) > 0.
(26)
For Eq. 6,
Seff. (a)) Seff. (S) = min dist (F (a) F (s:)) s.88
since effective element a are selected as much as possible, the value e will be small,
(S (a+)) Scff. (S) = min dist. (F (a), F (s:)) > 0.
(27)
siES
For Eq. 7, assuming that the searched a is valid,
(S) &
Scons. (S + a, fe) Scous. (8,fs) ~
[I( > 0,
(28)
likewise, for Eq. 8,
VG S).a 0. | (I 5) fll
Scolla. (S + a. fz) Scolla. (5,I,fs)
(29)
Combining Eq. 26, 27, 28, and 29 get:
(())
(30)
hence, we can prove that Eq. 9 monotonically non-decreasing.
C IMPLEMENTATION DETAILS
We primarily employed CNN-based architectures to explain our models. For face datasets, we evaluated recognition models that were trained using the ResNet-101 (He et al., 2016) architecture and the AreFace (Deng et al., 2019) loss function, with an input size of 112 x 112 pixels. For the CUB-200-2011 dataset, evaluated a recognition model trained on the ResNet-101 architecture with a cross-entropy loss function and an input size of 224 x 224 pixels. It's worth noting that all the uncertainty models denoted as (.).) in Sec. 4.2 were trained using the ResNet101 architecture. Given that the face recognition task primarily involves face verification, we use the target seman- tic feature f. in functions Scons. (S, f.) and Scolla. (S, I, f.) adopt features computed from the original image using pre-trained feature extractor. For the CUB-200-201 dataset, we directly im- plement the fully connected layer the classifier for specified class. For the two face datasets, we set N = 28 and m 98. For the CUB-200-201 dataset, we set N = 10 and m = 25. We conduct our experiments using the Xplique repository, which offers baseline methods and evaluation tools. To evaluate the Deletion and Insertion AUC scores, we set & to be the same as the set size m to evaluate the importance ranking of different sub-regions. These experiments were performed on an NVIDIA 3090 GPU.
Xplique: https:/ //g///ttuuu titthuu..... com/ deel a /xp1 qqui que
16

---

Published as a conference paper at ICLR 2024
Table 6: Evaluation on discovering the cause of incorrect predictions for different network back- bones.
Average highest confidence (1)
Method
Insertion (t)
(0-25%)
(0-50%
(0-75%)
(0-100S)
Grad-CAM++ (Chattopadhay al,, 2018)
0.1323
0.2130
0.2427
0.2925
0.1211
Grad Grad-CAMe CAM++ (w/ ours)
0,1395
0.2615
0.3821
0.4263
0.1304
VGGNet-19
Score-CAM (Wang et al., 2020)
0.1349
0.2125
0.2583
0.3058
0.1057
(Simonyan Zisserman, 2015)
Score-CAM (w/ ours)
0.1649
0.2624
0.3452
0,4224
0.1186
HSIC-Attribution (Novello al., 2022)
0.1456
0.1743
0.1906
0.2483
0.1297
HSIC-Auribttion (w/ ours)
Grad-CAM++ (Chattopadhay et al. 2018)
0.1988
0.2447
0.2544
0.2647
0.14
Grad-CAM++ (W/ ours)
0.3934
0.1672
ResNet- 101 (He et 2016)
Score: CAM (Wang et al.,, 2020)
0.1896
0.2323
0.2449
0.2510
0.1073
Score-CA ours)
0.3796
0.1622
HSIC ASSiC-nttibutionn (Novello et al., 2022)
0.1709
0.2091
0,2250
0.2493
0. 0.1446
HSIC-AAribttin (w/ ours)
0.1772
Gra-CAM-+ (Chattopadhay et al., 2018)
0.1584
0.2820
0.3223
0. 0.3462
1284
(w/ ours)
MobileNetV2
Score- (Wang et al., 2020)
0.1574
0.2456
0.2948
0.3141
0.1195
(Sandler al., 2018)
Score-CAM (w/ ours)
0.1667
HSIC-Atributtin (Novello et al., 2022)
0.1648
0.2190
0.2415
0.2914
0.1635
HSIC-Atrribtiio (w/
0.2460
0.4142
0.4913
0.5367
0.1922
Grad-CAM++ (Chattopadhay ct al. 2018)
0,2338
0,2549
0.2598
0.2659
0.1605
Grad-CAMe+ (w/
0.2502
0,3038
0.3146
0.3214
0,1795
EfficieniNe1V2... (Tan & Le. 2021)
Score: CAM (Wang et al., 2020)
0.2126
0.2327
0.2375
0.2403
0.1572
Score CAre (w/ (wr)
0.2442
0.2900
0,3029
0.3115
0,1745
HSIC-Atrriution (Novelio et al., 2022)
0,2418
0,2561
0.2615
0.2679
0.1611
HSIC-Atrribution (w/ ours)
0.2616
0,3117
0.3235
0.3306
0,1748
D DIFFERENT NETWORK BACKBONE
We verified the effect of our method more CNN network architectures, including VGGNet- 19 (Si- monyan & Zisserman, 2015), MobileNetV2 (Sandler et al., 2018), and EfficientNetV2-.. (Tan & Lc, 2021). We adopted the same settings the original network, with an input size of 224 224 for VGGNet-19, 224 x 224 for MobileNetV2, and 384 x 384 for EfficientNerV2-MM We conduct the experiment under the same setting as those employed for ResNet-101. When employing VGGNet- 19, our method outperforms the SOTA method Astri-Attibuttinn achieving 70.2% increase in the average highest confidence and a 5.2% improvement in the Insertion AUC score. While utilizing MobileNetV2, our method outperforms the SOTA method HSIC-Attribution, achieving an 84.2% increase in the average highest confidence and a 17.6% improvement the Insertion AUC score. Similarly, with EfficientNetV2-M our method surpasses HSIC-Attribttion, achieving 23.4% in- crease the average highest confidence and an 8.5% improvement in the Insertion AUC score. This indicates the versatility and effectiveness of our method across various backbones. Note that the 400 incorrectly predicted samples used by these networks in the experiment are not exactly the same, given the differences across the networks.
E HYPERPARAMETER SENSITIVITY ANALYSIS
We explored the effects of different score function weighting the shown in Table 7, we find that only when increasing the weight of Eq. 7, the Insertion AUC score will be slightly improved, and other indicators have obvious effect. If the weight set too high, it may cause other score functions to fail, resulting in a significant decline in indicator performance. Therefore, we do not need pay too much attention how weigh these score functions. The default weight is set to
dataset. As
F ABLATION ON MORE DETAILS
Ablation on division sub-region number N x N and set size m In this section, we explore the impact the size the divided sub-region number x N and the size of the set m on attribution performance. The experimental settings are set by default unless otherwise specified. We conducted experiments on the Celeb-A dataset and the CUB-200-222 shown in Table 8, found that the more granular the division the face dataset, the better the attribution results can be achieved, however, the attribution effect at the pixel level will be relatively poor. For the face dataset, better choice is set N 28 and 98. On the CUB-200-201
dataset. For the Celeb-A dataset, as
17

---

Published as a conference paper at ICLR 2024
Table Ablation study on the effects different score functions on the CUB-200-2011
validation
set.
Conf. Score
Eff. Score
Cons. Score
Colla, Score (Equation 8)
Deletion ())
Insertion (1)
(Equation 4)
(Equation 6)
(Equation 7)
0.0613
0.7262
20
0.1333
0.5979
50
0,2529
0.4681
100
0.3198
0,4059
0.0753
0.6725
50
0.0781
0.6614
100
0.0793
0.6537
1
20
0.0645
0.7276
1
50
0.0664
0.7276
100
0.0689
0.7221
0.0597
0.7174
50
0.0615
0.7001
100
0.0657
0.7010
Table Ablation study on division sub-region number N and set size m on the Celeb-A valida- tion set. The input image size is 112 x 112, the base attribution method is al., 2022).
(Novello
Number of divided areas element
Division method
Collection size
Deletion (L)
Insertion (1)
Patch 28x28
8
98
0.1054
0.5752
Patch 14x 14
98
0.1
0,5703
Patch 14x 14
4
49
0.1133
0.5757
Patch 10x10
2
50
0.1288
0.5621
Patch 10x10
25
0.1235
0,5645
Patch 8x8
2
32
0.1276
0,5542
Pixel 112x
98
0.1258
0,5404
Pixel 112x 112
256
49
0.1148
0,5615
Pixel 112x 112
392
32
0.1136
0.5664
dataset, shown in Table 9, the attribution effect at the pixel level is still relatively poor. Since the positions of birds the CUB-200-2011 dataset vary greatly and the background is complex, the images are not suitable for too fine- fineggainee division. For the CUB-200-201 dataset, better choice to set
10 and m 4.
Ablation of score function the attribution of incorrectly predicted samples Since the tribution of mispredicted samples requires specifying the correct category, we mainly discuss the impact of Eq. and Eq. 8 the attribution of mispredicted samples because these equations in-
Table 9: Ablation study on division sub-region number x N and set size m on the CUB-200- 2011 validation set. The input image size is 224 224, and the base attribution method is HSIC- Attribution (Novello et al., 2022).
Number of divided areas for element
Division method
Collection size
Deletion (L)
Insertion (1)
Patch 14x 14
2
98
0.0769
0.6343
Patch 14x 14
4
49
0.0652
0.6661
Patch 10x10
50
0.0689
0.6917
Patch 10x 10
4
25
0.0613
0.7262
Patch 8x8
2
32
0.0686
0.7079
Pixel 224x224
256
49
0.0859
0,5818
Pixel 224x224
392
32
0.0768
0.6398
18

---

Published as a conference paper at ICLR 2024
Table 10: Ablation study on submodular function score components for incorrectly predicted sam- ples in the CUB-200-2011 dataset.
Cons: Scure (Equation
Colla. Score
Average highest confidence
Method
Insertion (t)
(Equation
(0-25%)
(0-509))
(0-75%)
(0-100%)
Grad-CAM+ (w/ ours)
0,0821
0.1547
0.1923
0.2303
0.1122
Grad-CAM++ (w/ ours)
X
0.1654
0.2888
0.3338
0.3611
0.1452
Grad-CAM+ (w/ ours)
0.2424
0.3575
0,3934
0.4193
0.1672
Score-CAM (w/ ours)
x
/
0.0742
0.1348
0.1835
0.2237
0,1072
Score-CAM (w/ ours)
x
0.1383
0,2547
0.3131
0,3402
0.1 1306
Score-CAM (w/ ours)
/
0.2491
0.3395
0.3796
0.4082
0.1622
HSIC-Attribution (w/ ours)
/
0.1054
0.1803
0.2177
0.2600
0.1288
ISSIC-Atribtiin (w/ ours)
x
0.2394
0.3479
0.3940
0.4220
0.1645
HSIC-Attributin (w/ ours)
0.2430
0.3519
0.3984
0.4513
0.1772
Table 11: Ablation study on the effect of sub-region division size N x N in incorrect sample attri- bution.
Average highest confidence (1)
Method
Insertion (1)
(0-25%)
(0-50%)
(0-75%)
(0-100%)
HSIC-Attribation (Novello et al., 2022)
0.1709
0.2091
0.2250
0,2493
0.1446
Patch 5x5
0.2597
0.3933
0.4389
0,4514
0.1708
Patch 6x6
0.2372
0.4025
0.4555
0.4720
0.1538
Patch 7x7
0.2430
0.4289
0.4819
0.4985
0.1621
Patch 8x8
0.1903
0.4005
0.4740
0.5043
0.1584
Patch 10x10
0,2020
0.4065
0.4908
0.5237
0.1519
Patch 12
0.1667
0.3816
0.4987
0.5468
0.1247
volve specifying categories. The network backbone studied is ResNet-101, and 400 samples in the CUB-200-2011 test set that were misclassified by this network were used as research objects. As shown in Table 10, when any score function is removed, regardless of the imputation algorithm it is based on, the average highest confidence and Insertion AUC score will decrease, with Eq. 7 having the greatest impact.
Ablation of division sub-region number N x N of incorrect sample attribution Without using a priori saliency map, we study the impact of different patch division numbers on the error sample attribution effect, where the number of patches division sizes on the results. As depicted Table 11, our results indicate that dividing the image into more patches yields higher average highest confidence scores (0- (0-100%). However, excessive di- vision can introduce instability in the early search area (0-25%) In summary, without incorporating priori saliency maps for division, opting for 10x10 patch division followed by subset selection appears to be the optimal choice.
N x N. We explore the impact of six different
G EFFECTIVENESS
In this section, we discuss the processing time effectiveness of the proposed algorithm. The size m of the divided set we are testing is 98, we
sequentially the processing time required by our algorithm under different settings of the number search elements k.
Computation time
As shown in Fig. 4, the more elements are searched, the longer times takes, and the re- lationship is almost linear. To use the proposed method efficiently, is better to control the di- vided set size m, the size of the search set can be controlled by controlling the number of di- vided patches N, or by controlling the parame- ter that controls the number of patches assigned to each element.
10
10
30
50
70
90
Number elements searched
Figure 4: Processing time of our method.
19

---

Published as a conference paper at ICLR 2024
H
VISUALIZING ATTRIBUTION OF CORRECTLY PREDICTED CLASSES
H.1 CELEB-A
The visualization results compared with the HSIC-Attribttion method are shown in Fig. 5. Our method has a higher Insertion AUC curve area and searches for the highest confidence with higher attribution results than the HSIC-Attribution method.
HSIC- Attributntt
Ourrs
Insertion
1.0
Recognition Scare
HSIC-AAtribuiion
0.8
Ours
0.6
0.4
0.2
0.2
0.4
0.6
0.5
1.0
Percentage image revenled
HSIC-Auribution
Ours
Insertion
Recogntton Score
IISIC-Auribution
Ours
0.6
0.4
0.2
08.0
0.2
0.4
0.6
0.5
1.0
Percentage image revenled
USIC-Auriibttin
Ours
Insertion
HStC-Attribttion
Rosugation Score
Qurs
0.2
0%.0
0.2
0.4
0.6
0.5
1.0
Percentage imagee revealed
Aiti--uti
Ours
Insertion
Recognttion Score
HSSC--trribttinn
Ours
0.000
0.4
0.6
0.5
1.0
Percentage of image revealed
HStC-Attibbttion
Ours
Insertion
IUSIC-Aurbution
Recognition Score
Ours
0.0
0.4
0.6
0.5
Percentage of image revealed
Figure 5: Visualization of the methods on the Celeb-A dataset.
20

---

Published as a conference paper at ICLR 2024
H.2 LUB-200-2011
The visualization results compared with the HSIC-Attribution method are shown in Fig. 6. Our method has higher Insertion AUC curve area and searches for the highest confidence with higher attribution results than the HSIC-Attribution method.
HSIC-Attribution
+ Ours
Insertion
1.0
I1SiC-Atribttion
Recognition Score
+ Ours
0.6
0.4
%0.0
0.2
0.4
0.6
0.8
1.0
Percentage of image revealed
HSIC-Attribution
+ Ours
Insertion
1.0
JISiC-Atributiin
Recognitton Score
0.8
0.4
0.2
00000
0.4
0.8
1.0
Percentage of image revealed
AStC-Aitribttiion
+ Ours
Insertion
1.0
USSCC-Atribution
Recognition Score
0.8
+ Qurs
0.6
0.4
0.2
0%.00
0.4
Percentage of image revealed
HStC-Attribttion
+ Ours
Insertion
LISIC-Atribution
Recognitton Score
0.8
+ Ours
06
0.2
0.000
0.2
0.4
0.6
0.8
1.0
Percentage of image revealed
AStt--Atribtttio
+
Ours
Insertion
1.0
Recognition Score
IISIC- -isIC-Atribution
Ours
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
Percentage of image revealed
Figure 6: Visualization the methods on the CUB-200-2011 I dataset.
21

---

Published as a conference paper at ICLR 2024
1 VISUALIZING ATTRIBUTION OF MISCLASSIFIED PREDICTION
Visualization of our method searches for causes of model prediction errors compared with the HSIC- Attribution method, shown in Fig. Our method has higher Insertion AUC curve area and searches for the highest confidence with higher attribution results than the HSIC- Astribattibttion method.
HStC-Attribution
+ Ourrs
Insertion
1.0
HSIC-Atrribation
Recognition Score
0.8
Ours
0.6
0.4
0.2
0%.00
0.4
0.5
1.0
Percentage of image revealed
Aitii-Atributionn
+ Ours
Insertion
1.0
HSIC-Atrribution
Recognition Scove
0.8
Ours
0.4
0.2
0/100
1.0
Percentage of image revealed
HSIC-Atriibution
+ Ours
Insertion
HSIC-Atrribaioo
Recognition Score
0.8
Owrs
0.5
0.2
0%000
0.
0.6
0.8
1.0
Percentage of image revealed
HSIC-Atribution
+
Ours
Insertion
1.0
HSIC-Attribution
Recognition Score
Ours
0.5
0.4
0.000
0.2
0.4
0.6
0.8
1.0
Percentage of image revealed
HSSC-Attribttion
+ Ours
Insertion
1.0
HSIC-Atrribation
Recognitton Score
0.8
Owrs
0.4
0.2
0%00
0.4
0.5
0.8
1.0
Percentage of image revealed
Figure Visualization of the methods on CUB-200-2011 misclassified dataset.
22

---

Published as a conference paper at ICLR 2024
J LIMITATION
The main limitation of our method the computation time depending on the sub-region division approach. Specifically, first divide the image into different subregions, and then we use the greed algorithm to search for the important subset regions. To accelerate the search process, we introduce the prior maps, which are computed based on the existing attribution methods. However, we observe that the performance of our attribution method is based on the scale of subregions, where the smaller regions would achieve much better accuracy as shown in the experiments. There exists trade-off between the accuracy and the computation time. In the future, we will explore better optimization strategy solve this limitation.
23


Reviews:
[{"review_id": "Lxt6EZp71k", "reviewer": "Reviewer_kDMz", "summary": "The paper points out two main issues with current State-of-the-Art (SoTA) image attribution methods: they ignore the impact of local, fine-grained attribution regions which may lead to incorrect explanations, and they may struggle to map the region/cause of a prediction error to image samples. \n\nTo address these problems, the authors propose a new method based on submodular functions that divides an image into smaller regions and then selects the most informative ones (subset selection) to explain the model's decision-making process. They also employ a regional search to expand on the search regions.\nThe paper also introduces a novel submodular function to provide clearer and more detailed explanations, especially for incorrect predictions based on four major clues - confidence, effectiveness, consistency and collaboration scores.\n\nThe effectiveness of this new method is supported by experiments on facial recognition and fine-grained image recognition datasets, where it is shown to perform better than the current SoTA methods.", "questions": "Most of the suggestions have been explained in detail in the weakness section. Additionally, some additional suggestions are listed below:\n1. Section 4.1 which highlights the Sub-Region Division should be presented as an algorithm for better clarity.\n2. It would be great to label the Input Image as $\\mathbf{I}$ and saliency / attribution map as $\\mathbf{A}$ in figure 1 for better clarity.\n3. It is unclear as to what $M$ signifies in the problem definition, which should be clarified in sections 3 and 4.2. \n4. Lemmas 1 and 2 alongside equations 10 and 11 are already discussed in the problem definition in section 3, thus should be removed with referencing.\n5. Theorem 1 is prior art (Nemhauser et al., 1978) and should just be cited.\n6. Although optional, it would be good to have a set of notations as this paper encapsulates multiple domains in Machine Learning.\n7. Experiments in section 5 indicate that the proposed method demonstrates significant improvements over existing methods. This shows the generalizability of the approach irrespective of the underlying model, which I believe can be highlighted for better impact.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper highlights the importance of fine-grained, local regions for image attribution alongside causation of erroneous predictions to image features.\n2. The novel submodular function proposed in the paper has been demonstrated to outperform several State-of-the-Art (SoTA) approaches.\n3. The paper also introduce several interpretability clues such as confidence $s_{conf}$, effectiveness $s_{eff.}$, consistency $s_{cons.}$ and collaboration $s_{colla.}$ to evaluate the significance of the selected subsets. These additions effectively demonstrate better interpretability and are well supported by theoretical and empirical results.", "weaknesses": "1. The idea of decomposing an input image $\\mathbf{I}$ into regions has been studied for several vision tasks like self-supervision (Noroozi et al., 2016 etc.), object detection (Redmon and Farhadi, 2018) etc. These should be cited and the differences should be called out.\n2. The use of saliency maps $A$ in sub-region division is unclear. The paper should highlight how saliency maps are used to evaluate patch importance.\n3. Most of the scoring functions like  $s_{eff.}$, $s_{cons.}$ etc.  rely on cosine similarity or distance metrics which have been studied extensively in literature (Deng et al., 2018, Wang et al., 2018 etc.) but have not been cited in the paper.\n4. The paper lacks the explanation regarding how individual scores contribute to achieving their respective objectives. For example, $s_{colla.}$ employs a cosine distance metric between the semantic feature vector of the target class $f_s$ and features extracted from the residual regions of the original image when the selected subset of regions $S$ is removed. To the best of my knowledge, maximizing this metric ensures that the collective impact of the selected region is sufficient to generate explainable representations.\n5. The proposed greedy search algorithm (section 4.3) has been studied for subset selection tasks (Wei et al., 2015) and is therefore prior art.\n6. The paper misses a critical reference in submodular optimization (Fujishige, 2005) and should include it in the related work.\n7. The experiments should include ablations on $k$ which is the number of sub-regions selected from $V$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4mUBMVyBEV", "reviewer": "Reviewer_2zii", "summary": "The authors of this paper propose a novel explainability method that restates the image attribution problem as a submodular subset selection problem. They suggest they can achieve better interpretability results with local regions. Moreover, they aim to obtain higher scores of interpretability with fewer regions.", "questions": "- How do the image attribution algorithms relate to attention in neural networks in general?\n- What is the meaning of fine-grained interpretation regions?\n- What does the \"validity\" of a submodular function mean?\n- Is there an explanation on why the blue curve on the left of Figure 1 dips to almost 0 around 0.8?\n- You claim your method can find fewer regions that make the model predictions more confident but I am seeing more highlighted regions in your setting. Am I missing something or misreading the images? Can you elaborate more on what I should be seeing?\n- How does your method compare to the SOTA HSIC-Attribution method in terms of algorithmic complexity and time?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- It is impressive that their method achieves a 81.0% gain in the average highest confidence score for incorrectly predicted samples. \n- Treating the interpretable region identification problem as a submodular subset selection problem is a novel and interesting idea.\n- Their method has the ability to find the reasons that causes the prediction error for incorrectly predicted images.\n- Adding the ablation study at the end is a good idea.", "weaknesses": "Some suggestions for minor improvements:\n- The phrase \"... at the level of theoretical aspects.\"at the introduction sounds a bit too wordy. It can be expressed more concisely.\n- This sentence in the introduction \"Image attribution algorithm is a typical interpretable method, which produces saliency\nmaps that explain how important image regions are to model decisions.\" can be better phrased, in my opinion, as \"... that explain which image regions are more important to model decisions.\"\n- I don't think fine-grainedness (page 2) is an actual word. Fine-graininess may be an alternative but I am not sure.\n- On page 2, the word \"...datasets.\" at the end of the first sentence of the second paragraph needs to be omitted.\n- Contrary to what has been said on the introductory sentence of the White-Box Attribution method paragraph, I don't think there is a THE image attribution algorithm. I advise the authors to state either the name of the specific algorithm they are mentioning or use the plural.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7sYVNVhCSF", "reviewer": "Reviewer_gx4h", "summary": "This article transforms the image attribution problem into a submodular subset selection problem and determines the importance of divided regions through a greedy search algorithm. The author designed the submodular function from four aspects, hoping to use fewer areas to obtain a higher interpretable area. The author verified the effectiveness of this method on two tasks: face recognition and fine-grained recognition. Experimental results show that this method can achieve better attribution effects and can better debug incorrectly predicted samples.", "questions": "Listed in the weakness of the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper reformulates the attribution problem as a submodular subset selection problem that achieves higher interpretability with fewer fine-grained regions.\n\n- The proposed method enables more accurate attribution and can help find the reasons for the model to produce incorrect prediction results.\n\n- It is meaningful to verify this interpretable method on face recognition and fine-grained recognition tasks, because these tasks are closer to practical applications.\n\n- The authors provide some theoretical guarantees.", "weaknesses": "- In Algorithm 1, I didn't see the use of variable k. Should n of line 3 be k?\n\n- In Table 1, why are some results of LIME and Kernel Shap not reported? Is it because these attribution algorithms have limitations on the CUB data set? Hope the author can explain it.\n\n- It would be better if the authors could discuss the limitations of this method.\n\n- Can the author further state whether the proposed method is white-box based or black-box based (assuming that the calculation of a priori saliency map is not considered)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kxqa9ZMzXZ", "reviewer": "Reviewer_2VHY", "summary": "This paper applies the submodular subset selection theory to the image attribution method, and can effectively improve the attribution ability of the baseline saliency map. The authors verified the effectiveness of this method on the Celeb-A, VGGFace2 and CUB datasets. Experiments show that the method proposed in this article can obtain more effective explanations with fewer regions than the baseline method. In addition, this method has great advantages in searching for regions that lead to model prediction errors.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- It is very interesting and practical to use interpretability to find the reasons for model prediction errors, which is helpful for humans to discover model defects and assist in improvement.\n\n- Different from perturbation-based and gradient-based methods, the author adopts a method based on searching image areas for image attribution and achieves better performance.\n\n- The author conducted a detailed analysis and included more experimental results and visualization results in the appendix.", "weaknesses": "- In Section 5.3, for the discover the causes of incorrect predictions, the author only verified it on ResNet and achieved good quantitative results. It would be more convincing if the author could try to add some backbone, such as VGGNet.\n\n- Why are the results of LIME and Kernel Shap under CUB data not reported in Table 1?\n\n- In Table 2, the saliency map without a priori seems to be better than the method of adding a priori saliency map in terms of average highest confidence evaluation metric. Can the author add an ablation experiment to observe the impact of different partition sizes on the results without adding a priori saliency map, such as 8x8, 12x12, etc.\n\n- In the introduction, “and a fine-grained dataset CUB-200-2011 (Welinder et al., 2010) datasets” -> “and a fine-grained dataset CUB-200-2011 (Welinder et al., 2010)”\n\n- In Algorithm 1, the input k is not used, please check carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)