A Dual-Perspective Approach to Evaluating Feature Attribution Methods
Authors: Yawei Li
Yang Zhang
Kenji Kawaguchi
Ashkan Khakzar
Bernd Bischl
Mina Rezaei
Venue: Accepted by TMLR ()

Abstract: Feature attribution methods attempt to explain neural network predictions by identifying relevant features. However, establishing a cohesive framework for assessing feature attribution remains a challenge. There are several views through which we can evaluate attributions. One principal lens is to observe the effect of perturbing attributed features on the model’s behavior (i.e., faithfulness).
While providing useful insights, existing faithfulness evaluations suffer from shortcomings that we reveal in this paper. To address the limitations of previous evaluations, in this work, we propose two new perspectives within the faithfulness paradigm that reveal intuitive properties: soundness and completeness. Soundness assesses the degree to which attributed features are truly predictive features, while completeness examines how well the resulting attribution reveals all the predictive features. The two perspectives are based on a firm mathematical foundation and provide quantitative metrics that are computable through efficient algorithms. We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature attribution methods.

Published in Transactions on Machine Learning Research (11/2024)
A Dual-Perspective Approach to Evaluating Feature Attribution Methods
Yawei Li LMU Munich
Munich Center for Machine Learning
Yang Zhang
yangzhang mu.mu.. mus. edu
National University of Singapore
Kenji Kawaguchi
kenji@comp. wws. edu..g
National University of Singapore
Ashkan Khakzar
University of Oxford
Bernd Bischl
LMU Munich
Munich Center for Machine Learning
Mina Rezaci
LMU Munich
Munich Center for Machine Learning
Reviewed on OpenReview: https: openreview. net/ net/forum? forum? id= TTtTPbR
Abstract
Feature attribution methods attempt to explain neural network predictions by identifying relevant features. However, establishing a cohesive framework for assessing feature altribu- tion remains challenge. There several views through which can evaluate attribu- tions. One principal lens is to observe the effect of perturbing attributed features the model's behavior (i.e., faithfulnees)). While providing useful insights, existing faithfulness evaluations suffer from shortcomings that we reveal in this paper. To address the limitations of previous evaluations, in this work, we propose two new perspectives within the faithfulness paradigm that reveal intuitive properties: soundness and compledeness. Soundness assesses the degree to which attributed features are truly predictive features, while completeness examines how well the resulting attribution reveals all the predictive features. The two perspectives are based firm mathematical foundation and provide quantitative metrics thtt are computable through efficient algorithms. We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature at- tribution methods. Our code is provided at battps://gitiie http//sndlllkeessoooooo git.
1 Introduction
Understanding predictions of machine learning models a crucial aspect of trustworthy machine learning across diverse fields, including medical diagnosis (Bernhardt et al. 2022 Khakzar el. al. 2021c b), drug discovery [Callaway 2022 Jiménez- Luna al.) [2200] Gündüü al. 2023 2024), and autonomous driving
Kaya et al. 2022 Can al. 2022 Feature attribution, indicating the contribution of each feature to
"Equal contribution.

---

Published in Transactions on Machine Learning Research (11/2024)
model prediction, serves fundamental approach to interpreting neural networks. However, the out- comes from various feature attribution methods can be inconsistent for given input (Krishma et al. 2022) necessitating distinct evaluation metrics to gauge how well a feature attribution elucidates the prediction.
Research has introduced various lenses through which we can evaluate attributions. One lens is assessing the methods through sanity checks 4Adebayo et al. 2018). For example, by checking if the attribution changes if network parameters are randomized. Another evaluates attributions against ground truth features (Zhang et al. 2018 Yang et al. 2022 Zhang et al., 2023). Each lens reveals different insights. However, one lens is of particular interest in our study, evaluation via faithfulness. Faithfulness measures the degree to which the attributions mirror the relationships between features and the model's behavior. For instance, how changing (e.g., removing) attributed features affects the model's performance. This analysis has taken several forms, for instance by perturbing features according to their rankings and checking the immediate effect on output (Samek et al., 2016 Ancona et al. 2018), or first perturbing features, then re-training the network from scratch on the perturbed features (Hooker et al., 2019; Zhou et al. 2022, Rong et al., 2022). However, one common property exists between all these forms of faithfulness analysis. The evaluations solely consider the ranking of attribution values and disregard the attribution values.
In this work, we leverage the notion of considering the value of attributions in addition to the order and introduce two complementary perspectives within faithfulness: soundness and completeness. They serve as an evaluation of the alignment between attribution and predictive features. Soundness assesses the degree to which attributed features are truly predictive features, while completeness examines how well the resulting attribution map reveals all the predictive features. These proposed metrics work in tandem and reflect different aspects of feature attribution methods. We first motivate the work by revealing issues within existing faithfulness evaluations. We further see that by considering the attribution value in addition to the order, our metrics are more distinctive compared to existing methods, enabling more precise differentiation between attribution methods. Through extensive validation and benchmarking, we verify the correctness of the proposed metrics and showcase our metrics' potential to shed light on cxisting attribution methods.
2 Related work
2.1 Feature attribution methods
Attribution methods explain a model by assigning a score to each input feature, indicating the importance of that feature to the model's prediction. These methods can be categorized as follows:
Gradient-based methods: These methods (Simonyan et al. 2014 Bachrens et al.
2010 Springenberg
et al. 2015 Khakzar et al. 2021a Zhang et al. 2018 Shrikumar et al.] 2017) generate attributions based on variants of back- propagation rules. For instance, Simonyan et al. (014) employs the ab- solute values of gradients to determine feature importance, whereas DecpLIFT (Shrikumar et al. 2017) calculates attributions by decomposing the output prediction butions from each input feature. It uses reference activation to compare against actual activations, measuring the difference between the neuron's 's activation for a given input and its activation at the reference point. This difference is then propagated backward through the network, layer by layer, using set of rules specific to the activation functions and network architecture.
neural network into contri-
Hidden activation- braee methods: CAM (Zhou et al. 2016 produces attribution maps for CNNs, which are equipped with global average pooling and linear classification head. CAM multiplies the last CNN layer activations with the weights in the lincar head associated with target class, and then computing the weighted average. GradCAM (Selvaraju el al., 2017) further generalizes this approach by weighting the activation maps using the gradients, and then computing the weighted sum over the activation maps. Importantly, GradCAM does not require the network to have a global average pooling followed by linear classification layer, thus can be applied to broader range of neural network architectures.
Shapley value-based methods: This class of methods approximates Shapley values (Shaley et al., 1953), considering features cooperative players with different contributions the predic-
2

---

Published in Transactions on Machine Learning Research (11/2024)
tions. DeepSHAP, which combines ideas from DeepLIF'T and SHAP (SHapley Additive exPla- nations) (Lundberg & Lee, 2017), leverages SHAP's approach of using Shapley values. Specifically, DeepSHAP approximates these Shapley values by integrating DeepLIFTTs rules for backpropagating contributions from the output to the inputs, while considering multiple reference values to account for the distribution of the input data, which enhances the accuracy and stability of the attributions. Integrated Gradients (TG) (Sundararajan et al., 2017) integrates the gradients of the network's out- put with respect to its inputs along a straight path from a baseline (typically zero vector) to the actual imput. This path integration helps in capturing the importance of each input feature across different scales. While not directly employing Shapley values, the integral in this method can be seen as an approximation of Shapley value calculations, attributing the output prediction fairly among all input fealures by considering their marginal contributions along the path. Smooth- Grad (Smilkov et al., p017] is an approach that reduces noise in IG by averaging the gradients of the imput with small amounts of random noise added multiple times, enhancing the visual sharpness and interpretability of the attributions. SmoothGrad extends this by squaring the gradients before averaging, which emphasizes larger gradient values more and can highlight features more strongly. VarGrad (Adebayo et al. 2018), another variant, focuses the variance of the gradients with noise added, aiming to capture arcas of the imput space where the model's predictions are most sensitive to perturbations, thereby identifying potentially important features.
Perturbation- based methods: This type of methods works by perturbing parts of the input or hidden activations and observing the impact on the model's prediction, specifically targeting those regions whose alternation leads to the most extreme change in output. Extremal Perturbation (Fong et al, 2019) employs optimization techniques to systematically search for a smooth mask identifying the most influential regions of an input image. IBA (Schulz et al. 2020) searches the smooth mask on hidden activations using information bottleneck. The core idea viewing the masked hidden features as a random variable Z, and maximizing the mutual information Z) between target label Y and features Z, while minimizing the mutual information [(X; Z) between the feature Z and input X. The optimized information bottleneck, parametrized by the optimal mask, only admits a subset of features through the downstream layers. These admitted features can best maintain the model's performance, therefore they are the most important features. InputIBA (Zhang et al. 202I) reveals hidden features typically have small spatial size (e.g., 7 7), hence the mask returned by IBA can suffer from over-blurriness when being resized to the imput spatial size. Furthermore, putting the IBA at early layers suffers from pver-estimate of the mutual information, as the features at early layers do not necessarily follow Gaussian distribution, violating the assumption employed by IBA. ImputIBA solves this challenge by using Generative Adversarial Networks (Goodfellow et al. Arjovsky et al. 2017) to approximate the unknown hidden activation distribution, thereby producing more sharper and more detailed attribution maps.
2020:
2.2 Evaluation metrics for feature attribution methods
Existing metrics for assessing the faithfulness of attribution methods can be categorized as follows:
Expert-grounded metrics: These metrics rely on human expertise to interpret and assess the qual- ity of attribution maps produced by attribution methods. Experts visually inspect (Yang et al, 2022) these maps to determine if they highlight the relevant regions, as expected by the model's focus. For example, if a classifier identifies an image as containing a basketball, the attribution map should highlight the basketball itself. Another approach involves the "pointing game" (Zhang) et al. 2018), where the goal is to see the attribution method assigns the highest value to the pixel within the bounding box that localizes the object in question. Additionally, human-AL AI collaborative tasks (Nguyen et al.) 2021) gauge the effectiveness of attribution maps in aiding human classification efforts. While these methods incorporate human judgment, their outcomes can be subjective and may lack consistency.
Functional- metrics: These metrics, which include works by (Petsiuk et al. 2018 Samek
et al.] 2016 Ancona et al. 2018 Hooker et al. 2019 Rong et al. 2022 operate by removing input
3

---

Published in Transactions on Machine Learning Research (11/2024)
Original Image
(1) P. Train
Original Image
Dp Train
(2)
1.00
0.88
Strategy:
Test Accuracy
0.75
Strategy: 2
0.60
0.50
0.29
0.24
0.25
0.00
Retrained
Original
(a) D
Praria
(b) Prarin
D
(c) Test accuracy
Figure 1: Analysis of retraining- based metrics. Compared to (a) P.Train (b) D P,Trrin introduces an additional class-related spurious correlation during perturbation, visible in the upper-right region of the sample. (c) Despite equivalent removal informative features (central portions of images) using both per- turbation strategics, the two retrained models demonstrate different test accuracy (0.66 vs. 0.88), suggesting (hat the Lest accuracy of the relrained model does not accurately reflect the quantity of information removal.
(1)
features and measuring changes in the model's predictions. The underlying premise that removing an important. feature should result. a significant prediction change. Initially introduced by Inser- con/Deletion (Samek al.) 2016), two feature removal orders are considered: Most Relevant First (MoRF), which starts with the feature having the highest attribution value, and Least Relevant First (LeRF), which begins with the feature having the least attribution value. Plotting the probability of the target class against the number of removed features results in two distinct curves for MoRF and LeRF. A faithful attribution method should correctly rank feature importance, resulting in a steep initial drop in the MoRF curve, followed by a plateau, whereas the LeRF curve should show a plateau at the start and sharp drop towards the end. Hooker al. (2019) introduced Remove and Retrain (ROAR) to mitigate potential adversarial effects of feature removal by retraining the model perturbed datasets. ROAD (Rong al. 2022 highlighted that perturbation masks could inadvertently leak class information, potentially misrepresenting feature importance. More recently, Zhou et al. (022)) proposed method to inject "ground truth" fcatures into the training dataset, forcing the model to learn exclusively from these features and then lesting the ability of attribution methods to recognize them.
Others: Khakzar et al. (2022) introduced empirical evaluation of axioms, and Adebayo et al. (2018) introduced sanity checks for saliency maps, further diversifying the landscape of attribution method evaluation.
3 Analysis of prior evaluation metrics
In this section, an empirical analysis of various feature attribution evaluations conducted, with the objective of delineating the advantages and disadvantager of existing evaluation metrics. Later, we design our metrics with care circumvent the potential pitfalls in the previous evaluations.
3.1 Retraining-bassd evaluation metrics
Retraining- evaluations, such ROAR Hooker et al. 2019). involve retraining the model per- turbed dataset and measuring the accuracy of the retrained model perturbed test set. A sharper decrease in accuracy suggests grcater information loss resulting from the perturbation of attributed fea- tures, thus indicating better feature attribution. Since the model is retrained, it is not subject to OOD effects instigated by perturbation, as observed in Insertion/Delee Deeletin. However, the perturbation may give rise other spurious features when the original ones are removed. The model might learn these newly introduced features, as there are no stringent constraints in the learning process prevent the model from doing Therefore, if the retrained model leverages these spurious features rather than relying exclusively

---

Published in Transactions on Machine Learning Research (11/2024)
0.92
0.93
Image
Rect
Pooling
0.8
(2)
1.0
S
(1)
0.8
0.6
0.60
Attp%
20
0.6
0.4
0.31
0.4
0.2
0.2
0.0
Rect
Pooling
(a) Semi-natural datasets
(b) Attr%
Figure 2: Analysis of evaluation semi-natural datasets. (a) Designed semi-natural datasets and altribution maps from crafted altribution methods. (b) Each method excels on the dataset for which it has prior knowledge, but it underperforms on the other.
the remaining ones to make predictions, the testing accuracy might not accurately represent the extent of information loss.
We illustrate the issue of retraining via a falsification experiment, following the ROAR setting. We perturb 70% of pixels in each image in the CIFAR-10 (Krizhevsky et al.) 2009a) training and test datasets. The model then retrained on the perturbed training set and evaluated on the perturbed test set. Specifically, we employ two perturbation strategies: Strategy 1, which perturbs a central circular region to remove class- related objects (Figure Tal, and Strategy 2, which perturbs the image center and small edge region based the class label (Figure 1bb, introducing spurious class correlation. We refer to the Perturbed Training set using strategy as DD Traiinn Analogously, we have P,
D( Pe and P. We then train a model
(D2)
on the original dataset, retrain it on PP,Trrii and 1D(2) Trraiiii and report the accuracy of both the original and (12)
retrained models on PeTss and (2(2) P,Test Additional details are in Appendix C As shown in Figure 1c the original model performs poorly on 1D Test and D due to the perturbation on substantial class-related pixels. However, the two retrained models achieve distinct accuracy on their test scts. While the model retrained on m Traiin still performs badly on D P,Tes the model retrained on m2 has almost 90% test accuracy D Test: the latter model learns the spurious correlation introduced by perturbation. Despite both perturbation strategics notably disrupting the central informative part of an image, the model retrained m Prain stiill achieves high test accuracy. Therefore, spurious features have great impact the evaluation outcome.
3.2 Evaluation on semi-natural datasets
If we have access to the features that are truly relevant to labels, compare them with attribution maps to evaluate attribution methods. [hhou et al. (2022) proposed training the model on dataset with injected ground truth features. However, our subsequent experiment reveals that the evaluation outcome can be affected by the design of ground truth features. Moreover, results from semi-natural datasets may diverge from those on real-world datasets, as utilizing semi-natural datasets changes the original learning task.
The way construct semi-natural datasets significantly influences the properties of the introduced ground truth, such as its size and shape. With prior knowledge of semi-natural dataset construction, can tailor attribution methods to outperform others on this datasct. To illustrate this, we create two Semi-natural
Dataset ( and (2(2) from CIFAR-100 KKizzhessy al. 2009b). Tn the case of (1). numeric watermarks
5

---

Published in Transactions on Machine Learning Research (11/2024)
0.15
GradCAM
0.70
Direction of Improvement
IG
DeepSHAP
0,65
Reet
0.10
Accuracy
Directioo of Improvement
Accuracy
0.60
0.55
0.05
GradCAM
0.50
IG
DecpSHAP
0.45
Reet
0.00
0.2
0.6
0.2
0.6
Mask Ratio by Area
Mask Ratio by Area
(a) ROAD on dataset (1)
(b) ROAD CIFAR-100
Figure 3: Evaluation on semi-natural datasets vs. on real-world datasets. Evaluation results on semi-natural and real datasct can be markedly different. On the semi-natural datasct D A "dummy" method Rect simply using the prior information about the dataset ID( performs the best, while has the worst performance on CIFAR-100.
that correspond to class labels are injected (Figure 2a first row, left column), whereas for D cach image is divided into seven regions, and stripes are inserted, acting as binary encoding for class labels (Figure 2a second row, left column). For example, watermarks are put in the 4th and 6th regions for class 40 (i.e., 01010002). Additionally, we design Rect attribution method to take advantage of the prior knowledge that
the watermarks in m are square patches, and we design Pooling attribution for m.²) by exploiting the fact that a stripe watermark in DG² forms a rectangle spanning an entire row. The name "Pooling" comes from the operation of averaging attribution values within each row-spanning region in D= then broadcasting this pooled value across the region. Further details can be found in Appendix D.2 Figure 2a visualizes
attribution maps for Rect and Pooling. Using the Attr% metric (Zhou et al. 2022 we evaluate both
methods on both datasets (Figure 2b). Each method excels on the dataset it was designed for but performs poorly on the other, retonstrating the inconsistency of evaluations on semi-natural datasets with different ground truth features.
Besides the insights from the previous experiment, we further show the inconsistency between evaluation results semi-natural and real datasets. Due to the absence of ground truth real datasets, we replace the Attr% metric with ROAD (Rong et al., 2022 to assess attribution methods. We utilize CIFAR-100 as the real dataset and evaluate four distinct attribution methods (details in Appendix D.I The ROAD results on semi-natural dataset mg and CIFAR-100 are depicted Figure 3a and Figure 3b. respectively. These figures demonstrate that our tailored Rect method excels in ROAD evaluation on the semi-natural dataset, particularly at high mask ratios, but underperforms CIFAR-100, indicating the bias in evaluat on setirnatural datasets. Similarly, non-customized attribution methods like GradCAM, IG, and DeepSHAP exhibit inconsistent performance across the two datasets, underscoring that evaluation on semi-natural and real datasets can yield distinet results.
3.3 Order-based evaluation metrics
Many evaluation metrics for feature attribution methods, such as those described by ROAD (Rong et al. 2022) and Insertion/Deletion (Petsiuk et al. 2018), operate by incrementally perturbing features based on their sorted indices, which are derived from the feature attribution values. These perturbed inputs are then fod into the model to compute predictions, which are compared with those from the original input. The variation in predictions serves as an indicator of the importance of the perturbed features. Despite the popularity of these order-based metries, emphasize that they only assess the relative attribution order of

---

Published in Transactions on Machine Learning Research (11/2024)
the features that is, whether one feature is more important than another. They do not account for the magnitude of differences between the attribution values of the features. Consider a synthetic example where an input has three features, (2,, Ta] T and attribution method A yields an attribution map [1.0, 2.0. 3.0" while method B produces [1.0, 1.1, 100 Employing an order-based metric like Delction (Petsiuk et al. 2018), the removal order for both maps would be identical, i.e., 33, I2, #1. This would result identical output variations for each removal step and produce two identical curves when plotting output change against the removed feature index, ultimately leading to the same AUROC value for both methods. However, the semantic information conveyed by the attribution maps differs significantly: method A suggests that 23 is slightly more important than and 22, whereas method B indicates that is far more important. In summary, although order-based metrics can compare the relative importance of features, their ability to distinguish allribution values is limited.
4 Method
As highlighted in Section 3 both model retraining and the construction of semi-natural datasets present shortcomings that hurting the faithfulness of evaluation. In response, we have developed alternative ap- proach that avoids model retraining and the creation of additional datasets. Our method employs the fixed trained model and does not inject "ground truth" features into the dataset. Consequently, this approach is free from the shortcomings identified in Scction 3 Specifically, our approach originates from the observation that misaligument between altributed features and ground truth predictive features occurs in two distinct ways: (1) non-preed con-predictive features are incorrectly attributed; (2) predictive features receive zero attribution. Motivated by this observation, we formalize two essential properties of feature attribution: attribution sound- ness and attribution completeness. The combined evaluation of these two properties offers more refined and comprehensive assessment of the faithfulness of an attribution method.
4.1 Problem formulation
Our evaluation scenario is restricted to a specific model and dataset. This focus stems from our demonstration in Section 3 that the performance of attribution methods can significantly vary across different models and datasets. To aid readers, include table of mathematical notations in this work in Appendix A Given model that takes set of features F as input, we define the predictive information measurement function & and attribution method as follows:
Definition 4.1 (Predictive information measurement s). For feature set F and a feature F F, f) E Rzo represents the amount of predictive information of F.
Definition 4.2 (Attribution method 17). An attribution method n for a model f is a function that assigns value FFFFF:)) R20 the attribution to each feature F in the feature set F. This value quantifies the importance or contribution of feature F to the predictions made by the model f.
Definition 4.1 and Definition establish the frameworks for measuring true information and the attribution process within model, using functions. We provide concrete illustrative example for better understanding. Consider model ʃ as an image classification model that takes an image as input. Here, the feature set. F represents the imput image, and the feature F is a pixel in the input image. Since we constrain our discussion to a particular model and input data, we omit parameters I and F and use r7(F) or (PF) in the following text for notation simplicity. For a model f, there only exists a unique yp that measures the predictive information for the model. However, &o is inaccessible since knowing it requires a complete understanding of the model and its inner working mechanism. In contrast, there exist numerous possible attribution methods n, We can define the optimality of attribution method as functional cquivalence:
Definition 4.3 (Optimality of attribution method). An attribution method rj is optimal, 17 equals y.
However, it is challenging to directly compare the attribution method n with the predictive information measurement & because their analytical forms are usually not accessible. Instead, we assess their outcomes. To do so, we focus on two specific subsets of the feature set F. For a model f, a feature set F, and an attribution method s,, the predictive feature set I and the attributed feature set. A are defined as follows:
7

---

Published in Transactions on Machine Learning Research (11/2024)
Foatures nn Dataset
All Features in Dataset
All Features Datasot
Attibutedd Fastures
Procictive
A
A'
I
A
A' I
Fealures
A
AnI
I
(a)
(b)
(c)
Figure 4: Graphical demonstration for a better understanding of (a) the relationship between two attri- butions (A and A'). Although A and A'' have equal soundness (1.0 in this case), A' has higher completeness. (b) Although A and A' have equal completeness, A' has higher soundness. (c) We compare A An I with A and I to measure soundness and completeness.
Definition 4.4 (Predictive feature set Z). I C F is a predictive feature set if = (F E F ((F) > 0}. Definition 4.5 (Attributed feature set A). A F an attributed feature set A = (F E (TF) > 0}-
In essence, I F represents the features that are utilized by the model for decision-making, while A C F encompasses features identified significant by the attribution method f). Therefore, can compare the alignment between A and I to determine the faithfulness of the attribution method 17 on certain feature set.
Definition 4.6 (Optimality of attributed leature set A). Given a predictive feature set I, an attributed feature set A is sound if A A I, complete if I C A, and optimal (sound and complete) if A AA=I. I.
Definition 4.6 outlines the necessary conditions for an attributed feature set to be deemed optimal, namely, the set must be both sound and complete. This condition is unique in that the elements in A and I-or, equivalently, the feature indices must match exactly. However, simply comparing the feature indices of two sets is insufficient to fully assess the faithfulness of attribution. This because different attribution methods may assign varying values to the same feature, and feature may carry different predictive information across various trained models. Therefore, it is crucial consider both the attribution value and the predictive information associated with feature to further assess the aligmment between A and I. To this end, we first introduce the operator | le to measure the "cardinality" of set. Subsequently, we define two metrics to gange the soundness and completeness of A.
Definition 4.7 (Operator Ig). Given a feature set F and a function g, FFgg = EFEF For 0, we define 10,, =
In other words, for set of features F, FFl computes the total attribution of all features in F as determined by the attribution method n, while FFlp computes the total amount of predictive information in F. Following our carlier definitions, we can finally define the two properties of attribution:
Definition 4.8 (Soundness). For set of attributed features A, the soundness is the ratio LANZIa TA,,
Definition 4.9 (Completeness). For a set of attributed features A, the completeness the ratio LAnZle 12
Note that we use two operators soparately Definition 4.8 and Definition 4.9 Soundness measures how much of the altributed features actually contain predictive information. Complementary to soundness, completeness evaluates how comprehensively the attribution captures all predictive features. Both metrics are necessary to evaluate an attribution method. Figure D. illustrates two sets of attributed features with equivalent soundness but divergent levels of completeness, whereas Figure 4p depicts two sets of attributed features that share equal completeness but exhibit different soundness. Figure 4 illustrates the relationship between A ZZ, A, and Z.
Given our formulation, one challenge of measuring soundness and completeness that we have no information about predictive feat I. Therefore, it is infeasible to directly calculate either [AnZ), or |AnI p Despite
8

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm Soundness evaluation at predictive level
1: Input: labeled dataset DD {x(*) J with attribution maps [A [==1 model f, predictive level v accuracy threshold c.
2: Initialize 8 = 0, [Á(IN Jii = {A LN inc i=1 = 0;
// Start with empty Aine-
3: while s < do
1.N {A4 me (AA()) N i=1
Expand {A IN i=
Expand by adding the features with the highest attribution from A Ame, and record newly included features.
5: = Accuracy(f. {A y 1m
if 8 <6 then
{4 Ji=1 = (A0) AA 1 |i
end if
= 8;
10: end while
11: (A = 14 \ A
(1)
// Find A" by excluding accumulated S (F Z).
12: " (14-00) hn/ /
13: Return 7 N EE=
// Average soundness for all samples.
this absence of explicit knowledge about the model's predictive features, we can leverage the model to provide indirect indications. We make the following assumptions throughout the work:
Assumption 4.10. Given a dataset DD let F be the set of all input features in DB f be a model, and be a performance metric to assess the performance model f.
VF. p(f(FF) < ((f(F2)), then [INFill < IN
where I C F is the set of predictive features for the model f.
Based on Assumption 4.10 we can compare [In with Inn F2le using the model performance given two sets of features Fi and Fz. Specifically, while model performance is useful for identifying feature sets that contain more information, it is not suitable for quantifying the precise difference in information. Such mcasurement requires much stronger assumptions than Assumption 4.10) We conjecture Assumption 4.10 to be true for models converged in training. Next, show how measure soundness and completeness based on our definitions and Assumption 4.10.
4.2 Soundness evaluation
Owing to the intractability of |A n Z]n, the direct calculation of the soundness of A in single step is infeasible. However, shall demonstrate that iterative approach can effectively gauge the soundness for subset Aue S A that is included within the input. To ensure a fair evaluation across various attribution methods, we define Anee in our implementation as a subset of the features that possess the highest attribution values, satisfying the condition p(f(Ame) = > O. essence, this subset encompasses predictive features that cumulatively attain a specified predictive level v. A predictive level can be mcasured in various ways. For instance, in a classification task, it can be measured by Accuracy. The subsequent theorem shows the methodology for identifying the truly predictive portion within Aine; well as the means to compute the soundness.
Theorem 4.11. Given set Aine A and Aine nI 0,, suppose that So {S S Aine p(f(S) p ((((Ain))) is not emply for any Ainc A. Lel A arg minses, 11,, Then, the soundness of Ainc. is Id'la
Amchy
Proof: For any S (Ame)) including A", Assumption 4.10 and the condition p(f(Amc)) p(f(S) imply that [Znn 1 |SnI p. Note that S _ Ame implies |SNI $ Ame with the equality being achieved when Sn I = Aine 0 I. Therefore, S Aine OII Then, the minimization problem minses. (S],
9

---

Published in Transactions on Machine Learning Research (11/2024)
All Features in Dataset
A
Aine C
A
A" Ane ni
I
Figure 5: Soundness evaluation. Computing the soundness of A in single step is unfcasible. Instead, we incrementally include a subset Aine in input and compute its soundness. This process involves identifying the optimal set A' and calculating 14" Ainel A particular A is associated with a specific predictive level (i.e., model performance). When comparing two attribution methods, we can standardize the predictive level, allowing us to evaluate the soundness at this fixed level.
minses. (Amm) (1S +|n I)n, boils down to (Am S (F)) I),, because S SnI = Aine NI implics IS S , = |mme n In, which is a constant with fixed Amee- Thus, according to the definition of A and Ane nI + 0, we have |A" n (F) 1) = and |A = Ame n n + |A" n (F) Z) = |Aiu n Im Therefore, the soundness of Aiee is |Ame 7I
A'la
| We can also prove that minimizer A is the set that contains all predictive information of Aine. (A*((F)I))
(Anch
0 indicates A II since ()F) > 0 holds for all E A C. Next, A CI and A C Aiee yields A S (Aine nZ). Combining with |A* = | nI we have A' Ame NI.
Theorem 4.11 shows that the soundness of Aine can be computed by finding an clement from (Aiuc) that has the minimum attribution. Figure 5 depicts the relationship between A", Aine and I. Additionally, it is crucial to recognize that our definition of Ane satisfies the conditions in Theorem 4.11 specifically Aine S. A and Aine nI * 0. This inequality holds because p(f(A)n)) > O, and Assumption 4.10 implies Aine nI > 10
To facilitate comparison between different attribution methods, we can compute and compare their soundness a fixed predictive level Algorithm shows how to compute the soundness predictive level v. We gradually include features with the highest attribution values in the input. During the set expansion of Aine, we simultaneously perform minimization predictive features based on the change in the model's performance. After reaching predictive level we can directly calculate soundness based on the set Aine and the optimized set A". Nonetheless, Algorithm [ only approximates the actual soundness. The iterative algorithm uses accuracy to identify informative features, which may introduce bias, as certain features could be more contributive in conjunction with different sets of features. To minimize this bias, our approach incorporates a batch of features rather than single feature at each expansion step. Moreover, halting at specific accuracy threshold before saturation to capture potentially important features. A comprchensive description of the soundness evaluation algorithm is provided in Appendix E. Notably, lincar imputation [Rong el al. 2022) is used in soundness (and completeness) evaluation procedures to miligate OOD effects caused by feature removal (as we progressively include a portion of features). It is crucial Insortion/Delction metrics. While soundness evaluates the ratio of attribution values associated with two feature sets, Insertion Insertion/Deletion Deletion assesses accuracy following feature removal and employs attribution values solely for feature sorting.
shown in Theorem 4.11 by examining and excluding non-
sequentially include features from highest to lowest attribution,
to recognize that the soundness evaluation differs fundamentally from classical Insertion Delction
4.3 Completeness evaluation
Completeness, as previously discussed, assesses the extent which all predictive features are detected within attribution maps. Per Definition 4.9. removing attributed features from complete attribution map might
10

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 2 Completeness evaluation at attribution threshold
1: Input: dataset DD = (-(²), y with attribution maps (A00) J model f, attribution threshold t. Initialize 8o = Accuracy(( (f. D);
3: (A = ( :=::
// Include features w/ attr. val. > t.
(A0ON suii= = (F (
5: D 1
Return Ast 80 (f
// compare As; for completeness comparison.
also remove predictive features. However, if the attribution method has low completeness. removing the attributed features would not eliminate all predictive features in the input. Theorem 4.12 tells how to compare the completeness using the remaining features after the removal of attributed features.
Theorem 4.12. Let A1 and A2 be the attributed features given by two attribution methods, respectively, If p((FF) A1)) < (F A2)), then the allribulion method associaled with Ai is more complete than the one. associated with A2.
Proof: The condition of p(f(F) \ A1)) p(f(F) A2)) together with Assumption 4.10 implics that IIN
(F Ai)le IIn (F) A2)lpp Here, for any set S, IIn (F {S)( Ere(In(Fis) (PF Erez (PP)
EFE(TNS) (FF) = Io - II inS) Slp. Using this for S = Ai and A2, we have that IIn (F Ai)le En(F)AA)le + |7 Ailp < Alp A2 IZnAal
1I < irAilee
12
Based on the above analysis, we present Algorithm for evaluating completeness at an attribution threshold 6. We start by removing input features with attribution values above t. Then we pass the remaining features along with imputed features to the model and report the difference in the model performance between the original and the remaining features. higher score difference means higher completeness. The detailed procedure of completeness evaluation shown in Appendix FF] The completeness evaluation diverges IaaliiinnInsertton Insertion approach in its method of feature removal: it removes features with altribution exceeding a specific value, whereas Insertion Delletoo removes features whose ranking is better than certain threshold (c.g. top 20%). Despite the subtlety of this distinction, discussed in Section 5.2 the completeness evaluation is capable of discerning differences in attribution values, a nuance that Insertion/ DDelttiinn may fail to capture.
from the Deletion/
5 Experiments
In this section, we begin by validating our proposed metrics in a controlled setting, serving as sanity check. We subsequently underscore the importance of considering attribution values during evaluation, rather than just focusing feature ranking order. This allows for differentiation between methods that rank features identically but assign differing attribution values. Lastly, we further demonstrate that using our two metries together provides a more fine-grained evaluation, enabling us to gain a deeper understanding of how an attribution method can be improved.
5.1 Validation of the proposed metrics
In this section, we first validate whether the metrics work as expected and reflect the soundness and com- pleteness properties. In other words evaluate whether the proposed algorithms follow the predictions of our theories. We empirically validate the soundness and completeness metrics using synthetic setting, Through a designed synthetic datasct and a transparent lincar model, we obtain ground truth attribution maps that are inherently sound and complete. These inherently sound and complete altribution maps are then modified to probe expected effects in completeness or soundness, allowing us to test how our proposed metrics behave different situations. By incrcasing the attribution values of non-predictive fcatures, we introduce extra attribution (termed as Introduce) which hurts soundness but improves completeness. Con- versely, removing attribution (denoted Remove) lowers completeness without influencing soundness. Our
11

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
Accuracy Difference
Soundness
0.8
Disestion Cmppoemmm
0.8
*Diection Tmmpooonnnn
0.6
0.6
Ground Truth
Ground Truth
0.4
0.4
Remove
Remove
Introduce
Introduce
0.2
0.2
0.4
0.2
0.6 Attribution Threshold
0.4
0.8
Accuracy
(a) Soundness
(b) Completeness
Figure 6: Completeness and soundness evaluations on the synthetic dataset. The curve and error bar respectively represent the mean and variance across 1000 trials. The curves of Remove and Ground Truth are overlapped in (a) as they both saturate at 1. Both soundness and completeness evaluations can reliably reflect the modifications in the attribution maps.
objective to evaluate these modified attribution maps to ensure our proposed metrics accurately capture changes in both soundness and completeness.
The synthetic two-class dataset consists of data points sampled from a 200-dimensional Gaussian NN I). Data points are labeled based the sign of the sum of their features. Let be the i-th imput feature and (-) is a step function that riscs at 0. The linear model, formulated as =[E, xz), is designed to replicate the data generation process and is transparent, allowing us to oblain sound and complete ground truth attribution maps. Appendix G.1 provides further details. We randomly add and remove attribution from ground truth feature maps 1000 times each. Then, we compare the soundness and completeness between modified and original attribution maps.
Statistical results in Figure D show that Remove consistently outperforms ground truth attribution in Com- pleteness, whereas Introduce underperforms ground truth attribution in Completeness. In Soundness evalua- tion, the ranking of the three methods inverses. Note that the optimum soundness of ground truth attribution is 1, which can be also reached by Remove. Tn conclusion, the evaluations behave as expected, validating our proposed metrics in this case.
5.2 Comparison with order-based metrics
Attribution methods aim to determine the contribution values of features beyond merely ranking them by importance. Consequently, evaluating these methods necessitates consideration of the actual attribution values. Both Completeness and Soundness metrics incorporate attribution values: the former uses value- based thresholds for feature removal, while the latter, denoted as Amel, attribution values. Next, we show that this consideration of attribution values results in a more refined evaluation.
14'1, inherently captures variations in
pre-trained on Tm- ageNet (Deng et al. (209)) and conduct feature attribution on the ImageNet validation set. We apply the Remove and Introduce modifications to the original attribution maps produced by a given attribution method, such as GradCAM, as visualized in Figure 7a These modifications are intentionally designed to slightly adjust the ordering of attributions, yet they significantly alter the attribution values. Consoquently, the original attribution maps and those modified by Remove and Introduce are differentiated not just in terms of attribution values but also in their visual presentation, ation metric must be capable of capturing these differences clearly. Therefore, for metric to be considered effective, the curves representing the evaluation results for the original, Remove- and Introduce-modiifee attribution maps should be distinct and non-overlapping.
For the following experiments, we employ a VGG16 (Sioonyan & Zisserman, 2015
illustrated in Figure 7a. A well-designed evalu-
As illustratd in Figure 7 the curves representing Remove and Introduce overlap ROAD and Deletion. This is attributed to the fact that these metrics are based solely on the order of attribution, which remains nearly unchanged between Remove and Introduce- modifid attribution maps. In contrast, the curves for the
12

---

Published in Transactions on Machine Learning Research (11/2024)
Image
Original
Remove
Introduce
(a) Samples of modified altribution maps
!Direction of Imprevement
1.0
(Direction Improoeeenn
Direction inprovement
Accuracy Difference
0.6
Original Remove Introduce
0.8
Diieession Improvereent
0.6
Original Remove Introduce
Original
Soundness
Accurucy
Accuracy
Remove
0.4
0.4
0.6
0.4
Introduce
0.2
0.4
Original Remove
0.2
0.2
Introduce
0.0
0.0
0.0
0.25
0.50
0.2
0.4
0.6
0.25
0.50
0.75
0.75 Mask Ratio Area
0.25
0.50
Attribution Threshold
Accuracy
Mask Ratio by Area
(b) Completeness
(c) Soundness
(d) ROAD (MoRF)
(e) Deletion (MoRF)
Figure Analysis of our metrics and order-based metrics. (a) Modified attribution maps. The modifications result in only minimal changes the feature order. These modified maps are noticeably different from the original. An evaluation metric should capture this distinetiveness. By taking attribution values into account, Completeness (b) and Soundness (c) aptly distinguish the modifications in the attribution maps. Conversely, the differences between curves are less obvious in ROAD (d) and Delction (c). A side note on (c) is that Remove might not always preserve soundness. This is because original attribution maps are not always the same as ground truth maps (inaccessible in the real world), and Remove can eliminate both predictive and non-predictive features.
value-sensitive metrics Completeness and Soundness are noticeably distinct, highlighting their sensitivity to changes in attribution values. To quantitatively assess the differences between evaluation curves, we calculate the minimal Hausdorff distance between pairs of curves, denoted as minp-g Hausdorf[pp 4), where p. E {Original, Introduce, Remove]. A minimal Hausdorff distance approaching zero signifies an overlap in the evaluation results, indicating that the metric fails to distinguish between the modified and original attribution maps.
We implement three pairs of different modification schemes for Introduce and Remove, which are elaboratod in Appendix G These modification schemes were applied to attribution maps generated by GradCAM, IG, and ExPerturb, and the process of modification and evaluation was iterated for each scheme. The resulting minimal Hausdorff distances were then averaged. As indicated in Table T the Hausdorff distances for Com- pleteness and Soundness metries are significantly greater than zero. This demonstrates that these metrics can effectively differentiate between the modified and original attribution maps, even when the changes in attribution order are minimal. Conversely, the order-based metrics, ROAD and Deletion, demonstrate over- laps their curves, indicating their inadequacy in discorning subtle distinctions. This contrast highlights the superior sensitivity of Completeness and Soundness metrics in evaluating the nuances of attribution maps.
5.3 Benchmark experiments
As previously discussod, the misalignment between attributed features and predictive features arises from two types of attribution errors. By employing both the Completeness and Soundness metrics, we can identify which type of error reduction contributes to the superior performance of one method.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Minimal Hausdorff distances across evaluation curves for Original, Remove, and Introduce-modd attribution maps. Near-zero distances in ROAD and Delction imply curve overlap, indicating limited dif- ferentiation between the attribution maps. Conversely, sigmificant distances in Completeness and Soundness highlight their ability to distinctively evaluate and differentiate attribution maps, showcasing their finer
granularity.
Metric
Completeness
Soundness
Deletion
ROAD
Hausdorff
0.503 + 0.112
0.183 + 0.046
0.019 + 0.011
0.014 + 0.009
Distance
KG
IG-SG
Accuracy Diffgracce
0.14
of Inppooetaar
1G
0.6
1G
Dinciion
1G
0.9
1G-SG
KG-S
1G-SG
Soundress
0.8
IG-
0.10
1G-SQ 16-Var
Ascuncy
0.4
IG-SQ
TG-
Accurccy
0.2
IG-SQ
1G-Var Var
0.06
0.7
0.2
0.1
0.02
0.6
ingromment
0.0
0.0
0.25
0.30
0.75
0.25
0.30
0.25
0.30
0.75
Accuracy
Attribution Threshold
Mask Ratio by Area
Mask Ratio by Aren
(a) Soundness
(b) Completeness
(c) ROAD
(d) Deletion
Figure 8: Benchmark of IG ensembles. By employing both the Completeness and Soundness metrics, we can see that the superiority of ensemble methods over IG is predominantly in their soundness.
Benchmark of ensemble methods Several ensemble methods have been proposed as a means to im- prove attribution methods. In this study, we focus specifically on three ensembles of IG: SmoothGrad (IG-SG) (Smilkov et al. 2017), SmoothGrad (IG-SQ) (Hooker et al. 2019). and VarGrad (TG-Var) (Ade- bayo al. 2018). Intriguingly, only IG-SG displays an enhancement in completeness (Figure ( consistent with visual results from carlier studics. We present supplementary visual results in Appendix G.4 for further scrutiny. Previous research (Smilkov et al. 2017 has observed that gradients fluctuate significantly in neighboring samples. Consequently, the aggregation of attribution from neighboring samples can mitigate false attribution specifically those arising from non-predictive features receiving attribution and notably enhance soundness, as depicted in Figure Sa However, the benefits of ensemble methods are not so clear in ROAD (Figure Sc or Deletion (Figure 8d).
Benchmark of various attribution methods We conduct a comparative analysis of multiple attribution methods using our metrics with the goal of guiding the selection of suitable methods for diverse applications.
As illustrated in Figure g most of the evaluated methods excel in one metric over the other, suggesting their suitability varies based on specific scenarios. For applications like clinical medicine, where capturing all relevant features is essential, methods with higher completeness, like ExPerturb, stand out. On the other hand, in situations where falsely identifying non-predictive features as significant could be detrimental, methods showcasing superior soundness, such as IBA or GradCAM, are preferable.
6 Conclusion and limitations
In this paper, we first revealed the potential pitfalls in existing faithfulness evaluation of attribution methods. Subsequently, we defined two important propertics of attribution: soundness and completeness. We also proposed methodologies for measuring and comparing them. The two metrics work in conjunction and offer a higher level of differentiation granularity. Empirical validation convincingly demonstrated the effectiveness of our proposed metrics. Furthermore, we undertook benchmark of ensemble methods, revealing that these methods can considerably improve the soundness of the baseline. Lastly, we extended the comparative analysis to broader range of attribution methods to provide guidance for selecting methods for different
14

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
Direction of ooopoovemettt
Direction of Improvement
Aecuracy Difference
0.6
0.9
Soundness
GidCAM
ExPerturb
0.8
0.4
DeepSHAP IG
InpulBA
0.7
GraCCAM ExPatuub
0.2
DeqSSHAP
0.6
InputBBA
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.2
0.4
0.6
0.8
Accuracy
Attribution Threshold
(a) Soundness
(b) Completeness
Figure 9: Benchmark of different methods. Although no method exhibits superior performance in both Completeness and Soundness, some of them perform well in one of these metrics, implying their suitability for applications which have high demand for the corresponding property.
practical applications. One limitation of our evaluations that their efficacy hinges on the assessment of model performance. Accuracy may not. be the appropriate performance metric some cases. Therefore, additional research in the future needed to find better performance indicators for different. tasks. In addition, Theorem [.11 does not limit the selection of Aine, and better expansion strategies for Aine could yield more precise evaluation outcomes.
7
Acknowledgment
This research is partially supported by the National Research Foundation Singapore under the Al Singa- pore Programme (AISG Award No: AISC2--TC-2022-000--GLL and the Singapore Ministry of Education Academic Research Fund Tier (Award No: T1 251RES2207), and UKRI grant: Turing AI Fellowship EP EP/W002981/1.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems. Curran Associates, Inc., 2018.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. In International Conference on Learning Represen- tations, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017. URL https://arxiv.cm abs 0bss/101.077 07875
David Bachrens, Timon Schrocter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:
1803-1831, 2010.
Mélanie Bernhardt, Charles Jones, and Ben Glocker. Potential sources dataset bias complicate investiga- tion of underdiagnosis by machine lcarning algorithms. Nature Medicine, 28(6):1157-1158, 2022.
Ewen Callaway. 'the entire protein universe:: Ai predicts shape of nearly every known protein. Nature, 608
(7921):15-16, 2022.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Topology preserving local road network estimation from single onboard camera image. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 17263 17272, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248- 248-255, 2009. doi: 20100//VP.2200052222
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2950-2958, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farloy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nelworks. Commmnicaatinns of the ACM, 63(11):
139-144, 2020.
Hüseyin Anil Gündüz, Martin Binder, Xiao-Yin To, René Mreches, Bernd Bischl, Alice C McHardy, Philipp C Münch, and Mina Rezaei. A self-supervised deep learning method for data-efficient training in genomics. Communications Biology, 6(1):928, 2023.
Hüseyin Amil Gündüüz, René Mreches, Julia Moosbauer, Gary Robertson, Xiao Yin To, Eric A Franzosa, Curtis Huttenhower, Mina Rezaci, Alice C McHardy, Bernd Bischl, et al. Optimized model architectures for deep learning genomic data. Communicatiins Biology, 7(1):516, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
José Jiménez-Lana, Francesca Grisoni, and Gisbert Schneider. Drug discovery with explainable artificial intelligence. Nature Machine Intelligence, 2(10):573 584, 2020.
Berk Kaya, Suryansh Kumar, Carlos Oliveira, Vittorio Ferrari, and Lue Van Gool. Uncertainty-aware deep multi-view photometric stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12.11200 12601-12611, 2022.
Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Scong Tae Kim, and Nas- sir Navab. Neural response interpretation through the lens of critical pathways. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 13528-13538, June 2021a.
Ashkan Khakzar, Sabrina Musatian, Jonas Buchberger, Icxel Valeriano Quiroz, Nikolaus Pinger, Soroosh Basclizadch, Scong Tae Kim, and Nassir Navab. Towards semantic interpretation of thoracic discase and covid-19 diagnosis models. In International Conference on Medical Image Compuling and Compuler- Assisted Intervention, pp. 199-508, Springer, 2021b.
Ashkan Khakzar, Yang Zhang, Wejdene Mansour, Yuezhi Cai, Yawei Li, Yucheng Zhang, Seong Tae Kim, and Nassir Navab. Explaining covid-19 and thoracic pathology model predictions by identifying infor- mative input features. In International Conference on Medical Image Computing and Compuler-Assisled Intervention, pp. 391-401. Springer, 2021c.
Ashkan Khakzar, Pedram Khorsandi, Rozhin Nobahari, and Nassir Navab. Do explanations explain? model knows best. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10244-10253, June 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations, 2015.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardsnn. Captum: A unified and generic model interpretability library for pytorch, 2020.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXic:220..0 01602, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In Citeseer. Citesecr, 2009a.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). 2009b. URL http:://ww.ccc twwootottoooooo www.att/o.dduuriii/cccoooo ciftr..irr.../////// .html
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34: 26422 26436, 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise; Randomized input sampling for explanation of black-box models. BMVC, 2018.
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In International Conference on Machine Leorning, pp. 18795. PMLR, 2022.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller, Evaluating the visualization of what deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660-256) 2016.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618 626, 2017. doi: 10. 10.1109/ICCV. 2017.74.
Lloyd S Shapley et al. A value for n-person games. 1953.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3143-3153. PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. In Workshop on Visualization for Deep Learning, 1CML, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplic- ity: The all convolutional net. In Yoshua Bengio and Yann LeCun (ods.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pp. 3319 3328.. PMLR, 2017.
Scott Cheng-Hsin Hsin Yang, Nils Erik Tomas Folke, and Patrick Shafto. psychological theory of explainability. In International Conference Machine Learning, pp. 25007-25021. PMT.R, 2022.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084 1102, 2018.
Yang Zhang, Ashkan Khakzar, Yawei Li, Azade Farshad, Seong Tae Kim, and Nassir Navab. Fine-grained neural network explanation by identifying input features with predictive information. Advances in Newral Information Processing Systems, 34:20040 200511, 2021.
Yang Zhang, Yawei Li, Hannah Brown, Mina Rezaei, Bernd Bischl, Philip Torr, Ashkan Khakzar, and Kenji Kawaguchi. Attributionlab: Faithfulness of feature attribution under controllable environments. In NewrIPS 2023 Workshop XAI in Action, 2023.
Bolci Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pallern recognition, pp.
2016.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 9623 9633, 2022.
18

---

Published in Transactions on Machine Learning Research (11/2024)
A
Notations
Table summarizes the notations used in this paper.
Table 2: Table of notations.
predictive level (i.e. a specific level of model performance)
f
a model to explained
P
performance metric to assess the performance of model f (labeled) dataset
ID
perturbed training set using perturbation strategy i
Train
a perturbed test set using perturbation strategy
Teat
semi-natural dataset constructed by modifying the original dataset using modification method set that contains all features in the dataset
F
A
set of attributed features
I
set of predictive features for the model
In
the operator to calculate the sum of attribution
the operator to calculate the sum of class-related information a single feature JF
F.
(P(F)
function that returns the information value of a feature F function that returns the attribution value of feature F subset of the most salient features that reach p(f(A for given set Aine; we define S. (AAic) = (S C Aine p(S(Ainc)) (Ainc))= (AAnn))=)((($))) minimizer of minses,(Amm)) |S],
m(F)
Aine
(Ainc)
A
B Broader Impacts
We believe that our proposed completeness and soundness evaluations open many innovative directions. For instance, we have shown that the ensemble methods can greatly enhance the soundness of baselines such as IG and DeepSHAP. However, the gain completeness is very marginal. would be interesting to investigate how to also improve the completeness of IG, DeepSHAP, or their ensembles. Extremal Perturbations demonstrate lower soundness than IBA and GradCAM that perform attribution on the hidden neurons. This might suggest. that the semantic information in hidden layers be utilized in the optimization process of the Extremal Perturbations to reduce false attribution.
addition,
C Additional experiments for revealing the issues with retraining- metrics
In this section, we report an additional experiment to further illustrate the issue with retraining based eval- uation. For this additional experiment, we use the CIFAR-10 (Krizhevsky et al. 2009a) dataset. The model is a tiny ResNet (He et al.) 2016) with only residual blocks. Training is conducted using Adam (Kingma & Bal (0155) optimizer with learning rate of 0.001 and weight decay of 0.0001. The batch size used for the training is 256, and we train model in 35 epochs. Next, describe how to construct the maliciously modified dataset for retraining.
In the retraining experiment shown in Figure 10 we generate modified dataset from the original CIFAR-10 dataset. In this additional experiment, we only perturb 5% of each training image and replace the perturbed pixels with black pixels. The perturbation is correlated with class labels. For different classes, we select different positions close to the edge of the image so the object (usually at the center of the image) is barely removod.
The result is shown in Figure 10
We summarize our findings in the caption of Figure 10
19

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
0.887
0.8
Test Accuracy
0.6
0.4
0.322
0.2
0.0
Retrained
Original
(b) Evaluation on unperturbed test
(a) Sample training images
Figure 10: Retrain the perturbed dataset with spurious correlation. (a) illustrates sample images from the perturbed training set. Only small portion of pixels on the edge is removed. Hence, the class object is usually intact after perturbation. However, the position of the removed region depends on the label of the image. (b) Test accuracy on the unperturbed test set. Although objects are not removed in the perturbed training set, the retrained model achieves much lower test accuracy on the original test set than the model trained on the original dataset. This means that the retrained model ignores the object but learns to perform classification based on the spurious correlation introduced by perturbation. We would like to further demonstrate the issue of retraining, that the retrained model fails to learn exclusively from remaining features in the perturbed dataset. Hence, we cannot use the model performance to measure the information loss caused by perturbation.
D
Additional Experiments and Experiment Configurations on Semi-natural Datasets
D.1 Evaluation with Models Retrained CIFAR-100, Semi-natural, and Pure Synthetic Datasets
Image
IG
Rect
Pooling
Image
IG
Rect
Pooling
1.0
10
29
54
0.2
0.2
57
0.0
0.0
(a) Semi-natural datasel with number watermarks (i.e. (
(b) Semi-natura datasel. with stripe watermarks (i.e. (22)
Figure 11: Semi-natural datasets with different watermarks. Two types of attribution maps crafted based on IG attribution maps by utilizing prior knowledge about the watermarks. Rect is designed to fit (1))) while Pooling is designed to fit To(2)
20

---

Published in Transactions on Machine Learning Research (11/2024)
0.70
GradCAM
GradCAM
GradCAM IG
IG
IG
0.65
DeepSHAP
0.20
DeepSHAP
DeepSHAP
0.4
Accuracy
0.60
Accuracy
0.15
Accuracy
0.55
0.2
0.50
0.05
0.45
0.00
0.0
0.2
0.4
0.6
0.2
0.4
0.8
0.2
0.4
0.6
0.8
Mask Ratio by Area
Mask Ratio by Area
Mask Ratio by Arca
(a) Original CIFAR-100
(b) Semi-natural dataset
(c) Pure synthetic dataset
Figure 12: ROAD evaluation on different datasets. The performance of attribution methods is very distinct across different datasets. We observe that the ROAD result on the original dataset is different from semi- natural dataset and pure synthotic dataset. However, the ROAD result on the semi-natural datasct is very similar to the result on the pure syuthetic dataset. This implies that the semi-natural dataset changes the implicit learning task from learning representations of the original images to learning representations of synthetic symbols introduced during dataset construction. Subsequently, the task is greatly simplified and substantially divergent from the real dataset.
we argue that the attribution methods can behave much differently when explaining the model retrained on semi-natural dataset. As a result, it is not faithful to use the evaluation result on the semi-natural dataset as an assessment for the feature attribution methods. In this section, we demonstrate this issue with an experiment.
In Section 3.2
We first show the datasets used in the experiment. re-assign the labels for CIFAR-100 KKiizhvssy al. 2009b) images suggested in (Zhou et al. 2022). Next, we inject two types of watermarks into the images and blank canvas, obtaining two pairs of semi-natural and pure synthetic datasets, respectively. Note that the semi-natural datasets are also used in Section 3.2 The watermarks are designed as follows:
Number watermark: depicted in Figure Iia left, first insert a black rectangular region in the image and then put the white number sign within the black region.
Stripe watermark: as depicted in Figure 1lb left, we first encode the label into 7-digit binary mumber and divide the image into 7 equal-height regions. Next, set the pixels in each region to 255 if the corresponding digit is 1; otherwise, we leave the pixels unchanged.
The following experiment is conducted the semi-natural and pure synthetic dataset with number water- marks. We first train VGG-16 the CIFAR-100, semi-natural, and pure synthetic datasets, respectively. After obtaining the classifiers, we apply GradCAM, IG, and DeepSHAP to them to get the attribution maps. In the end, we benchmark the three attribution methods on each dataset using ROAD (Rong et al. 2022
The models achieve 70.4% on CIFAR-100, 99.4% on the semi-natural dataset, and 99.9% on the pure synthetic dataset, respectively. The difference accuracy shows that the learning tasks are differently complex across three datasets. Furthermore, as depicted in Figure [2 GradCAM outperforms IG and DeepSHAP on CIFAR- 100, while IG and DecpSHAP are much better than GradCAM on the semi-natural and pure synthetic datasets. The evaluation resulls the semi-nalural dalasel cannol correclly reflect allribution methods' performance on the real-world dataset.
D.2 Details Designing Rect and Pooling Attribution Maps
In this section, show how design attribution maps by leveraging prior knowledge about datascts. The target is to craft two types of attribution maps, with one performing well on the semi-natural dataset with number watermarks (i.e., IT(2)) in Section 3.2 and another performing well the semi-natural
21

---

Published in Transactions on Machine Learning Research (11/2024)
(2) in Section 3.2p. Both altribution maps are generated based IG dataset with stripe watermarks (i.e., D3 attribution maps. The following are the design details:
Rect is designed to fit ) where the modified pixels (i.e., Effecttve Region in (Zhou et al. 2022) are the black rectangular region (where the numbers are located) that we injected into an image. To craft attribution maps, we can also put a rectangular region full of value 1.0 on a background of value 0.0. The question where to put such a rectangular region. For each IG attribution map, we average across the pixels' spatial locations using the attribution values as weights, obtaining a "weighted center" of the attribution map. Then, we put the rectangular region of spatial size 60 x 60 at the weighted center. More samples are shown in Figure 11a
Pooling is designed to fit D where the modified pixels are the equal-height regions associated with digit 1. After knowing the shape of watermarks, we design attribution maps composed of 7 equal-height regions. To do so, apply average pooling to each IG attribution map, obtaining a 7-element attribution vector. Next, we fill each region in the crafted attribution map with the corresponding value in the attribution vector. More samples are shown in Figure 11b
E Implementation Details of Soundness Metric
Values in attribution maps are usually continuous. Hence, it is possible that an attribution method only has satisfactory performance only in a certain attribution value interval. To evaluate the overall performance of an attribution method, we use Algorithm [ as a basic building block to establishing the progressive evaluation procedure. Specifically, we evaluate soundness at different predictive levels indicated by the model performance. How soundness is calculated at specific predictive level has been explained in Section
Since the attribution method considers features with higher attribution values to be more influential for the expand our evaluation set by gradually including the most salient features that are not yet in the evaluation set. As shown in Algorithm 3 the expansion of the evaluation set happens by decreasing the mask ratio. For the soundness metric, the mask ratio means that the top v pixels in an attribution map sorted in ascending order are masked (i.e., area- based LeRF). We start from v = 0.98 and decrease v by the step size of 0.01. This is equivalent to first inserting 2% of the most important pixels in blank canvas and inserting 1% more pixels each step. If the accuracy difference between the current step and the previous step is smaller than the threshold 0.01, then the attribution of newly added pixels is deemed to be false attribution and will be discarded. Algorithm 3 demonstrates a more detailed procedure compared to Algorithm in Section
model prcision-making,
4.2
Note that some notations are overloaded.
F Implementation Details of Completeness Metric
To obtain the overall completeness performance of an altribution method, we again select subsets of an attribution set and evaluate the completeness of these subsets.
demonstrates the full computation process. For the completeness metric, the attribution thresh- old l means that the pixels with attribution between [t, 1] will be masked (i.e., value-based MoRF). We start from v = 0.9 and decrease by the step size 0.1. Compared to Algorithm iin Section [.3] the pooudo-code in Algorithm 4 is more detailed and closer to the actual implementation. Note that some annotations are overloaded.
Algorithm !
G Validation and Benchmark Experiments
G.1 Validation Tests
We create a two-class dataset of 1000 sample data points, and each data point has 200 features. The data point is sampled from 200-dimensional I) Gaussian distribution. If all features for a sample point sum up to be greater than zero, we assign positive class label to this sample. Otherwise, negative class label is
22

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 3 Soundness evaluation with accuracy (sm) performance indicator
1: Input: f: model; D = ,,'"),( p0 labeled dataset with attribution maps (A(()) ) Φ: perturbation function; v:: noisy linear imputation function; M = {0.99, 0.98, 0.97.... 0.01 mask ratios (by area); Accuracy: accuracy evaluation function. e: accuracy threshold; NewAdded: function that identifies the included features the current step and newly added features compared to the last step. Output: P: A list of tuples with each tuple being the accuracy and the soundness value.
Initialize F - [];
[
Ji=1 < {0];; // Initialize the set of features with false attribution.
5: mo 1; // Initialize the mask ratio at the previous step.
6: 80 +0; 0; // Initialize the accuracy at the previous step.
7: for m in
M do
8: Dm
f 0
// Initialize imputed dataset
for (x(1) (.) in (D, (A do
10:
200
(xx1) A00, m); // Perturb image in LeRF order.
11:
200 (8(0)); // Impute image.
12:
inc AAO + NewAdded(.A)), m, mo): // Identify included features at the current step and newly added feature compared to the last step.
Append(ID,, m. (,()))() ( AA());; 14: end for
13:
15: 8m f AAcuuaccyfF.. Dm); Accuracy on the imputed dataset.
16: if 8m 50 < 6 then
for ((+61, y()) A A, A in (Dm, ( do
17:
A + At) UAA(;; Update the features with false attribution. 19: end for
18:
20: end if
21: for ((x(), y(0) Ance AA A()) in (D.. {4 do
(00) 14
119
22:
23:
end for
Attribution ratio at the current step. 25: Append(IP, (5m:9) 9)) // Update results.
24: q+ 1/ E (())
26: So Smi
// Update the accuracy at the previous step. // Update the mask ratio at the previous step.
27:
28: end for
Return: P
assigned (as described in the main text). The model is linear model and can be formulated y = mi). where is the i-th feature, and a(-) is a step function that rises at x = 0, a(z) = -1 if r < 0, and o(x) = if x > 0. In other words, the model also sums up all features of the input and returns positive value if the result is greater than zero. Hence, the model can classify the dataset with 100% accuracy. Lastly, we describe how we create ground-truth attribution maps for this model and dataset. As the model a lincar model, and each feature x; is sampled from zero-mean Gaussian distribution, the Shapley value for æ; with (2,) 1 is then 1 (x, E[z]) z;, Similarly, the Shapley value for x, with o(x;) = -1 is -2;- We confirm that attribution maps generated by Shapley values are fully correct for linear models. As result, for positive samples, the attribution values are the same as feature values. For negative samples, the attribution values are the negation of feature values, which means that negative features actually contribute to the negative decision. Finally, we modify the attribution maps to be compatible with our soundness and completeness evaluation. Since our evaluation only supports positive attributions, we clip negative attribution values zero. This conversion step has no negative effect on the actual evaluation. The rest of the evaluation setup is identical other experiments.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 4 Completeness Evaluation
1: Input: f: model; D = {-(1, y labeled dataset and attribution maps { Ji=1 for each (x)
Φ: perturbation function; the noisy linear imputation function; T {0.9, 0.8, ..7,...,, 0.1}: attribution thresholds; Accuracy: accuracy evaluation function.
Output: SA: Accuracy differences associated with attribution thresholds T.
3: Initialize S <- [];
4: < (f))racy(f.)); // Accuracy on the unperturbed dataset
5: for in T do
[Ji // Initialize dataset of imputed images
for (xl1), .y()) 400 in (D, (A())) do
100)= + ((x() A ,t):
Perturb pixels whose attribution exceed z(0, ((20)); // Impute the perturbed image
(p, (I(1) y0)))
10:
11: end for
12: St + Accuracy (J, 1,;;; // Accuracy on the imputed dataset
13: Append(Ss, 8o st); // Accuracy difference at the current step
14: end for
15: Return SA
Implementation
Removing attribution
Introducing attribution Add the attribution map with a con- stant 0.6.
Constant
Subtract the attribution map by a constant 0.6.
Random
Sample shilt from u(-0.6, 0) for each pixel independently, and add the attribution of each pixel with the cor- responding shift.
Sample shift from U(0, 0.6) for each pixel independently, and add the at- tribution of each pixel with the corre- sponding shift.
Partial
Sort the attribution map in ascending order and select the pixels in the in- dexing range [0.6N, 0.8.], where N is the number of pixels. Then set the at- tribution of these pixels to O.
Sort the attribution map in ascending order and select the pixels in indexing range [0, 0.4], where N is the mum- ber of pixels. Then set the attribution of these pixels to qo.8, where ge denotes the t-th quantile of attribution values.
Table 3: The modifications of attribution maps on ImageNet.
G.2 ImageNet images for feature attribution
We randomly select 5 images for each class in the ImageNet validation set, obtaining a subset with 5000 images. When performing attribution, the images and attribution maps are resized 224 224 before being fed into the pretrained VGG16 model.
This subsection presents the configurations for generating attribution maps ImageNet. For GraCAM, We resize the resulting attribution maps to the same size as the corresponding input images. We use the implementations of GradCAM, DeepSHAP, IG, IG ensembles in Captum (Kokhlikyan et al. 2020 Some hyper-parameters for producing attribution maps are:
GradCAM We perform attribution on the features. 28 layer of VGG16 (i.c. the last convolutional layer).
DeepSHAP,, IG and IG ensembles We choose as the baseline for attribution. We clamp the attribution to [0, 1].
24

---

Published in Transactions on Machine Learning Research (11/2024)
Image
Original
Constant (R)
Constant (1)
Random (R)
Random Purtial(R)
Partial
Figure 13: Examples of modified attribution maps. (R) and (T) denote the Remove and Introduce modifica- tion, respectively. The authors ensure that samples are not cherry-picked.
G.3 Modifications of attribution maps on ImageNet
We implement three different modification schemes and modify the attribution maps. The details thereof are presented in Table 3 We show some examples of modified attribution maps in Figure 13 The images randomly selected.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Image
IG
IG-SG
IG-SQ
IG-Var
ONLY
Figure 14: Examples of IG and IG ensembles attribution maps. The authors ensure that samples not cherry-picked.
G.4 Benchmarking Ensemble Methods of IG
In each ensemble method, we use an isotropic Gaussian kernel (1(0,, 0.3 I) to sample 20 noisy samples for a given input sample. Figure 14 shows additional examples of IG and its ensemble methods. The images are randomly selected.
In addition, we also compare IG and IG ensembles using ROAD in LeRF order, and the result is shown Figure 15 For convenience, copy the figures of other metrics from Figure g and paste them in Figure 15
26

---

Published in Transactions on Machine Learning Research (11/2024)
1.0
IG
0.14
Direction of Improvement
Accuracy Difference
IG
IG-SG
IG-SG
0.9
IG-SQ
0.10
IG-SQ
Soundness
IG-Var
IG- Var
0.8
0.06
0.7
0.02
0.6
Direction of Improvement
0.25
0.50
0.75
0.2
0.4
0.6
Attribution Threshold
Accuracy
(a) Completeness
(b) Soundness
Direction Improvement
Direction of improvement
0.6
IG
IG-SG
IG-SQ
Accuracy
0.4
IG-Va
IG
0.2
IG-SG
0.2
IG-SQ
IG-Var
0.0
0.2
0.4
0.25
0.50
0.75
Mask Rutio by Area
Mask Ratio by Area
(d) ROAD (LeRF)
(c) ROAD (MoRF)
Direction of Improvemenn
0.3
IG
IG-SG
Accuracy
IG-SQ
"a-Var
0.2
0.1
0.0
0.25
0.50
0.75
Mask Ratio by Area
(e) Deletion (MoRF)
Figure 15: Evaluations of IG ensembles. Not all ensembles improve the completeness (a) of IG, but they significantly improve soundness (b). However, the advantage ensemble methods over IG is not notable ROAD (d) or Deletion (e) compared to that soundness.
Similar ROAD in MoRF order, IG ensembles do not show considerable advantage in the results of ROAD in LeRF order.
27


Reviews:
[{"review_id": "NO2hwd67vo", "reviewer": "Reviewer_RoJR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "See 'official comment' already filed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "See 'official comment' already filed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See 'official comment' already filed.", "broader_impact_concerns": "See 'official comment' already filed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lsjAobbDRH", "reviewer": "Reviewer_67M7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n* The authors reveal limitations with existing methods that are based on retraining with removed attributes and on injected ground-truth features. \n* The proposed metrics can capture differences in terms of the assignment of attribute importance, which existing methods that depend on the importance ranking fail to detect. \n* The proposed method does not require retraining of models and creating additional datasets. \n\n## Weaknesses \n* It is not clear how the limitations of the existing work have been addressed by the proposed metrics (the connection between Section 3 and 4 are not so clear). \n\n* The paper is not as self-contained as it could be: \n   1. Evaluation strategies like ROAD and Deletion are not explained at all.  \n   2. The same applies to feature attribution methods used in the experiments (such as IG) -- the ROAR paper does a good job in this regard. \n\nMinor comments: \n* The definition of \"predictive information\" is ambiguous. \"Predictive\" can be interpreted as being useful to achieve high performance in some prediction task or features that are used by a model to make certain decisions (but are not necessarily helpful in a given task). It appears that Assumption 4.10 effectively defines what informative means in terms of a performance metric $\\rho$.  \n* Definition 4.2 does not seem to define what a attribution method is (is just a function on $\\mathcal{F}$ that outputs nongevative values?). \nThere is also a typo: \"a feature $F$ and a feature $F$\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper propose two evaluation metrics (soundness and completeness) for feature attribution methods that quantify the contribution of input features to a model's output. \nSoundness measures the degree to which attribution methods assign importance to \"predictive\" features, whereas completeness quantifies the extent to which the attribution method collects and assign values to \"predictive\" features. \nThe authors provide algorithms to compute these metrics and clarify required assumptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please address the weaknesses points above\n* It is unclear what Expand in line 4 does. \n* There are many citations that should be parenthetic (e.g., the very first sentence). These should be corrected. \n* What is the attribution method used for Figure 7.(b)-(e)? \n* Appendix G.1:  Is the Shapley value the attribution method?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9mdYNl2ZZj", "reviewer": "Reviewer_CkYd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n  - Important problem, which is particularly challenging\n  - Decomposition in precision/recall metrics looks relevant\n  - Rather simple method to implement \n\nWeaknesses: \n   - Some parts are the paper are not sufficiently detailled or not enough from my point of view. Competitor methods should be further detailed to help the reader with a self-contained paper. Motivating experiments could be more clear. \n    - Implemented methods are not fully in line with motivations, as considering accuracy as a proxy for assessing a set of features (assumption 4.10) can induce misleading rankings. Also, approximation made to assess the soundness metric look really not tight. I am not fully confident with proposed methods. \n    - I cannot fully capture the difference between the proposed completeness and the LeRF method", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"A Dual-Perspective Approach to Evaluating Feature Attribution Methods\" proposes a new method for evaluating feature attribution methods. It proposes to decompose the evaluation in two distinct metrics, namely soundness, which refers to precision where we look at the ratio of attributed features that is included in the true predictive feature set, and completness, refering to recall which assess the rate of true predictive features that are attributed by the method. It looks at better ranking attribution methods than existing methods from the litterature.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As said in previous section, I which authors could further detail competitor methods (such as ROAD, ROAR and deletion), with som minimal formalization of their principle to help the reader to better understand differences. Also, the global objective of the task should be formalized clearly at the begining of the paper. The procedure \"to mitigate OOD effects\" should also be detailled in the main paper, as it is core of the proposed method. \n\nAlso, the differences with classical MoRF and LeRF should be further hilighted. \n\nOne important concern I have lies in the approximation method for finding soundness, which is based on an iterative algorithm (that could be clearer) that sequentially introduces features in a set until reaching a threshold accuracy value. Such an algorithm is clearly not guaranteed to find the set A* targeted by the method. Sequentially discarding features that did not contribute enough together with already selected features (less than epsilon, which is not introduced and discussed anywhere in the text) clearly do not guarantee that they would have not contributed more with other  selected components. It assumes sub-modularity of the accuracy, which clearly does not look as a realistic assumption. Please discuss. \n\n\nMinors : \n   \n - At the bottom of page 6, I did not understand sentence \"Specifically, the model...  \", which look counter-intuitive to me\n - I did not well understand why authors state in the proof of theorem 4.11 that S\\interI=A_inc \\inter I, as S is a subset of A_inc... \n - Evaluation with semi-natural datasets consider a caricatural example to illustrate inconsistency. But I suppose that it can exist some clever methods to introduce ground truth features in the input, which would be more consistent. Also, the motivating experiment with retrain is not fully convincing. I wish we could get a working example together with such setting that induces spurious correlations", "broader_impact_concerns": ".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I think this works is interesting and (especially) useful. While I agree with some of the reviewers' comments that the method doesn't fully disambiguate the confounding effects (weakness of evaluation method vs. weakness of the underlying model), the bar that is set there is too high IMO. I view this work not as a self-contained method that gives us all the answers, but as simply another tool that a researcher or practitioner can use to understand and verify their hypotheses.

I think the work can be accepted and the authors should heed the reviewers' comments on clarifying the underlying methods more (specifically, would be good to address Reviewer RoJR's comments about Sections 3 & 4).
Decision: Accept (poster)