Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks
Authors: Stefan Bluecher
Johanna Vielhaben
Nils Strodthoff
Venue: Accepted by TMLR ()

Abstract: Feature removal is a central building block for eXplainable AI (XAI), both for occlusion-based explanations (Shapley values) as well as their evaluation (pixel flipping, PF). 
However, occlusion strategies can vary significantly from simple mean replacement up to inpainting with state-of-the-art diffusion models. 
This ambiguity limits the usefulness of occlusion-based approaches. 
For example, PF benchmarks lead to contradicting rankings. 
This is amplified by competing PF measures: Features are either removed starting with most influential first (MIF) or least influential first (LIF).

This study proposes two complementary perspectives to resolve this disagreement problem.
Firstly, we address the common criticism of occlusion-based XAI, that artificial samples lead to unreliable model evaluations. 
We propose to measure the reliability by the R(eference)-Out-of-Model-Scope (OMS) score. 
The R-OMS score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings.
Secondly, we show that the insightfulness of MIF and LIF is conversely dependent on the R-OMS score.
To leverage this, we combine the MIF and LIF measures into the symmetric relevance gain (SRG) measure.
This breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings.
This resolves the disagreement problem of PF benchmarks, which we verify for a set of 40 different occlusion strategies.

Published in Transactions on Machine Learning Research (6/2024)
Decoupling Pixel Flipping and Occlusion Strategy for Con sistent XAI Benchmarks
Stefan Blücher
Mluecher@tutherlinn
BIFOLD- Berlin Institute for the Foundations of Learning and Data Machine Learning Group, TU Berlin
Johanna Viclhaben
Eaplainable Artificial Intelligence Group
Prounhofer Heinrich-Heri Hertt--ntttteee
Nils Strodthoff
nils. mil..strudthof@@uo...c de
Division A14Health
Carl von Ossietzky Universität Oldenbury
Reviewed on OpenReview: /ttps://openreeee net/ forum? <d- bIoLXdtUW
Abstract
Feature removal central building block for eXplainable AI (XAI), both for occlusion- based explanations (Shapley values) well their evaluation (pixel flipping, PF). However, occlusion strategies vary significantly from simple mean replacement up to inpainting with state-of-tho-art diffusion models. This ambiguity limits the usefulness of occlusion- based approaches. For example, PF benchmarks lead to contradicting rankings. This is amplified by competing PF measures: Features are either removed starting with most influ- ential first (MIF) or least influential first (LIF).
This study proposes two complementary perspectives to resolve this disagreement problem. Firstly, we address the common criticism of baatussion-bbsed XAI, that artificial samples lead to unreliable model evaluations. We propose to measure the reliability by the R(eference)- Out-of-Moddl-Scope (OMS) scorc. The R-OMS score enables systematic comparison of occlusion stateggees and resolves the disagreement problem by grouping consistent PF rank- ings. Secondly, we show that the insightfulness of MIF and LIF is conversely dependent on the R-OMS score. To leverage this, we combine the MIF and LIF measures into the symmetric relevance gain (SRG) measure. This breaks the inherent connection to the un- derlying occlusion strategy and leads consistent rankings. This resol ves the disagreement problem of PF benchmarks, which we verify for set of 40 different occlusion strategies.
1 Introduction
Explainable AI (XAI) reveals the reasoning structure of black-boo machine learning (ML) models [Lundberg
& Lee 2017
Montavon al. 2018 Samek al.] 2019 Covert et al. 2021 Samek al. 2021 A Appropriate
usage of XAI
methods enables new research avenues
various scientilic domains (Holzinger al. 2019:
Blücher et al.
2020 Binder et al. 2021 Anders et al.
2022 Klauschen et al. 2024 However, multitude
of possible XAI methods leads to practical challenges Freiesleben & König 2023). For example the incon-
clusive scenario of multiple, contradictory explanations for single model prediction was recently dubbed as disagreement problem (Neely et. al. 2021 Krishna el al. (022). However, such disagreement problem not only arises in explanations itself, but extends to their evaluation. In pixel flipping (PF) (Samek et al. 2016)) which assesses the faithfulness of XAT methods by removing fcatures from the model prediction depending on their explanations, the final ranking of XAI methods depends on the specific PF setup (see Table On one side, either removing most-in fnnnntiial features (MIF) least-influential features (T.IF) first when performing

---

Published in Transactions on Machine Learning Research (6/2024)
Table 1: Which ranking do you pick? The choice of measure (most influential first. (MTF) vs. least influential first (LIF)) and occlusion strategy (train set vs. diffusion) influences the ranking of XAI methods. Saliency, layer-wise relevance propagation (LRP) and integrated gradients (IG) denote three widely used attribution methods.
Disagreement problem of PF
MIF
LIF
MIF
LIF
Setup
Train set
Diffusion
IG
LRP
LRP
IG
Ranking
LRP
Saliency
Saliency
LRP
Saliency
IG
IG
Saliency
PF benchmarks leads disagrecing rankings of XAI methods. On the other side, the underlying occlusion strategy can be implemented in various different, ways. This leads to further disagreeing PF rankings and also multiple (contradictory) occlusion-based explanations (Covert et al. 2021
This study provides two-fold contribution: Firstly, we address the main criticism of occlusion- baaedd XAI approaches, which states that occluded samples are artificial and thus their evaluation is potent ially not reliable (Gomez et al.) 2022). We quantify this concern via the score and thereby enable an objective comparison of occlusion strategies. Secondly, we thoroughly analyze the disagreement problem of PF benchmarks. Here, sorting PF setups based on the R-OMS score groups consistent rankings. Moreover, MIF and LIF ranking complementary, in the sense that rankings are consistent for either measure across the R-OMS spectrun. Based this intuition, relevance gain (SRG), which combines both measures. The SRG measure provides consistent rankings across all occlusion strategies and thereby resolves the disagreement problem of PF benchmarks.
(R-OMS)
propose the symmetric
2 Crucial role of occlusion strategies for XAI
Before investigating occlusion strategies in detail, we first discuss their usage in the context of both XAI methods and their evaluation. We introduce notation in Table 2
Table 2: Notation.
(1,2)...,n}
Feature set
S S N
Coalition
[S]
Cardinality
x (II....,Z.)
Specific sample: x; might be aggregated input features (superpixels)
X (Xs,
Generic (random) sample spit into complements Š
N
π [ T.]
Feature ordering
(T(s) = { S $} Leading features in π
x² ,n...., m1]
Reverse feature ordering
fc(x)
Classification model p(c(ar) [0
fc(ss)
Occluded model restricted S
2.1 Understanding model reasoning with XAI
Generally, XA1 research is concerned with verifying and revealing the reasoning structure of machine learning
models (Lauusckkin et al.] 2019). Various approaches have boen proposed such as attribution methods
[Baehrens et al. [2010] Bach et al.
2015 Ribeiro et al. 2016 Zint el. al. 2017 Lelzgus et al. 2022)
concept discovery (Kim et al. 2018 Ghorbani et al. [2019 Vielhaben et al.) 2023)] Achtibat et al. 2023) or
global (model-wide) analysis tools (PDP) (Hastic al. 2009) (Apley & Zhu (020) /ICE Goldstoii
et al. 2015). Here, we focus on model-agnostic approaches, which all build on occluding features and observing changes in the model prediction (Covert et al. 2021
2

---

Published in Transactions on Machine Learning Research (6/2024)
Reatures sorted by attributions
Least
Least -> Most
Prodiction /R- RROMSS
SRG
Random baseline
MIF
Wider better
LIF
Depends on
occlusion strategy
Higher better
better
0%
50%
100%
Occlusion fraction
Figure 1: Pixel flipping benchmarks of XAT moth Both MIF and LIF are affected by the random baseline. Using the complete symmetric relevance gain (SRG) introduced in Section 5.2 breaks the inherent dependence on the occlusion strategy.
Modol-agnostic XAI methods rely on occluding features Here, Shapley values are a widespread approach. They build on an abstract value function v : 2" - R, which maps a feature set to a scalar payout. The attribution of feature i given by average marginal gain:
Φi = Σ (()))((ui))odl - SCNY
(1)
The unique normalization. N(s) =1(n-8-1) ensures the Shapley axioms symmetry, linearity, efficiency and n! null player (Shapley] 1953). To deal with the binomial growth in coalitions S. we approximate Equation (1)
by uniformly sub-sampling coalitions (Strumbelj & Kononenko 2010 Lundberg & Lee 2017). Competing
XAI methods such PredDiff (Robnik-Sikonjp & Kononenko 2008 Zeiler & Fergus 2014 Zintgraf et al,
2017 Blücher et al. 2022 only remove the target feature introduce the target feature into a fully occluded
sample such as Attribbtee (Tsang et al. 2020). PredDiff and ArchArtribute correspond to the first and last marginal contribution in Equation (1) respectively. The last step is to connect the abstract value function with the occluded model prediction via v(S) = log fe(s)) (Blücher et al. 2022 Therefore, ambiguities related to the occlusion strategy (called design choices Section 3) are contained within the resulting XAT method. This leads multiple Shapley values despite the underlying uniqueness property (Sundararajan
Najmi 2020
2.2 Evaluation of XAI methods
Pixel flipping for attribution evaluation To systematically judge the quality of XAI methods, variety This ranges from pixel flipping (Samek et al. 2016
of approaches has been proposed (Nauta et al. 2023
Rieger & Hansen 2020 Samek et al. 2021] Gevaert et al.
2022 Hedström et al. 2023 Li et al. 2023 over
sanity checks (Adebayo et al.] 2018 Binder et al.] 2023) to synth datasets with ground truth kndwledge
(Yang & Kim
2019 Kayser al. 2021 Budding et al. 2021 Arras al. 2022).
general and widespread solution for quantitative evaluation of XAI methods (Samek et al. 2016). PF measures the faithfulness of altributions: Is the actual model behavior captured by the attributions? To this end, one summarizes the changes in model prediction after successively removing fcatures depending on some ordering m:
Here, focus pixel flipping,
AUC| Σ PF (I(()).
(2)
In analogy to Shapley values, the value function PF fe(#s) denotes the occluded model prediction. A given explanation & induces a unique feature ordering T arg sort, di. Then the faithfulness of the underlying XAT method is assessed by the PF measures
3

---

Published in Transactions on Machine Learning Research (6/2024)
(Samek al.
2016
Petsiuk et al.
2018
Samek et al.
2021
Gomez et al 2022 Brocki & Chung 2023
MIF [ AUC [
(higher better)
[T]
(3)
LIF AUC
(lower better)
Faithful attributions lead to steep (most relevant first, MIF) or flat (least relevant first, LIF) descent for the two opposing measures (colored curves in Figure [] Complementary literature proposed to occlude
fixed number of features (sensitivity-n) and measure the calibration (sum of attributions) compared to the
actual drop in model prediction. (Ancona et al.) 2017 Yeh et a..
2019 Bhatt et al. 2020).
PF rankings rely on occluding features Like all model-agnostic XAI methods, PF benchmarks share all design choices of the underlying occlusion strategy (Tomsett et al. 2020). Thus, PF setups are ambiguous Gevaert et al.] 2022 Rong et al. 2022; Barr et al.) 2023) and lead to disagreeing rankings (Table I Cnfortunately, PF is commonly invoked to demonstrale the superiority of newly proposed XAI method (Friieseeeen & König 2023). For practical reasons, studies naturally focus single PF setup and neglect the inherent variability. Therefore, this study investigatos influential factors of occlusion strategies that can effect the final method ranking for MIF/LIF benchmarks.
2.3 Occlusion strategies in the literature
Previous research has investigated different possibilities for occluding features. Here, baselines that mimic feature absence by constant value are a widely used option. The impact of the specific value has been investigated and visualized in various studies (Sturmfels et al.] 2020 Haug et al. 2021 Mamalakis et al.,
2022 Complementary, Izzo et al.
( Shi et al. (2022): ; Ren et al. (2023) proposed criteria to fix the 2022
baseline value. However, Jain et al.
showed that simple baselines can lead to undesirable out-of-model-
scope biases. Exposing the model artificialy occluded samples during training can circumvent this problem
(Hooker et al. 2019: Hase et al. 2021
Brocki & Chung 2023). Alternatively, improved occlusion strategics
can be employed Lo create realistie in- in-distribution samples (Kim el al. 2018 Chang el al.
2019 Agarwal & Olsen et al.] 2022 Rong et al.) 2022 Augustin et al.
Nguyen 2020
Sivill & Flach 2022
2023 Lastly, XAI
methods can be adjusted to componmmmnnmmm compensate for OMS effects (Dombrowski et al. 2019 2022 Qiu et al.) 2021
Fel et al. 2023 Taufiq et al.
2023
Dombrowski et al. 2023) and prevent adversarial vulnerabilities (Anders
et al. 2020 Slack et al. 2020
However, the central question of how to choose reliable occlusion strategies
remains still unsolved.
3 Design choices for occlusion strategies
The occlusion strategy commonly identified with the imputer distribution. This is insufficient as more design choices can impact the reliability of occluded samples such as size and shape of superpixels. In particular, occlusion strategies are inherently connected to the underlying model, which relies on both architecture and training procodure. This study addresses computer vision, but similar considerations apply to other domains.
3.1 Design choice I: Imputer
Feature removal paradigms Occluded model predictions are ubiquitous component of XAI (see Sec- tion 2 Generally, it is not possible to omit features S for the occluded prediction fe(æs), but one has to shield the model prediction from their impact. Here, the only model-agnostic option is to construct occluded samples (us,Xg) based on an imputer g, which generates artificial values X3- Then the occluded model prediction is given by fe(as)) = Exyrg fe Xz). There are two principled possibilities for the imputer: Firstly, the conditional distribution = p(Xglas) allows to exactly marginalize the complementary fea- tures S, i.e. fc(ws) = p(c(os) = dd ddXxp(c)xs Xg)p(X (Bücche et al. 2022). Secondly, the marginal distribution q = p(Xg) explicitly breaks the relation between the features S and S. Due to this independence, marginal imputers are easily accessible and enable causal interpretations (Janizek et. al. 2020). In our experiments, we consider the set of imputers listed in Table
This relation lies at the heart PredDiff
4

---

Published in Transactions on Machine Learning Research (6/2024)
The alternative model-specific option is to leverage internal structures remove features S: Some models allow to directly omit. features in meaningful manner (from the perspective of the model). Here, examples are tree- based models (Lundberg et al. 2020) or transformers (Jain et al. 2022).
Table 3: Practical imputers considered in experiments, ranging from simple to complex.
Imputer
Example
Description
Mean marginal deterministic
Occludes superpixels with constant channel-wise data set mean.
Train set marginal probabilistic
The imputed features drawn from random training set sample (optimal marginal imputer).
Histogram condilional probabilistic
(Wei et al. 2018) Occludes superpixels with constant value sampled from colors contained in image (conditional analogue of the mean imputer).
cv2 conditional delerministic
Inpaints occluded superpixels basod on the surrounding pixel values Telea 2004).
Diffusion
Inpaints using (class-unconditionaa)) state-of-the-art diffusion model. Imputations are visually more aligned since the remaining features are used as reference (Lugmayr et al. 2022
conditional
probabilistic
3.2
Design choice II: Superpixel shape and number
From arbitrary features to superpixels
We now consider the case of images æ RuxxXxReaamde with width w, height and Rehannels color channels. Here, single pixels share redundant information with neighboring pixels. It is therefore more meaningful to consider a collection of pixels (superpixels) as individual features. Super- pixels are obtained by segmenting a complete image into disjoint patches = (1,2,..., n). Computational costs of the attribution method then depend on the number of superpixels n < w nehannels- All considered segmentation algorithms are listed Table
Number of superpixels
3.3 Design choice III: Model
How occluded samples are perceived by the model under consideration, can depend on the model architecture or its training procedure. Therefore, we use three different models in this study: Firstly, we use the standard- ResNet50 (He et al.) 2016) as provided by torchvision. Secondly, we compare to the same architecture but trained with a state-of-tho-art training procedure (Wightman et al. 2021) as provided by the timm library Nesmm)),) Lastly, we investigale vision transformer (ViT) model, which was already used in (Jain)
et al. 2022 to demonstrate the effects of occlusion strategies.
4 Comparing occlusion strategies via the R-OMS score
In this section, we derive a quantitative approach to characterize artificial samples solely relying on model predictions. This enables a systematic comparison of occlusion strategics thereby judging the impact of the underlying design choices.
5

---

Published in Transactions on Machine Learning Research (6/2024)
Table 4: Superpixel can be generated by simple or more advanced segmentation algorithms
Rectangular patches: Simple fixed segmentation mask, which is independent of the image.
Classical segmentations: Segmentation aligned to the semantic image content to some degree. Here, we use the classical SLIC algorithm (Achanta et al. 2012) with default compactness X = 1.
Semantic segmentation: A meaningful, semantic segmentation for each image is the ad- vanced end of the spectrum, Here, we build on the Segment Anything Model (SAM) (Kirillov
et al. 2023
4.1 Quantitative strategy to assess occluded samples
Artificial samples are not in-distribution Occlusion-based XAI relies imputed samples, which are necessarily artificial. This is common point of criticism, the model needs to some extent extrapolate away from the original data manifold (Hooker & Mentch 2019 Kumar et al. 2020). This is closely related to the out-of-distribution detection community, which aims to detect unreliable predictions to enable monitor- ing and employment of ML for safety-critiens applications (Salehi et al. 2022). Here, many studies strived to quantify whether a given sample
with respect to the underlying data distribution. How- ever, this perspective neglects the specific model under considcration. Therefore, a recent study suggested (Guérin et al., 2023) that it more meaningful to characterize the oul-of-moodel-coope-eess (OMS-ness) of samples. We also adopt this perspective here, as the model is the crucial component underlying any XAI application.
Relying on the reference samples is essential Conventional OMS scenarios monitor the reliability of arbitrary samples. This in stark contrast to occlusion- XAI, which interested in the reliability of artificial imputations of single fixed original sample. Therefore, the original sample serves as a reference and one interested in the difference with respect the occluded sample. In analogy the image quality assessment literature (Kamble & Bhurchandi 2015), refer to scores that leverage the available side- information about the original sample, as Reference- OMS (R-OMS). In contrast, conventional OMS-measures are denoted as Oo-Reference-OM (NR-OMS) scores. Conceptually, this distinction allows for characterizing and potentially adapting any OMS measure.
Original class prediction as R-OMS score A simple NR-OMS-score is the maximum softmax probability (MSP) max. fe(#s) (Hendrycks Gimpell 2016). For a low MSP score, the model not confident about its prediction and thus the input sample is unreliable. To obtain the related R-OMS score, we focus on the original class prediction (z by leveraging the original sample and its label č. From here on, R-OMS- score refers to this measure. The R-OMS score tracks how much information about the original sample is still accessible to the model. A high R-OMS score is indicative of reliably occluded samples.
Comparison of R-OMS and NR-OMS For occluded samples, the R-OMS and NR-OMS score agrec as long as there is no flip the predicted class label. This changes for severely altered imputations. We exem- plified this in two distinct cases. Firstly, for samples that are imputed with some natural image content (train set diffusion), the generated features inevitably correspond to some other output class (t NR-OMS). Thus, the occluded sample is biased towards the class that the train set im- puter draws or the diffusion imputer inpaints. possible workaround might be average over multiple samples when calculating the occluded prediction. However, this introduces linearly increasing computation costs. Secondly, the missingness bias (Jain et al. 2022 trig-
Natural image bias
Missingness bias Jain et al., 2022
Model oulput
Dog
Macimuun
Duck
Checkerboard
R-OMS
Rnference class
Predisted
NR-OMS
class
0%6 Occlusion
fraction
100%
0%
Occlusion fraction 100%
Figure 2: R-OMS vs. NR-OMS.

---

Published in Transactions on Machine Learning Research (6/2024)
Model dependence of occlussion strategies
Summary
Mean
Train set
-
Histogram
---
e12
Diffasion
0.8
standard BesNer3u
Design
Variation of I-OMS
choice
0.6
R-OMS
Emputerr
0.188
0.4
0.4
0.074
Model
0,053
0.2
0.2
Stape
0.057
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.8
1.0
Occlusion fraction
Occlusion fraction
B
Varying number of superpixei. It
C
Shape of segmentation mask
Imputer
25
50
Squares
Default
Plexible
Histogram
100
200
Diffassion
0.5
0.5
0.5
R-OMS
*
0.3
0.3
3
0.3
0.1
0.1
0.1
Mean
Train set
Hist.
Difi.
Mean Train
Hist.
Diff
Squares
Default
SAM
Figure 3: (A): Occlusion strategy and model interact. (B, C, D): Visualize variation of each design choice for fixed Seandard-ResNet500 (Summary) reports the average variation (interquartile ranges) associated with each design choice. (B) Granularity of segmentation (C) Shape of segmentation. (D) Imputer choice.
gers specific classes (e.g. checkerboard) based on the occlusion strategy (t NR-OMS). The R-OMS is not affected by both effects as these are unrelated to the reference class (L R-OMS). Therefore the R-OMS-score typically decreases monotonically whereas the NR-OMS-score potentially rises again when occluding more patches.
4.2 Characterize the impact of design choices
R-OMS coincides with the PF target In the following, we characterize occlusion strategies by measuring the R-OMS score at various occlusion fractions (n s)/n n for a random set occluded superpixels S. A sleeper descent reflects less reliable occlusion strategies. Importantly, this approach is independent of the considered XAI method. The AUC of this random PF curve is identical to the random baseline in PF benchmarks. To stress this inherent connection, this value is denoted as R-OMS = Euni(=) [AUn(=])UU[T]]]
Occlusion strategies depend on the model choice In Figure 3 (A), compare the standard ResNet55 vs. timm-ResNet50. Even though both models rely on the same architecture, the R-OMS scores for identical imputers (mean and histogram) vary significantly (Crothers et al. 20233 observed this in the NLP context). Occluded samples with imputed superpixels are not similar to natural images (see example in Table 8 Therefore, one naively expects low R-OMS score as it occurs for the standard ResNet50. However, the timm-ResNet50 confidently predicts the correct class even for heavily occluded samples. This change in model response originates from the different training schemes. Timm-training invokes claborate augmen- tation procedure that boosts model performance (Wightman et al. 2021). Thereby, the timm-ResNet50 learned to ignore non-informative constant patches and solely focus on the remaining original image content. Importantly, the R-OMS detects unreliable samples, as judged by the trained model, without knowledge about the original training strategy. Alternatively, one can enforce reliability by manually pre-training mod- cls with occluded samples, however at the price of a large computational overhead (Hooker al. 2019 Hasc et al. 2021). The above observation exemplifies, that human judgment of occhuded samples inherently flawed, as it lacks any connection to the underlying model.

---

Published in Transactions on Machine Learning Research (6/2024)
Diffusion ensures reliable model predictions The subfigures (A) and (D) in Figure 8 show that the diffuusin imputer consistently leads to the highest. R-OMS scores. This is expected, as generative diffusion models are trained to inpaint realistic patches for the masked superpixels. Therefore, occluded samples are similar to images seen during training for both models. This is in contrast to the marginal train set imputer, which leads to unrealistic samples (low R-OMS). Interestingly, the imputed superpixels themselves are drawn from the original data manifold and are therefore natural to the model. However, the model confused by the contradictory (imputed vs. original) information. As a consequence, the model cannot leverage the remaining information from the original image. Thus, the train set imputer, which is the optimal marginal approach, leads to steep descent in model confidence.
Diffusion imputer resemble internal strategy To further characterize the diffusion imputer, we compare it to the internal occlusion strategies of a ViT model. Here, we summarize our findings and defer the exact results to the supplementary Figure The internal imputer a neutral approach (not relying on artificial samples), as the masked superpixels are directly omitted (Jain et al.) 2022). Based on the R-OMS score, we find close alignment between the internal and diffusion strategy. This is very unexpected since both occlusion mechanisms are conceptually very different. To further strengthen this point, we confirmed that this similarity extends to the intermediate hidden activations. Overall, this alignment is an interesting argument in favor of the diffusion imputer as a natural replacement strategy.
Number of superpixels significantly impacts all imputers We compare the occlusion strategies de- pending on the number of superpixels in Figure 3 (C). We observe that simple imputers (mean, histogram and train set) are more reliable for larger superpixels (small n). Contrary, for smaller superpixels, a miss- ingness bias (see also Figure 2 reduces the R-OMS score. An opposing trend is visible for the conditional imputers (ev2 and diffusion), for which artificial samples are increasingly realistic to the model with smaller superpixels. This is an expected behavior, since for fixed occlusion fraction the imputation task for a smaller number of superpixels is comparably simpler than the same task for a larger number of superpixels, where larger segments have to be inpainted in a semantically meaningful fashion.
Naive imputations are more meaningful for semantic superpixels Next, we investigate how the segmentation algorithm (shape of superpixels) impacts the occlusion strategies (C). As outlined in Table 4 wc compare squares, defA (A = =),, flexiblo-SICC (A 0.1) and semantic SAM superpixels (Ych) et al. 2019 Yu el al. 2023 To ensure a fair comparison we filter for images with similar number of superpixels n 2 75. We observe that semantic SAM superpixels increase the R-OMS score for all three simple imputers. This aligns with (Rong et al. 2022), who discussed a similar phenomenon as information leakage through the segmentation mask, an effect which could also be framed as a positive missingness bias (Jain et al. 2022). In contrast, the conditional cv2 and diffusion imputer struggle to meaningfully embed semantic patches into the local neighborhood. Thus, the R-OMS score decreases for increasingly flexible superpixels. Overall, semantic superpixels reduce the inffuence of the imputer choice apparent from (D). Here, SAM superpixels clearly show the least variation.
Relative importance of design choices So far, we discussed cach design choice individually. In particular, we aimed to vary each dimension between its two extremes (small vs. large superpixels, square vs. semantic superpixel shapes, simple vs. complex imputers). Each design choice therefore induces an inherent degree of variation into the resulting occlusion stratogy. To quantitatively assess this variation, we calculate the maximal R-OMS spread for each design choice (columns in B, C and D) and report the interquartile range (ICR) over all experiments in the summary table in Figure 3 The supplementary Table A1 shows details about the underlying setups for the model. Based on this analysis, dominant design choice of the occlusion strategy. The secondary effects are associated with the number of superpixels and the underlying model. Lastly, the shape of superpixels has the least impact on the occlusion strategy. As a consequence, it is generally not worth invoking expensive SAM superpixels to obtain more reliable occlusion strategies. In fact, this can even be detrimental when employed in conjunction with the diffusion imputer.
conclude that the imputer choice is the
8

---

Published in Transactions on Machine Learning Research (6/2024)
MIF
LRP Saliency
NT-IG
Random
LIF
Benchmurk results
NT-Sallency
InpurXCradients
Benchmark resulls
Rankking
Reference
Disagreement problem
Disagreement problem
Hanking Py
Agreermnnt
X
Low
High
Low
Hligh
Sorted by I-OMS
Sorted by I-OMS
Figure 4: PF benchmarks based on varying occlusion strategies lead many disagrocing rankings for both MIF and LIF. Sorting rankings based on the R-OMS groups consistent rankings. The lower panel visualizes the disagreement problem as the deviation from the most frequent ranking (reference). The consistency of MIF (high) and LIF (low to medium) are complementary when sorting based the R-OMS.
5 Impact of occlusion strategies on PF benchmark
Setup This section explores the impact of different occlusion strategies on PF benchmarks. All results are based on 100 randomly selected imagenet samples. Based on Section 3 we construct a diverse set of occlusion strategies, varying all design choices (n: 25, 100, 500, 5000; imputer: mean, train set, histogram, cv2, diffusion; model: standard-ResNet500, Simm-ResNet50).. Using all 40 PF sctups we rank several standard XAI methods: Saliency, (Na))Saliency with Noise-tunnel, Integrated gradients (IG), In- putXGradients, layer-wise relevance propagation (LRP) and the random baseline (R-OMS). Pixel-wise attributions are averaged within superpixel to obtain superpixel-based attributions. Gradient-based attributions are calculated using captum (Kokhlikyan et al. 2020), LRP using zennit (Anders et al. 2021) and model-agnostic attributions based on custom implementation. Our code is available on at https://github. http:///lleeeeerr/////eeeaa-aa-p
5.1 Disagreement problem of MIF and LIF
Occlusion strategies lead to many rankings Based all 40 PF setups we perform both MIF and LIF benchnarks. Varying the occlusion strategy leads to 17 (MIF) and 23 (LIF) different rankings (lop panel in Figure 4). At this stage, it is not obvious which ranking is the most trustworthy. In principle, an adversary can advocate any method, by selecting the PF sctup for which the mothod performs best. This very troublesome and prevents a fair comparison of XAI methods terms of faithfulness. This exemplifies the prevalence of the disagreement problem for PF benchmarks (lower panel).
Identifying consistent MIF and LIF rankings Next, analyze the collection of all rankings in detail. To this end, sort the PF setups based on the underlying R-OMS score and visualize all rankings accordingly in Figure Interestingly, the rank of the random baseline seems to be also sorted by this. For low R-OMS the random baseline consistently outperforms established methods for both LIF as well as MIF, This verifies that occlusion strategics with low R-OMS score are indeed not reliable. For the top-ranked methods LIF and MIF behave conversely. This phenomenon is rooted the opposing orientation of the insight.flless of both measures, which we discuss in next section Scction 5.2 MIF rankings are most consistent. for a high R-OMS whereas the top-ranking LIF methods agree for low score. The lower panel in Figure visualizes this consistency based on the deviation to the most frequent ranking, which is identical for MIF and LIF. Clearly, MIF rankings are fully consistent for large R-OMS. For the LIF measure the situation is not conclusive. Here, a medium score seems to lead to the most consistent. rankings. Overall, the R-OMS can be viewed as an observable, which characterizes the outcome of PF benchmarks.
Quantitative characterization of PF benchmarks Our qualitative analysis showed, that the R-OMS score groups consistent rankings for both MIF and LIF benchmarks. To quantify this notion, we define (ground truth) consistency for all rankings the normalized discounted cumulative gain (nDCG)

---

Published in Transactions on Machine Learning Research (6/2024)
Table 5: Sorting rankings based on different variables. A high score means that a variable groups similar
rankings (high consistency) and therefore characterizes the PF setup. Ground truth sorting is defined based the nDCG. Variance a indicates the consistency of randomly sorting rankings (zero average J = 0).
Design choices
MIF
LIF
Observables
MIF
LIF
# superpixels n
0.57
0.53
R-OMS
0.79
0.45
Imputer
0.57
0.41
NR-OMS
0.59
0.11
Model 0.23
0.22
Random [=o]
0.16
0.16
(Järvelin & Kekäläinen 2002) with respect to the most frequent ranking. The nDCG penalizes misses in the leading position (winning XAT methods) more severely as changes at later position (around the random ranking). In Table 5 we report the correlation between the nDCG score and variables associated with the PF setup. Variables with a high correlation are predictive of the resulting ranking of the PF benchmark. The number of superpixels n is the most indicative design choice. However, design choices do not provide an objective criterion to distinguish PF setups. For example, consider the imputer choice, which does not have an inherent ordering. To circumvent this, we probed for all possible imputer orderings and reported the maximum correlation. In contrast, the (N)R-OMS scores are observables that naturally order PF setups. From Table 5 it clear that using the reference sample (R-OMS) to characterize the occlusion strategy is beneficial and leads to a more consistent sorting. Using naive non-reference mcasure (NR-OMS) is less insightful and even leads to nearly random performance for the LIF benchmark.
5.2 Consistent PF benchmarks with the SRG measure
Insightful MIF/LIF setups depend on the R-OMS score We just saw that the R-OMS characterizes the PF benchmark. This is expected since a high baseline corresponds to reliable occlusion strategies. However, there is deeper connection between the occlusion strategy and PF benchmark. This originates from the fact, that a random explanation leads to non-zero values R-OMS for both MIF and LIF. Thus, it is conceptually more meaningful to focus the relevance gain (RG), as the improvement over the random baseline:
[RG[ø] = R-OMS- MIF
LRG[0] [o - R-OMS. (higher better)
(4)
Importantly, MRG and LRG are now directly comparable without changing the final ranking of XAI methods. The theoretical optimal score (area below/abor /abvve the random baseline) now explicitly depends on the random baseline (max(MRG) = R-OMS and (a.(LGG) = 1 R-OMS). In other words, the insightfulnes of MIF benchmarks directly depends on the R-OMS score of the occlusion strategy (left panel in Figure 5
Experimental verification We can also observe this phenomenon empirically by measuring the spread between the performance of different XAI methods. A larger spread (separation of PF curves) indicates a more insightful benchmark. To quantify this spread, we calculate the absolute pairwise differences between the individual PF curves of all XAI methods and average over all pair-wise differences]] Then we relate this separation to the R-OMS score of the PF setup. We obtain a positive Pearson correlation for the MIF measure (0.88) and a negative correlation for LIF (-0.92). This shows that the insightfulness of the MIF and LIF measures are conversely dependent on the R-OMS score. For LIF, the reliability and insightfulness of the PF setup are not aligned (left panel Figure 5) Consequently, a medium score most beneficial and leads to consistent rankings, as visible in Figure
Combining most and least relevance gains Previous studies observed that MIF and LIF can lead to disagreeing rankings (Tomsett et al. 2020 Rong et al.) 2022 Our analysis revealed that this disagreement is complementary: depending on the random baseline either MIF or LIF are insightful. This motivates to eveuly combine both measures 2 The relevance gain (Equalion (4) allows to aggregate the most and least
The difference cancels the offset in Equation
4 Thus, MIF/LIF and MRG/LE LRRG are interchangeably.
Samek al. (0166) discuss a similar measure in the appendix. However, it has not been picked up in following literature.
10

---

Published in Transactions on Machine Learning Research (6/2024)
Benchmark results
SRG
Ranliing i BRaninng 5
High
LRG (LIF)
SRG
MRG (MIF)
maxpp meusure)
Insightfulness
Agreement across PT selups
Reliability
Zero
Low
High
Occlusion strategies [R-OMS]
Sorted by R-OMS
Figure 5: Consistency of SRG measure independent from the occlusion strategies. Left pancl: theorctically achievable improvement over random baseline. Right panel: SRG rankings of XAI methods (legend in
Figure 4).
relevant sides of the attribution spectrum into the symmetric relevance gain (SRG)
SRG[Ø] = [RG[ + [RG = [1F - MTF(ø],
(5)
which corresponds to the area between the PF curves in Figure random explanation, as the relevance gains MRG and LRG default to zero.
|
Trivially, the SRG measure is zero for a
LRG and MRG measure same notion of faithfulness Attributions highlight the relevant model rea- soning into single relevance score per feature. This can be regarded as locally approximating the model by an additive function Lundberg & Lee| 2017). Interestingly, Hama et al. (2023) showed that LRG and MRG converge to the same optimum for such an additive function. Thus, both measures capture the same notion of faithfulness (between attributions and underlying model) and should thus be treated symmet- rically. The simplest way of implementing this is the SRG measure in Equation (5) Going beyond this additive (first-order) model structure, requires estimate interaction effects (Blücher et al. 2022: Bordt & von Luxburg 2023). Since interactions attribute multiple (joint) relevance scores a single feature, novel targeted measures required to capture their faithfulness.
Theoretical consequence for SRG measure The theoretical optimal score for both LRG and MRG depends on the R-OMS of the occlusion strategy (as discussed below Equation (4). This induced the disagrecment problems into the final PF rankings (Figure 4 By combining both measures into the SRG measure we can alleviate this undesired dependence. The theoretical optimal SRG score not bounded by the random baseline and always one (left panel in Figure 5). This leads to consistent and quantitative stable PF benchmarks (as verified below). In other words, the SRG measures decouples PF and occlusion strategy.
Consistency of the novel SRG measure We show SRG rankings for all 40 PF setups in Figure (upper right panel). This leads to only 7 different rankings, which is approximately 3 times less compared to the one-sided MRG and LRG measures. Moreover, the remaining disagreement (lower right panel) is limited to neighboring and low-ranked XAI methods. contrast, both MIF and LIF disagree within the top-ranked XAI methods (Table Thus, combining both one-sided measures into the SRG measure leads to consistent rankings across the full spectrum of occlusion strategics.
Quantitative stability of SRG measure In addition to this consistency, the SRG measure is also quan- titatively stable. This means, that the performance of each XAI method is not affected by the value of the random bascline R-OMS. To analyze this, we consider variance for all three mcasures across PF experiments. Based on the variance of the LRP method across all 40 PF setups, we find that the SRG measure is up to ten times more s To visualize this, we show a boxplot of all three measures the supplemen- tary Figure A3, which validates the above conclusion. In summary, the quantitative stability of the SRG measure allows for aggregating multiple PF setups without losing the underlying ranking of the individual benchmarks. This is not. the for both MRG and LRG.
Summary Disagreeing rankings limit the usefulness of PF a benchmark for XAI methods. The random baseline R-OMS allows to identify insightful PF setups and groups consistent rankings. This re-establishes
Variance of the LRP method: Var(MRG) 0.0110, Var[LRG] 0.0128, Var[SRG] 0.0005
11

---

Published in Transactions on Machine Learning Research (6/2024)
Table 6: The SRG measure enables trustworthy PF benchmark of XAI methods, which resolves the dis- agreement problem from Table [ Higher is better and random explanations yield score of zero. Results are consistent for full range of design choices (PF setup: cv2, Hlandard-ResNetan and n 25) and with a trustworthy MRG benchmark (see supplementary Table A3).
Occlusion-based methods
Pixel-wise attributions
Method
SRG (1)
Method
SRG (1)
Shapley values (ev2)
0.47
LRP
0.35
Shapley values (Mean)
0.40
Saliency (NT)
0.25 (0.30)
PredDiff (cv2)
0.33
IG (abs NT) 0.23 (0.24 / 0.12)
ArchAttribute (cv2)
0.25
InputXGradients (abs) 0.05 (0.19)
trust in the conventional MIF /LIF measures. Using the full PF information (most + least) leads to the SRG measure. This obviates the disagreement problem, as the SRG measure is decoupled from the random baseline.
6 Trustworthy PF benchmarks for XAI methods
Having discussed all the prerequisites for reliable and insightful pixel flipping experiments, we are now ready to perform a final benchmark comparing all major XAI methods. To this end, we show the results for the SRG measure in Table G. These results align with the MRG measure for reliable and insightful occlusion strategy, i.e. high R-OMS score, which are presented in the supplementary Table A3 This singles out the diffusion imputer, however at the cost of drastically increased computational costs. In contrast, the SRG measure does not depend the occlusion strategy and can build on a cheaper strategy (e.g. ev2). We restrict to n = 25 superpixels to ensure fully converged Shapley value attributions.
Occlusion-based vs pixel-wise attributions The final ranking in Table t clearly shows that occlusion- based attributions are significantly more faithful to the model than pixel-wise attributions. This advantage comes at the price of higher computational cost. To emphasize this, note, that LRP only requires a single backward pass. In contrast, the cheapest occlusion-based attribution method, PredDiff, scales linearly with the number of superpixels with two-orders of magnitude pro-factor (Blücher et al. 2022). All other occlusion-based approaches are siguificantly more expensive. LRP is considerably more elficient but still achieves faithfulness scores in the same range as PredDiff.
Matching occlusion strategies for Shapley values and PF For model-agnostic XAI methods, is possible to match the occlusion strategy to the PF setup. Thus, a natural question is how much this alignment improves the PF performance. To estimate this effect, we calculate Shapley values using all mumerically fcasible imputers and report the worst performing imputer in Table We report the results for all possible imputer combinations of the SRG and MIF measure in supplementary Table A2 This analysis confirms that using matching imputers for both attribution and PF benchmark leads to the best performance. However, Shapley values with a mismatching imputer are still superior to the best-performing alternative XAI methods (PredDiff and LRP). This is very reassuring since it verifies that PF benchmarks are not dominated by the choice of occlusion strategy.
Pixel-wise attributions We now focus on the differences between the pixel-wise attribution methods. Here, the most surprising observation is that saliency-basse based explanations are more faithful than the axiomatically grounded 1G counterpart. Interestingly, employing a noise tunnel only improves the faithfulness of saliency whereas it is detrimental for IG. Note, that we restrict to the default zero bascline for IG and do not test alternatives (Sturmfels et al. 2020). As saliency builds on the absolute-valued gradients, we also report the basolute-valued IG performance, which is slightly improved (see also supplementary Figure Overall, we suspoct that the superpixel average of gradients is sufficient to provide a clear signal for the feature relevance (Kapishnikov et al., 2019 Muzellec et al., 2023). We stress, that this also holds for many small superpixels 5000), which incorporated in the previous set of occlusion strategies (Figure 5
12

---

Published in Transactions on Machine Learning Research (6/2024)
7 Conclusion
This study analyzed the inherent connection between PF benchmarks and the employed occlusion strategies. This connection leads to contradicting rankings and thereby limits the usefulness of PF as a tool to identify laithful XAT methods. We resolve this disagreement problem by disentangling two central ingredients: reliable occlusion strategies and insightful PF setups.
We propose to characterize occlusion strategies based on the R-OMS score. The R-OMS score measures how much information about the original sample is still contained in the occluded samples as perceived by the model. Thereby, we capture dominant differences between strategies across all relevant design choices. This allows to identify reliable occlusion strategies without prior knowledge about the model architecture or training procedure. Additionally, the R-OMS score indicates insightful PF setups when removing either the most or least inftuential features first. This groups consistent rankings for both conventional MIF and LIF measures. Importantly, insightfulness and reliability of the occlusion strategy are aligned for the MIF measure, which leads to an overall trustworthy PF setup. Moreover, symmetrically combining both possible feature orderings into the SRG measure, entirely breaks the troublesome connection between the occlusion strategy and the PF benchmark. This circumvents expensive PF setups, required for trustworthy MIF benchmarks, which require building the faithfulness of XAI methods across all design choices. We expect that this insight will improve the comparability of future studics and will thereby foster sustainable progress toward a generally acknowledgod understanding of XAI.
diffusion imputer. The SRG measure consistently evaluates
Limitations and future work This study focused on image classification as the prototypical use case of XAI methods. To explore possible limitations systematically, first consider other input domains and then move forward to different output domains (tasks). Generally, PF benchmarks require custom imputers for each new input domain. This means that the particular design choices of the occlusion strategy depend on the data domain. For example, it can be challenging to identify appropriate analogues of superpixels for tabular or time series data. Thus, a systematic exploration plausible occlusion strategies based on the R-OMS score seems to represent a promising direction for future research. Complementary, within the output domain, our proposed methodology tiod to the classification scenario, as we usod the remaining class probability to characterize occlusion strategies. This might not. be possible for other largets such multi-class predictions physical regression targets (Hama et al.) 2023). Here, a single output score, with the simple interpretation of lower is less faithful, might not be applicable. However, symmetrizing the PF measure (as done for the SRG measure) should still be beneficial. Going beyond a single score for characterizing imputers could also be interesting even within a classification scenario. Two imputers with similar model outputs can still trigger different intermediate representation, which means that the samples are perceived differently by the model. To capture such effects one can mcasure the similarity of imputed and reference sample based on intermediate features. Here, other measures such as the Mahalonobis distance Mahhllaobis 1936 concepts activations (Kim et al. 2018 Vielhaben et al. 2023 are imaginable. To keep the presentation self-contained, we focused on the simple R-OMS score based on output probabilities, but first exploratory results in this direction can be found in Appendix A.3 In particular, invoking a R-OMS score based on intermediate features also alleviates the restriction to the classification setting.
Acknowledgments
SB and JV gratefully acknowledge funding from the German Federal Ministry of Education and Research under the grant BIFOLD22B, BIFOLD23B and BIFOLD (01IS18025A, 011S1803711). The authors thank K.-R. Müller, S. Letzgus, L. Linhardt and S. Salehi for helpful comments on the manuscript.
13

---

Published in Transactions on Machine Learning Research (6/2024)
A Additional experiments
A.1 Details on characterizing occlusion strategies
The model variance is estimated based on the setups provided in Table A1 Each setup corresponds to one column a single figure of the lower panel (B, C, D) in Figure
Table A1: Sctups for variance of the model design choice in Figure 3
2
5
Imputer (PF)
Train set
cv2
cv2
Diffusion
Mean
25
75
75
200
5000
Shape
Standard
Squares
Semantic
Standard
Standard
A.2 Matching imputer for Shapley values and PF assessment
Ocelusion-based explanations can match the occlusion strategy of the PF setup. This provides an inherent advantage over XAI methods. To estimate this effect, compare matching versus non-matching imputer distributions in Table A2 We report both SRG and MTF measure for the standard-ResNett00. Clearly, matching imputer distributions lead to the best results.
Table A2: Ranking Shapley values based on different imputers (n 25, Seandard-ResNettiiy with all possible PF setups. Matching the imputer for PF assessment and attributions is the most superior.
Imputer (PF)
Mean
Train set
Histogram
cv2
SRG
Ranking
Mean
Train set
Train set (-1.5)
Cv2
attributions
Train set (1.0)
Histogram (60.9)
Histogram
Train set (47.6)
(A SRG)
Histogram (10.6)
Mean (65.2)
Mean (6.6)
Histogram (62.1)
[10-9]
cv2 (13.5)
cv2 (72.4)
cv2 (35.8)
Mean (70.1)
MRG
Ranking
Mean
Train set
Histogram
Train set (-0.7)
attribations
Train set (1.4)
Mean (10.2)
Train set (5.2)
cv2
(A MIF)
Histogram (1.4)
Histogram (11.2)
Mean (6.0)
Histogram (10.5)
cv2 (11.4)
cv2 (14.7)
cv2 (14.5)
Mean (11.1)
A.3 Diffusion imputer matches internal strategy
We analyze occlusion strategics for a ViT model in Figure A1 The left panel builds on the R-OMS score. The general behavior comparable to the timm-ResNet50 in Figure (A). For the ViT model, we can additionally compare to an internal stratogy (omitting tokens). Interestingly, the R-OMS score of internal and diffusion strategy closely aligned. This is unexpected since both occlusion mechanism are conceptually different.
This raises the question whether both strategies perceived similar by the VIT model. To analyze this, we go beyond the R-OMS score, which quantifies the impact of the imputer through single number. However, this can lead to identical R-OMS-score even though the imputed samples differ qualitatively. For a more detailed evaluation, we propose to compare feature representations of the original image and occluded samples (Crohhers al. 2023). The right panel Figure A1 shows that diffusion and internal strategy are also aligned for the intermediate hidden features. This alignment is an interesting argument in favor of the diffusion imputer natural replacement strategy.
14

---

Published in Transactions on Machine Learning Research (6/2024)
Output domain
Feature space
1.0
0.9
Cosine similarity
0.8
R-OMS
0.4
0.7
0.6
Mean
cv2
0.5
Train set
Diffusion
Histogram
Internal
0.0
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Occlusion fraction
Occlusion fraction
Figure A1: ViT model: Comparing internal strategy (omitting tokens) with external imputations. Left panel: comparison based on R-OMS score. Right panel: similarity feature space of remaining tokens. The diffusion imputer closely resembles the internal strategy.
cv2
Mean
Saliency
0.6
0.6
InpuiXGradients
abs-InputXGraddents
R-OMS
0.4
R-OMS
0.4
abs-IG
0.2
0.2
T
0.0
0.0
0.2
0,4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Occlusion fraction
Occlusion fraction
Figure A2: Using the signed or absolute valued gradient attributions has consistent effect on the perfor- mance. (n = 500 and standard-ResNett5)
A.4 Effects of absolute value for gradient-basse attributions
Figure A2 provides a more details on the effect of using the absolute valued attributions. Clearly, the performance signed and non-signed attributions are more aligned. In particular, this also holds when IG is more faithful Saliency (right panel).
A.5 Quantitative stability of SRG
The SRG measure combines the insightfulness of MIF and LIF. Therefore, it is independent of the random baseline R-OMS and leads to quantitative stable scores for fixed method across several PF setups. This is shown in Figure A3 by averaging across design choices. Here, we condition the boxplot on the imputer the dominant design choice. For the LRG and MRG measure, we observe a large spread for the top-ranked methods across the different imputers. This caused by the strong dependence on the random baseline. As consequence the original (most frequent) ranking is only visible for insightful imputer, i.c., reliable imputers (cv2, diffusion) for the MRG measure and simple imputers (mean, train set, histogram) for the LRG measure. This is to be contrasted to the SRG measure in the middle pane. The ranking is recognizable
15

---

Published in Transactions on Machine Learning Research (6/2024)
Most relevant
Two-sided PF
Least relevant
LRP
NT- ST--aliency
Saliency
IG
NT-IG
Diffusion
cv2
Histogram
InputX
Gradients
Train
Mcan
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.4
0.0
0.2
0.4
MRG
SRG
LRG
Figure A3: Average (over design choices) performance of XAI methods for the three different PF measures. XAI are sorted according to the most frequent ranking in the y-label. When methods (boxes) are separated horizontally we can infer ranking. The SRG measure is stable and the underlying ranking is still visible. In contrast, the LRG and MRG measure are numerically unstable.
Table A3: The MRG measure with a high R-OMS ensures trustworthy PF benchmark of XAT methods, which resolves the disagroement problem from Table T Higher is better and random explanations yicld score of zero. (PF setup: diffusion, standard-ResNet5500 and n 25).
Occhusion-based methods
Pixel-wise attributions
Method
MRG (t)
Method
MRG (t)
Shapley values (Train set)
0.22
LRP
0.21
Shapley values (Mean)
0.20
Saliency (NT)
0.15 (0.18)
ArchAttribute (TTain set)
0.19
IG (abs NT) 0.10 (0.14 / 0.06)
PredDiff (Train set)
0.18
InputXGradients
0.02
for all five imputers. This means that the SRG measure is numerically stable with regard varying design choices. Phrased differently, the SRG measure is independent from the random baseline R-OMS.
A.6 Trustworthy MRG benchmark
The SRG measure allows to perform trustworthy benchmarks independently from the occlusion strategy. However, the MRG measure can still be used if an occlusion strategy with high R-OMS score is invoked. This ensures insightful measure and reliable occlusion strategy. Therefore, present an additional overview benchmark based on the MRG measure in Table A3 Note, that this result relies on the expensive diffusion imputer.
16

---

Published in Transactions on Machine Learning Research (6/2024)
References
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pottern Analysis and Machine Intelligence, 34(11):2274-222) 2282, 2012.
Reduan Achtibat, Maximilian Dreyer, lloaa Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojciech Samck, and Schastian Lapuschkin. From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006 1019, 2023.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505 9515, 2018.
Chirag Agarwal and Anh Nguyen. Explaining image classifiers by removing input features using generative models. In Asian Conference Computer Vision, 2020.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. preprint arXiv:1711.06101, 2017.
Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference on Machine Learning, pp. 314-323, 2020.
Christopher Anders, David Neumann, Wojciech Samek, Klaus-Robert Müller, and Sebastian Lapuschkin. Software for dataset-wide xai: from local explanations to global insights with zennit, corelay, and virelay. preprint Xiv:22106. 13200, 2021.
Christopher J Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus-Robert Müller, and Sebas- tian Lapuschkin. Finding and removing clever hans: using explanation methods to debug and improve deep models. Information Fusion, 77:261-290 2022.
Daniel W Apley and Jingyu Zhu. Visualizing the effects predictor variables in black box supervised learning models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(4):1059-100 1086, 2020.
Leila Arras, Ahmed Osman, and Wojeiech Samek. Clevr-xai: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 40, 2022.
Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Analyzing and explaining image classifiers via diffusion guidance. arXie preprint arXiv: 2023.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):e0,30140, 2015.
David Bachrens, Timon Schrocter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How explain individual classification decisions. Journal of Machine Learning Research, 11: 1803-1831 2010.
Brian Barr, Noah Fatsi, Leif Hancox-Li, Peter Richter, Daniel Proano, and Caleb Mok. The disagreement problem in faithfulness metries. preprint arXiv:2311.07763, 2023.
Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model expla- nations. preprinl ar xxxx:20055.000 2020.
Alexander Binder, Michael Bockmayr, Miriam Hägele, Stephan Wienert, Daniel Heim, Katharina Hellweg, Masaru Ishii, Albrecht Stenzinger, Andreas Hocke, Carsten Denkert, al. Morphological and molecular breast cancer profiling through explainable machine learning. Nature Machine Intelligence, 3(4):355-36) 366, 2021.
17

---

Published in Transactions on Machine Learning Research (6/2024)
Alexander Binder, Leander Weber, Sebastian Lapuschkin, Grégoire Montavon, Wojeiech Samek. Shortcomings of top-down randonization-basel sanity checks for evaluations of deep neural network explanations. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 16143-16152, 2023.
Müller, and
Stefan Blücher, Lukas Kades, Jan M. Pawlowski, Nils Strodthoff, and Julian M. Urban. Towards novel insights in lattice field theory with explainable machine learning. Physical Review D, 101:094507, 2020.
Stefan Blücher, Johanna Vielhaben, and Nils Strodthoff Proddiff: Explanations and interactions from conditional expectations. Artificial Intelligence, 312:103774, 2022.
Sebastian Bordt and Ulike von Luxburg. From shapley values to generalized additive models and back. In International Conference on Artificial Intelligence and Statistics, pp. 709-745, 2023.
Lennart Brocki and Neo Christopher Chung. Feature perturbation augmentation for reliable evaluation of importance estimators in neural networks. Pattern Recognition Letters, 176:131-139, 2023.
Céline Budding, Fabian Eitel, Kerstin Ritter, and Stefan Haufe. Evaluating saliency methods on artificial data with different background types. preprint orXiv:2112.44888 2021.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations, 2019.
Ian C Covert, Scott Lundberg, and Su-In Lec. Explaining by removing: A unified framework for model
explanation. Journal of Machine Learning Research, 22(1):9477-9566, 2021.
Evan Crothers, Herna Viktor, and Nathalie Japkowiez. Faithful to whom? questioning interpretability measures in nlp. preprint orXiv:2308.0677 2023.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Roburt Müller, and Pan Kessel. Explanations can be manipulated and geometry to blame. In Advances in Newral Information Processing Systems, pp. 13589 13600, 2019.
Am--aathri Dombrowski, Christopher J Anders, Klaus-Robert Müller, and Pan Kessel. Towards robust explanations for deep neural networks. Pattern Recognition, 121:108194, 2022.
Ann-Kathrin Dombrowski, Jan E. Gerken, Klaus-Robert Müller, and Pan Kessel. Diffeomorphic counter- factuals with generative models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. doi: N0.1109/TPAMI
Thomas Fel, Mélanie Ducoffe, David Vigouroux, Rémi Cadène, Mikael Capelle, Claire Nicodème, and Thomas Serre. Don't lie to me! robust and efficient explainability with verified perturbation analysis. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 16153-16163, 2023.
Timo Freiesleben and Gunnar König. Dear XAI Community, We Need Talk! Fundamental Misconcep- tions in Current XAI Research. In Explainable Artificial Intelligence, pp. 48 65, 2023.
Arne Gevacrt, Axcl-Jan Rousscau, Thijs Becker, Dirk Valkenborg, Tijl De Bic, and Yvan Sacys. Evaluating feature attribution methods in the image domain. preprint arXi::2002.1129 2022.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based expla- nations. In Advances in Neural Information Processing Systems, pp. 9277 -286, 2019.
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. Peeking inside the black box: Visualiz- ing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics, 24(1):44 65, 2015.
Tristan Gomez, Thomas Fréour, and Harold Mouchère. Metrics for saliency map evaluation of deep learning explanation methods. In International Conference on Pattern Recognition and Artificial Intelligence, pp. 84 24-95, 2022.
18

---

Published in Transactions on Machine Learning Research (6/2024)
Joris Guérin, Kevin Delmas, Raul Ferreira, and Jérémie Guiochet. Out-of-distribbttion detection is not all you need. In AAAT Conference on Artificial Intelligence, pp.
2023.
Naofumi Hama, Masayoshi Mase, and Art B Owen. Deletion and insertion tests in regression models. Journal of Machine Learning Research, 24(29)), 2023.
Peter Hase, Harry Xie, and Mohit Bansal. The out-of-distribuution problem in explainability and search methods for feature importance explanations. Advances in Neural Information Processing Systems, pp. 3650-3666, 2021.
Trevor Hastic, Robert Tibshirani, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Johannes Hang. Stefan Zurn, Peter El-Jiz, and Gjergji Kasneci. On baselines for local feature attributions. preprinl ar xrx::21011000000 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770 (700778, 2016.
Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojeiech Samek, Scbastian Lapuschkin, and Marina Marina M.-. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34),
2023.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distributtioo examples in neural networks. In International Conference on Learning Representations, 2016.
Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal, and Heimo Müller. Causability and explain- ability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, (4)xe133,, 2019.
Giles Hooker and Lueas Mentch. Unrestricted permutation forces extrapolation: Variable importance re- quires at least one more model, or there is no free variable importance. preprint arXiv:1905.031 2019.
Sara Hooker, Dumitru Erhan, Picter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, pp. 9737-9748, 2019.
Cosimo Izzo, Aldo Lipani, Ramin Okhrati, and Francesca Medda. A baseline for shapley values in mlps: From missingness to noutrality. preprint arXiv:2006. 200.040006 2020.
Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Aleksander Madry. Missingness bias in model debugging. In International Conference on Learning Representations, 2022.
Joseph D. Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. preprint arXiv:2002.041338 2020.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated rain-based evaluation of ir techniques. ACM Transac- tions on Information Systems (TOTS), 2044):4224446, 2002.
Vipin Kamble and KM Bhurchandi. No-reference image quality assessment algorithms: A survey. Optik, 126(11-12):1000-1099 2015.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In IEEEInternationa Conference on Computer Vision, pp. 4948 4957, 2019.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynop Akata, and Thomas Lukasiewicz. o-vil: A dataset and benchmark for natural language explanations in langunela tasks. In JEFE International Conference on Computer Vision, pp. 1244-1254 2021.
19

---

Published in Transactions on Machine Learning Research (6/2024)
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp. 2668 2677, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. preprint arXiv:2304.000 02643. 2023.
Frederick Klauschen, Jonas Dippel, Philipp Keyl, Philipp Jurmeister, Michael Bockmayr, Andreas Mock, Oliver Buchstab, Maximilian Alber, Lukas Rulf, Grégoire Montavon, et al. Toward explainable artificial intelligence for precision pathology. Annual Review of Pathology: Mechanisms of Disease, 19:541-570, 2024.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. preprint (tt::2009007890 2020.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreementt problem in explainable machine learning: A practitioner's perspective. preprint ar Xiv:22202. 01602, 2022.
Elizabeth Kumar, Suresh Venkatasubramanian Carlos Scheidegger, and Sorelle Friedler. Problems with sagley--valuebbasee explanations as feature importance measures. In International Conference on Machine Learning, pp. 5491-5500, 2020.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10:1096, 2019.
Simon Letzgus, Patrick Wagner, Jonas Lederer, Wojeiech Samek, Klaus-Robent Müller, and Gregoire Mon- tavon. Toward explainable artificial intelligence for regression models: A methodological perspective. IEEE Signal Processing Mogazine, 39(4):40-58, 2022.
Xuhong Li, Mengnan Du, Jiamin Chen, Yekun Chai, Himabindu Lakkaraju, and Haoyi Xiong. M A unified xai benchmark for faithfulness evaluation of feature attribution methods across metrics, modalities and models. In Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Andreas Lugmayr, Martin Danclljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE Conference on Compuler Vision and Pallern Recognition, pp. 11461- 1144111 2022.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Newral Information Processing Systems, pp. 4765-4774, 2017.
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence, 2(1):2522 5839, 2020.
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Indian Journal of Statistics, 12: 49-55, 1936.
Antonios Mamalakis, Elizabeth A Barnes, and Imme Ebert-Uphoff. Carefully choose the baseline: Lessons learned from applying xai attribution methods for regression tasks in geoscience. preprint arXiv:2208.09/ 2022.
Grégoire Montavon, Wojeiech Samek, and Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Sabine Muzellec, Leo Andcol, Thomas Fel, Rufin VanRullen, and Thomas Serre, Gradient strikes back: How filtering out high frequencies improves explanations. preprint ar Xixx2337.00599, 2023.
20

---

Published in Transactions on Machine Learning Research (6/2024)
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys, 55(13s), 2023.
Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, and Ana Lucic. Order in the court: Explainable AI methods prone to disagreement. preprint arXin:2105.09255 2021.
Lars Henry Berge Olsen, Ingrid Kristine Glad, Martin Jullum, and Kjersti Aas. Using shapley values and variational autocncoders to explain predictive models with dependent mixed features. Journal of Machine Learning Research, 23(1:95533-99) 9900,, 2022.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. preprint arXiv: 07421, 2018.
Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yucyuan Zheng, Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. Resisting out-of-distribution data problem in perturbation of xai. preprinl arXiv:2107.14000 2021.
Jie Ren, Zhanpeng Zhou, Qirui Chen, and Quanshi Zhang. Can faithfully represent absence states to compute shapley values DNN? In International Conference on Learning Representations, 2023.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should i trust you?": Explaining the pre- dictions of any classifier. ACM International Conference Knowledge Discovery and Data Mining, 22, 2016.
Laura Rieger and Lars Kai Hansen. Trof:
low resource evaluation metric for explanation methods. In Workshop Al for Affordable Healtheare at ICLR 2020, 2020.
M. Sobnik-Sikonjj and I. Kononenko. Explaining classifications for individual instances. IEEE Transactions on Knowledge and Data Engineering, 20(5):589 600, 2008
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In International Conference on Machine Learning, pp. 18770-18795, 2022.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokron. A unified survey on anomaly, novelty, open-set, and out of-distribution detection: Solutions and future challenges. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
https:///ppenreviev. wwt.://orrumm/dratttt//..o http:s///ppeenrrrrrreee//oooooor
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Claus-RRobet Müller, Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2660-6677 2016.
Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller, Explain- able AI: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, (37272277- 2021.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games II, pp. 307-317, 1953.
Rui Shi, Tianxing Li, and Yasushi Yamaguchi. Output-targeted baseline for neuron attribution calculation. Image and Vision Computing, 124:104516, 2022.
Torty Sivill and Peter Flach. Limesegment: Mcaningful, realistic time series explanations. In International Conference on Artificial Intelligence and Stalistics, pp. 3418 3433, 2022.
Dylan Slack, Sophic Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In AAAI/ACM Conference on AI, Ethics, and Sociely, pp. 180, 2020.
21

---

Published in Transactions on Machine Learning Research (6/2024)
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11, 2010.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International Conference on Machine Learning, pp. 9269 9278, 2020.
Muhammad Faaiz Taufig, Patrick Blöbaum, and Lenon Minorics. Manifold restricted interventional shapley values. preprint arXiw:2301.0000 2023.
Alexandru Telea. An image inpainting technique based on the fast marching method. Journal of Graphics Tools, 9(1):23-34, 34, 2004.
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In AAAI Conference on Artificial Intelligence, pp. 6021 6029, 2020.
Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attri- bution for feature interactions. In Advances in Neural Information Processing Systems, pp. 6147-6159, 2020.
Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimennional concept discovery (MCD): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023.
Yi Wei, Ming-Chine Chang, Yiming Ying, Ser Nam Lim, and Siwei Lyu. Explain black-box image classifi- cations using superpixcl-based interpretation. In International Conference Pattern Recognition, 2018.
Ross Wightman, Hugo Touvron, and Hervé Jégon. Resnet strikes back: An improved training procodure in limm. preprinl 0rXiv:211000041 70.00476, 2021.
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance. preprint arXiv:1907.09700 2019.
Chil-Kuan Yeh, Cheng-Yo Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. In Advances in Neurol Information Processing Systems, pp. 10967-10978, 2019.
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. preprind arXis:2304006790 2023.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European
Conference on Computer Vision, pp. 818 833, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling, Visualizing deep neural network decisions:
Prediction difference analysis. In International Conference on Learning Representations, 2017.
22


Reviews:
[{"review_id": "pDlUoSmbPB", "reviewer": "Reviewer_FCxG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper convincingly demonstrates the disagreement problem in PF benchmarks and traces its root cause to the choice of occlusion strategy. This is an important issue that needs to be addressed.\n2. The proposed R-OMS score provides a principled way to compare occlusion strategies and identify reliable ones. The experiments systematically analyze the impact of various design choices.\n3. The SRG measure is a clever way to decouple PF rankings from the occlusion strategy by combining MIF and LIF. The experiments show SRG provides stable rankings across occlusion setups.\n4. The paper conducts extensive experiments, considering 40 occlusion strategies. The final XAI method ranking using SRG on a reliable occlusion strategy is a valuable benchmark for the community.\nWeaknesses:\n 1. While the SRG measure works well empirically, a deeper theoretical analysis of why symmetrically combining MIF and LIF leads to occlusion-independent rankings would strengthen the paper. \n2. The paper focuses on image classification. Discussing the applicability of the approach to other data modalities would be useful, such as language and graphs. \n3. The diffusion-based occlusion strategy performs best but has a high computational cost. A more efficient alternative that works nearly as well would be practically valuable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses an important issue with pixel flipping (PF) benchmarks used to evaluate feature attribution methods in explainable AI (XAI). The authors point out that PF rankings of XAI methods are sensitive to the choice of occlusion strategy used to remove features, leading to contradictory rankings. To resolve this disagreement problem, they propose to characterize the reliability of occlusion strategies using a Reference-Out-of-Model-Scope (R-OMS) score that measures how much information about the original sample is retained after occlusion. This enables comparing occlusion strategies objectively. In addition, they combine the most-influential-first (MIF) and least influential-first (LIF) relevance gains into a symmetric relevance gain (SRG) measure. The SRG breaks the dependence on occlusion strategy and leads to consistent XAI method rankings across occlusion setups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses in the revision.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vn9DZlQZm5", "reviewer": "Reviewer_9bnw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The study validates its proposed methodologies across a wide range of occlusion strategies, demonstrating their effectiveness in resolving inconsistent rankings and benchmarks. This empirical validation enhances the credibility and applicability of the proposed approaches.\n\nWhile the submission discusses the strengths of the proposed methodologies, it could benefit from a more thorough exploration of potential limitations or assumptions underlying these approaches. A deeper discussion on the practical constraints or scenarios where the proposed methods might not be suitable would enhance the robustness of the study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The study dissects the connection between pixel flipping (PF) benchmarks and the various occlusion strategies employed in XAI. By revealing the inherent link between these two components, the submission provides a pathway to resolving inconsistent rankings and benchmarks.\n2. Validation Across 40 Occlusion Strategies. The submission provides empirical validation of its proposed methodologies across 40 different occlusion strategies, demonstrating their effectiveness in resolving the disagreement problem and enhancing the trustworthiness and comparability of PF benchmarks in XAI.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is critical to include a comprehensive discussion of potential limitations or assumptions underlying the proposed methodologies.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VkzwN2CGYz", "reviewer": "Reviewer_xdiZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First let me state that I am not an ideal reviewer for this paper. I do not work in explainable AI, so I had to dig through the references to see what is going on. \n\nThe paper has a variety of claims that I think are wrong. \n\nThe paper says \"many competing XAI methods exist, which has sparked criticisms from both theoretical and practical perspectives\" (references omitted for brevity). I checked the references here, and they do not seem to support this claim. For example, Doshi-Velez & Kim (2017) propose a taxonomy of evaluation approaches for interpretability. This is very different from criticizing the fact that multiple methods exist. I also checked Rudin (2019), and this paper argues for creating inherently interpretable models instead of searching for explanations post-hoc. Again, this is very different from the stated claim. \n\nThe paper goes on to state that the criticism toward competing XAI approaches has led to \"the disagreement problem, which summarizes the inconclusive scenario of multiple, contradictory explanations\". I checked the paper referenced here, and this is not quite the disagreement problem as defined by that paper. It has nothing to do with summarizing an inconclusive scenario. It actually refers to the issue where explanations obtained by different methods disagree with each other. Also, I could not find support in that paper for the idea that the disagreement problem came from criticism of multiple competing XAI methods. \n\nThe paper then claims \"this naturally arises\" for occlusion-based explanations (without defining what occlusion-based explanation means, or what exactly arises), and states that the problem \"extends\" to pixel flipping benchmarks (again without defining these). The paper also claims that removing most-influential or least-influential features is \"complementary to the ambiguity\", which I don't understand. \n\nThe largest claim is that this work \"resolves the disagreement problem\". Given the scope of the problem as defined in Krishna et al. (2022, which introduced this term), and given that the current paper defines the problem in a different (or at least odd and confusing) way, I am extremely skeptical of this huge claim. Even if the errors in definitions are somehow typographical errors and the results should be interpreted as tackling the actual disagreement problem, there is no way that this paper actually resolves the problem since it does not consider the same set of methods and tasks as in Krishna et al.: LIME, Kernel SHAP, Gradient Input, SmoothGrad, Integrated Gradients, and GradCAM. In particular, GradCAM seems critical to include, since that one has broken through to broader use in computer vision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies, and claims to finally resolve, \"the disagreement problem\" for explainable AI methods. The paper makes a slightly odd definition of the disagreement problem, but in the related work this is defined as the issue where multiple AI explanation methods arrive at inconsistent explanations. The paper focuses especially on two \"feature removal\" methods: occlusion, and pixel flipping. The paper proposes a new scoring system for occlusion methods, evaluating \"reference\" \"out-of-model-scope-ness\", and also a score for measuring feature relevance, called \"symmetric relevance gain\".", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "If I were working on the paper, I would start by rewriting the introduction. I recommend greatly narrowing the scope, simplifying the writing, clarifying the story and the key ideas, and supporting each claim with appropriate references or results. In particular, it is very bad to make a claim and then cite a paper that does not support the claim.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript tackles the discord surrounding Pixel Flipping (PF) benchmarks in the context of consistent XAI. It introduces a more robust model evaluation metric, the Reference-Out-of-Model-Scope (R-OMS) score, and devises a Symmetric Relevance Gain (SRG) measure to mitigate conflicting rankings resulting from competing PF approaches: most influential first (MIF) or least influential first (LIF). Following detailed feedback, the authors have adeptly revised the manuscript, swaying both the reviewers and AE. The AE finds the work valuable and recommends its publication in TMLR, advocating for its acceptance.
Decision: Accept (poster)