F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI
Authors: Xu Zheng
Farhad Shirani
Zhuomin Chen
Chaohao Lin
Wei Cheng
Wenbo Guo
Dongsheng Luo
Venue: ICLR 2025 Poster (2025)
Keywords: Explainable AI
Evaluation Metric
Computer Vision
Primary area: interpretability and explainable AI

Abstract: Recent research has developed a number of eXplainable AI (XAI) techniques, such as gradient-based approaches, input perturbation-base methods, and black-box explanation methods. While these XAI techniques can extract meaningful insights from deep learning models, how to properly evaluate them remains an open problem.  The most widely used approach is to perturb or even remove what the XAI method considers to be the most important features in an input and observe the changes in the output prediction. This approach, although straightforward, suffers the Out-of-Distribution (OOD) problem as the perturbed samples may no longer follow the original data distribution. A recent method RemOve And Retrain (ROAR) solves the OOD issue by retraining the model with perturbed samples guided by explanations. However, using the model retrained based on XAI methods to evaluate these explainers may cause information leakage and thus lead to unfair comparisons.  We propose Fine-tuned Fidelity (F-Fidelity), a robust evaluation framework for XAI, which utilizes i) an explanation-agnostic fine-tuning strategy, thus mitigating the information leakage issue, and ii) a random masking operation that ensures that the removal step does not generate an OOD input. We also design controlled experiments with state-of-the-art (SOTA) explainers and their degraded version to verify the correctness of our framework. We conduct experiments on multiple data modalities, such as images, time series, and natural language. The results demonstrate that F-Fidelity significantly improves upon prior evaluation metrics in recovering the ground-truth ranking of the explainers. Furthermore, we show both theoretically and empirically that, given a faithful explainer,  F-Fidelity metric can be used to compute the sparsity of influential input components, i.e., to extract the true explanation size.

Published as a conference paper at ICLR 2025
F-FIDELITY: A ROBUST FRAMEWORK FOR FAITHFUL- NESS EVALUATION OF EXPLAINABLE AI
Xu Zheng', Farhad Shirani! X Zhuomin Chen', Chaohao Lin', Wei Cheng 3
Wenbo Guo Dongsheng Luo
Florida International University, Miami, United States
NEC Laboratories America, Princeton, United States University of California, Santa Barbara, United States (xzhen01 9, fsh rani, zchen051, c 11in227, dluo fffu fuduu we seichengenee-Lass. com henrygwbeucsb.edu
ABSTRACT
Recent research has developed number of eXplainable AI (XAI) techniques, such as gradient-based approaches, input perturbation-bart base methods, and black- box explanation methods. While these XAI techniques sights from deep learning models, how to properly evaluate them remains an open problem. The most widely used approach is to perturb or even remove what the XAI method considers to be the most important features in an input and observe the changes in the output prediction. This approach, although straightforward, suffers the longer follow the original data distribution. recent method RemOve And Re- train (ROAR) solves the OOD issue by retraining the model with perturbed sam- ples guided by explanations. However, using the model retrained based on XAI methods to evaluate these explainers may cause information leakage and thus lead to unfair comparisons. We propose Fine-tuned Fidelity (F-Fidelity), a robust eval- uation framework for XAL, which utilizes i) explanation-agun aagnostic fine-tuning strategy. thus mitigating the information leakage issue, and ii) a random masking operation that ensures that the removal step does not generate an OOD input. We also design controlled experiments with state-of-the-art (SOTA) explainers and their degraded version to verify the correctness of framework. We conduct experiments on multiple data modalities, such as images, time series, and natu- ral language. The results demonstrate that F-Fidelity significantly improves upon prior evaluation metrics in recovering the ground-truth ranking of the explainers. Furthermore, we show both theoretically and empirically that, given a faithful ex- plainer, F-Fidelity metric can be used to compute the sparsity of influential input components, i.e., to extract true explanation size. The source code available at httpss//trustais--laa. lab.tiiuds.aaaii.ubbu.. wwt.tg.ii/ffieeeelll..e. io/ffieliity
extract meaningful in-
(OOD) problem as the perturbed samples may no
INTRODUCTION
EXplainable AI (XAI) methods have been widely used in many domains, such as Computer Vision
(CV) (Chattopadhay et al. 2018, [mmikov et al. 20177] Jiang et al. 2021
/hou al. 2016
Sel-
varaju al. (017). Neural Language Processing (NLP) (Lyu et al. 2024 Luo et al.) 2024
Zhao
ct al. (2024). Graph Neural Networks (GNNs) Ying et al.
2019 Luo et al. 2020 Yuan et al. 2020
Vu & Thai 2020), and Time Series (Liu et al. Queen al. 2024). There varioss types of expla- l- nation methods, in which the most predominant one is post-hoe instance-level explanation. Given a pre-trained classifier and specific input, these methods aim to identify the most important features of the model's output. For instance, such explanations map to subset of important pixels in image
classification (Ribeiro al. 2016 Selvaraju et al. 2017 Lundberg. 2017). Existing research has proposed a number of XAI methods to draw post-hoc explanations - such as Integrated Gradi-
Corresponding authors

---

Published as a conference paper at ICLR 2025
ents (Sundararajan et al. 2017). CAM-based approaches (Selvaraju et al. 2017). and SmoothGrad Smilkov et al. 2017).
Despite extracting useful insights about model decisions, how to faithfully evaluate and compare planation methods remains an open challenge. There have been some existing efforts to address this issue. The preliminary works create datasets with "ground-truth" explanations for XAI evaluation, such as the MS-CoCo dataset in CV, (Lin et al. 2014), the Mutag dataset in graph (Debnath et al.) 1991]. and the e-SNLI dataset in NLP (Camburu et al. 2018). However, these datasets are still lim- ited to certain tasks and cannot be generalized. More importantly, these "ground-truth explanations are created based on humans' understanding of a data sample, which may not reflect an ML model's decision-makin processes, and thus may still give unfaithful evaluations. More recent works pro- pose the removal strategy, which does not rely on "ground-truth" explanations (Zhou et al. 2016] Selvaraju et al., 2017; Rong et al. 2022, Hooker et al., 20199) Madsen et al. 2023 [022] Hase et al., 2021 Zheng et al.j [0244] Yuan et al., 2022). Technically speaking, the removal strategy removes certain parts of input that are deemed as inon-)important and records the changes in the model prediction. A larger drop in accuracy when removing the important parts reflects the removed parts as indeed important and thus validates the faithfulness of the corresponding explanation method. For example, in computer vision, removal means setting important pixels to zero/black; in natural language processing, it means replacing chosen tokens with "[MASK]" token zero embeddings. Specific metrics designed based on the removal strategy include the Most Relevant First (MoRF) and Least Relevant First (LcRF) (Samek et al. 2016 Yuan al. 2022) in CV and Importance Mea- sures (Madsen et al. 2021 2023) in NLP. However, these removal- the Out-Or-Distribution (OOD) issue (Hooker et al. 2019). as the perturbed samples with removed features may no longer follow the original distribution. As a result, the model predictions of the perturbed samples are unreliable, regardless of the explanation faithfulness.
explanations suffer from
To address the OOD problem, RemOve And Retrain (ROAR) (Hooker et al. 2019), proposed re- training the model for evaluating the explainer, where the model is trained using explanation-gu guided removal. However, retraining the model based on the explainer output leads to an information leak- age issue [Rong et al. 2022). An additional issue, which has been overlooked in prior works, is that the removal strategy is highly dependent on explanation size ( (o sparsity). That is, the size of the explanation part that is removed affects the evaluation output. In the absence of ground-truth explanations, it is not clear what fraction of the input elements should be removed as part of the explanation, which leads to inconsistent evaluations.
To address the aforementioned challenges, introduce Fine-tuned Fidelity (F-Fidelity), novel framework to evalute the performance of explainers. F-Fidelity leverages i) an explanation-agnosst fine-tuning strategy to prevent information leakage and ii) a controlled random masking operation to overcome the OOD issues observed in the application of prior evaluation metrics. The fine- tuning process employs stochastic masking operations, such as randomly dropping pixels in images, tokens in language, or time steps in time series data, to generate augmented training samples. This augmented data is then used to fine-tune a surrogate model. Unlike previous approaches such as ROAR (Hooker et al., 2019). our strategy effectively mitigates the risk of information leakage and stochastic masks, while also offering improved efficiency
label bias by using explanation-agnowe as it eliminates the need to retrain the model for each individual explainer. During the evaluation phase, F-Fidelity implements a removal strategy that generates stochastic masks conditioned on the explainer output, designed to be in-distribution with respect to the masks used in the fine-tuning step, which mitigates the OOD issues observed in the application of prior evaluation metrics.
To verify the effectiveness of F-Fidelity, we provide comprehensive empirical faithfulness eval- uations on a collection of explainers that are systematically degraded from an original explainer through controlled random perturbations. Thus the correct (ground-truth) ranking of the explainers, in terms of faithfulness, is known beforehand. The experiments on multiple data modalities demon- strate the robustness of F-Fidelity through both macro and micro correlations. Macro correlations measure the Spearman rank correlation between the ground truth and the overall performance across all sparsity levels, while miero correlations capture the average Spearman rank correlation between the ground truth and performance at individual sparsity levels. In both aspects, F-Fidelity consis- tently outperforms existing methods. Furthermore, we show both theoretically and empirically that, given faithful explainer, the F-Fidelity metric can be used to compute the size (or sparsity) of ground-truth explanations. To elaborate, we show that the F-Fidelity metric output, when evaluated
2

---

Published as a conference paper at ICLR 2025
function of mask size, produces piecewise constant function, where the length of the constant pieces depends on the explanation size. The following summarizes our main contributions:
We introduce novel evaluation framework for measuring explanation faithfulness with strong theoretical principles that is robust to distribution shifts.
We design and implement a rigorous experimental setting to fairly compare metrics. Our com- prehensive evaluations across multiple data modalities (images, time series, and natural language) demonstrate the superior performance and broad applicability of F-Fidelity.
We theoretically and empirically analyze the relationship between the explanation size and the F- Fidelity metrics and demonstrate F-Fidelity, the ground truth explanation sizes can be inferred.
2
PRELIMINARIES
2.1 NOTATION
Sets are denoted by calligraphic letters such as X. The set (1,2,...,,7 is represented by [n]]- Multidimensional arrays (tensors) are denoted by bold-face letters such Upper-case letters such as X represent random variables, and lower-case letters such as x represent their realizations. Similarly, random tensors denoted by upper-case letters such X.
2.2 CLASSIFICATION MODELS AND EXPLANATION FUNCTIONS
Let f X - y be (pre-trained) classification model such neural network which takes an input X E gexd and outputs a label Y E V, where y is finite set of labels. In CV tasks,
x w, where h, w are the height and width, and d is the number of channels per input pixel. Analogously, in NLP and time series classification tasks, t E N represents the time index, and d is the feature dimension. An explanation function (explainer) consists of pair of mappings (), where Φ X + Rixd the score function, mapping each input element to its (non- negative) important score, and mask function & <(X) -> M, mapping the output of the score function to binary mask M E {0, 1]txd The masked input X is called the explanation for the input X and model f(-), where @ represents elementwise multiplication. We denote the explanation I1 is the l norm. That is, the explanation size S is the number of non-zero elements of M. In general, the size may be deterministically set to constant value or alternatively, may depend on the output the score function, e.g.. input elements receiving a score higher than given threshold are included in the mask and the rest are removed. Let us assume that the (ground-truth) data distribution is Px,.- Then, a 'good' explainer is one which minimizes the total variation distance drv (Pyix, PyjxeM), while satisfying an explanation size constraint Ex [[] S s. where s E N is the desired average explanation size. The minimization of the total variation essentially enforces that the posterior distribution of the classifier output be mostly determined by the masked input explanation, implying that the subset of input components which are removed by the mask have a low influence on the classifier output.
size by S |M]] I. where
2.3 QUANTIFYING THE PERFORMANCE OF EXPLAINERS
A key challenge in explainability research is to quantify and compare the performance of various explainers. The performance of an explainer can be formally quantified in terms the total variation distance drv (Pyix, Py PYIXoM) as a function of the average explanation size Ex M|])) However, in most problems of interest, the underlying statisties Px,y not available. and hence direct evalu- ation of the aforementioned total variation distance is not possible. As discussed in the introduction, some datasets are accompanied by ground-trutt explanations, which enables the use of measures such AUC and loU for evaluating the quality explainers. However, the ground-truth explanations are available only for limited collection of datasets, and even when moound-truthe explanations are available, they may not accurately reflect the model's internal decision-making processes. To ad- dress the aforementioned issues, a widely used set of metrics have been proposed in the literature which are based on the removal strategy. In CV, two removal orders have been considered (Samek) (et al. 2016)) Yuan et al. 2022): MoRF, which evaluates explanations by removing the most influen- tial pixels first, and LeRF, which begins with removing the least influential pixels. These approaches
3

---

Published as a conference paper at ICLR 2025
provide complementary perspectives on feature importance- MoRF assesses whether removing im- portant features significantly impacts predictions, while LeRF verifies if retaining important features is sufficient for model performance. In the graph domain, alternative but conceptually related re- moval strategy is used, where first the size of explanation subgraphs is determined according to a sparsity parameter, and then edges removed either from the explanation subgraph of the desired size, or the non-explanation subgraph which is its complement. In this paper, we use metrics based on this graph domain removal strategy, called the Fidelity (Fid) metric in the literature (Pope al.) 2019 [Bjaj et al. 2022 Yuan et al.) 2022). The Fidelity metrics align with MoRF and LeRF prin- ciples through Fid and Fid respectively. Formally, given the input and label pair (x. y) and binary mask m, the Fidelity metrics are defined as follows:
Fid+ (v)
1
Σ (X..U))T
1 = f(x,)) 1y: = =f(x; x, o m.))
(1)
[T|
Fid =
1
Σ
11 = f(xi)) - (vv = f(x; o m))).
(2)
I
(X..)))T
where T is the dataset used for evaluating the performance of the explainer, n. is the size of the dataset, 1 (-) denotes the indicator function, and m; = (xxx) is the explanation corresponding to x;
produced by the explainer (-))) Here, Fid+ measures prediction changes when removing important features (similar to MoRF), while Fid evaluates model performance when keeping only important
features (similar to LeRF).
A significant limitation of explanation evaluation through removal strategies is the OOD prob- lem (Hooker et al.) 2019 /Zhnn al. 2024). When we remove elements from an input whether they are pixels in images, time steps in time series, or edges in graphs the modified input may longer follow the original data distribution that the model was trained on. For instance, when eval- uating image explanations by zeroing out important pixels, the resulting images with black patches are unlikely to resemble natural images. Consequently, the model's predictions on these modified inputs may be unreliable, due to low quality of the explanation itself, but because the model is operating outside its training distribution.
This OOD problem has been analyzed through different approaches. In ROAR (Hooker et al. 2019), which focused on CV tasks, the solution was to retrain the model on perturbed data where important input elements identified by the explainer are removed. While this approach mitigates the OOD issue, Roog et al. (2022) identified that it suffers from information leakage through the binary masks of removed pixels the pattern of removals itself can contain enough class information to affect the evaluation outcome. Moreover, ROAR requires retraining separate model for each explainer be- ing evaluated, making it computationally expensive and time-consuming when comparing multiple explanation methods.
Similarly, In R-Fidelity (Zheng et al. 2024), which considered the evaluation of GNN explainers, it was argued that the Fidelity metric highly relies on the robustness of the underlying classifier to removal of potentially large sections of the input, t.g.,the removal of a large subgraph explanation for Fidt or its complement for F That is, the classifier should be robust OOD inputs for the Fidelity metric to align with those of the (theoretically justified) saaall-vriiii-baaaeee metric dis- cussed in the previous sections. Rather than retraining the model, R-Fidelity introduces stochastic removal strategy that addresses the OOD issue by controlling the size of removed sections and ran- domly sampling which elements to remove, thus limiting the distribution shift of perturbed inputs. Specifically, the following Robust Fidelity metrics (RFid) was introduced:
RFid (w,at, at,ss=))=+))))
Σ
1 f(x:)) P(yi = f(xt f(xx(xx;x"" ax .ss))
(3)
[
(TTW))T
RFid (+) a ,8)
Σ (TT
1 = f(x)) P(ys = f(x (xx((x;,, a,, ,)),,
(4)
T
where X" (x;, 8) is a sampling function which randomly, uniformly, and independently removes [sa elements from the highest scoring elements of x; based on the scores produced by o(x:), and x (x,, a removes [(d - s)a elements from the lowest scoring td s elements. If a = a 1, then the RFid metric reduces the Fid metric. On the other hand, a and a
4

---

Published as a conference paper at ICLR 2025
decreased, fewer input elements removed, hence requiring lower OOD robustness to ensure the accuracy of the evaluation output.
ROBUST FIDELITY VIA FINE-TUNING AND STOCHASTIC REMOVAL
As discussed in the previous section, significant limitation of prior explanation evaluation metrics is the loss in accuracy due to the OOD nature of the modified inputs generated by the application of removal strategies. For instance, the probability difference P(Y f f(X) - - may be large, even for low-quality explanations. This occurs because the modified input X 1x-Xa O OOD for the trained classifier f((), despite Py and Pyxx-XeM -oM being close to each other. Consequently, this yields a high Fid+ score despite the explanation's 's low quality with respect to the theoretically justified total variation metric. For example, in the empirical evaluations provided the next sections, we demonstrate that the Fid measure sometimes assigns better evaluations to completely random explainers than to those whose outputs align with ground-trune truth explanations.
A partial solution in the graph domain addresses this issue by removing only an at fraction of the explanation subgraph and a fraction of the non-explanation subgraphs (Zheng et al. 2024). However, we argue that two issues degrade the evaluation quality of the RFid metric. First, the classifier may lack robustness and produce unreliable outputs even when the input is only slightly perturbed. Second, if the original explanation size is large (small), then removing an at (a-) of the explanation (non-explanation) part of the input, this would still yield OOD inputs.
In this work, we introduce a simple yet effective framework, F-Fidelity, for robust evaluation of XAI methods. Our strategy can summarized as fine-tuning and stochastic removal. Specifically, we first fine-tune the model with randomly masked inputs to improve its robustness to perturbation. Then employ a controlled stochastic removal process that ensures the perturbed inputs remain within the distribution seen during fine-tuning.
To achieve reliable predictions on partially removed inputs, we design fine-tuning process that randomly removes up to 3 E [0, 1] ratio of imput elements. To elaborate, we introduce a stochastic mask generator P3 d) + Mş, which takes the input dimensions (1, d) as input and outputs mask M3 E {0, 13xd of size fitd, i.e. with up to 3td non-zero elements. For instance, in image classification, the mask generator is designed to select random image pixels or patches for removal. Formally, we define the fine-tuning loss as:
( - Y)).
L = =
(5)
where L is the loss function used during training (c.g., cross-entropy).
In the evaluation process, we modify the RFid metric to ensure consistency with our fine-tuning
strategy by upper-bounding the total number of removed elements by Btd the same bound used
during fine-tuning. That is, for a fixed 8 and RFid parameters a upper-bounded RFid parameters
aorig aorig E [0, 1], we set the
a min (atign Btd and a min(a (td s)
3td
(6)
so that the sampling functions x and x remove the minimum of a o (based explanation size) and 3td (based on input size) elements for x and the minimum a (td - s) and Std elements for _ providing absolute upper bounds on the number of removed elements.
The pipeline of our method shown in Algorithm [ We denote the resulting metrics, which use the fine-tuning process and the RFid+ and RFid metrics with sampling rates that are truncated based on equation 6). as Fid (1h, trrggg 3, and Fiid (1), Ooorig B3, 8), respectively.
is worth noting that both Fid and FFidd can take negative values in certain cases. This occurs when the accuracy after masking exceeds the original prediction accuracy. Alternative formulations could enforce positive values by only reporting masked accuracy like ROAR (Hooker et al. 2019) and deletion/insertion scores (Petsiuk al. 2018 Pan et al. 2021).
5

---

Published as a conference paper at ICLR 2025
4 DETERMINING THE EXPLANATION SIZE VIA FIDELITY METRICS
A critical challenge in explainable AT is determining the appropriate size or scope of explanations. Ideally, ground truth explanations would be discretized into distinct clusters representing different levels of importance. For instance, in image classification, pixels associated with the target object tend receive high importance scores. Conversely, pixels corresponding irrelevant regions receive low scores. However, in many practical scenarios, even good explainers that produce accurate explanation masks - as measured by the Fid and RFid evaluation metrics may yield explanation scores that are not discretized into distinct clusters.
the background or
In this section, we theoretically demonstrate that our proposed evaluation metric can recover the cluster sizes given an explainer that outputs the correct explanation mask (i.e., correctly ranks the importance of input elements). Thus, provided the explainer outputs an accurate mask function, our metric can recover the explanation size (also known sparsity). To provide concrete theoretical analysis, we first consider a classification problem under set of idealized assumptions that gener- alize the above observations classification task defined by joint distribution Px and a classifier f x + y. We assume that the input elements can be partitioned into several influence fiers. That is, for any given input x, there exists a partition Ce (x),, k E [r] of the index set [4] x [d], where (x) represents the set of indices of the input elements belonging
the clustering of explanation scores. Specifically, let consider
tier k, and C2 (x)) are the (fixed) tier sizes. For given mask m, the probability of correct classification based on the masked input x @ m depends only on the counts of unmasked elements in each influence tier. Formally, P(Y xx O m) 92fr)2,,... I.), where g [c1] x [c2] X ... x [c,l + [0. 1] is function monotonically increasing with respect to the lexicographic ordering on its input, and jk [cs]] is the number of elements in Cx) whose cor- responding mask element in is non-zero (unmasked). We further focus on Shapley-value -alub---aa explanations, which provide theoretical foundation for our analysis. Recall that, given label y. the Shapley value associated with element (i, E [t] x [d] of is given (Lundberg 2017):
| (mtlli-(tdd
||m|| - 1)! (P(Y = y x m') - P(Y = y @ m)), (td)!
Sx (x.))) = Σ
where m' is the mask obtained from m by setting m²,; 1 (unmasking the (i,j) element). Under the aforementioned influence tier assumption, it is straightforward to verify that input elements within the same influence tier receive equal Shapley values. Specifically, for any l [r] and any (i,5), (') E (x), we have Sx(i,.) = (x')))").
Theorem 1. For the classification task described above, and a given pre-trained classifier f(), consider a Shapley-value based explainer (t)). For arigg E [0, 1] and 3 E [0, ab,tt.,,, ), let
e(s) = Ex,y (Fidd (tb, at t 3,8),, s SE td].
Then, e(s) is monotonically increasing for 8 E [0,c]] and monotonically decreasing for E [max( 3 (d,.)))td),
The proof is provided in the Appendix.
This theorem shows that F Fidd can recover the size of the most influential tier (i.e., the first cluster size) when the explainer's 's ranking is close to that of an ideal chapley-based explainer. Specifically, the value of s in which FFid' changes direction corresponds to the size of an influence tier. This is verified in the empirical evaluation provided in Section D.8 This result implies that even when an explainer provides continuous scores without distinct clustering, our metric can infer the underlying discrete structure of the ground truth explanations and recover the explanation size.
EXPERIMENTS
To demonstrate the robustness of the F-Fidelity framework, we conduct comprehensive experiments across multiple domains, including image classification, time series analysis, and natural language processing. Our evaluation strategy builds upon the concept introduced by Rong et al. (2022). which posits that an ideal evaluation method should yield consistent rankings in both MoRF and LeRF set- tings. To further establish controlled experimental setting with ground truth (GT) rankings,

---

Published as a conference paper at ICLR 2025
introduce novel approach for fair comparison using a degradation operation good expla- nation, such as an explanation obtained by Integrated Gradients (IG) (Sundararajan et al. 2017), generating series of explanations with varying levels of random noise. Specifically, we first obtain initial (good) explanations using well-established explainers, then systematically degrade them by adding different ratios of random noise. Since explanation quality naturally decreases with increased noise, this creates a ground truth ranking where explanations with less noise should rank higher.
We evaluate the performance of F-Fidelity against established baselines such as (Fidelity), ROAR (Hooker et al. 2019), and R-Fidelity (Zheng et al., 2024) across a wide range of sparsity levels, from 5% to 95% 5% intervals. Throughout our evaluation, we focus on three key Spear- man rank correlations that measure how well different evaluation metrics align with ground truth and each other. Specifically, "MoRF vs. GT measures how well Fid+ rankings align with ground truth rankings when removing important features first, while "LeRF vs. GT measures the correlation be- tween Fid- rankings and ground truth when retaining important features. The "MoRF vs. LeRF" correlation assesses the consistency between Fidt and Fid evaluations, where strong negative correlation indicates that features identified as important by one metric are consistently identified as important by the other. These correlations allow us to assess the methods' performance under vari- ous conditions, from highly sparse to nearly complete explanations. To provide a thorough analysis, employ both macro and micro correlation metrics:
Following Rong et al. (2022)'s approach of evaluating overall explainer consistency, and inspired by the AUC-based aggregation methods in Zhu et al. (2024) and Pan et al. (202T), we compute the AUC with respect to sparsity across the entire 5-95% range for each explanation method. The macro correlations are then calculated using these AUC values, providing an overall performance measure across all sparsity levels.
Macro Correlation:
To capture fine-grained performance differences, we calculate micro cor- relations at each sparsity level. In the main body of the paper. we report the averaged micro correlations across all sparsity levels, as well as the average rank of each method.
Micro Correlation:
5.1 IMAGE CLASSIFICATION EXPLANATION EVALUATION
Setup. We use CIFAR-100 Krizhevskyy et al. 2009 and Tiny-Imagenet (Deng et al. 2009 as
the benchmark datasets. To obtain pre-traeeeee trainer model to be explained, we adopt ResNet (He) al. 2016). More experiments with Vision Transformer(ViT) (ViT) (Dosovitskiy et al. 2020) can be found in Appendix [.I] To generate different explanations, we first use two explanation meth- ods, SmoothGrad Squared (SG-SQ) (Smilkov et al. (017) and GradCAM (Selvaraju et al., 2017) to obtain the explanations. Then use set proportion of noise perturbations in the image, [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]. We provide the implementation detail in Appendix C
Results. As shown in Table[aand F-Fidelity achieves superior performances across different explainers and correlation metrics. Traditional Fidelity and ROAR methods show inconsistent and often poor performance, particularly in MoRF scenarios. Fidelity suffers from the out-of-distribuu (OOD) issue, leading to unreliable evaluations when features are removed. ROAR, while addressing the OOD problem through retraining, faces challenges with information leakage and potential con- vergence issues, resulting in suboptimal correlations. In contrast, for both SG-SQ and GradCAM, F-Fidelity consistently achieves optimal or near-optimal performance in ranking explanations. In CIFAR-100, F-Fidelity achieves perfect Macro and Micro correlations for all three cases with SG- SQ. For GradCAM, it shows strong negative correlation (-0.60 to -0.71) in MoRF comparisons, significantly outperforming other methods. Tiny ImageNet results further reinforce its effectiveness with perfect correlations (-1.00) across all metrics for both explainers. Notably, F-Fidelity consis- tently ranks first in the micro rank evaluation for "MoRF vs. GT and "MoRF vs. LeRF" correlations across both datasets and explainers, indicating its robust performance across various sparsity levels.
5.2 TIME SERIES CLASSIFICATION EXPLANATION EVALUATION
Setup. We use two benchmark datasets for time series analysis: PAM for human activity recognition and Boiler for mechanical fault detection (Queen 2023). For PAM, we use 534 samples across
7

---

Published as a conference paper at ICLR 2025
Table 1: Spearman rank correlation and rank results on CIFAR-100 dataset with ResNet.
SG-SQ
GradCam
Correlation
Fidelity
ROAR
R-Fidelity
F-Fidelity
Fidelity
ROAR
R-Fidelity
F-Fidelity
Macro
MoRF vs GT 4
20.6
-0.662mm
-1.00
-1.00.00
0.81
0.99
0.20
-0.60
LeRF vs GT
g.00
1.00
1.00
1.00
1.00 swnr
1.00
1.00
100
McRF vs LeRF
-0.6823re
40.66.mm
-1.00
-1.00.001
0.8
0.99
0.20gem
-0.601am
Micro
Corr.
MoRF vs GT J
-0.36
10.54
-1.00
-1.00
0.57
0.76
0.09.m
-0.71
LeRF'vs GTT
g.98
2.99
g.00
1.00
2.99
a.99
1.00
1.00gom
MoRF vs LeRF
-0.23
-0.33
-1000.ma0
0.57
0.75
0.01
-1.00
Micro Rank
MoRF GT
3.68
3.26+s04
e
1.0
3.00
3.74
2.1
=00
LeRF vs GT |
1.11 Loci
1.21
1.42
2.05
1.68 11:0
1.21 Low
1.11 goas
MoRF vs LcRF|
3.68 tomeo
3.32+045
1.11
+.00
3,00
3.74
2.16
1.00=000
Table 2: Spearman rank correlation results on Tiny-Imagenet dataset with ResNet.
SG-SQ
GradCam
Correlation
Fidelity
ROAR
R-Fidelity
F-Fidelity
Fidelity
ROAR
R-Fidelity
F-Fidelity
Macro
Conr.
MoRF vs GT I
=0.69
-0.83
-1.00
-1,00
-0.97
0.63=om
-1,00
-1.00=mm
LeRF vs GT
1.00
1.00_om
1000
1.00_
1.00
1.00
1.00
1.00
MoRF ys LeRF
=0.69
-0.83saes
-1.002000
-1.00
-0.97aam
0.63=om
-1.00
-1.00
Micro
Corr.
MoRF vy GT 4
-0.50gon
-0.99.am
-1.00
-0.42aa1
0.54gnic
-0.99
-1.00
LeRF vs GT
1.00
1.00-co
1.00
1.00=com
1.00
0,99=com
=.00
1.00
MoRF vs. LeLL 4
g0
-0.50101
.0.9.am
-1.00
-0.42ga1
0.55gn
-1.00gmm
Miero Rank
MoRF vs GT 4
3.741am
3.21
1.16_am
1.00
3.05
1.84
1.21
1.00
LeRF ys GT
1.11
1.00_om
1.16
1.16
1.47
1,58
1.662a
1.11
MoRF vs Le.R]]
3.74
3.21 g004
1.10
1.00..com.
3.16
3.84
1.21 124
1.00
activity classes, with each sample recorded using a fixed segment window length of 600 from 17
sensors. The Boiler dataset, used for mechanical fault detection, consists of 400 samples with 20 dimensions and fixed segment window length of 36. We employ IG from the Captum library to obtain initial explanations. To generate different explanations, we apply noise perturbations to the importance of each timestamp, using proportions of [0.0, 0.1, 0.2, 0.3, 0.4, 0.5] for PAM and [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] for Boiler.
Results. 3 demonstrates the robust performance of F-Fidelity across different evaluation met- rics. Specifically, in the PAM dataset, F-Fidelity and R- R-Fidelity outperform Fidelity and ROAR in miero correlations and ranks, with F-Fidelity slightly edging out R-Fidelity in "LeRF vs GT" and "MoRF vs LeRF" micro ranks. The Boiler dataset results reveal more significant differences among the methods. F-Fidelity substantially outperforms other methods in "LeRF vs GT and "MoRF vs LeRF" macro correlations. In micro correlations and ranks for the Boiler dataset, F-Fidelity sistently achieves the best or near-best performance across all metrics. These results underscore the effectiveness and stability of F-Fidelity in evaluating explanations for time series data.
Table Spearman ranks correlations and ranks on time series datasets with LSTM as classifier. The best performance is marked as bold. the "_" means the correlation can't 't be obtained because of the same rank of different explanations.
Correlation
PAM
Boiler
Fidelity
ROAR
R-Fidelity
F-Fidelity
Fidelity
ROAR
R- R-Fidclity
F-Fidelity
Maero Corr.
MoRF GT
11000
-1.00_om
-1.00_mmm
-1.00_om
00.99
-1.00_em
-1.00
LeRF vs GT +
1.00:0ee
1.00
1.00
1.00
0.22
0.36:am
0.53
0.86
MoRF LeRF
-1.00
-1.00_eo
_m0
-1.00_em
0.27
1038
-0.33_
10.8
Micro
Corr.
MoRF GT
LcRF vs GT f
0.74
10.8
-0.97
0.80=am
-1.00
1.00
-1.00
1,00=aa0
-0.79
0.69
-0.78
0.79
MoRF vs LeRF
-0.65zo4
-0.77
-1.00 swo
-1.00
-0.81 1:277
-0.97
Micro
Rank
MoRF vs GT
LeRF vs GT I
2.32
1.53eexs
1.37
2.37
1.00
1.16203s
1.05:az:
1.05
2.95+as:
2.59
2.74
2.73-um
2.21
1.21+sa:
1.52
1.68 2008
MoRF vs LeRF
2.37+um
2.42+au2
1.66
1.11+am
2.95+a12
2.95-022
1.89
1.05-r22
8

---

Published as a conference paper at ICLR 2025
5.3 NATURAL LANGUAGE CLASSIFICATION EXPLANATION EVALUATION
Setup. We use two benchmark datasets for NLP experiments: the Stanford Sentiment Tree- bank (SST2) (Socher et al. 2013) for binary sentiment classification and the Boolean Questions (BoolQ) (Socher al. 2013) dataset for question-answeriin tasks. For SST2, utilize 67,349 sentences for training and 872 for testing. BoolQ comprises 9,427 question-answer pairs for train- ing and 3,270 for testing. We employ two popular model architectures: LSTM networks and Transformer-based models (Appendix D.I). We follow the setting in image classification. Specifi- cally. to generate explanations, we first use IG to obtain initial explanations, then apply noise per- turbations to the importance of each timestamp at levels of [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]. We compare our F-Fidelity method against baselines including Fidelity and R-Fidelity, evaluating performance using both macro and micro Spearman correlations. We use the Adam optimizer (Kingma & Ba, 2013) with a learning rate of le-4, keeping other hyperparameters at their default values.
Table 4: Spearman rank correlation and rank results with LSTM model on SST2 and BoolQ datasets.
SST2
BoolQ
Correlation
Fidelity
R-Fidelity
F-Fidelity
Fidelity
R- R-Fidelity
F-Fidelity
Maero
Corr.
MoRF vs GT |
-1.00.0m
-1.00
-1.00.0.0
-1.00
-1.00
-1.00
LeRF vs GT
1.00
1.00
1.00....00
1.00
1.00
1.00
MoRF vs LeRF 4
-1.00
-1.00
-1.00
-1.00
-1.00
-1.00
Micro
Comr.
MoRF vs GT L
+1000
=1.00
-0.99
-1.00
=1.00
=1.00
LeRF vs GT
1.00
2.00
1.00
1.00
1.00
1.00
MoRF vs LcRF
-1.00
-1.00
-0.99
-1.00
-1.00
-1.00
Micro Rank
MoRF GT J
1.00
=.00
1.05+
1.00
1.00
1.00=0..0
LeRF vs GT
1.05
1.05 2..22
1.051.0.22
1.00
1.00
1.00
MoRF vs LeRF
1.00
1.00.0.00
1.1 +0.31
1.00
1.00
1.00=0000
Results. The results are shown in Table 3 Our experiments in the NLP domain reveal an inter- esting phenomenon that NLP models demonstrate remarkable robustness to OOD inputs, which reflected in the performance of various fidelity metrics. Notably, the vanilla Fidelity (Fidelity) method achieves excellent results, often matching or closely approaching the performance of more sophisticated methods like R-Fidelity and our proposed F-Fidelity. This is evident from the near- perfect correlations (-1.00 or 1.00) across most metrics in both datasets. This strong performance of vanilla fidelity suggests that the OOD problem, which often necessitates more complex evaluation frameworks in other domains, may be less pronounced in NLP tasks. The robustness of NLP models to perturbations in input tokens likely contributes to this phenomenon, allowing simpler evaluation methods maintain their efficacy. However, it's worth noting that our proposed F-Fidelity method still demonstrates consistent top-tier performance across all metrics and models, reinforcing its ver- satility and reliability even in scenarios where simpler methods perform well.
5.4 PRACTICAL GUIDELINES FOR F-FIDELITY USAGE
We provide in-depth analysis on the relation between model robustness and performance of F- Fidelity in Appendix D.2 The observed robustness patterns exhibit a direct correlation with F- Fidelity's 's effectiveness, which provides practical guidance. For tasks where models exhibit signifi- cant sensitivity to perturbations, such as computer vision and time series models, F-Fidelity should be the preferred choice over traditional fidelity metries to ensure reliable explanation evaluation. In these domains, the substantial performance degradation under perturbation indicates vulnerability to OOD issues, making F-Fidelity's 's robustness-enhaneing properties particularly valuable. However, for tasks where models demonstrate inherent robustness to feature removal, such many NLP tasks, simpler fidelity metrics are preferred. This relationship between model robustness and evaluation method choice enables one make informed decisions based on their specific domain characteris- tics and accuracy requirements. In Appendix D.22, we provide a simple robustness test by measuring model accuracy under different masking ratios to determine the most appropriate evaluation metric.

---

Published as a conference paper at ICLR 2025
6 RELATED WORK
Existing methods for evaluating explanations can be generally divided into two categories according whether ground truth explanations are available. Comparing the ground truth is intuitive way for explanation evaluation. For example, in time series data and graph data, there exist some syn- thetic datasets for explanation evaluation, including BA-Shapes (Ying et al. a [2019). BA-Motif's (Luo) et al.] 2020), FreqShapes, SeqComb- SegComb-MY and Low Var (Queen et al. 2024 Liu et al.). In computer Vision and Natural Language Processing. the important parts can also obtained with human annotation. However, these methods suffer from the heavy labor of ground truth annotation.
The second category of evaluation methods assesses the explanation quality by comparing model outputs between the original input and inputs modified based on the generated explanations (Zheng ctt al. (2024). For example, Class Activation Mapping (CAM) (Zhou et al. (016) first compares the classification performance between the original and their GAP networks, which makes sure the explanation is faithful for the original network. FraddCAMM(ellvrrjji et al. 2017)) uses image oc- clusion to measure faithfulness. In the following work, Grad-CAM++ (Chattopadhay et al. 2018)
uses three metrics to evaluate the performance of explanation methods, "Average Drop % " crease in Confidence", and "Win %%. In Adversarial Gradient Integration (Pan et al. 2021 Petsiuk iet al. 2018), the authors use "Deletion Score" and "Insertion Score" to measure the faithfulness, where "Deletion Score" is delete the attributions from the original input and "Insertion Score" is to insert attributions into one blank input according to the explanations. In (Samek et al. 2016), the LeRF/MoRF method proposed to measure the importance is consistent with the accuracy of the model. However, this group of metrics does not consider the effect of the OOD issues. In (Hooker) al. 2019), the author propose a new evaluation method ROAR to calculate the accuracy, which avoids the OOD problem by using retrain. ROAD (Rong et al. 2022) is introduced to solve in- formation leakage and time-consuming issues. In the graph domain, GinX-Eval extends ROAR by introducing fine-tuning strategy (Amara et al. 2023).
" In-
In the field of NLP, various methods are employed for evaluating faithfulness, as outlined in (Ja covi & Goldberg 2020). These methods include axiomatic evaluation, predictive power evaluation, robustness evaluation, and perturhation-base evaluation, among others. Axiomatic evaluation in- volves testing explanations based on predefined principles (Jacovi & Goldberg 2020 Adebayy al., 2018 Liu ct al. 2022. Wiegreffe et al./ (2020). Predictive power evaluation operates on the premise that explanations do not lead to the corresponding predictions, they are deemed unfaithful (Ja- covi & Goldberg 2020 Sia et al., 2023 Ye et al. (02T). Robustness evaluation examines whether explanations remain stable when there are minor changes in the input (Ju et al., 2021 Yin et al., 2021 Zheng et al., (022)) such as when input words with similar semantics produce similar outputs. Perturbation- evaluation, one of the most widely used methods, assesses how explanations change when perturbed (Atanasova 2024 Jain & Wallace| 2019). This approach is akin to MoRF and LCRF, where YDeno et al. 2019) measures prediction sufficiency and comprehensiveness by removing both unimportant and important features. For further information, please refer to the survey paper (Lyu et al. 2024).
CONCLUSION
In this paper, we introduced F-Fidelity, a robust framework for faithfulness evaluation in explainable AI By leveraging a novel fine-tuning process, our method significantly mitigates the OOD problem that has plagued previous evaluation metrics. Through comprehensive experiments across multi- ple data modalities, we demonstrated that F-Fidelity consistently outperforms existing baselines in assessing the quality of explanations. Notably, our framework revealed a relationship between eval- uation performance and ground truth explanation size under certain conditions, providing valuable insights into the nature of model explanations. In the future, we plan to explore alternative pertur- bation strategies, such Gaussian blur. Additionally, we will investigate fine-tuming models with limited training data to further enhance the applicability and practicality of F-Fidelity, particularly in cssutrceconstraiied settings.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project was partially supported by NSF grants IIS-2331908 and CCF-2241057. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Kenza Amara, Mennatallah El-Assady, and Rex Ying. Glnx-eval: Towards in-distribution evaluation of graph neural network explanations. XAI in Action: Past, Present, and Future Applications, 2023. URL http:::/ http:///pppeeeeeeeeee net http://www....tttrroroaoaaaaoaoo
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1rXiv:1907 2019.
Pepa Atanasova. A diagnostic study of explainability techniques for text classification. In Account- able and Explainable Methods for Complex Reasoning over Text, pp. 155-187. Springer, 2024.
Mohit Bajaj, Lingyang Chu, Zi Yu Xue, Jian Pei, Lanjun Wang, Peter Cho-Ho Lam, and Yong Zhang. Robust counterfactual explanations on graph neural networks. Advances in Neural Infor- mation Processing Systems, 34:5644-5655 2021.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-116, 1995.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances Neural Information Process- ing Systems, 31, 2018.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018.
Asim Kumar Debnath, Rosa Lopez de Compadre, Gargi Debnath, Alan Shusterman, and Cor- win Hansch. Structure-activinn relationship of mutagenic aromatic and heteroaromatic nitro com- pounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786-797, 1991.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jay Younn Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition scale. In International Confer- ence on Learning Representations, 2020.
Peter Hase, Harry Xie, and Mohit Bansal. The out-of-distribution problem in explainabil- ity and search methods for feature importance explanations. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 3650-3666. Curran Associates, Inc., 2021. URL httpss://proeeddnggss http:///oooeedn.......ieeeeeeiie

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. Advances neural information processing systems, 32, 2019.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nip systems: How should define and evaluate faithfulness? arXiv preprint arXi::2004.0368. 2020.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint 1rXiv:1902.10188 2019.
Peng- Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Pro- cessing, 30:5875-5888, 2021.
Yiming Ju, Yuanzhe Zhang, Zhao Yang, Zhongtao Jiang, Kang Liu, and Jun Zhao. Logic traps in evaluating attribution scores. arXiv preprint arXlv:2109.05463, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htht::
7/ //arxiv.. wtggabs///aaalaa.// 6980
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Yibing Liu, Haoliang Li, Yangyang Guo, Chenqi Kong, Jing Li, and Shiqi Wang. Rethinking attention- model explainability through faithfulness violation test. In International Conference on Machine Learning, pp. 13807-13824. PMLR, 2022.
Zichuan Liu, Tianchun Wang, Jimeng Shi, Xu Zheng, Zhuomin Chen, Lei Song, Wenqian Dong, Jayantha Obeysekera, Farhad Shirani, and Dongsheng Luo. Timex++: Learning time-series explanations with information bottleneck. In Forty-first International Conference on Machine Learning.
Scott Lundberg. unified approach to interpreting model predictions.
arXiv preprint
1rX1v:1705.07874, 2017.
Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information pro- cessing systems, 33:19620-198 2020.
Siwen Luo, Hamish Ivison, Soyeon Caren Han, and Josiah Poon. Local interpretations for explain- able natural language processing: A survey. ACM Computing Surveys, 56(9):1-36. 2024.
Qing Lyu, Marianna Apidianaki, and Chris Lallison-Burchh Towards faithful model explanation in nlp: A survey. Computational Linguistics, pp. 1-67, 2024.
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness of importance measures in nlp by recursively masking allegedly important tokens and retraining. arXiv preprint 10.00:2100.08/11 2021.
12

---

Published as a conference paper at ICLR 2025
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness of
importance measures in nlp by recursively masking allegedly important tokens and retraining. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1731-1751, 2022.
Andreas Madsen, Siva Reddy, and Sarath Chandar. Faithfulness measurable masked language mod- els. arXiv preprinl arXiv: 23.00078 2023.
Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In Thirtieth International Joint Conference on Artificial Intelligence (IJCAI), 2021.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of 2018.
black-box models. arXiv preprint
Phillip E Pope. Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. Ex- plainability methods for graph convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10772-10781, 2019.
Owen Queen. Replication Data for: TimeX, 2023. URL htt.:ttts////0..rr//10.. /1111///0000////0001//00///
BODEQJ
Owen Queen, Thomas Hartvigsen, Teddy Koker, He Huan, Theodoros Tsiligkaridis, and Marinka Zitnik. Encoding time-series explanations through self-supervised model behavior consistency. In Proceedings of Neural Information Processing Systems, NeurIPS, 2023.
Owen Queen, Tom Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, and Marinka Zit- nik. Encoding time-series explanations through self-supervised model behavior consistency, Ad- vances in Neural Information Processing Systems, 36, 2024.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 2016.
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. consistent and efficient evaluation strategy for attribution methods. arXiv preprint arXiv:2202.0044: 2022.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(1 (1) 1):2660-2673, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Suzanna Sia, Anton Belyy, Amjad Almahairi, Madian Khabsa, Luke Zettlemoyer, and Lambert Mathias. Logical satisfibiility of counterfactuals for faithful explanations in nli. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9837-9845, 2023.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv: 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad:
removing noise by adding noise. arXiv preprint arXiv: arXiv: (70.::/70000252 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-16444 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, ((1):1929-1953) 2014.
13

---

Published as a conference paper at ICLR 2025
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 5319-3328, PMLR, 2017.
Minh Vu and My T Thai. Pgm-explainer: Probabilistic graphical model explanations for graph
neural networks. Advances in neural information processing systems, 33:12225-12233 2020.
Sarah Wiegreffe, Ana Marasovié, and Noah Smith. Measuring association between labels and free-text rationales. arXiv preprint arXiv:2010. 2020.
Xi Ye, Rohan Nair, and Greg Durrett. Connecting attributions and qa model behavior realistic counterfactuals. arXiv preprint arXiv:2104.0455,,222 2021.
Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, and Kai-Wei Chang. On the sensitivity and stability of model interpretations in nlp. arXiv preprint (0ti::2004087785 2021.
Zhitao Ying. Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.
Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang Xgnn: Towards model-level explanations of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 430-438, 2020.
Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: A taxonomic survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey, ACM Trans- actions on Intelligent Systems and Technology. 15(2):1-38, 2024.
Xu Zheng, Farhad Shirani, Tianchun Wang. Wei Cheng, Zhuomin Chen, Haifeng Chen, Hua Wei, and Dongsheng Luo. Towards robust fidelity for evaluating explainability of graph neural net- works. In The Twelfth International Conference on Learning Representations, 2024. URL
https: http:://peeeeeeeeeee net. meet:/oorrumll=upphrhmie
Yiming Zheng, Serena Booth, Julie Shah, and Yilun Zhou. The irrationality of neural rationale models. arXiv preprint arXiv:2110.07550 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Zhiyu Zhu, Huaming Chen, Xinyi Wang. Jiayu Zhang, Zhibo Jin, Jason Xue, and Jun Shen. It- erative search attribution for deep neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 62332-62348. PMLR, 21-27 Jul 2024. URL https: http:://pcceeeings.. mlr. www.erss/seec v235/ zhu24a. ht.ml
14

---

Published as a conference paper at ICLR 2025
A DETAILED ALGORITHM
In this section, we provided the detailed pipeline of our evaluation method F-Fidelity as follows:
Algorithm Computing FFid+ FF.FFFFiic Fid
I: Input: model to be explained f. train set Dr, explanation test set D. with n = [D,] samples, explainer th explanation size original parameters orig orig fixed fraction 3, training epochs E, sampling numbers
2: Output: FFid*, FFFid
3: copy f" -
4: for in range(EE) do
5: for (x:: 34) in Dr do
update f by using [(f(x; P3 xi).
7: end for
8: end for
9: for (x:; me) in D. do
m (x.)
obtain the explanation
11: for k range( N) do
12:
at min((a
8td Rid
13:
min(o orig td-a
14:
FFid
(y) = fx(x)) 1n f" axx, s)))
15:
FFid k
1 = f (x:))
1y:: (x (xi, ,
16: end for
FFid [ 18: FFid [i] -- N A= FFid [i,
17: FFid [1]
N EN k=
19:
end for
20: FFudt [Pol Σ DE, FFid* [t]
21: FFid + [Dy] DEx, FFid
22: Return FFid", FFid FFid
B PROOF OF THEOREM
We provide the proof by considering the following cases:
Case 1: E [max( _id,
ld, (1),
Corig
We have:
e(s) Ex.y (FFid (ip, atrigs B,
= =
Σ
P(Y: = f+(x +Xi, atrigg 8))),
(X.Y)TT
where f+(-) represent the finetuned model Algorithm
which assumed to be robust up to mmll 2 id -td. Consequently,
3ld removals, i.e., (xfr(x m) = YY = P(Y|x C m) it
e(s) Px,v = (X))) (Y,x((=(((X) PX
S+(x (X, (trgg))))))))))
g((,, C2 ... ,c) Σ
in, 02 j2,...,0 jr)
=g(c, ... ,4) -((g(c J1, O2 - J2, ... , Jc)),
where J" a multivariate hypergeometric vector with parameters (n1. .2, nr, Btd), where:
C
Ersioe Ss,
n4 = s Σ Eveier if Eecice S Ers
otherwise
To explain the last equation, recall that FFid+ first ranks the elements of the input based on Shap values, and chooses the top element. Then, it randomly and uniformly samples Std ele- ments from the top elements (as long [ Bld). So, if the i-th influence tier satisfies
15

---

Published as a conference paper at ICLR 2025
Ev s, then all of its elements will be chosen among the top influential elements. If Eves Co S S S Cv, then Ei'cs Cyr elements from the tier would be chosen among the top 8, and otherwise, no elements will be chosen from the tier. So, in general, n, elements from each tier will be chosen, where n, is defined in equation 7 Next, FFid samples Std of these top elements and masks them. Thus, the number of elements masked in each tier follows multivariate hypergeometric distribution, modeling the sampling of 3Ld elements from Th, f22, :.. My elements of types 1, 2, .... ,r, respectively. As increases, the choice is diluted by the inclusion of the lowest tier elements in the top (since only for the tier with Ei' Ci' S S Eiss Cy increases as is increased to + 1). So, J is decreasing (in terms of lexicographic ordering) as a function of s, and c" J" is increasing as function of Since e(s) is linearly related to B[g(c" - J")), it follows by the fact that g(i,,J2, ... (r) is increasing with respect to the lexicographic ordering that e(s) is decreasing in s.
Case 2: E [1]
The proof follows by similar arguments as in the previous case. We have:
e(s) g(1,, 02, ,G, g(c - La *e 0,, O).
It follows by the fact that 9 ". .Ir) is increasing with respect to the lexicographic ordering that c(s) is increasing in s.
C DETAILED EXPERIMENT SETUP
In F-Fidelity, the hyperparameter 3 is used in both fine-tuning and evaluation. We select 0.1 by default. For the sampling number, N, a larger number helps reduce the standard deviation but leads increasing evaluation time. In this paper, and a are designed to alleviate the OOD problem, we choose a a a = 0.5. We provide hyperparameter sensitivity studies in Section D.9 and show comprehensive results in Section E.4 For the key parameter B, we further provide an in-depth analysis in Section D.3 For baselines, we use their default values as suggested in their original papers or codes.
select a balance point of 50. The hyperparameter a
We explore two architectures, ResNet and ViT for the image classification tasks. We use ResNet- 9 and 6-layer VIT as backbones for both CIFAR- 100 and Tiny-Imagenet. In the ResNet architecture, the hidden dimensions are set to [64, 128, 128, 256, 512, 512, 512, 512]. For the ViT model, we use a patch size of 4, with a patch embedding dimension of 128. The backbone consists of 6 attention layers, each with 8 heads for multi-head attention. The final output hidden dimension of the ViT encoder is 256. In the training stage, we set the learning rate and weight decay to IE-4 for ResNet and set the learning rate to 1E-3 and weight decay to I IE-4 for VIT. We use Adam as the optimizer and the training epochs are 100 for ResNet and 200 for VIT. During fine-tuning, we use the same hyperparameters.
For the time series classification task, we follow the TimeX (Queen et al., 2023) to use a simple LSTM for both Boiler and PAM datasets. The simple LSTM model contains 3 bidirectional LSTM layers with 128 hidden embedding sizes. We use AdamW as the optimizer with a learning rate of 1E-3 and weight decay is 1E-2.
For the natural language task, we use LSTT (Hochreiter & Schmidhuber 1997) and Transformer as the backbone. the LSTM has one hidden layer with the dimension set 128. In Transformer, the hidden dimension is set to 512. The number of Transformer layers for SST2 and BoolQ are 2 and 4. The head of the Transformer layers for SST2 and BoolQ are and 8. For all datasets and architectures, we use Adam optimizer (Kingma & Ba 2015) with default learning rate 1E-4, training epochs 100.
D EXTRA EXPERIMENTAL STUDY
D.1 EXPERIMENTS WITH OTHER BACKBONES.
VIT as Backbone for Image Classification. To further validate the robustness our F- Fidelity framework across different model architectures, conducted additional experiments us- ing ViT (Dosovitskiy et al. 2020) as the backbone for image classification on both CIFAR-100 and
16

---

Published as a conference paper at ICLR 2025
TinyImageNet datasets. We utilize the SG-SQ (Smilkov et al. 2017) to generate explanations. For the settings, we follow our main experimental setup. As shown in Table F-Fidelity consistently achieves best correlations across all metries for both datasets, outperforming other methods. This strong performance on ViT models further emphasizes the versatility and effectiveness of F-Fidelity in evaluating explanations across different deep learning architectures.
Table 5: Spearman rank correlation results on CIFAR-100 and Viny-Imagenan dataset with VIT.
CIFAR 100
Tiny Imagenet
Correlation
Fidelity
ROAR
R-Fidelity
F-Fidelity
Fidelity
ROAR
R- R-Fidelity
F-Fidelity
Macro
Corr.
MoRF's vs GT
-0.94_mm
-1.00
-1.00_mm
-0.82
-0.83gare
-1.00_om
-1.00gare
LeRF vs GT f
1.00.m00
1.00
1.00
1.00
1000
1.00
1.00
1.00
MoRF vs LeRF
-0,941am
-0.94_mm
-1.00
-1.00_cmm
+0.82.00
-0.83
-1.00_om
-1,00
Miero
Corr.
MoRF vs GT
-0.68404
-0.91 =000
-1.00
-1.00=0m
-0,74
-0 0.766a2
-1.00
-1,00
LRRF vs GIT f
1.00gcm
0.94
1.00
1.00
1.00
1.00
1.00
1.00
MoRF LeRF
10.69
-0.77
-1.00
-1.00
-0.74
-0.762019
-1.00_ccm
-1.00
Micro
Rank
MoRF GT
LeRF vs GT 1
3.47
1.16
1.11 20:
3. 3.162007
1.05gam
1.32
1000
1.32 10.77
3.26
1.00
g.
1.00 2ooM
1.00
1.00
1.00
1.00 g
MoRF vs LeRF!
3,58
3.16:107
+.05+012
1.00
3.26+004
3.63
1.00+s00
1.00=100
CNN as Backbone for Time Series Classification. To verify the effectiveness of F-Fidelity on
Time Series, we conduct experiments with CNN as the classifier by following the setting in Section 5.2 As Table [ shows, F-Fidelity consistently outperforms baselines across all metrics including macro and micro correlations. Similarly, we observe the same phenomenon as shown in Table 3 the Fidelity and ROAR suffer OOD problem. Under some sparsity, the correlation is not available for the same rank while F-Fidelity and R- R-Fidelity perform well. These results on CNN models highlight the versatility and effectiveness of F-Fidelity in assessing explanations across various deep learning architectures.
Table 6: Spearman ranks correlations and ranks on time series datasets with CNN as the classifier. The best performance is marked as bold. The *_" means the correlation can't an't be obtained because the different explanations have the same rank.
PAM
Boiler
Correlation
Fidelity
ROAR
Fidelity
F-Fidelity
Fidelity
ROAR
R. Fidelity
F- F-Fidelity
Maero
Co..
MoRF GT
LeRE vs GT f
0.78
0.962an
-0.71_mm
-0.90=tm
-1.00
1.00
-1.00=co
1,00
1.91
-100
0.86
-1.00
-1.00=cm
1.00
-1.00
1.00
MoRF vs LeRF
0.68
0.61
-1.00 swor
-1.00_....
-0.91
-0.861am
-1.00_cm
-1.00
Micro Corr.
MoRF GT
.0.021ap
-0.39_mm
-0000
-1.00_0mm
-0.94_015
.0.92ga
LeRF vs GT
0.93
0.68 ja5:
1.00.cm
1.00
0,93
0.93
MoRF vs LcRF
-0.03++as
-0.12moa1
=000
-1.00-cmm
-0.93=ou2
-0.94
Micro Rank
MoRF vs GT
3.00
2.00
1.00
1.1s
3.161am
2.63
1.11
1.32
LcRF vs GT
2.68-122
2.84-1142
1.00aaco
1.00+aa0
3.11+a4
3.53-030
1.47+aaa
1.21-041
MoRF vs LeRF
3.682946
2.79
1.00
1.11
3.21
3.05
1.32an4s
1.16
Transformer as Backbone for Natural Language Classification. To further validate our findings with modern architectures, we conducte experiments using Transformer models both SST2 and BoolQ datasets. As shown in Table / the results demonstrate interesting patterns. On SST2, while all methods perform well, F-Fidelity shows slight advantages, particularly in miero correlations and rankings. For BoolQ, all methods achieve nearly perfect correlations, suggesting that Transformer models on this dataset exhibit strong robustness perturbations, making the differences between evaluation methods less pronounced. These results align with our earlier observations about the inherent robustness of NLP models, while still demonstrating the reliability and slight advantages of F-Fidelity in more challenging scenarios.
D.2 RELATIONSHIP BETWEEN MODEL ROBUSTNESS WITH F-FIDELITY PERFORMANCE
To understand how model robustness influences F-Fidelity's 's effectiveness, we conduct a system- atic evaluation across three domains: computer vision (CIFAR- 100), time series (Boiler), and NLP (SST2). For each domain, we test multiple architectures - ResNet and Transformer for CIFAR- 100, L.TM and CNN for Boiler, and LSTM and Transformer for SST2. We measure model robust-
17

---

Published as a conference paper at ICLR 2025
Table 7: Spearman rank correlation and rank results with Transformer model SST2 and BoolQ
datasets.
SST2
BoolQ
Correlation
Fidelity
R-Fidelity
F- F-Fidelity
Fidelity
R- R-Fidelity
F- F-Fidelity
Macro Corr.
MoRF vs GT J
+0,94
=1.00
-1.00+..o
=1.00
-1.00=0.00
-1.00
LeRF vs GT
1.00
=.00
1.00
1.00
=.00
1.00
MoRF vs LeRF
-0.94
-1.00
-1000...0
-1.00
-1.00
-1.00
MoRF vs GT +
+0.93
-0.97
=1.00
=1.00
-1.00=0.01
-1.00
Micro
Corr.
LeRF vs GT
0.98
0.99
0.99
1.00
=.00
1.00
MoRF vs LeRF
+0.92
00.77
-0.99 10002
-1.00
=1.00
-1.00
Micro
Rank
MoRF vs GT +
2.05
1 2.32
1.05
1.00
1.00
1.00.000
LeRF vs GT
1.16+0.0
1.26
1.16+0.36
1.00 +0.00
1.00=0.00
1.00 #:000
MoRF LeRF
2.05+
1.32+0.46
1.11
1.00
1.00
1.00
Table 8:
Classification accuracy (%) comparison between original and fine-tuned models under different perturbation ratios.
Dataset
Architecture
Model
0%
5%
10%
Original
73.23
56.70
41.81
ResNet
Fine-tuned
73.10
73.12
72.97
CIFAR- 100
Original
50.25
46.19
39.87
VIT
Fine-tuned
46.91
47.05
47.30
Original 80.14
67.78
61.53
LSTM
Fine-tuned
77.22
72.64
73.19
Boiler
Original 82.22
68.89
57.92
CNN
Fine-tuned
86.39
74.03
63.89
Original 82.45
80.28
79.82
LSTM
Fine-tuned
82.68
80.16
79.24
SST2
Original 80.85
77.29
73.28
Transformer
Fine-tuned
80.73
81.08
78.90
ness through classification accuracy under different perturbation levels (0%, 5%, and 10%), where perturbations are implemented through domain-specife masking operations: zeroing pixels for im- ages, masking feature values for time series, and replacing token with zeros for text. For each model-architecture pair, compare the performance between the original model and its fine-tuned version under these perturbation conditions.
Table reveals distinet robustness patterns across different domains and architectures. In com- puter vision, models exhibit high sensitivity to perturbations the original ResNet on CIFAR-100 shows dramatic accuracy degradation under even small perturbations, while its fine-tuned counter- part maintains stable performance. Time series models demonstrate moderate sensitivity, with orig- inal models showing notable but less severe accuracy drops under perturbations, which fine-tuning helps to stabilize. In contrast, NLP models display inherent robustness, maintaining relatively stable performance even under perturbations regardless of fine-tuning.
These robustness patterns directly correlate with F-Fidelity's 's effectiveness. The performance im- provements provided by F-Fidelity are most pronounced in domains where models show high sen- sitivity to perturbations (computer vision and time series). This is because traditional fidelity met- rics in these domains suffer from unreliable evaluations due to OOD inputs, which our fine-tuning strategy effectively mitigates. However, in NLP tasks where models possess natural robustness to perturbations, the OOD problem is less severe, consequently diminishing the relative advantages of F-Fidelity. This relationship provides practical guidance for choosing evaluation metrics F-Fidelity should be preferred in domains where models show high sensitivity to perturbations, while simpler metrics may suffice for naturally robust domains.
18

---

Published as a conference paper at ICLR 2025
D.3 ANALYSIS OF DECISION BOUNDARY PRESERVATION UNDER RANDOM MASKING FINE- TUNN-TNNIG
This section addresses whether fine-tuning the classification model with random masking (up to fraction 3 of input elements) significantly changes the model's decision boundaries. We examine this by looking at both global classification patterns and local decision characteristics. We consider two aspects of the model's decision boundaries as follows.
Global Decision Boundary: The global decision boundary refers the overall separation be- tween different classes in the input space. It captures the main structure of the data distribution and represents the classifier's 's ability to distinguish between classes on a broad scale.
Local Decision Boundary: The local decision boundary characterizes how the model responds to small perturbations around individual data points.
$=0.00
10
$=0.00
$=0.01
8=0.01
0.6
0.8
8=0.05
$=0.05
0.5
8-0.10
--0.10
0.6
0.4
8=0.20
P=0.20
0.3
$=0.30
$=0.30
0.2
$=0.40
$=0.40
0.2
8=0.50
5=0.50
0.1
0,05 0.1 0.2
0.4 0.5
0.05 0.1 0.2 0.3
0.5
Perturbation Size
Perturbation Size
(a) Classification accuracy
(b) Prediction agreement with original prediction
Figure 1: Effects of random masking fine-tuning on model behavior. Higher fine-tuning 3 values improve robustness perturbations but reduce agreement with the original model's predictions.
Our random masking fine-tuning aims to maintain the model's ' global classification behavior while improving its stability to small input changes. The random masking process applies uniform noise across all inputs, regardless of their class. This helps the model learn to handle missing information while keeping its main classification patterns intact. Random masking fine-tuning maintains model performance through several key principles. First, the random masking operation applies uniform noise distribution across the input space. Since this noise is class-agnostic and uniformly distributed, it preserves the statistical properties of the underlying class distributions. Second, random masking acts as form of dropout regularization (Srivastava al., 2014), which has been shown im- prove model generalization while maintaining core decision boundaries. Third, the introduction of masked inputs during training encourages the model to develop smoother decision boundaries in local neighborhoods, as demonstrated by Bishop (1995) in the context of noise injection.
To empirically evaluate random masking fine-tuning, we conduct experiments with ResNet on CIFAR-100. We examine both global decision boundary preservation and local boundary character- isties through systematic perturbation analysis. For global boundary preservation, we compare clas- sification accuracy between the original and fine-tuned models on clean test data, using 0% masking as the baseline reference point. We also measure prediction agreement between these models to assess consistency in their decision-making To analyze local boundary characteristics, we apply in- cremental perturbations during testing by randomly masking input elements, with perturbation sizes ranging from 0.05 to 0.50. At each perturbation level, we track both classification accuracy and prediction agreement with the original model. Classification accuracy reveals the model's robust- ness to perturbations, while prediction agreement quantifies the consistency of decision boundaries under noise. The masking ratio 3 during fine-tuning controls the balance between preserving global boundaries and smoothing local ones.
Our experimental results demonstrate that random masking fine-tuning effectively balances global decision boundary preservation with local boundary smoothing. The fine-tuned models maintain
19

---

Published as a conference paper at ICLR 2025
comparable baseline accuracy the original model while showing significantly improved robustness to perturbations. On clean test data, the accuracy difference between original and fine-tuned models remains small, indicating preserved global classification behavior. The high prediction agreement between original and fine-tuned models on clean data further confirms the preservation of global decision patterns. Meanwhile, the smoother local decision boundaries manifest through sustained performance under perturbations, with fine-tuned models showing notably accuracy change com- pared the original model.
The masking ratio 3 plays crucial role in model behavior under perturbations, as evidenced in Figure [1] In terms of accuracy (Figure [(a], models fine-tuned with larger values demonstrate remarkable resilience, maintaining accuracy above 60% even under 50% input perturbation, while the original model (3 0) drops below 40% accuracy with just 10% perturbation. The prediction agreement results (Figure l show trade-off. Higher & values lead to better accuracy under perturbations and result in lower agreement with the original model's predictions. This suggests that stronger fine-tuning with larger & values causes the model to learn more robust but slightly different decision boundaries compared to the original model. Models with moderate values (like 0.10) maintain better balance between perturbation resistance and consistency with the original model's behavior.
D.4 PRESERVATION OF EXPLANATION AFTER FINE-TUNING
To further validate that our fine-tuning strategy preserves meaningful explanations, we conduct qualitative analysis on Tiny-Imagenet. We visualize and compare the attribution maps generated for both the original and fine-tuned models using GradCAM. As shown in Table ! the attribution maps from both models highlight similar regions of importance, with only minor variations in the highlight intensity. This visual consistency suggests that while our fine-tuning process improves model robustness perturbations, it maintains the model''s 's original attention to meaningful features, thereby preserving the validity of the generated explanations.
Table 9: The explanation visualization comparison between original and fine-tuned model.
ResNet
VIT
Input
Original
Fine-tuned
Original
Fine-tuned
20

---

Published as a conference paper at ICLR 2025
D.5 EVALUATION WITH LAYERCAM AS ALTERNATIVE EXPLANATION METHOD
To further validate the robustness of our method across different explanation techniques, we conduct additional experiments using LayerCAM (Jiang al. ) alternative explanation method to GradCAM. Following the same experimental setup as in Table I we evaluate the performance using ResNet architecture on the CIFAR-100 dataset. As shown in Table LO F-Fidelity maintains its supe- rior performance with LayerCAM, exhibiting consistent patterns with our GradCAM results. This consistency across different explanation methods reinforces the generalizability of our evaluation framework and its ability to assess various types of explanation techniques reliably.
Table 10: Spearman rank correlation and rank results on CIFAR-100 dataset with ResNet explained by LayerCAM.
Correlation
Fidelity
ROAR
R-Fidelity
F- F-Fidelity
Macro
Corr.
MoRF GT
0.77+006
0.98
0.26
-0.26
LeRF vs GT f
1.00
1.00
1.00
1.00
MoRF LeRF 4
0.77:0006
0.98
0.26
-0.26
MoRF vs GT
0.53
0.8 1:000
-0.02.20.2
-0.50
Micro
CoIr.
LeRF vs GT f
0.99
0.99
1.00
1.00
MoRF vs LeRF| +
0.53
0.81 +:0.0
-0.02+0.2
0.50
MoRF vs GT +
2.95
3,95
1.89
1.21 =042
Micro Rank
LeRF vs GT +
2.53+1339
1.95 18
1.05
1.00.00:00
MoRF vs LeRF 4
2.89
3,951023
1.89
1.21 1:042
D.6 CASE STUDY
To provide the intuitional verification of our method, we conduct a case study with an image from the Tiny-Imagenet dataset. We first use GradCAM to obtain the explanation. To get various degraded explanations, we follow the previous setting to add noise, which is a list of [0.2, 0.4, 0.6, 0.8, 1.0], to the explanation obtain the ground truth ranking of explanations. We compare baselines and F-Fidelity and show the results in Figure 2 We observe that the Fidelity and ROAR methods fail differentiate explanations with noise levels of 0.2 and 0.4 when the sparsity is below certain thresholds (e.g., 25%). However, the R-Fidelity and F-Fidelity have a good performance in such cases and F-Fidelity has a better performance than R-Fidelity from the micro correlation results.
D.7 ABLATION STUDY
To evaluate the effectiveness of our proposed F-Fidelity framework, conduct ablation stud- ies comparing three evaluation setups: the original Fidelity metric, Fidelity with fine-tuning (Fidelity+Fine-tune)). and F-Fidelity. We utilize the CIFAR-100 dataset for these experiments, em- ploying a ResNet architecture as the backbone for the classifier. The explainers are based on SG-SQ.
As shown in Figure 3 across all three correlation Pestrics-MMofe vs GT", "LeRF GT", and "MoRF vs LeRF Fidelity consistently outperforms the other methods. Notably, Fidelity+Fine- tune shows a marked improvement over the original Fidelity, indicating that the fine-tuning step enhances the robustness of the evaluation. F-Fidelity further advances this by effectively addressing the OOD issue, resulting in the most faithful and reliable rankings of the explainers.
D.8 EMPRICAL VERIFICATION OF DETERMINING THE EXPLANATION SIZE
In this section, we demonstrate empirically that the Fiid metric can be used to deduce the size or sparsity of explanations, complementing the theoretical analysis of Section 4 Specifically, we consider the colored-MNIST dataset (Arjovsky et al. 2019), where the explanation corresponds to the pixels representing the digit in each image. To control the explanation size, we rescale and crop the image samples that the digit pixels occupy Y E {0.11 0.15, 0.2, 0.25] fraction of the total image area. Our goal is to show empirically that the FFid metric can recover the value of y, thus revealing the explanation size. A thrre-comvolution-layee model is used the pre-trained model.
21

---

Published as a conference paper at ICLR 2025
Image
Explanation
Degruded Explanations
noise-0.2
noise-0.4
noise-0.6
noise-0.8
noise-1.
Fidelity
ROAR
R-Fidelity
F-Fidclity
MoRF
LcRF
Macro
Correlation
-0.54
0.11
-0.94
-1.00
MoRF vs LeRF
Micro Rank
2.37
2.63
1.74
1.00
MoRF LeRF
Figure 2: Case study of sample in Tmny-mmagenet. We use GradCAM to obtain the explanation and get degraded explanations by adding random noise to the explanation. The noise level is a list of [0.2, 0.4, 0.6, 0.8, 1.0]. We visualize the Fidelity score and Correlation on different sparsity.
Fidelity
Fidelity
Speamann Comelation
Fine tune + Fidelity
5pearman Correlation
1.00
Spearman Correlation
Fine Tmee-tnn- Fidelity
F-Fidelity
F-Rdelity
0.98
0.96
-0.6
0.94
Fidelity
-0.8
0.92
Fine-tune Ridliiy
F-Fidelity
-1.0
-1.0
0.90
Macro
Micro
Macro
Micro
Macro
Micro
(a) MoRF vs GT
(b) LcRF vs GT
(c) MoRF vs LeRF
Figure 3: The ablation study between ground truth explanation size and FFid in F-Fidelity.
Following the terminology of Section the pixels can be divided into two tiers: those belonging the digit (forming the explanation) and those in the background. The explanation size is thus G = yld. According to Theorem [ we expect Fiid to increase for < Ci and decrease for 8 max (1l, xld)
To verify this behavior, we evaluate FFid across three settings a and five settings of 3. As shown in Figure 4] the ground truth explanation size can be identified from the flat area in Ridd curves. Two critical points emerge on the x-axis: the explanation size y and the point where the removal attributions equal the explanation size (3/a+). The region between these points forms plateau that enables reliable comparison of explanations. When o td C1, the point where FFid changes direction corresponds to the explanation size y, allowing us to recover the true explanation size by evaluating F Fidt over different values.
22

---

Published as a conference paper at ICLR 2025
While our theoretical analysis assumes uniform importance weights across explanation components, real-world upplications show some deviation from this idealized behavior, manifesting as smooth transitions rather than sharp changes in the FFid curves. Nevertheless, the empirical results val- idate the theoretical predictions and demonstrate the utility of F Fidd metrics in determining expla- nation sizes.
5.4
8-0.05
8-0.5
5-0.05
2-0.30 6:2
5-0.5
504
6-0.00
$w0.10
5-0.10
p-0.15
$.0.15
$o0.15
$50.15
p=0.0
2039
p-020
5-0.0
p-0.20
8-0.5
8-025
6310
8-025
56
(a) 7 = 0.1,a = -03
(b) 7 = 0.15, a+ 0.3 (c) 7 = 0.2, a 0.3 (d) 7 = 0.25, a
= 0.3
8-0.05
pe0.50
9-0.10
5-0.10
6:00
9-0.15
2-0.13
0-0.15
$50.15
8 am
252
0-2.0
p-020
B-025
2-025
5-0.0
0
$5022
2-0.00
009
631
00
630
21
30
4
(c) 7 0.
0.5
7 0.15, at 0.5 (g) 7 0.2, a+ 0.5 (h) 7 0.25, a+ 0.5
p-2:05
4-0.05
035
314
2.2
4-325
B0025
2-0 25
306
300
Leparatoe 201
n. Expanatio
1e 10 wate co
(i 7 0. 0.7
(k) 7 0 0.7 ( 0.15, a 0.7
(1) 0.25, a 0.7
Figure 4: The relationship between ground truth explanation size y and RFid in F-Fidelity. We compare different a and | in the evaluation stage. We observe that the ground truth can be inferred from the RFid+.
D.9 HYPERPARAMETER SENSITIVITY STUDY
In this part, assess the robustness of F-Fidelity to hyperparameter choices. We select subset of 400 samples with an explanation size ratio of 0.2 from the colored-MNIST dataset (Arjovsky et al.) 2019. we have three key hyperparameters: the number of sampling iterations N. the ratio a²//ar and B. For each hyperparameter, we explored range of values to understand their impact on the performance of F-Fidelity. We evaluated the method's performance using three Macro Spearman correlations, with additional detailed results on RFid RFid and Micro Spearman correlations provided in Appendix E.4
Figure presents the results of our hyperparameter sensitivity study. The findings demonstrate that F-Fidelity exhibits remarkable stability across a wide range hyperparameter settings. This con- sistency is observed across all three Macro Spearman correlations, indicating that the performance of F-Fidelity is not overly sensitive to specific hyperparameter choices.
D.10 COMPARING DIFFERENT EXPLANATION METHODS
To demonstrate F-Fidelity's 's ability to differentiate between explanation methods of varying quality, we conduct a comparative study between GradCAM and Saliency Map (Simonyan et al., 2013) on both Tiny-Imagend and ImageNet. GradCAM typically produces more focused explanations by utilizing class-specific gradient information at higher-level feature maps, while Saliency Map operates directly on pixel-level gradients.
23

---

Published as a conference paper at ICLR 2025
1.:
Speammn Carrelattion
0.5
MoRF LeRF
MoRT VS LeRF
MoRF VS LeRT
Speamen
0.
MoRF VS GT
00:
MoRF VS GT
0.0-
MORF VS GT
VS GT
-10
-1.
-10
30
50
150
0.55
0.30
0.15
0.20
0.55
020
0.50
0.40
0.50
0.00
0.70
0.80
Sampling Narmber
Hbyperperameterr F
[lppeperameter
(a) Ablation study on N
(b) Ablation study on B
(c) Ablation study on a
Figure 5: Ablation study sampling number N, 8, and a at a-. We report the Macro
Spearman rank correlations.
Our evaluation on Tiny-Imagene confirms this expected difference. Across our set, GradCAM achieves better performance with higher FFid (0.2286 vs 0.2020) and lower FFid (0.0547 vs 0. 0.1050) scores than Saliency Map. Note that both explanation methods are used to generate explanations with respect to the ground truth label. Thus, the FFid values are negative if the original prediction incorrect.
Table 11: Comparative evaluation of GradCAM and Saliency Map explanations using F-Fidelity on Tiny-lmagenet. We show five cases with their original images and corresponding explanation visualizations. The FFid and FFid scores averaged over different sparsity levels, with bold scores indicating better performance. The F Fiid and FFid scores of case are negative because the original image is misclassified. The bottom row reports metrics averaged across all samples in
the test set.
Explanation Visualization
Evaluation Metrics
Image
Original
GradCAM
Saliency Map
Method
FFid
FFid
GradCAM Saliency Map
0.8743 0,8638
0.1267
Case 1
0.3342
GradCAM Saliency Map
0.3838
0.029
Case 2
0.3143
0,0048
GradCAM Saliency Map
0.2095
0.0190
Case 3
0.0752
0,0314
GradCAM Saliency Map
0.4552
0.0486
Case 4
0.1095
0.1000
GradCAM Saliency Map
-0.0057
-0,0048
Case 5
-0.0086
-0.0029
GradCAM
0.2286
0.0547
Average over Images in The Test Set
Saliency Map
0.2020
0.1050
To verify our method's effectiveness on high-dimensional data, we further evaluate a subset of ImageNet (5000 images, 224x224x3 resolution) using ResNet-18 as the backbone. Unlike our pre- vious experiments where fine-tuned on training data and evaluated explainers set, here we fine-tune and evaluate on the same image set to enable controlled comparison between explainers. Note that this fine-tuned model is specifically optimized for this dataset and should only used to compare explainer performance within this set. Following the same protocol as our Imag-ema experiments, fine-tune the model using Adam optimizer with default settings and
separate test
24

---

Published as a conference paper at ICLR 2025
evaluate 500 sampled images from this set. As shown in Table 12 The results consistently show GradCAM outperforning Saliency Map, with higher FFid (0.1123 vs 0.0672) and lower Fidd (0.0046 vs 0.0091) scores. It shows that F-Fidelity successfully captures the qualitative differences between explainers, such as GradCAM's more focused attribution maps compared to Saliency Map's noisier explanations.
Table 12: Comparative evaluation of GradCAM and Saliency Map explanations using F-Fidelity on ImageNet dataset with ResNet-18. The FFid* and F Fid scores averaged over different sparsity levels, with bold scores indicating better performance.
Explanation Visualization
Evaluation Metrics
Image
Original
GradCAM
Saliency Map
Method
FFidt 1
FFid
GradCAM Saliency Map
0.3476 0.2362
0,0019 0.0057
Case
GradCAM Saliency Map
0.3981
0.0076
Case 2
0,3571
0.0105
GradCAM Saliency Map
0.5438
0.0257
Case
0.2162
0.0590
GradCAM Saliency Map
0.6190
0.0029
Case 4
0.1019
0,0162
GradCAM Saliency Map
-0.0010
-0.2467
Case 5
-0.1477
-0.0895
GradCAM
0.1123
0.0046
Average Over 500 Images
Saliency Map
0,0672
0.0091
D.11 COMPUTATIONAL EFFICIENCY ANALYSIS
Following the comparative setup between GradCAM and Saliency Map on Tiny-Imagenet, we duct a detailed computational efficiency analysis. Table [3] breaks down the computational time requirements for both ROAR and F-Fidelity. The key advantage of F-Fidelity lies its one-time fine-tuning cost while ROAR requires separate explaining and training processes for each explainer (totaling 7140s for two explainers), F-Fidelity needs only a single fine-tuning process (2925s) that can be reused across all explainers.
For the Imagenet dataset with ResNet backbone, ROAR's per-explainer cost includes generat- ing explanations for training samples (approximately 650-700s), model retraining (around 2900s), and evaluation (about 120s). In contrast, F-Fidelity requires only single fine-tuning step (2925s), followed by evaluation for each explainer (approximately 550s). This translates to total processing time of 4026s for F-Fidelity compared to 7378s for ROAR when evaluating two explainers, with the gap widening further more explainers are evaluated.
This efficiency gain becomes particularly significant when evaluating multiple explainers or con- ducting extensive ablation studies, making F-Fidelity more practical for real-world applications while maintaining superior evaluation quality as demonstrated in our previous experiments. The advantage is even more pronounced for computationally intensive explainers, ROAR requires
25

---

Published as a conference paper at ICLR 2025
'able 13: Computational time analysis for evaluating GradCAM and Salieney Map Tiny- Imagenet using ResNet as the backbone. While ROAR requires separate explaining and training processes for each explainer, F-Fidclity's fine-tuning cost is one-time.
Explaining Training Samples (100k)
Model Training/
Evaluation (2000 samples)
Total Time
Method
Fine-tuning (100 epochs)
ROAR
647s + 721s
2911s + 2861s
114s+124s 541s+560s
7378s
F-Fidelity
2925s
4026s
generating explanations for the entire training set (100k+ samples in our case), while F-Fidelity only needs explanations for the evaluation set. Moreover, our hyperparameter analysis shows that we can further reduce the evaluation time by decreasing the number of samples with N 10, the evaluation time per explainer drops from approximately 550s to 180s while maintaining robust performance. This flexibility allows users to balance between evaluation speed and precision based on their specific needs.
E DETAILED EXPERIMENTAL RESULTS
IMAGE CLASSIFICATION EXPLANATION EVALUATION
Experiments on CIFAR-100. We provide the detailed experiment results on the CIFAR-100 dataset. The CIFAR- 100 dataset is collection of 60,000 color images, each sized 32x32 pixels, classified into 100 distinct categories, with 600 images per class. contains 50,000 training images and 10,000 test images. For the ResNet backbone, in Table 14 compared different methods the Fid+ and Fid- metrics. From the figures, we found R-Fidelity and F-Fidelity methods have the advantage of distinguishing explanations with larger margin scores.
Similar results can be observed for the ViT backbone. We provide the results in Table 15
Table 14: Fidelity results on CIFAR-100 dataset with ResNet as the model to be explained. Fidelity
ROAR
R-Fidelity
F-Fidelity
500 snc 1:40
wabed.00
rewe
Fid+
3.8)
vde 116
Jilo 11
wb.3.00
10003.00
rebo1.00
Fid
3004.0
5005-.20
1000420
X60 140
802 0.40
1500 140
reve 0.4)
Vi
1001.00
5000-.00
1056100
Experiments on Tiny-Imagenet. Similar to the CIFAR-100, we provide the detailed experiment results Tiny-Imagenet in this section, which is scaled-down version of the ImageNet dataset, designed for image classification tasks. Tiny- Tiny-Imagenet contains 200 classes and 500 training images per class, 50 validation images, and 50 test images per class. The image resolution 64x64. From Table T6 and [7] we observe that compared to R-Fidelity and F-Fidelity, F-Fidelity and R-Fidelity have the best performance.
26

---

Published as a conference paper at ICLR 2025
Table 15: Fidelity results on CIFAR-100 dataset with ViT the model be explained.
Fidelity
ROAR
R-Fidelity
F-Fidelity
035
Fid
Fid
rebo0.3
Table 16: Fidelity results Tiny-Imagent dataset with ResNet the model to be explained.
Fidelity
ROAR
R-Fidelity
F-Fidelity
1ibe5.40 540
Fid
220
1..0.00
Nos..0 5.00
10
Fid
reled 66
reled 63
vbs110
Table 17: Fidelity results -iay-Im dataset with ViT the model be explained.
Fidelity
ROAR
R-Fidelity
F-Fidelity
wibe..c 1004-50
100e--000
rolbe-.0 co
Fid+
5009-.20
1056-22 20
1ec 1.40
robee E6
580-300
Sar Mepoetii
5a% Mal./eerr...oo mi
612
5.0
1000-.00
413
Fid
van 246
27

---

Published as a conference paper at ICLR 2025
E.2 DETAILED RESULTS ON TIME SERIES DATASETS
We conduct the experiments on the PAM and Boiler datasets with LSTM architecture. As Table [8] and show, our method distinguishes explanations with larger margin than R-Fidelity. Moreover, Fidelity and ROAR method fail to distinguish different explanations in the Boiler dataset.
Table 18: Fidelity results on PAM dataset with LSTM as the model to be explained. Fidelity
ROAR
R- R-Fidelity
F-Fidelity
10996-10
8:0
Fid
1-0
40
0.5
Fid
seheed 20
Table 19: Fidelity results on Boiler dataset with LSTM as the model to be explained. Fidelity
ROAR
R- R-Fidelity
F-Fidelity
Fid+
100 E40
N000-.00
robe- 60
reledd
400
Fid
7Sle-0.20
1200 0.40
(236-049
3.40
1319-100
(030-1.0
E.3 DETAILED RESULTS ON NLP
and with LSTM and Transformer, we find the Fidelity has consistent result which quite different than other tasks. Compared to R-Fidelity and F-Fidelity, the results are almost the same both in the SST2 and BoolQ datasets, which indicates in these two cases, the OOD problem can be ignored.
In Table 20
21
28

---

Published as a conference paper at ICLR 2025
Table 20: Fidelity results on SST2 dataset with LSTM as the model to be explained.
Fidelity
R-Fidelity
F-Fidelity
de 0.07 S0.0.2
nolce 0.00
0330
notee 0.00
Polsg-0.2 0.20
nolse 0.20
nater-0.00
Fid
50
30
notse
nose
nolse 0:40
nosse 60
0.00
0.36e:0.30
Fid
nelio-0 0.80
nosso 60
nakw-0..0 0.30
notie 8:80
0.00
noise 0.00
0.40
60
nclse-1.00
nollo 1.00
-216
-22
-0.00
Trp % Ebr unaiee See
To (eclaetts
Tap 5/ (Do andton Sex
Table 21: Fidelity results on SST2 dataset with Transformer as the model to explained.
Fidelity
R-Fidelity
F-Fidelity
Fid+
0.20
0.20
0.00
nolso: 0.00
3.20
6335
nokve-0 0.50
nolce 0.40
0033
nolse 0.30
noice 1.00
606
1.00
0.0
0.00
0.00
0.00
032
-0.02
Fid
Tokoe 0.20
ratee-0.4
Potse-1
nolse 0
nokis 0.50
-189
-0.00
75
Table 22: Fidelity results BoolQ dataset with LSTM as the model to be explained.
Fidelity
R-Fidelity
F-Fidelity
molie-n.00 Do
nale--..00
0.20
6:35
Fid
n0 ce 0.90
3.0
nakve-0.01
noise- 0.00
cotee-3
neise 0.00
msse-0000
nclue-0.70
nolue-2.20 20
-0.7%
notse 0:40
nolco 0.40
Fid
nntsse
0.80
-0075
nokas
-0225
notee-
nokse-1.00 1.00-.000
29

---

Published as a conference paper at ICLR 2025
Table 23: Fidelity results on BoolQ dataset with Transformer as the model to be explained. Fidelity
R-Fidelity
F-Fidelity
nolse-8.00
nolie-n on
molee-1220
notac 0:20
0.2
noise 0.40
noice 0.40
Fid+
nolse-0 60
315
n 0.00
nolse 0.30
nokve-0..0 0.50
3.0
1200-1100
noise 1.00
0.59
nolte-0.00
0.00
Dosc
Fid
nokoe 0.20
nolse
0.4e
-22
nolce-1.00
-0.15
noke: 0.50
nolse 0
-2.22
2
30

---

Published as a conference paper at ICLR 2025
E.4 DETAILED RESULTS OF HYPERPARAMEIN SENSITIVITY STUDY
In this section, we provide the detailed ablation study experiment results on the Colored-Mnist dataset. The experiment settings described Section D.9 We provide the detailed Fid+
Fid-, and three Spearman rank Correlation, same as the subsection E. As Table 24 show, our method is robust to the choice hyperparameters.
25 and 26
Table 24: The detailed ablation study results on sampling number N, with B = 0.1, a a- 0.5.
Sampling
Local Spearman Correlation
Fidt
Fid
Number N
nekss 0.00
noice-0.22
nascc 40
rass--..0
MORF VS LoRF Misll GIT
10
6:12
notes-0
rois-0.20
LeRP
0:5
0.00
-1.0
Top %. avarice Sob
Doperatton 20e
0399
30
030
nolee-di.00
noise 1.00
nolse 0.30
8333
8333
RORAB-G.20 0.20
rolse-0.8 0.44
ralse-0.50
ToGC 0.00
MORF vS
MARE Lest
Lesir VS GI
nokve-dl dU
2.0
0.55
noliz 0.20
0331
notis-
8220
MORF Leke
50
032
rolso-0.50
rolce-0
25
roice11.00 000
0.00
-2.0
58
15
53
25
832%
0.35
nolee-0.20
80
0.38
notie 1.00
nokes 0.50
nobse 0.40
notse-0.00 0.50
6357
0335
1al66-0.20
1000-0000
MORF VS GT
MoRT VS LWFT
GI
8331
0.00
-1.0
notee-0.00
0.29
0.35
8320
150
0.22
8315
1ole-0.20
VS
ndies-
FolsD-C.40
Toice-0.00
0300
relsc-0.50
0.00
31

---

Published as a conference paper at ICLR 2025
Table 25: The detailed ablation study results on hyperparameter 3 with N 50, a = a 0.5.
Local Spearman Correlation
B
Fidt
Fid
0.05
333
3.4
0.0000
0.00
MORF VS Lefk Manti un VS
3.2
00002
0.1
3.90
notse-1.00
Top (5 Cocianntion Soe
Top 55 Coslenriin
Top do Era
notse 00
20sde-0
0.10
3.00
mndee-0.20 0.20
2.200.20
VS.
notse 0.40
notee-O 0.40
Luff vs
neise 0.60
noliz 0.50
notse-0.00
3.00
nots-1100
nofse-1.00
LARF
0.15
notse-0.0
notse-O
nobse 0.80
nolse-0
nobe-11.0
notve-1.00
0.20
0.20
3.55
3.70
nodse 0.20
notee-- 60
nolse 8.00
colce 0.00
0.20
MoRF VS GT
Lattt OI
Molli Vs Latt
3.23
0:25
Rotee-B 80
01100
0.25
3.5%
holie 20
0.055
notoe 0.20
notse 0.00
noise- 2.40
8.000
nolce 0
nolse-0..4 0.40
LoBF
nelse 0.80
n
0.00
nolse-1
25
32

---

Published as a conference paper at ICLR 2025
Table 26:
The detailed ablation study results on hyperparameter a at with N 50,
B 0.1.
Local Spearman
Fid+
Fid
Correlation
0000
0.0
00
Musii Vi Laff! Mode VS
0.2
0.0
0.1
0000
0.00
0.00
no 1.00
0.00
noise 1.00
-1.0
0.00
0A05
0.3
rolse-0.00
0002
1.00.000
0.70
Noli VS
MoRF VS LefF
0.027
00 0.40
oren
netue-1
0.80
noise 0.40
LeBT GT
0.00
-0.05
Cellanation
Collenaton
0.54
1.0
0002
0.10
0.4
0.00
0002
001-
no.se
10:00-0.00 80
nobe- 0.50
nolse
-0.5
0.00
no 1.00
-0.00
notse 1.00
Explaration 2:0
28
50
dar
50
0005
nalSe 0.00
0.00
MofF VS LeRF
0.5
0.50
0002
Mollf VS or
0.00
0000
0.00
10:00-1.00
ncow-11000
-10
0.1%
0005
0.0000.00
Latt"
0.6
0.00
0.50
n0.00 0.20
1o5S--0.0 0.60
10:00-000
0005
nebe: 0.60.60
notoe 0.00
0.20
03
MoRF VS
LORF VS GI
nete--..0
0.90
-0000
10.00
0005
VS
0.7
0.50
10300-..40
0000
nobse
0.70
(0) 0,80
nokic 0.80
0.00
no mm 1.00
0.00
nobon 1.00
/Coleration
50
23
50
0.8
0.50
0.15
10/ba-0.20 20
0012
0045
1.20-02.0 0.20
Robee 0.00
Mott VS leff
MoFF VS GT
Ludft VS.
0160
33


Reviews:
[{"review_id": "uv7xRhK8ZQ", "reviewer": "Reviewer_NNY1", "summary": "The authors propose a fidelity evaluation framework which utilizes an explanation agnostic fine tuning and a random mask generator which ensures that the generates input is not OOD. The efficacy of the proposed method if shown in various data types like images, time series etc.", "questions": "Questions are as below\n\n#### W1:\n- a) Provide specific examples of how \"removal\" is implemented for different data types like images, tabular data, and text.\n- b) Give a more detailed explanation of how bounding the removed portion to a fraction of the input, rather than the explanation, helps mitigate OOD issues.\n\n#### W2:\n- a) Elaborate on how the random masking process relates to or simulates small perturbations, particularly in the context of image data.\n- b) Provide examples or visualizations that demonstrate how this masking approach compares to traditional perturbation methods.\n- c) Present empirical evidence or theoretical justification for why their masking approach is an effective way to improve classifier robustness.\n\n#### W3:\n- a) Provide empirical evidence or theoretical justification for why their fine-tuning process preserves the original classifier's decision boundaries.\n- b) Compare the original and fine-tuned classifiers' outputs on a test set to demonstrate the degree of change.\n\n#### W4:\n- a) Provide theoretical or empirical evidence demonstrating the equivalence between explanations of the original and fine-tuned classifiers.\n- b) Include a comparison of explanations generated for both classifiers on a set of test examples.\n\n#### W5:\n- a) Provide a step-by-step description of how they generate or obtain the ground truth (GT) explanations, including any assumptions made in this process.\n- b) Include examples of GT explanations for different data types.\n\n#### W6:\n- a) Provide a detailed analysis of why F-Fidelity underperforms on this specific dataset and model combination.\n- b) Investigate potential reasons, such as dataset characteristics, model architecture specifics, or hyperparameter settings that might contribute to this result.\n- c) Discuss the implications of this finding for the broader applicability of their method.\n- d) Consider conducting an ablation study to isolate the factors contributing to this performance difference.\n\nThe paper addresses an important aspect of fidelity evaluation and the issue of OOD while doing it. Overall, I like the central theme of the paper and that’s why I am leaning towards mild acceptance but I would like the authors to address the points in the weakness section and the questions in this section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) The paper deals with important aspect of fidelity evaluation\n2) The authors discuss about OOD issue during fidelity evaluation\n3) The authors propose an approach to handle the issues and show the performance in different data types like images, time series and NLP.", "weaknesses": "W1) In section 3, it is mentioned in lines [215 to 222] that the size of the removed part is upper bounded by a fraction of the input and not a fraction of the explanation. It would be helpful to the readers if the authors could explain what they mean by \"removed,\" as it might mean different in tabular data and images. Further, the authors should also explain how their strategy of selecting a fraction of input and not explanation addresses the OOD issue. \n\nW2) In lines[224:235], the authors use fine tuning to address the problem of robustness of the classifier for small perturbations(mentioned in lines [213:214]). However, at least from the perspective of images, it is unclear how applying a mask of {0,1} on random patches of an image can be considered as a perturbation.\n\nW3) Further, the fine-tuning process mentioned in lines[224:235] seems like retraining the classifier with occlusion (for image classifiers). The authors should explain how their method won't change the decision boundary drastically so that their approach can be called fine-tuning and not training a different classifier altogether.\n\nW4) The authors should explain why explaining the fine-tuned classifier is equivalent to explaining the original classifier (as mentioned in line[224])\n\nW5) For the ease of readers, the authors should explain more about GT and how they extract it(in Section 4)\n\nW6) It needs to be clarified to see why F-Fidelity performs worse than vanilla Fidelity and R-Fidelity for SST2 dataset with LSTM (Table 8). The authors are requested to explain this phenomenon in detail for the ease of readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p9oVWdeNNB", "reviewer": "Reviewer_S4br", "summary": "The authors propose a new attribution evaluation framework which attempts to solve existing issues in current metrics such as the OOD problem from perturbation metrics, and the information leakage of fine-tuning methods like ROAR. To do this, the authors propose limiting the size of the perturbations made to each image and performing fine-tuning on images randomly masked under this perturbation size limitation.", "questions": "The ablation study provided regarding B selection suggests this value can approach 1 (i.e. this is the same as if the value was not implemented) and the correlation scores would be good. Is this true? If so, what use does B have? In addition, what value of B is used and how was it selected? \n\nWhat is the intuition for fine tuning with the image perturbation limitation? I do not see why fine-tuning under image perturbed by random masks instead of perturbed explanations is better either. It is not obvious that this skirts the OOD problem. \n\nWhat is the value of being able to extract the explanation size? The motivation for this is lacking and it is not clear why it is important or interesting.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "From their experiments which cover a broad range of applications and datasets, it seems this new method does improve over existing methods in rank correlation tests, indicating its ability to fairly and consistently sort attributions under evaluation. \n\nThe solutions presented are simple.", "weaknesses": "I do not think the intuition behind the solutions is entirely clear and I do not think the presentation of the solutions is clear.  \n\nThe choices for multiple hypermeters are not indicated. \n\nMore attribution methods should have been employed for a better representation of the existing space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3690AS2Tk", "reviewer": "Reviewer_9DPh", "summary": "The authors presented F-fidelity, a new metric for evaluating the faithfulness of attribution methods. Traditional evaluation methods such as MoRF and LeRF suffer out-of-distribution(OOD) issue, and ROAR method suffers information leakage issue. F-fidelity overcomes these limitations through two strategies. 1. explanation-agnostic fine-tuning, which fine-tunes the model using stochastic masking operations. 2. a controlled random masking operation to overcome the OOD issue, by applying a random masking operation conditioned on the explainer's output so that removal of input features does not produce OOD samples. The authors conducted controlled experiments with other explainers. Moreopver, they showed that F-fidelity can be used to determine the sparsity of influential input components.", "questions": "1. I couldn't find how Pan et al., 2021, or Zhu et al., 2024 utilized macro correlation in their work. Can you explain how they employed macro correlation and how it relates to your citation of their work?\n\n2. In Section 6, it seems that both MoRF and LeRF conditions would be satisfied equally. This is because deleting or inserting features based on the gamma value would maximize the MoRF-LeRF difference. I'm uncertain whether this experiment effectively demonstrates that F-Fidelity can distinguish explainers that are closer to the ideal Shapley explainer than other metrics. Could you clarify how this experiment supports that claim?\n\n3. Regarding computational resources, I would like to know how much computational effort your method requires. One of the primary reasons ROAR isn't widely adopted is not just due to information leakage but also because of its significant computational cost. It would be helpful if you could provide details on the computational resources required for F-Fidelity and how they compare to those needed for ROAR.\n\n** Typo : in figure 1, the y-axis label is RFid.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well written, especially introduction section and preliminary section.\n\nThe paper effectively tackles the OOD problem inherent in traditional removal-based evaluation methods. This is a crucial issue because OOD inputs can lead to unreliable assessments of an explainer's faithfulness.\n\nThe authors conducted extensive experiments across broad data, such as images, time series, and natural language processing.\n\nThe method is well mitigate limitation of prior methods; Addresses the information leakage issue and OOD issue at once.\n\nThe paper provides theoretical analysis demonstrating that F-Fidelity can recover the sparsity of the ideal SHAP explanations.\n\nThe paper introduces a novel approach by assuming that an explainer is better the closer it is to an ideal Shapley value explainer. This assumption is appealing because Shapley values provide a theoretically sound basis for feature attribution.", "weaknesses": "Lack of Visualizations Demonstrating F-Fidelity's Effectiveness. The paper does not include figures or visualizations that illustrate the effectiveness of F-Fidelity. Providing attribution maps or examples where there is a significant discrepancy between F-Fidelity and other evaluation metrics would enhance understanding and make the results more tangible. Visual comparisons could help readers better appreciate the advantages of F-Fidelity over existing methods. For example, Attribution map between samples that have discrepancy between F-Fidelity and MoRF - LeRF.\n\nNovelty issue. Though theoretical availability of extracting the size of the ground-truth explanation is novel, the core difference between F-fidelity and R-Fidelity is introducing an upper bound on the fraction of input elements removed. This extension seems an incremental refinement rather than a fundamental novel idea.\n\nLimited Explanation of Ground Truth. The paper's approach to establishing ground truth rankings involves using Integrated Gradients with systematically added noise levels. However, Integrated Gradients is known to be less faithful than some state-of-the-art explanation methods, such as the LRP (Layer-wise Relevance Propagation) family or LayerCAM. This raises concerns about whether the ground truth used is an appropriate benchmark. As the authors discuss ideal Shapley value explainers in Section 5, utilizing Shapley values as the ground truth might have been more appropriate to align with their theoretical framework.\n\nInsufficient Clarification on the Importance of Macro Correlation. The paper could benefit from a more thorough explanation of why macro correlation between different evaluation methods is important. It was not immediately clear how macro correlation demonstrates the usefulness of F-Fidelity. Including figures or visual aids that depict the relationship between macro correlation and the effectiveness of F-Fidelity would help clarify this point and strengthen the argument.\n\nLimited Variety of Explanation Methods Evaluated. The experiments involve a relatively small number of explanation methods. Comparing F-Fidelity scores across a wider range of explainers would provide a more comprehensive evaluation. For instance, methods like LayerCAM and LRP family are known to have higher fidelity than GradCAM, and GradCAM typically outperforms the pure gradient. Demonstrating whether F-Fidelity can reflect these known differences in fidelity would enhance the paper's validity.\n\nPractical Challenges in Determining Explanation Size (c₁). In the section 6, the size of the most influential tier (c₁) is assumed to be known. However, in real-world applications, c₁ is not readily observable. It would have been beneficial if the authors had demonstrated how c₁ could be estimated or inferred from practical data, such as through experiments on datasets like ImageNet. Providing practical guidance on determining c₁ would make the theoretical contributions more applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)