The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus
Authors: Anna Hedström
Philine Lou Bommer
Kristoffer Knutsen Wickstrøm
Wojciech Samek
Sebastian Lapuschkin
Marina MC Höhne
Venue: Accepted by TMLR ()

Abstract: One of the unsolved challenges in the field of Explainable AI (XAI) is determining how to most reliably estimate the quality of an explanation method in the absence of ground truth explanation labels. Resolving this issue is of utmost importance as the evaluation outcomes generated by competing evaluation methods (or ``quality estimators''), which aim at measuring the same property of an explanation method, frequently present conflicting rankings. Such disagreements can be challenging for practitioners to interpret, thereby complicating their ability to select the best-performing explanation method. We address this problem through a meta-evaluation of different quality estimators in XAI, which we define as ``the process of evaluating the evaluation method''. Our novel framework, MetaQuantus, analyses two complementary performance characteristics of a quality estimator: its resilience to noise and reactivity to randomness, thus circumventing the need for ground truth labels. We demonstrate the effectiveness of our framework through a series of experiments, targeting various open questions in XAI such as the selection and hyperparameter optimisation of quality estimators. Our work is released under an open-source license (https://github.com/annahedstroem/MetaQuantus) to serve as a development tool for XAI- and Machine Learning (ML) practitioners to verify and benchmark newly constructed quality estimators in a given explainability context. With this work, we provide the community with clear and theoretically-grounded guidance for identifying reliable evaluation methods, thus facilitating reproducibility in the field.

Published in Transactions on Machine Learning Research (06/2023)
The Meta-Evaluation Problem in Explainable Al: Identifying Reliable Estimators with MetaQuantus
Anna Hcdström 1,6,1
hema.heestroemmeee Werelllinn de philinee. bommer atu-berlin. de kristoffer. k. wickstrom @ust. no wojciech. samek samek@hhi. frounhofer. de sehastion. lapuschhin @hhi, frounhofer de mhochne@olt Botts-otsdooo-....oooooooo de
Philine Bommer 1.6
Kristoffer K. Wickstrøm
Wojciech S.m
Sebastian Lapuschkin
Marina M.-.. Höhne
Department of Electrioal Engincering and Computer Science, TU Berlin BIFOLD Berlin Institute for the Foundations of Learning and Data Department of Physics and Technology, UtI the Aretic University of Norway Department of Artificial Intelligence, Fraunhofer Heinrich-Herte--Ne Institute Department of Computer Science, University of Potsdum
UMI Lab, Letbniz Institute of Agricultural Engineering and Bioeconomy e. V. (ATB)
corresponding authors
Reviewed OpenReview: httpp:///ppaaaerreer.eoo/oorrttp
Abstract
One of the unsolved challenges in the field of Explainable Al (XAI) is determining how to most reliably estimate the quality of an explanation method in the absence of ground truth explanation labels. Resolving this issue of utmost importance as the evaluation outcomes generated by competing ovaluation methods (or "quality estimators"), which aim at mea- suring the same property of an explanation method, frequently present conflicting rankings. be challenging for practitioners to interpret, thereby complicating
Such disagreements their ability to select the best-performing explanation method. We address this problem through meta-evaluation of different quality estimators in XAI, which we define as "the process of evaluating the evalualion melhod". Our novel framework, MetaQuantus, analy- ses two complementary performance characteristics noise and reactivity to randomness, thus circumventing the need for ground truth labels. We demonstrate the elfectiveness of our framework through a series of experiments, largeting various open questions in XAI such the selection and hyperparameter optimisation of quality estimators. Our work is released under an open-source license* to serve as devel- opment tool for XAI- and Machine Learning (ML) practitioners to verify and benchmark newly constructed quality estimators in a given explainability context. With this work, we provide the community with clear and theoretically groounddd guidance for identifying reliable evaluation methods, thus facilitating reproducibility in the field.
quality estimator: its resilience to
1 Introduction
Since Explainable AI (XAI) is intended to increase trust and transparency AI systems, is necessary evaluate the performance of proposed explanation methods to ensure their reliability. In the context of black- box Machine Learning (MT.) models such neural networks (NNs) -where the input to output mapping is not explicitly known or interpretable by user (Benitez et al., 1997; Bellido & Fiesler, 1993; Lipton, 2018) there is generally an absence of ground truth explanations to understand the model's decision. This makes it difficult. to evaluate the performance explanation methods since the exact outcomes of explanations
Code available the Hub repository: http::///jtttt.cccmmmmmaaatttttt

---

Published in Transactions on Machine Learning Research (06/2023)
The Problem Meta Evaluatvlnnn
Evaluating P:0(s,,,::2 =)
Explaining (1)
Modeling f(x: 6) y
"digit 2"
GR
GS
IG
FC (1): 0.001 (R=3)
0.150 (R=2)
0,175 (R=1)
PF (1)= 2,855
(R-1) 5,121
(R-3) 3,433 (R-2)
Figure 1: An illustration Meta- Evalnation through three phases: (i) Modeling, (ii) Explaining and (iii) Evaluating. (i) A ResNet-9 model (He et al., 2016) is trained to classify digits from 0 to 9 on Customised-MMNIST dataset (Bykov et al., 2022) (i.e., MNIST digits pasted on randomly sampled CIFAR-10 10 backgrounds). (ii) To understand the model's prediction,
several explanation methods including Gradient (Morch et al., 1995; Baehrens et al., 2010), Inte- grated Gradients (Sundararajan et al., 2017) and GradientShap (Lundberg & Lee, 2017), which are distinguished by their respective colours. (iii) To evaluate the quality of the explanations, apply different estimators of faithfulness such as Failkfuiness Corvelation (FC) (Bhatt et al., 2020) and Pinel-Flipping (PF) (Bach et al., 2015), which return correlation coefficient and an AUC score, respectively. However, since the scores vary depending on the estima- tor, both in range and direction, with lower or higher scores indicating more faithful explanations, interpreting the resulting faithfulness scores remains difficult for the practitioner.
oftentimes remain unknown and thus unverifiable (Dasgupta et al., 2022). Without consensus around how to define the quality "correctness" of an explanation method, varicty of XAI evaluation procedures have been proposed. These efforts most commonly involve (i) measuring the extent to which desirable properties are fulfilled by the explanation method, e.g., through faithfulness or robustness analysis (Samek et al., 2017; Sundararajan et al., 2017; Montavon et al., 2018; Agarwal et al., 2022a), (ii) generating well-defined, synthetic settings where explanation labels are simulated (Yang & Kim, 2019; Arras et al., 2022; Liu et al., 2021a) or, (iii) evaluating explanations based on visual aligmment with a human prior (Smilkov et al., 2017). Most relevant our work is the first category of evaluation methods or "metrics" (Hedström al., 2023), where the goal is to estimate the quality of an attribution-ban importance method. Henceforth, we refer to these evaluation methods as "quality estimators", or simply "estimators" of explanation quality.
explanation method, also known feature-
The abundance of explanation methods and an ever-growing number of quality estimators, combined with little guidance on how to use them, have caused practitioner confusion within the XAI and ML communities Krishna al. (2022). Strong claims of identified failure modes for explanation methods with assertions of which methods pass or fail (Adebayo et al., 2018; Dombrowski et al., 2019; Sixt et al., 2020), followed by rebuttals (Sundararajan & Taly, 2018; Yona & Greenfeld, 2021; Binder et al., 2022), are ever-present. To answer the question of "which explanation method to use for given task", we must first be able to answer "how to reliably define and mcasure the relevant qualitics that an explanation method should fulfill". While preliminary efforts exist to address this issue (Brunke el al., 2020; Tomsett el al., 2020; Gevaert el. al., 2022; Rong et al., 2022), to the best of our knowledge, there is currently no comprehensive solution that thoroughly evaluates the various estimators used compare, seleet and reject different explanation methods XAI. Previous efforts at addressing this issue have been limited in scope and do not provide a thorough theoretical motivation. With this work, aim to fill this critical yet largely neglected research gap.
2

---

Published in Transactions on Machine Learning Research (06/2023)
In this work, characterise the problem of meta-evaluation, which we refer to as "the process of evaluating the evaluation method". This problem arises we select and quantitatively evaluate different explanation methods for a given model, dataset, and task and where conflicting evaluation outcomes are produced. As illustrated in Figure 1, we can apply various estimators to compare the explanation methods' faithfulness, which measures how closely the explanations align with the predictive behaviour of the model (the experi- mental details are described in Appendix A.4). However, the estimators rank the same explanation methods differently, e.g,, the Gradient method (Morch et al., 1995; Baehurns al., 2010) is both ranked the highest (R=1) and the lowest (R=3) depending the estimator used. With disagreement in evaluation outcome about which explanation method is superior (Krishna et al., 2022) coupled with little to no guidance on how to identify a high-quality estimator (Wang& & Wang, 2022), practitioners may unknowingly choose an infe- rior estimator which ultimately results in a selection of an explanation method that presents a less faithful explanation to the end user.
To address the issue of ovaluation disagroement in XAI, we propose a simple yet comprehensive framework which primary purpose is provide an objective, independent view of the estimator's performance by mela- evaluating it against two failure modes: resilience to noise (NR) and reactivity to adversary (AR). Similar to how software systems undergo vulnerability- and penetration tests before getting deployed in a larger system, we apply this framework to stress test the estimators. If vuinerabilities in the quality estimator are discovered, e.g., high sensitivity to noise in input or low reactivity to randomness, appropriate actions can be taken to improve the estimators. The contribution of this work is three-fold.
First, we provide clear argument for why meta-evaluation of quality estimators is challenging (Sec- tion 2.2), emphasising the importance of reliability analysis as a tool to analyse estimator behaviour.
Second, based on these findings, we propose a framework to meta-evaluate the performance of XAI estimators (Section 3), with a sound theoretical foundation and agnosticism towards various network architectures and attribution-based explanation methods.
Third, through experiments on various explanation methods, datasets, and models, we demonstrate that our framework can solve a range of XAI-related tasks such as estimator selection and hyperpa- rameter optimisation, generating novel insights into estimators' behaviour (Section 6).
We find it important to point out that we have no interest in developing yet another quality estimator. The real need in our community lies in developing mota-evaluation schemes to validate and determine the reliability of the quality estimators that already been developed. It is surprising to us that very little effort has so far been directed towards this important area of analysing their performance. With this work, we aim to support the process of selecting a quality estimator in a given explainability context in order to provide more clarity and guidance on how to effectively evaluate explanation methods.
1.1 Related Works
Despite much activity towards the development of estimators to evaluate explanation quality (Bach et al., 2015; Sundararajan et al., 2017: Bhatt et al., 2020; Nguyen & Martinez, 2020; Rieger & Hansen, 2020; Arias- Duart et al., 2021) and benchmarking tools (Liu et al., 2021a; Agarwal et al., 2022b; Hedström et al., 2023), limited attention has thus far been given to analysing the performance of the estimators themselves. Only recently, increased attention has been raised on the intricacies that come with XAI evaluation, for example, the contributions of Brunke al. (2020); Brocki & Chung (2022); Rong et al. (2022) emphasise the difficulty that comes with parameterising estimators. Another issue with evaluation was brought to light by Neely et al. (2021); Krishna et al. (2022), which revealed that explanations frequently disagree in their ranking of fcatures. Additionally, several independent rescarch groups were able to identify empirical "confounders" (Sundararajan & Taly, 2018; Kokhlikyan et al., 2021; Yona & Greenfeld, 2021; Binder et al., 2022) affecting the well-adopted Model Parometer Randomisation test (Adebayo et al., 2018). From these publications, it seems worryingly "casy to get it wrong" when comes to evaluating explainable methods empirically. There still remains lot of ambiguity when it comes to determining what makes up a good or bad quality estimator (Wang& & Wang, 2022).
3

---

Published in Transactions on Machine Learning Research (06/2023)
Within the scope of evaluating quality estimators, preliminary efforts exist but a unified effort is required. Since there is little to no consensus on how to determine the true value of quality estimator- whenn new estimator is introduced, it is often assessed based on single perspectives, e.g., by randomisation experiments (Rieger & Hansen, 2020; Arias-Duart et al., 2021) or ranking consistency (Rong et al., 2022). All of these mentioned works are undoubtedly steps the right direction, but what is missing is a broader, more comprehensive framing of what a quality estimator ought to fulfil. With this work, we aim to fill this gap.
2 Preliminaries
In the following, we derive a mathematical definition of the evaluation problem in XAI by outlining the key elements required to perform quality estimation on given explanation method. In the succeeding section, we discuss the Challenge of Unverifaabiitty (CoU) which explains why meta-evaluation is theoretically difficult. All notation used throughout this paper can be found in Appendix A.7.
2.1 The Evaluation Problem
Consider supervised classification problem² where we have black-box model f parameterised by 0 that has been trained on given training dataset Xur = {(01, v1)...., (ZN, (z), output class y E (1,..., C), with a trained functional mapping such as:
} to map an input æ E RD to an
[(x; = 0.
(1)
More generally, can define the model function f f X + Y that maps inputs from the instance space X to predictions in the label space Y with æ E X and 9 E Y. Let UF denote the function space such that f E F. To quantitatively estimate the performance of model f, we compute the prediction error on given test dataset X where there exists a label y for each prediction 9. To understand the reasoning of the model ʃ behind a certain prediction ú, we can apply one of the many proposed local explanation methods (Smilkov et al., 2017; Zeiler& & Forgus, 2014; Sundararajan al., 2017; Selvaraju et al., 2020; Bykov et al., 2022) as follows:
(<x f. é
where &: RR x x RD is an explanation function that is parameterised by À and which distributes an altribution to each individual feature in x according to its importance, typically visualised in an explanation map è E RR. A broad variety of explanation methods fall within the scope of & such as gradient- grasee (Smilkov ct al., 2017: Sundararajan et al., 2017; Bykov ct al., 2022), back peeee--oopggatin-bbeeee based (Bach et al., 2015), model-agnostic (Zeiler & Fergus, 2014; Lundberg & Lee, 2017), local surrogate (Ribeiro et al., 2016), attention-based (Chefer et al., 2021; Covert et al., 2022), well prototypical explanation methods (Si- monyan et al., 2014). Let IE denote the space of possible explanations with respect to the model such that E
Similar to how we compute the prediction error to estimate the performance of a model f, to evaluate the quality of the explanation function 4, we compute the explanation error, requiring a ground truth explanation e. These labels are, however, generally not available for black-box ML models and in particular NNs² since their inner workings are considered uninterpretable (Bellido & Fiesler, 1993; Benitez et al., 1997; Dasgupta et al., 2022). Thercfore, XAI rescarchers and ML practitioners must resort to indirect approaches to estimate the quality of a given explanation, e.g., by measuring the explanation's 's relative fulfilment of certain human-defined properties. Recent work by Hedström et al. (2023) has proposed to group these properties of explanation quality into six categorics; (a) faithfulness, (b) robustness, (c) localisation, (d) randomisation, (e) complexity, and (f) axiomatic estimators which provide a natural framework to compare and analyse explanation quality. A summary of these explanation quality categories can be found Appendix A.2 (see Equations 11-14).
Since classification lasks commonly encountered in the XAI community, is chosen to illustrate the Evaluation Problem. However, as discussed in Appendix A.1.1, our statements also apply other prediction scenarios.
Even toy- expert-annotated ground truth XAI datascts where features are known priori (Arras al., 2022; Yang & Kim, 2019), the explanation labels provided may not accurately reflect the model's decision-making process. Only with perfect knowledge of the model can true ground truth explanation labels be obtained.
4

---

Published in Transactions on Machine Learning Research (06/2023)
Verifiable spaces n E (3),0,,0
Unverifiable spaces U (F),O),EE
fg
V
e=e
4-
Tigure 2: A visual representation of the conditional dependencies between variables in quality estimation (Equation 3). The information flows from modelling to explaining and evaluating the explanations, ie., & e f, which is indicated by the direction of the arrows in the directed acyclic graph (DAG). The colours indicate if the spaces have verifiable (black) or unverifiable outcomes (red).
We provide generalised notation for quality estimation of attribution- based explanation methods follows. Let V, E RD x F x Y R be quality estimator that is parameterised by t and takes one explanation and returns one scalar value ("quality estimate") to indicate the quality of the explanation. The evaluation of an explanation, i.e, quality estimation, can be written as follows:
20 v. -
where Vi represents the quality estimalor and the whole space of possible estimators is denoted V E O. Mathematical descriptions such estimators can be found in Appendix (see Equations A.3).
2.2 The Challenge of Unverifiability
The goal of quantitative XAI evaluation is to provide an objective measure of the quality of an explanation. However, due to missing ground truth, the quantitative assessment of neural network explanations remains non-trivial. To clarify where this difficulty arises, we represent the process of quality estimation as a directed acyclic graph (DAG), as seen in Figure
Here, each node represents a random variable and the edges represent the relationships between the variables, with the uncertainty of a parent node propagating to its child node. We separate the nodes between verifiable- and unverifiable spaces. The verifiable spaces are spaces where ground truth labels are available, i.., Ω E (1X), {x, (x.F)) and the unverifiable spaces include spaces where there is an absence of labels, i.c., U ({E), (0), ().
As indicated by the direction of the arrows, a key observation is that in quality estimation there exists a conditional dependency between the variables of modelling, explaining and evaluating (the explanations). This further means that since the evaluation function applied to the results of the unverifiable explanation function, the evaluation outcome also renders unverifiable. We refer to this phenomenon as the Challenge of Unverifiability. Another key observation is that cannot determine the accuracy validity of an estimator (i.e., whether it actually measures the intended quality) since such assessment requires access to ground truth labels. However, as reliability analysis does not depend on the availability of ground truth labels, it is still possible to study the reliability of an estimator, which refers to its overall consistency ("does this estimator produce similar results under consistent conditions?"). This can be achieved by repeatedly measuring the evaluation outcomes that result from fixing the unverifiable parameters and functions and only varying the elements of the verifiable spaces. In the following, we will use the distinction between verifiable- and unverifiable spaces systematically and controllably mcasure the performance of quality estimators.
3 A Meta-Evaluation Framework
While the Challenge of Unverifiability makes meta-evaluation of quality estimators challenging, it is still possible to study the performance characteristics of an estimator through the lens of reliability. To this end, we developed threc-step framework, which higher-level evaluation scheme that examines quality istimators that themselves have been used to evaluate particular explanation method.
5

---

Published in Transactions on Machine Learning Research (06/2023)
3.1 Defining Failure Modes
Without ground Iruth information, we cannol validate or optimise the quality estimators against what we want them to fulfil, but we can instead articulate edge-case scenarios or behaviours that we do not want them to exhibit. For this purpose, we formulate failure modes which are described in the following. Failure Mode 1 (Noise Resilience). A quality estimator should be resilient to minor perturbations of its imput parameters.
Similar to the stability (or "robustness") property of explanation functions (Agarwal et al., 2022a; Montavon et al., 2018) and especially Lipschilz Conlinuily (A (Alvarez-Melis & Jaakkola, 2018; Yeh et al., 2019), where small changes in the input should only lead to small changes in the explanation, noise resilience (NR) evaluates the extent to which a quality estimator is robust towards minor perturbations of its inputs. Following our general perturbation Definition in Appendix A.2, we define a minor perturbation PM of any verifiable space as follows:
Definition 1 (Minor Perturbation). Let Po(w) be perturbation function of E 0, 0 f(x; 0) be the original prediction of the network and y' be the prediction after the perturbation. Then Pn(w) is minor PM (PM())) (0)) (f(x (z): PF (0))}, Ece e < such that:
if 0, o E H (z); (f(=; PM
|1 v'llp c
For classification, we employ L1-norm with p 1, thus, Definition states that the predicted label y' stays unchanged after the perturbation, i.c., = y'. Similar to works by Brunke et al. (2020); Brocki & Chung (2022); Rong et. al. (2022), measure the vulnerability of quality estimators to variations or "minor confounds" in the estimator. However, in contrast to these aforementioned works, we only perturb in the verifiable space by mcans of mcasuring the change in the model decision on a sample before and after perturbation, and thus we can control and directly measure the strength of the perturbation. Accordingly, to quantitatively examine Failure Mode 1, we expose the estimator to perturbations with small or minor impacts. Complementary testing an estimator's resilience noise, we also formulate a second failure mode to test whether quality estimator produces significant change when exposed to disruptive perturbation, i.e., randomisation to any of its inputs.
Failure Mode 2 (Adversary Reactivity). A quality estimator should be reactive to disruptive perturbotions of its imput parameters.
Previous research has noted that the estimators' scores should be conceivably different when produced for a random explanation (Rieger & Hansen, 2020) or a randomly initialised model (Arias-Duart et al., 2021). Our approach is similar in that it also seeks to disrupt the explanation process. However, since perturbation strength in the verifiable spaces, we can make more well-grounded claims about the expected outcomes of perturbation. Thcoretically, we define disruptive perturbations PP contrary to Definition 1. Definition 2 (Disruptive Porturbation). (P.(w) be a perturbation function of w E n, f = f(x; f) be the original prediction of the network and 3/ be the prediction after the perturbation. Then Po(w) is disruptive (x)P)))))))) (z); (=; pp((P))))))) (0)) (f(P) (x)) (x): (0)))) Bee R, 1 such that:
control the
Pf. if 9. y' (If(P)
|10 - 11--'11b >c
In a classification context, Definition 2 implics a change in the predicted class label. Figure 3 illustrates the main difference between minor and disruptive perturbations, which is that the decision boundary remains uncrossed or crossed, respectively. In Appendix A.1.1, we expand the Definitions and 2 other problem settings such as multi-label classification and also discuss how adversarial attacks relate to these definitions.
Using Definitions 1 2, can generate perturbed quality estimates g' by applying minor disruptive perturbation on the verifiable spaces in the input, model, or input- and model spaces simultancously
(4) where the superscript of the perturbation function, {M, D) indicates the perturbation strength. For simplicity, we omit the hyperparameters t, A from Equation 4. By repeating this perturbation (Fquation 4)
e ( Ps()) ( (O), ( (x), 0)) }.
6

---

Published in Transactions on Machine Learning Research (06/2023)
x
Input space LX]
Model space [F]
Figure 3: An illustration of minor versus disruptive perturbation in the different spaces (left: X, right: F) for classification task. The direction of the arrows shows how the respective perturbations are realised, where blue and red colours indicate minor- or disruptive perturbation, respectively. The minor perturbation keeps the decision boundary intact, either by perturbing a sample x= M (left) or perturbing the model itself fd (right). The disruptive perturbation implies that the decision boundary is crossed either through sample xD (left) or model fo (right).
multiple times, we gather sets of perturbed estimates for meta-evaluation analysis. In the next section, we provide a detailed description of how this analysis performed.
3.2 Formulating Consistency Criteria
To determine whether a quality estimator appropriately circumvented a failure mode, we can measure the similarity of its quality estimates before and after the perturbation. After minor perturbation, when testing for noise resilience, perturbations, when testing for reactivity to adversary, we would anticipate a large response to information annihilation of the explanation process by means of scores being dis-similarly distributed. We formalise this idea in our Intra- Consistency (IAC) criterion as follows:
would expect that the scores are similarly distributed. Conversely, for disruptive
1 K IAC
Σ K
d(s gk),
where ğ refers to unperturbed estimates and q E R.N k == (I.....) is a set of perturbed quality estimates, replicated K times for N test samples (see Equation 4) such that Q = lq',..., 1kk] E RNNK Here, d refers to a statistical significance measure d : RR x R + R that takes a set of unperturbed- and perturbed estimates high p-value indicates that & and qf are similarly distributed and a low p-value value means that the estimates are differently distributed. Accordingly, Equation 5 returns the average p-value across all perturbed samples over K perturbations, with IAC [0, 1]. Since the nominal values of quality estimators can vary and often have little to no semantic meaning, use the non-paramtrric Wilcozon signed-rank test (Wilcoxon, 1945) which does not carry strong assumptions about the data distribution, only about its ranking. In addition to the intra-consistency analysis, we also measure whether quality estimators exhibit consistent behaviour in terms of ranking. This type of inter-consistency analysis is commonly used in Explainable AT research (Tomsett et al., 2020; Gevaert al., 2022; Hedström el al., 2023; Rong el. al., 2022) and complements the aforementioned by involving more than one explanation method. Let Q Nx denote matrix for the unperturbed estimates á for L explanation methods and Q' RNZ be matrix for the perturbed estinates de which are both averaged over K perturbations. We formulate the Conss-Conss (IEC) criterion as follows:
and returns a p-value.
IEC
U
(6)
X
L
where U,, E [0, 1] are entries of binary ranking agreement matrix U that takes quality estimates from Q and Q' and populates the entries according to the interpretation of ranking. Here, IEC = indicates perfect ranking consistency and IEC = the absence of it, where IEC E [0, 1]. The perturbation strength is indicated in the superseript DM. The interpretation of ranking is different depending on the perturbation strength, i..,, minor disruptive. For minor perturbations, we measure if the quality estimator
7

---

Published in Transactions on Machine Learning Research (06/2023)
Step 2. Scoring
Moasure efferts of the perturbations via LAC and IEC criteria
GR.
16
Step 3. Integrating
Step 1. Perturbing
Fvaluate meta consiteecy performance by combining the failure modes
Depending on faiture mode, initiate a or disruptice perterbation
dla dh). (
TAC
IBC
NxL 1-1
EEE, U/ (6)
(1771
((mr))
Tf
IACAN
(9)
PA
Noise Resilience
TECxn
33, 2, 1]
IEC
d =
[ 3, 2, 1
K..
AYR
E3 -Fj ocherwise,
(F)
IACAR
AECA.M
d= (0.6, 0.7,
PB
Adversary Reactivity
(0.30 0.5,0.11
1 U2
Qu (8)
ocherwiso.
Figure evtaaavalu of quality estimators performed three steps: (i) Perturbing, (ii) Scoring and (iil) Integrating. (i) First, minor or disruptive perturbation is induced depending the failure mode, ie., Ph for NR and Pf for AR. (ii) Second, the estimator's intra- and inter-consistency are calculated to assess each performance dimensson. The IAC score caplures the extent that the estimator produces similar dis-similar scores with respect to and of. which is illustrated through the distribution plots, where for NR and AR, the score distributions are overlapping and non-overlapping, respectively. The IEC score expresses ranking consistency. NR measures how consistently the estimator ranks different explanation methods and AR calculates how consistently the perturbed scores lower than the unperturbed scores. (iii) In the final step, we integrate the previous steps and produce MC score that summarises the estimator's performance: its resilience to noise and reactivity to adversary.
ranks different explanation methods similarly. We define UM for minor perturbations with entries such as:
=T otherwise,
UM
(7)
0
where MM = r( with Q=:=Q and r(Q.)) are ranking vectors given a ranking measure r R² RL that takes each row in Q and Qi... respectively and sorts the values in descending order. Each entry FMM E N corresponds to integers indicating their relative rank. For example, suppose we have one sample a, three explanation methods and their corresponding quality estimates, such as Q = [0.76, 0.86, 0.66]. Then the estimator would provide
results obtained from applying would be PM = [2, 1. An pptimally-performme the same rankings for FM as F for all N inputs, resulting in IEC = 1. However, as discussed Section 6, the reality that many estimators often conflict with the optimal.
For disruptive perturbations, we interpret ranking consistency differently. Here, as explained in-depth in Appendix A.1.2, we measure how consistently the quality estimator ranks estimates from Q higher than Q" Q'. We define U for disruptive perturbations with entries such as:
U2 - 02200 otherwise,
{0
(8)
where the quality estimates Q are generated for an explanation with respect to the same class as the predicted for the unperturbed estimate Quy- For some estimators, e.g., in the robustness category, lower values are considered better than higher values, for which we invert the comparison symbol in Equation
3.3 Quantifying Meta-Consistent
To conclude the framework, we want characterise the performance of a quality estimator with a single Meta- Meta-Consistency (MC) score. To capture both the estimator's resilience to noise (NR) and its reactivity to
8

---

Published in Transactions on Machine Learning Research (06/2023)
adversary (AR), average over the two criteria for both failure modes:
IACNR IACAR IEC
MC
|m
m T m where
m=
(9)
IECAR
and m² 1 represents optimally performing quality estimator as defined by the all-one indicator vector. A good quality estimator should produce an MC score close to 1 as higher values indicate better performance on the tested criteria", where MC E [o, 1]. An estimator that demonstrales a ba balance of resilience against minor perturbations and reactivity towards disruptive perturbations as evidenced through its score distribution and ranking of different explanation methods would achieve high consisoony scores with our framework. Our proposed score has the advantage of being both concise and comprehensive, as provides summary of the performance characteristies of an estimator while also taking into account multiple criteria. For full overview of the framework, please see Figure 4.
4 Practical Evaluation
Within the framework of meta-ovaluation, it is necessary to generate perturbed quality estimates for analysis. To accomplish this, we developed a series of practical tests. The methodology behind these lests is simple and thus easily extensible through the tests made available in the repository First, the space in which perturbations will be applied is selected, with options being either the input or the model. Second, based on the chosen space, an appropriate type of noise defined. To ensure that the perturbations are meaningful and relevant to the task at hand, the noise type should be chosen caltextually with respect to the data domain. For example, when perturbing the input space for images, we define a test as follows:
Input Perturbation Test (IPT). Apply i.d additive uniform noise such that a; x with d; U( B) where for noise resilience, à; fulfills Definition and for adversary reactivity, d; fulfills Definition 2
where a, 3 have to be chosen according to the data domain and respective failure mode (e.g., set -0.001, 9 = 0.001 for NR and a 0.0, = 1.0 for AR). To maintain the statistics of the data distribution, we clip B to the maximum and the minimum value of the test set, respectively. Moreover, when perturbing the model space, to maintain the variance of the network, we follow an established methodology by Bykov et al. (2022) and scale the learned weights 0 of the model f as follows:
Model Perturbation Test (MPT). Apply multiplicative Gaussian noise to all weights of the network, i.e., = is with is ~ (n,, Σ) where μ = and for noise resilience, Ó fulfills Definition and for adversary reactivity, 0 fulfills Definition 2
where for Σ to be consistent with either Definition 1 or 2, it is set based on the specific context of the model and task being considered (e.g., Σ = 0.001 for NR and Σ = 2.0 for AR). Third, to collect sets of perturbed quality estimates for intra- and inter-consistency analysis, we repeat the process of perturbation (as outlined in IPT and MPT) and subsequent evaluation (using Equation 4) under K runs. Finally, we compute the MC score. For sanity-checking experiments of the tests, see Appendix A.5. Moreover, third testing scenario, it is theoretically possible to perturb both the input- and model spaces simultaneously, iee, Px(x), PP( well as their respective latent spaces. This we leave for future work.
5 Experimental Setup
In this section, we give a brief account of the experimental setup, including the datasets, models, explanation methods and estimators used in this work. Further details can be found in Appendix A.4.
When computing intra-consisteucy scores for AR, apply reverse scoring, i..., 1 IACAR that all elements in the meta-evall evaluation vector (Equation 9) can be interpreted in the same way, Le., that higher values better. 5 Code available at the Hub repository: btaps://githinn wtcchhu..cmm//aaa www./anaaedstrreeeeeoooooo htwirrrreeeeettaaaatuut
9

---

Published in Transactions on Machine Learning Research (06/2023)
In our experiments, we benchmark five different categories of explanation quality and within each category, we selected two estimators as follows: Complexity (CO) (Bhatt et al., 2020), Sparseness (SP) (Chalasani et al., 2020), Faithfutness Correlation (FC) (Bhatt et al., 2020), Pizel-Flippine (PF) (Bach et al., 2015), Maz-Sensifivity (MS) (Ych et al., 2019), Local Lipschitz Estimate (LLE) (Alvarcz-Melis & Jaakkola, 2018);
Pointing- Game (PG) (Zhang et al., 2018), Relevance Mass Accurucy (RMA) (Arras et al., 2022), Model Parameter Randomisation Test (MPR) (Adebayo et al., 2018) and Random Logit (RL) (Sixt et al., 2020).
Each estimator evaluates explanations from a popular category of post-hoc attribution methods, including both bradient-based- and model-agnostic techniques: Gradient (Morch et al., 1995; Bachrens et al., 2010), Saliency (Morch et al., 1995), GradCAM (Selvaraju et al., 2020), Integrated Gradients (Sundararajan et al., 2017), Inputx Gradient (Shrikumar et al., 2016), Occlusion (Zeiler & Fergus, 2014) and GrudientSHAP (Lundberg & Lee, 2017) from which we generate explanations with respect to a sample's predicted class. For comparability, we normalise the explanations by dividing the attribution map by the square root of its average recond-moment estimate (Binder et al., 2022). The mathematical definitions of the estimators are described in Appendix A.3. For estimator implementations, we use Quantus library (Hedström el al., 2023).
We use four image classification datasets for our experiments: ILSVRC-15 (i.e., ImageNet) (Russakovsky et al., 2015), MNIST (LeCun et al., 2010), fMNIST (Xiao et al., 2017) and customised (i.c., cMINST) (Bykov et al., 2022) with different black- box NNs such as LeNets (LeCun et al., 1998), ResNets (He et al., 2016) and transformer-base Transformer (Liu et al., 2021b).
models such as Vision Transformer (ViT) (Dosovitskiy et al., 2021) and SWIN
6 Results
Many open questions remain in the field of XAI. In this section, we show how meta-evaluation can solve a subset of those problems such as (i) estimator selection, (ii) optimising hyperparameters of an estima- tor, (ifi) evaluating the category convergence, i.e., the extent that estimators within the same category of explanation quality measure the same concept and (iv) comparing estimator choice for different network architectures including transformer-baset architectures. We prioritise the topic of estimalor selection in the main manuscript and provide detailed analysis and discussion of the experiments addressing questions (ii), (fii) and (iv) in Appendix A.6. Instructions to reproduce the experiments can be found in the repository.
Table 1: Mutaiovaluation benchmarking results with ImageNet, aggregated over 3 iterations with k= 5. IPT results are in grey rows and MPT results are in white rows. MC denotes the averages of the MC scores over IPT and MPT. The top-performing MC- or MC method each explanation category, which outperforms the nottom-performing method by at least 2 standard deviations, is underlined. Higher values are preferred for all scoring criteria.
Category
Estimator
MC (t)
MC (t)
IACNR (t)
IAC (t)
IECaR (f)
IEC AR (t)
0.697 = 0.048
0.519
0.229
0.745 : 0.145
0.870 + 0.060
0.653 + 0.064
Sparseness
0.636 +:2 0.055
Complexity
0.575
0.062
0.588
0.171
0.445 + 0.164
0.862
0.047
0.403
0.036
Complexity
0.562 + 0.033
0.590
+ 0.043
0.170
0.093
0.965 + 0.092
1.000
0.000
0.225
+ 0.068
0.534
0.023
0.221
0.086
0.792 = 0.088
1.000
0.000
0.122
0.012
0.417
0.078
0.518
0.241
0.550 = 0.138
0.312
0.063
0.470
0.051
Faithfullosss Corr.
0.480 + 0.050
Faithfulness
0.490
0.082
0.497 + 0.216
0.586 + 0.149
0.329
0.074
0.546
0.039
Pixel- Pixel-Flipping
0.698 + 0.095
0.623
0.128
0.518
0.232
0.691
+ 0.405
0.707
0.052
0.574
0.125
0.774
+ 0.062
0.515 + 0.224
1.000 = 0.000
0.719 = 0.060
0.863 = 0.011
0.591
+ 0.066
0.685 + 0.121
0.579 + 0.159
0.903 = 0.0-19
0,196 - 0.032
Pointing-Game
0.537 + 0.013
U.483 + 0.020
0.868 + 0.068
0.102 + 0.098
0.935 + 0.026
0.026
+ 0.012
Localisation
Relevance Mass Acc.
0.580 + 0.070
0.619 + 0.067
0.524 + 0.213
0.633 + 0.172
0.750 + 0.044
U.570
+ 0.047
0.541 = 0.074
0.558 + 0.125
0.476 = 0.199
0.767 = 0.052
0.364
- 0.037
0.644 = 0.063
0.481
+ 0,268
0.631 + 0.217
0.939 = 0.035
0.524
- 0.083
Random Logit
0.601 - 0.032
Randomisation
0.564
+ 0.000
0.393
+ 0.000
0.400
+ 0.000
0.960
+ 0.000
U.505
+ 0.000
Model Param. Rand.
0.697 + 0.042
0.762
+ 0.041
0.454
+ 0.170
0.909
+ 0.000
0.967
+ 0.020
0.720
+ 0.022
0.631
# 0.044
0.465
+ 0.200
0.563
= 0.077
0.954
0.017
0.541
= 0.016
Max-Sensitivity
0.610 + 0.065
0.621
= 0.068
0.507
0.144
0.593
= 0.284
0.938
= 0.026
0.416
= 0.089
Robustness
0.600
0.061
0.454
0.251
1.000
+ 0.000
0.921
0.020
0.023
0.007
Local Lipschitz Est.
0.615 + 0.071
0.635
+ 0.112
0.565
0.039
0.630
+ 0.561
0.916
0.031
0.431
+ 0.197
0.030 0.594
0.498
0.144
0.485
= 0.073
0.895
0.027
0.498
0.016
10

---

Published in Transactions on Machine Learning Research (06/2023)
0.8
1.0
1.0
MNISST
PPA
IMNIST
cMNIST
ImageNet
AR
IECan
0.5
0.5
MC
0112
0.6
2.00
0:20
IPT
IPT
0.5
0.0
MPT
MPT
0.0
1.0
0.0
1.0
SP
CO
FC
PF
PG
RMA
RL
MPR
MS
LLE
IACUR
IECNR
Figure Left: A visualisation MNIST benchmarking results (Table 3), particular LAC and IEC scores for noise resilience (x-axes) and adverse reactivily (y-axes). The colours indicate the estimator and the symbols show the lest type, i.e. TPT and MPT, respectively. Right: comparison of averaged meta-consistency performance quality estimators using MPT and IPT, aggregated over 3 iterations with K 5, across different models (LeNets, ResNet-9, 18) and datasets (MNIST, fMNIST, cMNIST, ImageNet). Higher values are preferred.
different
6.1 Benchmarking
As first example, will demonstrate how mela-evaluation can be used to select a certain quality estimator for given category of explanation quality, To this end, we set up benchmarking experiment, where we take two popular estimators from five different explanation quality categorics and evaluate six explanation methods L = GGrudient, Saliency, GradCAM, Integrated Gradients, Occlusion, GrudientShap) for MNIST, IMNIST and cMNIST datasets and evaluate three explanation methods I =[Saliency, Inpulx Gradienl, GradientShap) for ImageNet dataset, using the MetaQuantus library. Since the choice of L has minimal influence on the MC scores (sce experiments in Appendix A.5.2), we omit results from other tested sets of explanation methods in the main manuscript.
6.1.1 Comparison XAI Estimators
The ImageNet results summarised in Table 1. The grey rows indicate the results from the Input Perturba- tion Test and the white rows show the results from the Model Perturbation Test. A more detailed discussion of the results, including additional datascts, can be found in Appendix A.6.3. From Table 1, we can observe that no lested estimator performs optimally, L.e. A MC < 1. From column MC, which displays the averaged MC scores (over IPT and MPT) we note that Sparseness, Flippigg, Relevance Mass Accuracy, Model Parameter Randomisation Test and Local Lipschitz Estimate are the best-performing estimators their respective category for ImageNet dataset. From Figure 5 (right), we can observe that this comparison of MC scores generally consistent across the tested datasets, which contributes to the generalisability of our findings. Certain categories such as localisation and randomisation exhibit higher variability with respect to the MC score, which we further discuss in Appendix A.6.5.
6.1.2 Comparison of XAI Evaluation Categories
The meta-evaluation framework can moreover be applied to gain insights into the performance characteristics of different estimators on a meta-evaluation vector as coordinates on a 2D plane and visualise the results as an area graph (see Figure 7). By inspecting the coloured areas of the respective estimators in terms of their size and shape, we can deduce the overall performance of both failure modes. Here, larger coloured areas imply better performance the different scoring criteria and the grey area indicates the area of optimally performing quality estimator, i.c., m 14. Each column of estimators represents a category of explanation quality, from left to right: Complexity, Faithfulness, Localisation, Randomisation and Robustness, which colour scheme we apply consistently across all figures.
basis. For this purpose, we represent the entries of the
As seen Figure 7 (third column), the localisation estimators exhibit notable deficiency in terms of adversary reactivity on the Input Perturbation Test. low IPT score for adversary reactivity means that the estimators insensitive to disruptive input perturbations, evidenced by similar score distributions
11

---

Published in Transactions on Machine Learning Research (06/2023)
Sparsaniss (0 (38557)
Fast fudesss Conmuatin (6.4997)
Garetig-Game (0.5364)
Randem Logil (0.8011)
Maxe Sas--Snnitttty (0.103))
MCM
0.5
05
05
C5
0L5
XCaa
LSC
0.5
0.5
0.5
0.5
Complesiyy
Poei-llpping (500)))
Accuracy (0.5002)
Parameeer
Uppotii Extimete (0.6140)
Mapx
0.5
0.0
0.0
0.0
X00
Miai
Figure 7: A graphical representation o the ImageNet benchmarking results (Table 1), aggregated over 3 iterations with K = 5. Each column corresponds to a category of explanation quality, from left to right: Complezity, Faithfulness, Locolisation, Randomisatiin and Robustness. The grey area indicates the area of an optimally performing estimator, ie., m = 1 The MC score (indicaled in brackels) is averaged over MPT and IPT. Higher values are preferred.
(low 1AC) and an inability to rank disruptively perturbed explanations lower than unperturbed explanations (low IEC). Based on the definitions of these estimators (described in Equations 24-25), which include the Pointing- Game method (Zhang al., 2018), which evaluates explanations by verifying that the highest attributed feature intersects with a given segmentation mask and the Relevance Mass Accuracy method (Arras et al., 2022), which calculates the amount of explainable mass intersecting with the segmentation mask we would expect that these estimators perform well on this test since disrupted input usually leads to scattered attributions. It may seem counterintuitive that these estimators lack reactivity to disruption, however, we posit that the reason for the poor reactivity to adversary is the estimators' inherent dependency on the segmentation mask. If the segmentation mask (relative to the object of interest, or the input) is large enough, high localisation scores are attainable irrespective of the "quality" of the explanation (Kohlbrenner et al., 2020). This finding is further validated by the increase in MC scores for cMNIST dataset, which generally has a smaller bounding box compared to ImageNet, MNIST and IMNIST (see details in Appendix A.4), where disruption evidently has an effect, as evidenced by higher AR scores, depicted in Figure 5 (right). Practitioners should be aware of this category's 's reliance (or oversensitivity) on the segmentation mask where relying solely or too heavily on this category in XAI evaluation may not be advisable.
The highest overall meta-connsistency scores are obtained by the robustness and ran- domisation categories, which can be observed in Figure 6 and by their respective areas in Figure 7. One potential explanation for this is that the estimators in these categories already include some element of stochasticity in their estimator definit ions (see Equations 18-19 and 20-21, respectively) which may make them more resilient as well as reactive to perturbations. For example, both robustness estimators, i.e., Maz-Sensitivity (Yeh et al., 2019) and Local Lipschitz Estimate (Alvarez-Melis & Jaakkola, 2018) rely on Monte Carlo sampling-basse approximation where expla- nation methods are evaluated by examining their response to minor perturbation of the input, aggregated over multiple runs. In the randomisation category, the Model Parameter Randomisation Test (Adebayo et al., 2018) evaluates explanations by incrcasingly perturbing the model weights and Random Logit (Sixt et al., 2020) evaluates explanations by a random selection of an explanation of a non-target class. As seen in Figure 6, the complexity category has the lowest overall MC scores across the tested datascts, which includes estimators such as Sparseness (Chalasani et al., 2020) and Complexity (Bhatt et al., 2020) that evaluate explanations by calculating their Gini coefficient and Shannon entropy, respectively. Given the simplicity of these calculations, this outcome is not surprising.
0.65
MC
0.6
0.55
CO FA LO RARO
Figure 6: Average MC scores per category over tested datasets (TmageNet, MNIST, JMNIST, cMNIST).
12

---

Published in Transactions on Machine Learning Research (06/2023)
Pording Game (0.5631)
Top-C Internsction (0.5015)
Pelevarce Mars Acoracy (0.5877)
Relevarnce Rank Accunaay (0,5156)
520
HUM
His
05
25
30
0.
Mia
05
3.5
05
Heit
05
0G
3.5
0.
055 0 05
05 35
05 05
06
Figure 8: A graphical representation of the ImageNet benchmarking results for different architectures aggregated 5. Each column represents the estimator, from left to right: Pointing- Game, Top-K
over iterations with Intersection, Relevance Mass Accuracy and Helevance Hank Accuracy, and each row the model type [HesNet-18, ViT, SWIN), ;, as indicated by the labels. The grey area indicates the area an optimally performing estimator, i.e., m" = 11. The MC score (indicated in brackets) is averaged over MPT and IPT. Higher values are preferred.
Another notable category of poor performance that is picked up by the meta-evaluation tests is faithfulness. Our results, which show lack of resilience to noise in the ranking of explanation methods (low IEC especially for Faithfulness Correlation (Bhatt et al., 2020), corroborate previous studies (Brunke et al., 2020; Brocki & Chung, 2022; Rong et al., 2022) that found that faithfulness estimators may rank explanation methods inconsistently when subjected to perturbation, such as changing the masking pixel strategy from, for example, "uniform" to "black". This trend is particularly evident in Figure 5 (left), where the points belonging to the faithfulness category have notably lower TEC scores compared to the other categories of explanation quality. A possible explanation is their well-documented sensitivity to parameterisation ( (Tomsett et al., 2020; Hedström et al., 2023; Rong al., 2022).
While certain estimator categories, such as faithfulness, may present challenges such as parameterisation, it is not advisable to disregard their evaluation. Compared to categories such as complexity which are well- defined and simple to calculate, they may not offer as much information as categories such as faithfulness, which can provide important insights into how the explanation- and model functions are related. Relying on only one category to estimate explanation quality therefore not recommended. This is espocially true since an explanation function may trade one calegory of explanation quality over another (Bhatt et al., 2020), for example, an explanation that is faithful may be too complex for the user to understand. Therefore, to avoid arriving at incomplete or incorrect conclusions about which explanation methods work (and not), it is of utmost importance for practitioners to approach quality estimation by incorporating multiple definitions of explanation quality.
6.1.3 Comparison of SOTA Vision Architectures
To investigate if the performance characteristics the different quality estimators revealed by the meta- evaluation endure across various architectural types, we conducted additional experiments with ResNet- and transformer-based model architectures. These architectures included the Vision Transformer (ViT) (Doso- vitskiy et al., 2021) and the SWIN Transformer (Liu et al., 2021b). For this experiment, we employed two
13

---

Published in Transactions on Machine Learning Research (06/2023)
explanation methods == {Saliency, GradientShap) and meta-evaluated four estimators in the localisation calegory, namely Poimläng-Game (PG) (Zhang et al., 2018), Relevance Moss Accuracy (RMA) (Arras et al., 2022), Relevance Rank Accuracy (RRA) (Arras et al., 2022) and Top-K Intersection (TK) (Theiner et al., 2022). These cstimators are all defined in Appendix A.3. We also conducted a comparable experiment with the complexity estimators Sparseness (Chalasani el. al., 2020) and Complerity (Bhalt et al., 2020). These results are found Appendix A.6.4, which yielded outcomes that are consistent with the findings presented here
In Figure 8, we present the meta-evaluation results area graphs. Each column represents localisation estimator and cach row represents a network architecture, as indicated by the labels. The performance attributes of each estimator, when examined across different network types, display a noticeable similarity. This suggests that the characteristics revealed through meta-evaluation analysis may be persistent across architecture types, thereby enabling general conclusions of individual estimators. In particular, Pointing- Game (Zhang al., 2018) performed poorly across all model architectures, with low adverse reactivity in the rankings (TEC) where Relevance Mass Accuracy (Arras el. al., 2022) generally demonstrated higher performance. Relevance Rank Accuracy (Arras et al., 2022) and Top-K Intersection (Theiner et al., 2022) showed the highest scores with narrow margins, possibly due to their similarity in definitions (sce Equations 26 and 27 in Appendix A.3). Overall, these results contribute to the robustness of our findings and provide evidence demonstrating the framework's agnosticism to the choice of network architecture
7 Conclusion
When we neither understand the general behaviour of the explanation methods nor the estimators that we apply to estimate their quality, we are bound to make mistakes. This problem in XAI is exacerbated by the fact that different estimators within the same category of explanation quality may rank the same explanation method differently (Neely et al., 2021; Krishna et al., 2022). Without an understanding of the performance characteristics of the estimators we employ, we risk presenting inferior explanation methods to the end user.
To address the problem of meta-evaluation, we propose a novel framework for identifying reliable quality estimators to evaluate explanation methods. We circumvent the Challenge of Unverifiability by evaluating perturbing the verifiable variables of XAI evaluation
the estimators through the lens of and thercafter analysing the estimator's outcomes under different failure modes, we can get objective and independent characterisation of its performance. We show, in a series of experiments, how to use the framework for estimator selection and for systematic meta-evaluation of the strengths and weaknesses of individual estimators (Section 6.1.1), general categorics of explanation quality (Section 6.1.2) as well as architecture types (Section 6. 6.1.3). Our findings show that (i) localisation estimators demonstrate a deficiency in terms of adversary reactivity, possibly due to their dependency on the segmentation mask, (ii) faithfulness estimators are inconsistent in their ranking and (iii) estimators within the randomisation- and robustness categories demonstrate the highest performance. It is advised that XAI practitioners take into account the limitations of various estimator categories and exercise caution when relying heavily on certain categories.
Evaluating the intrinsic value or "validity" of a quality estimator is, however, still an open and important question to consider. It is essential to keep in mind that the reliability of an estimator does not necessarily imply any intrinsic validity, e.g,, an estimator's theoretical soundness (Sundararajan et al., 2017; Binder et al., 2022). As we make the meta-ovaluation tests readily available, we inadvertently create the risk of the tests being misused and the results being misinterpreted. We encourage the reader to exercise caution when interpreting the outcome of the tests and not rush to conclusions regarding the overall reliability of quality estimators, which should always be judged with respect to their specific experimental context.
Our experiments are limited to image classification applications. To fully rate the generalisability of MetaQuantus, we plan to extend our experiments more broadly in the sciences and medicine and include other explanation method types as well as other data domains such as tabular, textual and time series data. This will require additional work to ensure that the estimators themselves support these data domains, which will be addressed in upcoming publications.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Acknowledgments
This work was partly funded by the German Federal Ministry for Education and Research through project Explaining 4.0 (ref. 01TS200551), BIFOLD (ref. 011S18025A and ref. 011S18037A), the Investitionsbank Berlin through BerDiBa (grant no. 10174498), the European Union's Horizon 2020 research and innovation programme through iToBoS (grant 965221) as well as European Union's Horizon 2022 rescarch and innovation programme (EU Horizon Europe) TEMA (grant no. 101093003). It was also financially supported by the Research Council of Norway (RCN), through the FRIPRO (grant no. 315029), and partially funded by its Centre for Research-baseee Innovation funding scheme (Visual Intelligence, grant no. 309439) and Consortium Partners, RCN IKTPLUSS (grant no. 303514), and the UiT Thematic Initiative "Data-Driven Health Technology".
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Tan J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Desachianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 9525 9536, 2018.
Chirag Agarwal, Nari Johnson, Martin Pawelczyk, Satyapriya Krishna, Eshika Saxena, Marinka Zitnik, and Himabindu Lakkaraju. Rethinking stabllity for attribution-base based explanations. CoRR, 2022a.
Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. OpenXAI: Towards a transparent evaluation of model explanations. In siith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b.
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Samy Bengio, Hugo Larochelle, Kristen Grauman, Nicolò Gesa-Bianche and Roman Garneu (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7786-7795, 2018.
Anna Arias-Duart, Ferran Pares, Dario Garcia-Gasulla, and Victor Gimenez-Abalos. Focus! Rating xai methods and finding biases. CoRR, abs/2203.02923 2021.
Leila Arras, Ahmed Osman, and Wojciech Samek. CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 2022.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Müller. How to explain individual classification decisions. J. Mach. Learn. Res., 11:1803-1831, 2010.
Naman Bansal, Chirag Agarwal, and Anh Nguyen. SAM: the sensitivity of attribution methods to hy- perparameters. In 2020 IEEE/CVP Conference Compuler Vision and Pallern Recognétion, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 11-21. Computer Vision Foundation IEEE, 2020.
I. Bellido and Emile Fiesler. Do backpropagation trained neural networks have normal weight distributions? In Stan Gielen and Bert Kappen (eds.), ICANN '93, pp. 772 775, London, 1993. Springer London. ISBN 978-1-4471-2063-66
Jose Manuel Benitez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes? IEFE Transactions on Neural Networks and Learning Systems, 156):1156-1166 1997.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature-based model explanations. Tn Christian Bessiere (ed.), Proceedings of the Twenly-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3016 3022. ijcai.org, 2020.
Alexander Binder, Leander Weber, Sebastian Lapuschkin, Grégoire Montavon, Rabus- Müller, and Wojeiech Samek. Shortcomings of top-down randomization-based sanity checks for evaluations of deep neural network explanations. CoRR, abs/2211.122486, 2022.
Lennart Brocki and Neo Christopher Chung. Evaluation of interpretability methods and perturbation arti- facts in deep neural networks. CoRR, 029228, 2022.
Lukas Brunke, Prateek Agrawal, and Nikhil George. Evaluating input perturbation methods for interpreting CNNs and saliency map comparison. In Computer Vision ECCV 2020 Workshops, pp. 120-134. Springer International Publishing, 2020.
Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M.-C Höhne, Noisegrad enhancing ex- planations by introducing stochasticity to model weights. In Thirty-Sizth AAAI Conference on Artificial Intelligence, AAAI 2022, Fourth-Fourth Conference
Innovative Applications of Artificial Intelligence, TAAT 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Vir- tual Event, February 22 March 1, 2022, pp. 6132 6140. AAAI Press, 2022.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of ncural networks using adversarial training. In Proceedings of the 37th International Conference on Muchine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1393. PMLR, 2020.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 782-791. Computer Vision Foundation / IEEE, 2021.
Tan Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate shapley values with vision transformers. CoRR, abs/2206.05282, 2022.
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local expla- nations. In International Conference Machine Learning, pp. 4794 4815. PMLR, 2022.
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Balché-Buc, Emily B. Fox, and Roman Garnett (eds.): Advances in Neurol Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13567-13578 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virlual Event, Austria, May 3-7, 2021. OpenReview.net net. 2021.
Arne Gevaert, Axel-Jan Rousseau, Thijs Becker, Dirk Valkenborg, Tijl De Bie, and Yvan Saeys. Evaluating feature attribution methods in the image domain. CoRR, abs/2202.12270., 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770 778. IEEE Computer Society, 2016.
Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojeiech Samek, Scbastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An explainable toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34):
1-11, 2023.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojeiech Samek, and Sebastian Tapuschkin. Towards best practice in explaining neural network decisions with lrp. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. LEEE, 2020.
Narine Kokhlikyan, Vivek Miglani, Bilal Alsallakh, Miguel Martin, and Orion Reblitz- Investi- gating sanity checks for saliency maps with image and text classification. CoRR, (tbs/2216.000 074477 2021.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. CoRR, 2bs/2202.01602, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to docu- ment recognition. Proc. JEEF, 86(11):2278 22222244223 1998.
Yann LeCun, Corinna Cortes, and CJ Burges, Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.deee wow...ccoo/eeeee endb/ cerdd//mot /mnist, 2, 2010.
Simon Letzgus, Patrick Wagner, Jonas Lederer, Wojeiech Samek, Claus-Roberr Müller, and Grégoire Mon- tavon. Toward explainable AT for regression models. CoRR, (bss2112.111409 2021.
Zachary C. Lipton. The mythos of model interpretability. Comman. ACM, 61(10))36-43, 2018, doi: 10.114 5/ 5/3233231.
Yang Liu, Sujay Khandagale, Colin White, and Willie Neiswanger. Synthetic benchmarks for scientific research in explainable machine learning. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9992- 10002. TERE, 2021b.
Scott M. Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4765 4774, 2017.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digit. Signal Process., 73:1 15, 2018
Niels S. Morch, Ulrik Kjems, Lars Kai Hansen, Claus Svarer, lan Law, Benny Lautrup, Stephen C. Strother, and Kelly Rehm. Visualization of neural networks using saliency maps. In Proceedings of International Conference on Neural Networks (ICNN'95), Perth, WA, Australia, November 27 December 1, 1995, pp. 2085-200 2090. IEFE, 1995.
Michael Neely, Stefan F. Schouten, Maurits J. R. Bleeker, and Ana Lucic. Order in the court: Explainable AI methods prone to disagreement. CoRR, hass/2105.0288, 2021.
An-phi Nguyen and Maria Rodriguez Martinez. On quantitative aspects of model interpretability. CoRR, abs/2007.07555 07584,5544444 2020.
Frederik Pahde, Galip Umit Yolcu, Alexander Binder, Wojeiech Samek, and Sebastian Lapuschkin. Opti- mizing explanations by network canonization and hyperparameter search. CoRR, Abs//221.11114. 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, peg-hmaeformance deep learning library. In Advonces in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should I trust you?": Explaining the pre- dictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Son Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144. ACM, 2016.
Laura Rieger and Lars Kai Hansen. IROF: a low resource evaluation metric for explanation methods. CoRR. abs/2003.0S 08047,, 2020.
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In Kamalika Chaudhuri, Stefanie Jegelka, Te Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 18770 18795. PMLR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Scbastian Lapuschkin, and Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Newral Networks and Learning Systems, 28(11):2600) 26660-26673 2017.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. Int. J. Comput. Vis., 12:33) 359, 2020.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. CoRR, abs/1605.01713,, 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun (ods.), 2nd International Conference on Learning Represenlations, ICLR 2014, Bansl, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified BP attribu- tions fail. In Proceedings of the 37th International Conference Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 9046 9057. PMLR, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706 03860338.. 2017.
Mukund Sundararajan and Ankur Taly. A note about: Local explanation methods for deep neural networks lack sensitivity parameter values. CoRR, abs/ abs/1806.044 1800.04205, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 94th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3319 3328. PMLR, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Jonas Theiner, Eric Müller-Budack, and Ralph Ewerth. Interpretable semantic photo geolocation. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloo, HI, USA, January 3-8, 2022, pp. 1474 1484. TEFE, 2022.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun D. Preece. Sanity checks for saliency metrics. In The Thirly-Fourlle AAAT Conference Artificial Intelligence, AAAT 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 6021 6029. AAAT Press, 2020.
Yipei Wang and Xiaoqian Wang. A unified study of machine learning explanation evaluation metrics. CoRR, 2ttp/2223...4466.. 2022.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 166):80 83, 1945. ISSN 00994987.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Bashion-mnise a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/ 0bs//708.00744 2017.
Mengjiao Yang and Been Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, abs/ 1907.099799 2019.
Chih-Kuan Ych, Cheng-Ya Yu Hsich, Arun Sai Suggala, David I. Inouye, and Pradcep Ravikumar. On the (in)sidelity and sensitivity of explanations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence MAlché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10965-10976, 2019.
Gal Yona and Daniel Greenfeld. Revisiting sanity checks for saliency maps. Colf,, abs/ 2110.14297, 2021.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Swilzerland, Seplember 6-12, 2014, Proceedings, Part I, volume 8689 of Lecture Notes in Computer Science, pp. 818 833. Springer, 2014.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):0084-110, 2018.
19

---

Published in Transactions on Machine Learning Research (06/2023)
A
Appendix
In this section, we include all the necessary details and information to support the claims and results presented in the main paper. In Section A.I, we present theoretical considerations for the mela-evaluation framework. In Sections A.2 and A.3, we outline the mathematical definitions explanation quality, for categories and estimators, respectively. In Section A.4, we describe the experimental setup and the following Section A.5, we describe the experiments that were performed to sanity-check the framework. In Section A.6, we provide supplementary results, in terms of additional applications and supporting experiments. We provide notation table at the end in Section A.7.
Broader Impact Statement
This research is important since we raise awareness and address the need for reliable evaluation methods in the Explainable AT (XAT) community. In the XAT community, the evaluation of explanation methods has often been neglected or clouded by the ambiguity that an absence of ground truth labels entails yet to foster sustainable progress in the field over time it is necessary to systematically define and meta-evaluate the methods used to measure the quality of explanations. This research takes the first step towards this goal by developing practical, quantifiable tools for reliable XAI evaluation. Without careful examination of the quality of explanations, the deployment of potentially beneficial machine learning algorithiss may be hindered, preventing the full potential of AI from being realised in important areas such as healthcare, education, finance and policy.
A.1 Theoretical Considerations
This section discusses the concept of minor and disruptive perturbations in the context of multi-label clas- sification and regression tasks in XAI. It also addresses the potential vulnerability of the framework to adversarial attacks and the motivation for the calculation of the IEC scoring criterion for disruptive pertur- bations.
A.1.1 Expanding Scope and Adversarial Attacks
Multi-label classification In this work, given the popularity of image classification lasks in XAI, we focused mostly on this application. In this application, the definitions of minor and disruptive perturbations (as given in Definitions 1 and 2) apply to I and y' which represent the predicted classes for truc and perturbed samples, respectively. However, our definitions can be easily extended to other types of classification tasks, such as multi-label classification. In this case, for a multi-label classification problem with C classes, the definitions of minor and disruptive perturbations (as given in Definitions and 2) would apply to binary prediction vectors タ E RC and y' E RC, rather than single classes. The distance between the two vectors can be denoted using the ||| Il| notation.
Regression As is the case with many explanation methods (Letzgus et al., 2021), the extension to the regression problem in XAT is not straightforward. Given y and y' as real-valued prediction outcomes, we would need to adjust Definitions 1 and 2 to encompass a derivation of a proper boundary e. We leave the task of adapting the meta-ovaluation framowork to regression problems to future work.
Adversarial attacks Adversarial attacks are techniques used to manipulate or deceive models (Szegedy et al., 2014) or their explanations (Dombrowski et al., 2019) by introducing perturbations to the input data that are imperceptible to humans but results in an incorrect prediction by the model. To adversarially attack Definition 1, it is theoretically possible to define perturbation that maximises the strength of the perturbation, while still remaining consistent with Definition 1, i.c., 110 - y'Ilp > In the same vein, to allack Definition 2, it is theoretically possible to define a perturbation that minimises the strength of the perturbation, while still remaining consistent with Definition 2, ie., ||0 1//lpp S e. While it may be a theoretical possibility to attack the framework through the definitions, performing such attacks would not serve any practical purpose. This is because contradicts the purpose of the framework, which is to assist practitioners in selecting and developing reliable XAI methods.
20

---

Published in Transactions on Machine Learning Research (06/2023)
A.1.2 Motivation for IECAR calculation
Contrary to the calculation of the inter-consistency criterion for minor perturbations, ie., IECNR, we cannot motivate IRCAR based on ranking consislency with respect to explanation methods, as disruptive perturba- tions implicate a change in the estimator score. While the expectation of changed rankings as a result of disruptive perturbations may appear intuitive, the imposed change the quality estimators could theoret- ically lead to symmetrical change across explanation method scores, which preserves the ranking across explanation methods. Since the behaviour of the explanation functions under disruptive perturbations lies in the unverifiable ble spaces U, we cannot exclude the possibility of a symmetrical response. Accordingly, for the calculation of IECAR: we relax the theoretical assumptions to a ranking comparison based on scores (as defined in Equation 7) which remain in the verifiable spaces O.
The assumption for the caleulation of the IEC score with respect disruptive perturbation is motivated by the scenario of an ideal estimator, which is expected to be able assess the true performance of an explanation method, denoted girue. In the ideal scenario, the real performance varies only slightly, i.e., oj of + e would therefore define an upper estimation bound ofrue s qj for each explanation method E [L....,L All estimates Q.j resulting from the AR scenario should differ from the unperturbed quality estimate Q. * Vi.- In the idealised scenario dfrue s gj we argue that Q7, R g and QP < Qi. leading to Equation 8. Note, however, that in practice quality estimates are subject to larger variations which mcans that the assumption qite = qgax and Equation 8 might not always hold. Therefore, practice, we do not expect IECAR & 1, which aligns with our results in Table 1. Nonetheless, further research on the inter-consistency criterion under disruptive perturbations is subject to future work.
A.2 Explanation Quality: Category Definitions
In the main paper, we described how a lack of explanation ground truth labels has led to diverse set of interpretations of explanation quality. In the following, we provide bricf summary of the most established categories of explanation quality, grouped into six categories; (a) faithfulness, (b) robustness, (c) randomi- sation, (d) complexity, (e) localisation and (f) axiomatic imators (Hedström et al., 2023). To establish mathematical ground for each category, we present a summarising equation. This means that all the nuances that typically exist within a category of explanation quality are not considered. For completeness, we, therefore, provide the exact mathematical descriptions of each of the individual estimators used this work in Appendix A.3.
Since many explanation categories do rely on perturbation, we define general perturbation function any real-valued space S C RRR,, e N in the following.
Definition 3 (Perturbation). Let Ps(s;)) S - S be perturbation function of S with parameters 17 R such that V& E S,
Ps(s;n)=8 =
(10)
For simplicity, we also write Ps(s) = -))ss;), which is used in the main paper.
Faithfulness (Montavon et al., 2018; Samek et al., 2017; Bach al., 2015; Bhatt et al., 2020; Nguyen & Martinez, 2020) quantifics the extent that explanations follow the predictive behaviour of the model, asserting that more important features affect model decisions more strongly. Given f, æ, y' and è, the change in the model output f(x) is measured the input features of are manipulated based on their attribution in é. The input manipulation is defined as a perturbation function Px(x, M) with E X where M is the number of input features that are perturbed. Since f denotes a trained model with parameters A, for brevity, we denote (ax; 0) as f(x) where possible.
1(f(2) ((f(=) ((Px(x,
(11)
Here, we present general theoretical considerations, but the specific claims for each estimator would require individual proofs.
21

---

Published in Transactions on Machine Learning Research (06/2023)
Robustness (Montavon et al., 2018; Alvarez-Melis & Jaakkola, 2018; Yeh et al., 2019; Dasgupta et al., 2022) measures the stability of the explanation ion function with respect to small changes in the input, requiring that those small perturbations in the input space [|px(x)) - æ|Ip < E, e.g., under an lp norm constraint upper bounded by some positive constant E, lead to only slight changes in the explanation - (.
assumiig that the model output approximately stayed the same f(x)) R [(Px(1)).
(R, D. é, P) = l|ē ((Px(x), 5 s.
A)ll
(12)
Randomisation (Adebayo et al., 2018; Sixt et al., 2020) measures the extent explanations deteriorate as randomness is introduced to the quality estimation. For example, Adebayo et al. (2018) measure the change in explanation as model paramcters 0 are increasingly randomised, roquiring large perturbations in the parameter space of the model, i.e., (0) >> E to result. in large changes the explanation, i.e., llē Po Po (0), (; >> E.
RRA ORA
||l 0(x, (T(0), 0:1) >
(13)
Complexity (Bhatt et al., 2020; Chalasani et al., 2020; Nguyen & Martinez, 2020) captures the conciseness of explanations, i.c., only a few features should be selected to explain a model prediction. The notion of complexity differs in how is empirically interpreted, e.g., by computing the Shannon entropy of attribution map (Bhatt al., 2020). Alternatively, Chalasani et al. (2020) quantifies complexity by calculating the Gini Index of the absolute value of the attribution vector è where D the length of the attribution vector.
c((e EP (2i D D-
(14)
D DELL é,
D
Localisation (Theiner et al., 2022; Kohlbrenner et al., 2020; Zhang et al., 2018; Rong et al., 2022; Arias- Duart et al., 2021) tests if the explainable evidence is centred around a region of interest, which may be defined around an object by a bounding box, a segmentation mask or a cell within a grid. It requires an additional segmentation mask g E RD, mostly binary mask of the input s.e {0, 10, serving as a simulation or "proxy" of ground truth. While many variations exist, the goodness of é can be defined by, e.g., their intersection divided by their union.
ensue get
M Li sot) = ê sst
(15)
Axiomatic (Sundararajan et al., 2017: Nguyen & Martinez, 2020) estimators measure to what extent explanation fulfil some axiomatic properties such completeness (Sundararajan et al., 2017) and non con-ssnsitivity (Nguyen & Martinez, 2020). Due to the ambiguity that ariscs when empirically evalu- aling estimators in this category, we do not study this calegory in detail.
A.3 Explanation Quality: Estimator Definitions
Within each of the five categories of explanation quality used in this work; (a) faithfulness, (b) robustness, (c) randomisation, (d) complexity and (e) localisation, we benchmarked two estimators per category in our benchmarking experiments. In the experiments that involved comparing meta-evaluation outcomes of estimators within the localisation category across different model architectures, we included two additional estimators.
Faithfulness Correlation (FC) (Bhatt et al., 2020) is defined in the following:
V FF = corr Σ
0(x 0: A)s, f(x) - f (I =.))
(16)
SE|sICd
YES
where |S] C D is a subset of indices of sample x, E is the chosen baseline value and -|,,=,,| is, therefore, the masked input, with indices chosen randomly. Since f denotes a trained model with parameters 0, for brevity, we denote f(x; 6) as f(x) where possible. Higher values indicate that the explanation method's assignment of attribution is correlated with the behaviour of the model and hence is preferred.
22

---

Published in Transactions on Machine Learning Research (06/2023)
Pixel-Flipping (PF) (Bach et al., 2015) returns curve of prediction scores iterative set of pixel replacements, which are sorted in descending order by the highest relevant pixel in the explanation faū A). To return one evaluation score per input sample, we calculate the area under the curve (AUC)
as follows:
T1 PF Σ
(0) D1+1) Pi+1 Ps
(17)
where n is the number of discrete perturbation steps, pi and Pi-1 are the x-values for the jth and (i +i+ 1)th perturbation steps and D1 and Di+1 are the prediction values. For faithful explanations, a steep degradation of prediction scores is expected when attributions are iteratively replaced in descending order. Therefore, a lower value of AUC is indicative of better performance.
Max-Sensitivity (MS) (Yeh et al., 2019) measures the maximum sensitivity of an explanation using Monte Carlo sampling-basse based approximation. It is defined as follows:
0 f. 0) o(x + 5. 0; A)1l
(18) where & defines the radius of a discrete, finite-sample neighborhood around each input sample æ. This neighborhood, denoted as Nr (x), includes all samples in the set X that are within a distance of from æ. A lower MS score is indicative of more robustness.
MC =
max ++)EN
ar
Local Lipschitz Estimate (LLE) (Alvarez-Melis & Jaakkola, 2018) works similarly to the Max- Sensitivity (MS) method and estimates the Lipschitz constant of the explanation, which is measure of how much the explanation changes with respect to the input under slight perturbation, defined as 8, The LLE method is defined as follows:
[0, J.D:
A) (xx + 6. A)ll
LLE
max
(19)
x (x 8)11
where lower values indicate less change with respect to the change in input, which is desirable.
Model Parameter Randomisation Test (MPR) (Adebayo et al., 2018) measures the correlation be- tween explanation from a randomly parameterised model Pr Pr (0;+)) f and the original model f for
each separate layer of the network. To generate one quality estimate per sample, we calculate the average of the correlation coefficients over all the layers in the network, denoted V:
V MPR corr( (Z, f.Di), (x (.;A)))
(20)
where f" f. is the explanation generated by the original model f for layer v and 4" 0s A) is the explanation generated by the randomly parameterised model f for layer v. The correlation between the two explanations is calculated for each layer and then averaged over all layers to generate the MPR score, where a lower correlation coefficient is desired.
Random Logit (RL) method proposed by (Sixt et al., 2020) is originally defined using the structural similarity index (SSIM) over the explanation of the ground truth label and an explanation of non-targe class y'. However, to make comparable with the MPR estimator, the SSIM calculation replacod with the Spearman Rank Correlation Coe(ficient as follows:
RL co f. A), ((x f.)
(21)
where (ax, A) the explanation generated for the prediction & and fx, is the explanation generated for non-target class y/ Lower values indicate that the explanations are not correlated which is desirable.
23

---

Published in Transactions on Machine Learning Research (06/2023)
Sparseness (SP) (Chalasani al., 2020) is method for evaluating the sparsity of explanations and is defined the Gini index of the explanation. It is calculated by summing the product of the ranks of the input features and their attributions and dividing by the sum of the attribution as follows:
(2i (2- -)
SP
(22)
D(D - -i
a higher sparseness score indicates lower complexity the explanation é, which is desirable.
Complexity (CO) (Bhatt et al., 2020) is defined using the Shannon entropy calculation which measures the amount of uncertainty or randomness in the explanation map. It is calculated by summing the product of the probabilities of the attributions and the logarithm of the probabilities of the attributions:
D
Co = E, (P = Σ (Pa()
(23)
(2, f.0:
with (1 = ΣjElq (x, f. A)K P = Po(d).,
where |. denotes the absolute value, (1) denotes the fractional contribution of feature æ, to the total quantity of the attribution. A higher entropy indicates a higher level of uncertainty or randomness, i.e., a higher complexity, A uniformly distributed attribution would have the highest possible complexity score.
Pointing-Gani (PG) (Zhang et al., 2018) captures whether the feature of maximal attribution lies on the ground truth mask, which is binary mask indicating the true features that contribute to the model's output. It is defined as follows:
PC {1
arg max, D.(x,f00: E ggt
(24)
otherwise
where (x, J, 0; A) represents the 8th input feature of highest atttribution and get E RR denotes the binary ground truth mask.
Relevance Mass Accuracy (RMA) (Arras al., 2022) quantifies the fraction of the sum of the attri- bution that intersects with the ground truth mask over the full explanation sum. It is defined follows:
Ee D, A) 8gt,i Ee (2, S, A)
V IMAA
(25)
where (x, f, 0; A) is the attribution of the 7th input feature and 8gt.i is the value of the ith element in the ground truth mask.
Top-K Intersection (TK) (Theiner et al., 2022) extends the Pointing-Gond Game method (Zhang et al., 2018) by measuring the fraction K highest ranked attributions that lic on the binary mask Syt:
81-K with S1-K = -1-K 0 Syt=
VTK
(26)
where 8x denotes the subset of indices of explanation è that corresponds to the K highest ranked features that intersect with Sgt, with r = Rank(é).
Relevance Rank Accuracy (RRA) (Arras et al., 2022) counts the number of features ranked by attri- bution value that intersects with syr
V VRAA-
with 88 0 8gt,
(27)
18t
where se represents the elements of the explanation è that intersect with each tth positive element in the ground truth mask 8gt, with Rank(ē).
24

---

Published in Transactions on Machine Learning Research (06/2023)
A.4 Experimental Setup
In this section, we describe the experimental setup more in detail, which includes the datasets, models, explanation methods and estimators used in this work. We keep this section short since most of the methods in the following have been widely discussed in previous works. For more details, we refer the reader to the respective original publications. The experiments can be reproduced following the instructions in the repository (https: //////tttubb// htwetttww.....tteeeeeeeeeeeeeeee http///mmmaassstreeeeetaaaaaaaaa
Datasets We use four image classification datasets in the experiments: ILSVRC-15 (ImageNet) (Rus- sakovsky et al., 2015), MNTST(LeCunn et al., 2010), IMNIST (Xiao et al., 2017) and cMINST (customised- MNIST) (Bykov et al., 2022). For MNIST and fMNIST, we randomly sample 1024 test samples. We randomly sample 384 test samples from cMINST (customised-MNIST) (Bykov et al., 2022) which consists of MNIST digits displayed on uniformly sampled CIFAR-10 (Krizhevsky et al., 2009) backgrounds. To under- stand the real impact of State-of-the-ar (SOTA), we also perform experiments on ILSVRC-15 (ImageNet) (Russakovsky et al., 2015), using 206 randomly selected test samples in batches of 50 samples.
We have chosen these datasets based on the availability of segmentation masks, since the estimators within the localisation category require such masks for computation. The bounding boxes for these datasets are designed to enclose the object of interest. For cMNIST, the bounding box covers 25% of the input and for MNIST and fMNIST, they cover approximately 35% (but up to 64%) of the imput features. For ImageNet, the bounding boxes vary in size depending on the class of interest.
Models The experiments are performed using different neural network models, including architectures such as LeNets (LeCun et al., 1998) and ResNets (He et al., 2016) which contributes to the robustness of our findings. For MNIST and IMNIST, we train LeNets to an accuracy of 98.14% and 87.44% respectively. For the eMNIST dataset, ResNet-9 is trained to an accuracy of 98.09%. The training of all models is performed in a similar fashion; employing SGD optimisation with standard cross-entropy loss, an initial learning rate of 0.001 and momentum of 0.9. All models are trained for 20 epochs cach. For ILSVRC-15 (Russakovsky et al., 2015), we use the ResNet-18 model (He et al., 2016) with pre-trained weights given the ImageNet dataset, accessible via PyTorch (Paszke et al., 2019). In Appendix A.6.4, we further performed experiments with transformers- based model architectures including Vision Transformer (ViT) (Dosovitskiy et al., 2021) and SWIN Transformer (Liu et al., 2021b), both with pre-traines weights using PyTorch (Paszke et al., 2019).
Explanation methods We employ explanation methods from widely used category of post-hoc at- tribution methods, both gradient-based and model-agnostic techniques, i.e., Gradient (Morch et al., 1995; Baehrens et al., 2010), Saliency (Morch et al., 1995), GradCAM (Selvaraju et al., 2020), Integrated Gradients (Sumdararajan et al., 2017), Inputx Gradient (Shrikumar et al., 2016), Occlusion (Zeiler & Fergus, 2014) and GradientSHAP (Lundberg & Lee, 2017).
In all experiments, we generate explanations for a sample's predicted class ÿ. Whereas certain estimators such the Saliency explanation ignore the signs of the explanations, we refrain from taking their absolute values, to preserve the explainable evidence in the attribution. For comparability, we normalise the explanations prior to their evaluation using the square root of its average second-momen estimate (Binder et al., 2022). which is defined as follows:
(88)
1/2
I HW
En No
where ê is the value of the explanation map at pixel location (h, w) and H, W denote the height and width, respectively".
TThis normalisation ensures that each score in the attribution map has an average squared distance to that is equal to one. Since this operation does not normalise the attributions into fixed range, it is not meant for visuallsations, rather it is meant preserve a quantity that is useful for the comparison of distances between different explanation methods.
25

---

Published in Transactions on Machine Learning Research (06/2023)
Estimators We select the most established estimators within each of the five categories of explanation quality: Complexity (CO) (Bhatt et. al., 2020), Sporseness (SP) (Chalasani et al., 2020), Failhfulness Corre- lation (FC) (Bhatt et al., 2020), Pizel-Flipping (PF) (Bach et al., 2015), Maz-Sensitions 2019), Local Lipschitz Estimate (LLE) (Alvarez-Melis & Jaakkola, 2018), Pointing- Game (PG) (Zhang et al., 2018), Relevance Mass Accuracy (RMA) (Arras et al., 2022), Model Parumeler Randomisation Test (MPR) (Adebayo et al., 2018) and Random Logit (RL) (Sixt et al., 2020). We have defined each of the individual estimators mathematically in Appendix A.3.
(MS) (Yeh et al.,
Parameterisation For the initialisation of the different estimators, we mostly followed the recommenda- tions stated in the respective original publications. However, to make the estimators within a certain explanation category as comparable as possible, alterations to certain hyperparameters were made. When applying Pizel-Flipping (Bach et al., 2015) on image datasets, it generally becomes computationally infea- 2ew sible to iterate over one pixel at a time. Therefore, we iterate over 7) where is the dimensions of the input and w and h are the width and height of the image, respectively (which are assumed to be the same). We also use this same value to choose the subset size for Faithfulness Correlation (Bhatt et al., 2020). For both faithfulness estimators, as the replacement strategy for masked pixcls, we use uniform sampling where we set the lower and higher bounds to the minimum and the maximun value of the lest set, respectively, For the robustness estimators, which both are based on Monte Carlo sampling-best based approximation, we let it run for 10 iterations. In the randomisation category, for comparability, we use the Spearman's 'ss Rank Cor- relation Coefficient to calculate the similarity between the original explanation and the explanation subject to randomisation. A full overview of the parameterisation of the estimators can be found in the GitHub repository https://githun www.comm/aaacc http///maaddssteee/e//aaaaarttte
Hardware All experiments were computed on GPUs where we used NVIDIA A100-PCIE. 40GB for the toy datasets and NVIDIA A100-PCIE 80GB and Tesla V100S-PCIE-32GB for ImageNet dataset.
A.5 Sanity-Checks of the Leta-Evaluation Framework
In this section, we conduct two sanity-checking experiments. In the first experiment, we create and meta- evaluate adversarial estimators to demonstrate the usability of the framework in practice and highlight how the two failure modes act complementary. In the second experiment, we examine the extent that the choice of L, i.e., the set of explanation methods, may influence the MC score.
A.5.1 Adversarial Estimators
To sanity-check the meta-evaluation framework, we created adversarial quality estimators that were intended to perform poorly in specific failure mode and thus, should indisputably fail the corresponding part of the testing scenarios of IPT and MPT. Specifically, we created an adversarial quality estimator that, independent of its given model, data and explanations, returns scores that are always the same (i.e., using deterministic sampling"). As such, this estimator should ultimately fail the reactivity to adversary tests (i.e., IAC and ARCAR) since those tests expect a response to disruption. We denote this estimator V=_. We create a second adversarial quality estimator that, independent of its inputs, returns scores that are drawn from different probability distribution (i.e., using stochastic sampling"). In this setup, we expect poor performance on the noise resilience tests (i.e., IACNR and IECxи) since these tests check that the quality estimates remain relatively unchanged after perturbation. We denote this adversarial estimator y.
Table 2 summarises the outcome of this exercise, which includes the four score elements TACNR, TACAR: IEC and IECAR, aggregated for 5 itcrations with K = 10 for the two tests, IPT and MPT. The expectation of the test outcome indicated by the value in brackets after the display of the actual score, including the
We assemble this adversarial estimator by repeatedly returning the same scores for of set of uniformly sampled scores > Lffa, (8) with a 0 and 8 = Herr, sample from normal distribution (2) with の² 1 but with different means for the unperturbed- and the perturbed estimates, respectively. For the unperturbed estimates 4. we sample z from wide range, L.e,, & [- [-100000,- -1] and for the perturbed estimates g', we set narrow range with y E [0. 1].
26

---

Published in Transactions on Machine Learning Research (06/2023)
standard deviation. Here, value of 0 indicates that the test should fail and any other value indicates the desired outcome of the test to be successful. From Table 2, we note that both estimators, V and == produced scores that align with the expected value. Since estimator ye, relies on stochastic sampling, the scores are approximate, nevertheless, the scores and expectation are close and the standard deviation is small, indicating that the sanity checks results are stable. Overall, we can observe that the expectation aligns with the empirical reality across the different test settings. Therefore, we conclude the sanity-checking experiment to be passed.
Another important insight that can be drawn from Table 2 is that the two failure modes complement each other in determining the performance of an estimator. For an estimator that is provably bad, i.c., returns scores that are completely unrelated to the model, data and explanation methods (such as demonstrated by V and =)), at least one of the failure modes (AR or NR) will reveal that the estimator is failing. To fully assess the performance of an estimator, both failure modes are therefore necessary.
The sanity-check exercise results show aggregated scores including std, over 5 iterations with = 5. The direction of the arrow, i.e,, t indicates if higher value is better. The expectation of the test outcome is indicated by the value in brackets, after the display of the actual score.
Table 2:
Test
Estimator
IACNN (t)
LAC (1)
IECNR (f)
IEC An (t)
IPT
V
0.015 + 0.023 (0.00)
0.983 0.011 (1.00)
0.248 + 0.004 (0.25)
0.0 + 0.0 (0.00)
V
1.0 = 0.0 (1.00)
0.0 0.0 (0.00)
1.0 = 0.0 (1.00)
0.0 + 0.0 (0.00)
MPT
V
0.014 1 0.010 (0.00)
0.973 0.019 (1.00)
0.248 1 0.003 (0.25)
0.0 d: 0.0 (0.00)
V
1.0 0.0 (1.00)
0.0 0.0 (0.00)
1.0 = 0.0 (1.00)
0.0 + 0.0 (0.00)
A.5.2 Dependency on L
The meta-evaluation framework is intentionally designed to take into account the set of explanation hods given in the setup. For example, in the cotrisconsit criterion (IEC) for noise resilience, we compute the estimator's ability to rank different explanation methods consistently when exposed to minor perturbations The resulting MC score of a quality estimator will, therefore, to a certain extent, be dependent on the choice of L: both in terms of ils cardinality and how similar the explanation functions are.
To understand how the performance of our quality estimator may vary depending the choice of L, we conducted an experiment where we computed the MC score while enumcrating various choices of L. In this experiment, we vary both the cardinality of L, by choosing values of {2,3, 4] and the methods included in the set. We selected both model-agnostic explanation methods such as Occlusion (Zeiler & Fergus, 2014) as well as gradient-based techniques such as GradCAM (Selvaraju et al., 2020), Integrated Gradients (Sundararajan et al., 2017) etc. For the sets of 2 explanation methods we included: { (Gradient, Occlusion), { Gradient, Imputx Gradient), { (Gradient, Saliency), (Gradient, Inputx Gradient)- For sets with 3 methods: { Gradient,
MNIST
eINIST
0.90
0.70
0,72372
0.67
0.3
0.68
0.68
3.05
0.50
MC
0,50
0.50
0.990.59
0.62
0.57
0:56
SP
RMA
MPR
Figure 9: Comparison of averaged meta-consistenny performance for different quality estimators using MPT and IPT, aggregated over 3 iterations with K = 5, across models [TeNet, ResNet) and different datasets (MNIST, JMNIST, cMNIST) with error bars showing the standard deviation.
10 exception the expected value of the inter-consistency score, IECxR, for estimator / not 0.0 but 0.25. This is becanse, for an estimator that assigns attributions randomly, Lee, independent of the explanation method, the likelihond of the condition AM = 1 is z.
27

---

Published in Transactions on Machine Learning Research (06/2023)
GrodCAM, GradientSHAP): (Gradient, Saliency, Integrated Gradients) and for methods: {Gradient, Saliency, Impulx Grudient, GradCAM), (Grudient, Saliency, Occlusion, GrudCAM).
In Figure 9, we show the aggregate values for different explanation sets across the datasets separately. Here, the error bars indicate the standard deviations. By comparing the MC scores category by category, we can observe that the error bars from the respective estimator do generally not overlap. This means that the choice of L has limited influence on the MC score, suggesting the measure's stability.
A.6 Supplementary Experiments
the following section, we present additional experiments conducted in the scope of this work. First, we demonstrate that MetaQuantus can be used for additional applications in Explainable AI. Here, we include two demonstrations, first, show how the MC score can be employed as target variable for optimising the hyperparameters of an estimator and second, we demonstrate how the framework can be used to analyse calegory convergence. At the end of this section, we discuss supplementary results for the benchmarking experiment which includes an additional analysis of ranking consistency.
A.6.1 Application - Hyperparameter Optimisation
It is practically well-known and increasingly publicly recognised (Bansal et al., 2020; Brunke et al., 2020; Brocki & Chung, 2022; Pahde et al., 2022) how difficult it can be to tune the hyperparameters in the ex- plainability domain. Unlike in traditional MI., in XAI, we generally do not have target variable to optimise against. As an additional experiment, we, therefore, investigated how the meta-evaluation framework can useful in solving the task of selecting the best set hyperparameters for a given estimator. For this, we choose an estimator with relatively many parameters, that Faithfulness Correlation (Bhatt et al., 2020) and performed grid-scarch on these using ImageNet. By exploring combinations of three bascline strategies ['Black', 'Uniform', 'Mean') and four subset sizes [28, 52, 102, 128], we created 12 hyperpa- rameter settings", We determined the performance of each of the estimator's parameterisation by storing the meta-evaluation vector m and the MC score at cach run. The objective of this excrcise is to determine the hyperparameter setting that optimises the performance of the estimator, i.e., its resilience to noise and reactivity to adversary.
IFC, NR
IAC,
IEC, AR
JAC_AR
MNIIT
MNIST
dMNIST
Relalive Scoring Criteria
3020 46600.48
0.54400.402
Spearmnn Rank Correlation
0.50
0.55
0.8
0.37
0.40
0.40
0.44
0.4
P2
P5
P10P11P12
0.0
Outside Calegory
Within Category
Figure 10: Left: The results of using the meta-evaluation framework to optimize the hyperparameters of FC (Bhatt et al., 2020) estimator across 12 parameterisations (P1-P12) on ImageNet dataset, averaged over 3 iterations with K = 3. The parameter selting P11 demonstrated the highest scores with small slandard deviation and thus is selected as the parameter setting, Right: The results from comparing the correlation coefficients between meta- evaluation vector scores for estimators within the same category versus those outside of the category, suggesting that the estimators the same category have more resemble with respect to its performance characteristics compared estimators outside.
The parameters combined in the following 12 scttings: P1: ['Black', 28), P2: ['Black', 52), P3: ['Black', 102), P4: (BBacck' 128], P5: ['Uniform', 28', P6: [Uniffomm, 52), P7: ['Uniform', 102], P8: ('Uniform", 128), ['Mean', 28), P10: ["Mean'; 62), P1:: (MMaan,, 102], P12: ['Meann, 128).
28

---

Published in Transactions on Machine Learning Research (06/2023)
From Figure 10 (left), we can observe that P11 has the highest meta-consistency score and as such, we recommend the associated parameter setting of "mean" as the replacement strategy with 102 features as the subset size. In contrast to previous works that found a relatively large difference in evaluation outcomes between different parameterisations of faithfulness estimators (Tomsett et al., 2020; Brunke et al., 2020; Rong et al., 2022; Hedström et al., 2023), we detect, that with the MC score- which provides a more comprehensive picture of the estimator's performance- there is not a considerable variability, as depicted by the similarity in LAC and IEC scores over P1 to P12.
A.6.2 Application - Category Convergence
The question of whether quality estimators within the same category are measuring the same underlying concept has been of significant interest to the communily (Tomsett el al., 2020; Gevaert el al., 2022). Based on the observed similarity of estimator shapes in Figure 5-that the estimators within the same category typically have a higher resemblance in area shapes compared to estimators outside of their categories we sought to employ the meta-evaluation framework to investigate whether estimators within a category exhibit a greater level of correlation than those outside of the category. To address this question, often referred to as "convergent validity", the prevalent technique has been to mcasure intra-corrclation, which simply involves correlating the scores of different estimators within the same calegory. This approach, however, has limitations, as it disregards the aspect of ranking consistency (IEC) and may not account for the fact that scores from different estimators may have different scales and interpretations, which may skew the results.
We improve upon the current methodology proposed in Tomsett et al. (2020); Gevaert et al. (2022) by calculating the correlation coefficient on the meta-evaluation vector m of different estimators, within- and outside of their category as produced in the benchmarking exercisc. This approach advantagcous as it: (i) yields scores in normalised range [0, 1] and (ii) provides a more comprehensive view of the estimator's performance characteristics by incorporating multiple failure modes and criteria.
Figure 10 (right) presents the results of this experiment, averaged over all estimators as described in A.4. Here, we can observe that the estimator's performance characteristics are more similar within a category, as seen in the higher correlation coefficient. (Spearman Rank: Correlation Coefficient) across all datasets. These observations contrast previous works by Tomsett et al. (2020); Gevaert et al. (2022) that found a low correlation coefficient (for faithfulness estimators in particular). We posit that this difference can be explained by the fact that the meta-evaluation framework considers multiple failure modes and criteria of
Table 3: Meta-evaluation benchmarking results with MNIST, aggregated over 3 iterations with K = IPT results are in grey rows and MPT results are in white rows. MC denotes the averages of the MC scores over TPT and MPT. The top-performing MC- or MC method each explanation category, which outperforms the Nottom-performing method by at least 2 standard deviations, is underlined. Higher values are preferred for all scoring criteria.
Category
Estimator
MC (t)
MC (t)
IACNR (t)
IAC (t)
IECaR (f)
IEC AR (t)
0.640 = 0.043
0.209
0.040
0.946 : 0.086
0.837 : 0.002
0.569 + 0.046
Sparseness
0.558 +: 0.028
Complexity
0.476
0.013
0.929
0.063
0.053 + 0.014
0.840
0.005
0.084
0.001
Complexity
0.521 + 0.003
0.541
+ 0.007
0.000
0.013
1.000 + 0.000
1.000
0.000
0.156
+ 0.014
0.500
0.000
0.167
0.000
0.833 = 0.000
1.000
0.000
0.000
0.000
0.537
0.003
0.477
0.032
0,900
= 0.023
0.190
0.003
0.579
0.008
Faithfulloss Corr.
0.540 + 0.015
Faithfulness
0.543
0.026
0.500
0.107
0.850
+ 0.005
0.190
0.002
0.584
0.005
Pixel- Pixel-Flipping
0.626 + 0.039
0.609
0.039
0.547
0.139
0.963
+ 0.034
0 0.299
0.001
0.626
0.046
0.644
= 0.038
0.485 + 0.141
1.000
= 0.000
0.294
0.006
0.796
= 0.006
0.672 + 0.020
0.977
+ 0.005
0.607
= 0.075
0.996 = 0.000
0,108
- 0.012
Pointing-Game
0.586 + 0.010
Localisation
0.500
+ 0.000
1.000
+ 0.000
0.000
+ 0.000
1 1.000
0.000
0.000
+ 0.000
Relevance Mass Acc.
0.552 + 0.015
0.613
+ 0.022
0.258
+ 0.062
0.793
+ 0.023
0.846 + 0.001
0.553
+ 0.032
0.491 = 0.007
0.940
+ 0.019
0.071
= 0.019
0.902 = 0.003
0.051
- 0.000
0.712 = 0.008
0.360
+ 0.041
0,969
+ 0.010
0.937 = 0.003
0.581
- 0.006
Random Logit
0.666 + 0.00
Randomisation
0.620
0.000
0.186
+ 0.000
0.874
+ 0.000
0.860
+ 0.000
0.562
+ 0.000
Model Param. Rand.
0.583 + 0.007
0.624
+ 0.005
0.264
+ 0.019
0.959
+ 0.000
0.764 + 0.002
0.510
+ 0.001
0.542
= 0.010
0.250
+ 0.065
0.500
= 0.028
0.647
= 0.003
0.463
= 0.004
Max-Sensitivity
0.619 :: 0.007
0.754
= 0.002
0.547
+ 0.064
0,938
+ 0.033
0.804
= 0.001
0.726
= 0.03S
Robustness
0.545
0.012
0.361
0.053
1.000
+ 0.000
0.806
0.005
0.011
0.001
Local Lipschitz Est.
0.741 + 0.030
0.726
+ 0.026
0.484
0.091
0.935
+ 0.088
0.736
0.002
0,750
+ 0.034
0.756
0.034
0.519
0.118
0.974
= 0.017
0.740
0.005
0.789
0.006
29

---

Published in Transactions on Machine Learning Research (06/2023)
Table 4: Aleta-evaluation benchmarking results with fMNIST, aggregated over 3 iterations with A 5. IPT results are in grey rows and MPT results are in white rows. MC denotes the averages of the MC scores over IPT and MPT. The top-performing MC- or MC method each explanation category, which outperforms the botton-performing method by at least 2 standard deviations, is underlined. Higher values are preferred for all scoring criteria.
Category
Estimator
MC (1)
MC (t)
IACvR (f)
IACAR (1)
IECvR (1)
IECAR (t)
0.506 + 0.012
0.145 + 0.039
0.915 = 0.045
0.831 : 0.004
0.492 + 0.082
Sparseness
0.536 + 0.011
Complexity
0.475
0.010
0.917 + 0.036
0.070 + 0.003
0.832
0.003
0.083 + 0.001
Complexity
0.516 + 0.007
0.532
= 0.014
0.050
0.017
0.990 = 0.027
0.999
: 0.000
0.086 = 0.02S
0.500
0.000
0.167
0.000
0.833
+ 0.000
1,000
0.000
0.000
0.000
0.524
0.021
0.527
0.030
0.857
= 0.072
0.198
0.008
0.515
0.004
Faithfuluess Corr.
0.530 + 0.021
Fuithfulness
0.536
0.021
0.448
0.087
0.984
= 0.003
0.196
0.004
0.504
0.002
Pixel-Flipping
0.530 + 0.021
0.573
0.025
0.447
0.050
0,938
= 0.088
0.329
0.002
0,558
0.032
0.649
0.018
0,453
0.073
1.000
+ 0.000
0.324
0.001
0.817
0,003
0.666
= 0.009
0,950 = 0.025
0.634
= 0.032
0.995 = 0.001
0,084 = 0,018
Pointing Game
0.583 + 0.005
0.500
+ 0.000
1,000 + 0..000
0.000
= 0.000
1,000 + 0.000
0,000 + 0,000
Localisation
0.587
= 0.024
0.231
= 0.102
0,506
= 0.056
0,850 = 0.007
0.460 = 0.048
Relevance Mass Acc.
0.538 + 0.023
0.490 + 0.022
0.944
+ 0.034
0.067
+ 0.067
0,894 + 0.003
0.055 + 0.003
0.717 = 0.010
0.234 = 0.039
1.000
= 0.000
0.955 = 0.005
0.680
= 0.005
Random Logit
0.689 + 0.005
0.660 + 0.000
0.062 + 0.000
1.000
= 0.000
0.902 = 0.000
0.677 = 0.000
Randomisation
Model Param. Rand.
0.570 + 0.010
0.622 = 0.010
0.355 + 0.042
0.925
7 0.000
0.755 = 0.005
0.451 = 0.000
0.518 + 0.010
0.098 + 0.008
0.902
+ 0.045
0. 0.657
+ 0.004
0.414 + 0.001
Max Mex-Seesittvity
0.639 + 0.036
0.699 = 0.037
0.515 = 0.097
0.961
= 0.021
0.810
0.007
0.501 = 0.058
Robustness
0.580 = 0.035
0.504
0.141
1.000
= 0.000
0.811
0.002
0.004 = 0.000
Local Lipschitz Est.
0.710 + 0.022
0.696 = 0.038
0.538
0.139
0.979
= 0.033
0.775 + 0.005
0,492
= 0.092
0.724 0.006
0.567
0.037
0.896 + 0.024
0.774 0.001
0.661 0.006
Table 5: Mela-evaluation benchmarking results with cMNIST, aggregated over 3 iteralions with K = 5. IPT results are in grey rows and MPT results are white rows. MC denotes the averages the MC scores over IPT and MPT. The top- perfformii MC- or MC method in each explanation category, which outperforms the bottom--erforming method by at least 2 standard deviations, is underlined. Higher values are preferred for all scoring criteria.
Category
Estimator
MC (1)
MC (1)
IACva (1)
IACAR (1)
IECyR (t)
IECaR (1)
0.706 + 0.013
0.352 + 0.061
0.989 : 0.017
0.814 + 0.001
0.670
0.016
Sparseness
0.616 + 0.015
Complexity
0.525 + 0.018
0.626 + 0.099
0.313 = 0.028
0.830 + 0.005
0.333
0.006
Complexity
0.541 +: 0.018
0.565 + 0.024
0.056 + 0.084
1.000 + 0.000
0.996 # 0.001
0.209
0.013
0.518 + 0.013
0.062 + 0.010
0.928 + 0.047
1.000 + 0.000
0.080
0.005
0,563 + 0.017
0.508 1 0.061
0.939 1 0.017
0.182 1 0.004
0.622
0.005
Paithfulness Corr.
0.562 + 0.014
Faithfulness
0.562
+ 0.010
0.490 + 0.031
0.934
+: 0.018
0.188 + 0.008
0.634
0.012
Pixel FlippiFiip
0.604 + 0.016
0.586
0.022
0.565
1 0.040
0.963
= 0.022
0.287
: 0.005
0.526
0.080
0.621
0.010
0.495
1 0.037
0.995
= 0.001
0.293
=2 0.012
0.701
0.002
Pointing-Game
0.687 + 0.006
0.873
0.010
0.967
+ 0.000
1.000
0.000
0.997
0.000
0.527
0.040
0.502
0.001
0.993
0.003
0.013
0.003
0.999
= 0.001
0.001
0.000
Localisution
Relevance Mass Acc.
0.621 + 0.011
0.856
0.020
0.751 + 0.008
0.358
0.055
1.000 = 0.000
0.791
0.012
0.491
0.014
0.640 + 0.028
0,306
0.032
0.796
0.003
0,223
0,005
Random Logit
0.713 + 0.005
0.723 + 0.010
0.530 + 0.065
0.834
+ 0.026
0.881 + 0.007
0.586
0.012
0.703 + 0.000
0.410 + 0.000
0.884 + 0.000
0.913 = 0.000
0,606
0.000
Randomisation
Model Param. Rand.
0.657 + 0.009
0.673 + 0.003
0.490 + 0.006
1.000 + 0.000
0.814 # 0.005
0.387
0.000
0.641 + 0.016
0.417 + 0.058
1.000 + 0.000
0.804 + 0.005
0.344
0.004
0.690 + 0.035
0.494 1 0.069
0.972 + 0.045
0.687 1 0.004
0.606
= 0.050
Max-Sensitivity
0.637 + 0.030
0.583 + 0.024
0.382 + 0.079
0.992 + 0.002
0.680 + 0.019
0.080
0.002
Robustness
Local Lipschitz Est.
0.697 + 0.020
0.689
0.026
0.548 1 0.077
0.971
: 0.049
0.628
= 0.007
0.609
0.042
0.706 0.014
0.308 1 0.041
0.999
= 0.000
0.630 = 0.007
0.685
0.005
what a quality estimator should fulfil and not only one, e.g., ranking consistency (Rong et al., 2022) and such, give more comprchensive answer. However, from the error bars in Figure 10, we also observe, that the correlation coellicients are greatly varying within each group. Further research is thus necessary to fully understand the extent to which estimators of the same explanation quality category measure the same underlying concopt.
A.6.3 Supplementary Results Benchmarking
Similar to Table 1, we present the results of the IPT and the MPT for the MNIST, fMNIST and cMNIST datasets in Tables 3-5, respectively. The grey rows indicate the results from the IPT and the white rows
30

---

Published in Transactions on Machine Learning Research (06/2023)
Sparsariss (0, 5584)
Fast fudress Conmuatin (6.5400)
Garetng-Game (0.5850)
Randem Lagit (0 (0,8955)
Max Sonsitityyy (0,8492)
MC
MCM
0.5
0.
05
C55
0L5
0.5
0.5
0.5
0.
0.5
Complesiyy
Poe--llpping (261)
Accuracy (0.5510)
Lipeciit Estimate
0.5
0.0
0.0
0.
0.5
MC
10
Figure 11: A graphical representation of the MNIST benchmarking results (Table 3), aggregated over 3 iterations with K K= 5, Each column corresponds to a category of explanation quality, from left to right: Complezity, Faithfulness, Locolisation, Randomisation and Robustness. The grey area indicates the area of an optimally performing estimator, ie., m = 1 The MC score (indicaled in brackels) is averaged over MPT and IPT. Higher values are preferred.
Sparseness (0.5256)
Faitiiness Comiiatio (6.5258)
Pointng-Came (0.5828)
Randam Logit (0 (0.8688))
Aus-Sensithtty (3 (3.8391)
XCa
MCa
KC
VCCa.
KSCAb
0.5
0.5
0.5
c.5
0.5
Complecly (O 51501
Positiipin (U (1.110))
Releaane Vars Accurapy (0:3390)
Modil Parameeer Ranstmm (0.2896)
Lotal Lipedhiiz Extimate (C: (C.7102)
R00
Hoal
MCA
MCM
0.5
0.5
05
0.5
0.5
0.
05
0.5
05
0.5
47//
0.5
2.5
0.5 8.5
25.5
13.5
13. 0.5
13.
0.5
Figure 12: A graphical representation of the fMNIST benchmarking results (Table 4), aggregated over 3 iterations with K 5. Each column corresponds to calegory of explanation quality, from left to right: Complexity, Faithfulness, Locolisation, Randomisation and Robustness. The grey area indicates the area of an optimally performing es estimator, i.e. m' 1 The MC score (indicated in brackets) averaged over MPT and IPT. Higher values are preferred.
show the results from the MPT. The results are consistent with those presented in the main manuscript, both in terms of individual score criteria, estimator- and category comparison.
Similar to Figure 7, we also represent Tables 3-5 as area graphs. With an exception of slightly higher localisation scores for cMNIST dataset (as explained in the main paper), the results as demonstrated in Figures 11-13 are completely consistent with those findings presented in the main paper. Recall that, larger coloured areas imply better performance on the different scoring criteria and the grey area indicates the area of an optimally performing quality estimator, i.e., m = 1 Each column of estimators represents a category of explanation quality, from left to right: Complerity, Faithfulness, Localisation, Randomisation and Robustness.
We further visualise the results (as shown in Tables 1, 4 and 5) as scatter plots for ImageNet, fMNIST and cMNIST datasets in Figure 14. In the main paper, we identified that the faithfulness category (blue points)
31

---

Published in Transactions on Machine Learning Research (06/2023)
Sparsariss (0 80)
Fast fudress Conmuatin (6.5622)
Coeetng-Game (0.66122)
Randem Logil (0.7128)
Son--Sesiitttyy (G (0.8388)
MCA
0.5
05
05
055
0L5
XCa
0.5
0.5
0.5
0.5
0.5
Complecty
Poe-Fllypig
Accuracy (0 (2022)
Estimate (..873))
0.5
0.0
0.0
0.0
10
Figure 13: A graphical representation of the eMNIST benchmarking results (Table 5), aggregated 3 iterations with K = 5. Each column corresponds to a category explanation quality, from left to right: Complexity, Faith- fulness, Localisation, Randomisation and Robusiness. The grey area indicates the area of optimally performing estimator, Le., m² = 1 The MC score (indicated in brackets) is averaged over MPT and IPT. Higher values are preferred.
ImageNet
ImageNet
FMNIST
INNIST
cMNIST
cMNIST
1.0
1.0
1.0
1.0
1.0
1.0
AA
AR
IACAR
|RC
IACaR
ARCAR
0.5
0.5
0.5
0.5
0.5
0.5
IPT
IPT
IPT
IPT
IPT
0.0
MPT
0.0
MPT
0.0
MPT
0.0
MPT
0.0
MPT
4
0.0
MPT
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
0.0
1.0
IA |ACNR
IECNR
IAC
IEC
IAC
IECNR
Figure 1:: A supplementary visualisation the benchmarking results (Tables 1, 4 and rettalllbeenchnaiinngennss in particular IAC and IEC scores for noise resilience (x-axes) and adverse reactivity (y-axes). The colours indicate the estimator and the symbols demonstrate the test: IPT and MPT, respectively. Higher values are preferred.
had a particularly low ranking consistency (TEC), which is also evident in these supplementary plots. From Figure 14, we can moreover observe how the estimators' scores on the respective failure modes are related. These plots also show that a higher resilience noise does not necessarily imply more reactivity to adversary and vice versa the performance characteristics of the estimators are more complex than that.
A.6.4 Supplementary Results - Comparing Neural Network Architectures
As part of our exploration of meta-evaluation outcomes across varying model architectures, including those basod on transformers, we embarked on additional benchmarking experiments. These involved comparing estimators from both the localisation and complexity categories of explanation quality among Vision Trans- former (Dosovitskiy et al., 2021), SWIN (Liu et al., 2021b), and ResNet-18 18 (He et al, 2016) models using the ImageNet dataset (Russakovsky et al., 2015). The outcomes concerning the localisation category have been discussed in the main manuscript 6.1.3. The results for the complexity category, including estimators such as Complexity (CO) (Bhatt et al., 2020) and Sparseness (SP) (Chalasani et al., 2020) are discussed below. For this experiment we used two explanation methods L =Inpux Gradient, Gradients.
To analyse the results, we represent the entries of the meta-evaluation vector as coordinates on 2D plane and visualise the benchmarking results as area graphs, as seen in Figure 15 (left). Again, larger coloured
32

---

Published in Transactions on Machine Learning Research (06/2023)
areas imply better performance the different scoring criteria and the grey area indicates the area of an optimally performing quality estimator, i.e., m = 1.. Each column represents the estimator and each row the network architecture, as indicated by the labels of Figure 15 (left).
Spanoertres (0,83722)
Complenty 20 50000
Undil
MNIST
IMNIST
CMNIST
1.4
%% Top Rank
75%
0.6
0.5
0.5
0.2
08
Sparseness (0.5721))
Complesty 20 5000)
1.
14
Mise
25%
05
05
Complexty Falthtuiness Localistion Randomisatio Robustness
5
MNIST
APP
IMNIST
CMNIST
100%
0:-0
070
Scorseness (0.5783)
Complexit 5000)
% Top Rank
75%
08
0-:0
Medi
0.
50%
500N
MGGA
AaCa
25%
03
05
Complexiy Faithluness Localisation Ramdommission Robusteess
0.5
0.5
35
0
0.5
Figure 15: Left: A graphical representation of the ImageNet benchmarking results for the complexity category, aggregated over 3 iterations with K = 5. Each column represents the estimator, from left to right: Sparseness and Complexilty, and each row the model type [ResNet-18, ViT, SWIN), as indicated by the labels. The grey area indicates the area of an optimally performing estimator, ie., m = 1". The MC score (indicated in brackets) is averaged over MPT and IPT. Higher values are preferred. Right: A visualisation of the benchmarking results (Tables 3, 4 and 5) showing the distribution of top rankings within each category of explanation quality. There higher variability in scores for the IPT tests (above), while it is lower for the MPT tests (below).
As can be observed in Figure 15 (left), the areas indicating the performance profiles of each individual es- timator across the network types demonstrate a noticeable similarity. In accordance with the observations made in the main manuscript, the Sparseness method (Chalasani et al., 2020) demonstrates superior perfor- mance compared to the Complerity method (Bhatt et al., 2020). Overall, this provide additional evidence in proving the framework to be agnostic with respect to the choice of network architecture.
A.6.5 Supplementary Results Ranking Consistency
In the main paper, we presented the average MC scores for each dataset in Figure 5, which showed that the best-performing estimator each category of explanation quality generally remained consistent across tested datasets. To further explore this consistency, we considered a margin of error of 2 standard deviations applied to the MC scores and re-caleulated the within-calegory ranking for each individual estimator in each category and visualised the results in Figure 15 (right).
Figure 15 (right) showeases the the frequency with which the different estimators within each category achieved the highest or the lowest average score, respectively. The color scheme, consisting of one lighter and one darker hue for each category, is consistent with the individual estimators in previous figures. A larger fraction of bar with a single huc indicates more "category wins" for a specific estimator. From Figure 15 (right), similar observations in the main paper, we find that estimators within the localisation and randomisation categories are prone to stronger score variability. In general, we anticipate variations in
33

---

Published in Transactions on Machine Learning Research (06/2023)
MC scores across all categories between different datasets. This is because all quality estimators evaluate explanation qualities, which depend on the specific dataset and network in use. Additionally, as seen in Figure 15 (right), the score variability for IPT (above) is higher than that of MPT (below). Therefore, in the main paper, we present the MC scores averaged over both MPT and IPT to capture a more comprchensive view of the estimator's performance.
A.7 Notation Table
In the following, we provide notation tables that encompasses all the notations employed throughout this paper. The table is sorted according to the section in which each notation initially appeared.
Preliminaries
f
A black-box model function that maps input æ output The parameters of the model function The training dataset on which the model f is trained The test dataset on which the model f is evaluated An input in the instance space X
X
An output class in the label space y A prediction made by the model f The number of output classes The number test samples The dimension of the input
C
N
D
X
The instance space The label space
F
The function space of all models
+
An explanation function that maps x, f, and ý to an explanation map
The parameter of the explanation function +
The explanation map produced by +
E
The space of possible explanations
V
A quality estimation function that takes é and returns scalar & to indicate its quality
The parameter of the quality estimation function W
A quality estimate made by the estimator V
The verifiable spaces the estimator's V input parameters Ω E
A Meta-Evaluution Framework
U
The unverifiable spaces of the estimator's input parameters U ([E), (0), (E,0)) The first failure mode, Noise resilience
NR
AR
The second failure mode, Adversary reactivity
y'
A prediction after perturbation on the input, model or both input and model spaces
A threshold e e R for determining the type of perturbation
The perturbation strength (M, D}
Pn(w)
A perturbation function of the verifiable spaces w E Ω
PM
A minor perturbation function of the input space X
PMM
A minor perturbation function of the function space
PD
A disruptive perturbation function of the input space
34

---

Published in Transactions on Machine Learning Research (06/2023)
Pp
A disruptive perturbation function of the function space IF The number of perturbations
K
L
The set of explanation methods The unperturbed quality cstimates q E RR
q
dh
The perturbed quality estimates, replicated K times for test samples A statistical significance function that takes q and dh and returns a p-value
A ranking function that takes nominal values and returns rankings in descending order
Q
A matrix of all perturbed samples over K perturbations
Q
A matrix for the unperturbed estimates q for L explanation methods, averaged over K
Q'
A matrix for the perturbed estimates Te for L explanation methods, averaged over K
QM
A matrix for the perturbed estimates under minor perturbation
Q
A matrix for the perturbed estimates under disruptive perturbation
U
A binary ranking agroement matrix that takes quality estimates from Q and Q' and populates the entries according to the interpretation of ranking
UM
A binary ranking agreement matrix with perturbed estimates under minor perturbation
UD
A binary ranking agrecment matrix with perturbed estimates under disruptive pertur- bation
A meta-consistency vector containing the IAC and IAC scores for both failure modes
m"
An optimally performing quality estimator V as defined by the all-one vector 14
IAC
The intra-consistency scoring criterion, with IAC E [0, 1]
IEC
The inter-consistency scoring criterion, with IEC E [0. 1]
MC
The meta-consistency score, with MC E [0,
Practical Evaluation
u
The uniform distribution with parameters a
The lower bound of the uniform distribution (d,,33)
8
The upper bound of the uniform distribution U( 33)
8,
Additive uniform noise applied to input space such that è; = x 5,
N
The normal distribution with parameters E. E
The mean the normal distribution N(, M)
>
The variance of the normal distribution N ((, E)
Multiplicative Gaussian noise applied applied to model parameters such that ô 0.v,
Vi
35


Reviews:
[{"review_id": "04Vni1ML5Q", "reviewer": "Reviewer_W9T7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Interesting method to capture explanation quality\n2 The motivation seems reasonable and paper is overall well written\n\nWeaknesses:\n1. Clearly this evaluation is designed for feature based explanations, although there are many types of explanations. I think the title and content should reflect this fact.\n2. Experiments are fairly limited. Not sure the hyper focus on MNIST is as desirable. As a result assessing generalizability of the metrics is a bit challenging. \n3. A bigger challenge of explanations is not that there is no unifying way to evaluate them, its that conceptually they are technically estimated in very different ways, making different assumptions, and prone to different types of failure models. Hence the biggest problem for a practitioner is actually assessing whether or not a particular explanation's assumptions are suitable in their problem context. On the other hand the proposed evaluation framework hides this issue even further. \nFor example, feature importance methods that sample perturbations from marginals, vs conditionals are assessing wildly different sensitivity of the model and their relevance is quite subjective depending on what the user wants to get out of an explanation. By proposing a framework and claiming to be fairly general, I think this will worsen the problem where the actual problems with explanations are now buried under the abstraction of these metrics. Given these concerns, I strongly urge the authors to highlight very clearly, the limitations of the method they are proposing and why it is useful to think of evaluation in this way. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is proposing an evaluation framework for explanation methods, particularly feature based explanation methods. The two main evaluations studied are around: i) how resilient an explanation method is to noise, i.e. small perturbations. ii) should result in small perturbations in explanations and ii) whether an explanation method significantly responds to dsruptive perturbations. Based on this, two \"meta\"-evaluations have been suggested, on data input and model parameters. Finally the main meta evaluation consists of Intra vs Inter consistency evaluations of these explanation methods to measure these metrics before and after the perturbations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my concerns above regarding the utility and scope of the evaluation framework proposed. \nI will reassess my evaluation based on how authors address those concerns. ", "broader_impact_concerns": "I believe the authors sufficiently address part of the broader impact concerns but see above for other comments.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HM1urHgpIC", "reviewer": "Reviewer_8A54", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\nThe paper is well written and motivated, including referencing relevant related work and discussing current issues of XAI evaluation metrics. \nXAI methods and their evaluation are highly relevant, keeping the impact of current deep learning systems in mind. The framework has the potential to be an important tool in future research. \nA wide range of XAI evaluation metrics is empirically investigated, providing novel insights into their advantages and disadvantages.\n\n**Weaknesses:**\n\nUnfortunately, the framework is not evaluated on SOTA deep networks and is limited to image classification tasks (and only convolutional networks), including syntactic datasets. This is, in general, not a major concern. However, especially conflicts with the claims contained in the introduction. \nE.g., in this regard, it confuses that the paper claims to evaluate metrics based on the ImageNet dataset. At the same time, this is not contained in the main text’s evaluation and only utilized in a hyperparameter optimization in Appendix Fig. 8. Statements in the introductions such as \"wide applicability across various data-, models-, explanation methods- and metric domains\" and \"a series of experiments on a variety of SOTA explanation methods, datasets and models\" are misleading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a framework to evaluate metrics measuring the quality of explainable AI (XAI) methods. \nThe framework aims to help users to select XAI methods based on given tasks and datasets. Further, it could help to optimize a metric’s hyperparameters.  \n\nThe present paper claims that the framework is widely applicable across various data-, models-, explanation methods- and metric domains. To this end, the framework measures the influence of \"minor\" and \"disruptive\" perturbations. Since different metrics ranking can strongly vary, two consistency criteria are introduced, finally leading to a single Meta-Consistency (MC) score. Empirical evidence is provided by experiments conducted on image classification tasks on the datasets MNIST and fMINST and variations. Multiply XAI evaluation metrics are investigated using the introduced framework. The framework is limited to post-hoc attribution methods. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the authors mention the limited evaluation in Section 7, other tasks and networks (e.g., transformers and their corresponding novel XAI methods) would strengthen the paper and its impact. Further, it is contrary to what is claimed in the introduction. So I recommend either lowering claims or providing additional evidence.\n\n2. In Sec 2.1:\n\"[…] we compute the explanation error, requiring a ground truth explanation e. These labels are, however, generally not available for complex ML models and in particular NNs, since their inner workings are considered unknown.\" Why are ground truth explanations not available for complex ML models? A ground truth explanation would depend on the dataset and task and not on the model. Further, how is complex defined? Is it a two-hidden-layer NN complex or a 175B parameter model complex? Is there a general difference, and how would ground truth explanations using attribution maps differ?\n\n3. Can you further clarify the statement \"since most explanation methods and metrics have been developed for the task of image classification\"? A large number of methods as well as metrics were also developed for or applied to language processing tasks as well as other vision tasks such as VQA.", "broader_impact_concerns": "There are no concerns present.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pssw2BbP0p", "reviewer": "Reviewer_pFfR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Across multiple explanation methods, datasets, and predictive models, the authors experimentally show that their proposed framework, MetaQuantus, can solve a variety of XAI-related tasks like selecting and optimizing the metrics hyperparameters.\n\n2. The paper provides a robust analysis of different combinations of methods and metrics. In particular, the results of comparing different estimators and performance categories are insightful.\n\n**Weaknesses**\n\n1. The authors can improve the paper writing. In particular, the novelty of the work is unclear as previous benchmarking efforts [1-5] in XAI have already discussed the challenges of the evaluation of XAI methods and the authors list this as one of their contributions.\n\n2. The authors mention that the proposed MetaQuantus has a useful theoretical foundation but it lacks novelty. For instance, the perturbation analysis with respect to input parameters and model parameters has been performed in previous XAI works, showing the sensitivity of explanation methods across different models, explanation methods, and evaluation metrics. Further, the paper does not provide any theoretical analysis of explanation methods for the two proposed failure modes.\n\n3. One of the major drawbacks of MetaQuantus is that it is fundamentally based on perturbation-based metrics (noise resilience and adversarial reactivity) but fails to talk about the advances in the stability analysis of explanation methods. For example, Agarwal et al. [6] refine the original stability metric, proposes three types of relative stability, and argue that the vanilla stability analysis only assumes black-box access to the predictive model, and doesn’t leverage potentially meaningful information such as the model’s internal representations\nto evaluate stability.\n\n4. The links to the code do not work.\n\n**References**\n1. Hedström et al. Quantus: an explainable ai toolkit for responsible evaluation of neural network explanations. JMLR, 2023\n2. Shap benchmark. URL https://shap.readthedocs.io/en/latest/index.html\n3. Yang et al. Synthetic benchmarks for scientific research in explainable machine learning. In NeurIPS Datasets and Benchmarks Track, 2021.\n4. Agarwal et al. OpenXAI: Towards a Transparent Evaluation of Post hoc Model Explanations. In NeurIPS Datasets and Benchmarks Track, 2022.\n5. Faber et al. When comparing to ground truth is wrong: On evaluating GNN explanation methods. KDD, 2021.\n6. Agarwal et al. Rethinking Stability for Attribution-based Explanations, ICLR 2022 PAIR^2Struct Workshop, 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "With the increasing use of neural network models in high-stakes applications, XAI methods have been used to describe the behavior of complex models. However, one of the key long-standing problems in XAI is estimating the performance of explanation methods for classification models on a plethora of evaluation metrics. In this work, the authors propose *MetaQuantus* -- a flexible framework that meta-evaluates two failure modes of explanation methods, i.e., noise resilience and adversarial reactivity. Further, the authors show the effectiveness of MetaQuantus by utilizing it to select and optimize hyperparameters of a given evaluation metric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be great if the authors can emphasize the novelty of their work in comparison to existing benchmarking efforts. Please look at the weakness section for more details.", "broader_impact_concerns": "The paper is timely and talks about the evaluation of explanation methods which lacks consensus in the XAI community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper first reveals an important issue in Explainable AI (XAI), i.e., the disagreement between different criteria. To address such an issue, the authors developed the meta-evaluation, which is agnostic to the groundtruth and exploiting the perturbation consistency. 

The authors did a good job in addressing the concerns raised from the reviewers, especially adding more empirical study on SoTA models with large datasets. 

All the reviewers acknowledge that the paper is technical correct and may be of interest to a group of machine learning researchers. 

Therefore, I recommend acceptance with minor revision. The authors should take the suggestions from reviewers to make the paper positioning clearly. Meanwhile, I suggest the authors can move the additional experiments on ImageNet with transformers-based model architectures to main text. 
Decision: Accept (poster)