Evaluating Interpretable Methods via Geometric Alignment of Functional Distortions
Authors: Anna Hedström
Philine Lou Bommer
Thomas F Burns
Sebastian Lapuschkin
Wojciech Samek
Marina MC Höhne
Venue: Accepted by TMLR ()

Abstract: Interpretability researchers face a universal question: without access to ground truth labels, how can the faithfulness of an explanation to its model be determined? Despite immense efforts to develop new evaluation methods, current approaches remain in a pre-paradigmatic state: fragmented, difficult to calibrate, and lacking cohesive theoretical grounding. Observ- ing the lack of a unifying theory, we propose a novel evaluative criterion entitled Generalised Explanation Faithfulness (GEF) which is centered on explanation-to-model alignment, and integrates existing perturbation-based evaluations to eliminate the need for singular, task-specific evaluations. Complementing this unifying perspective, from a geometric point of view, we reveal a prevalent yet critical oversight in current evaluation practice: the failure to account for the learned geometry, and non-linear mapping present in the model, and explanation spaces. To solve this, we propose a general-purpose, threshold-free faithfulness evaluator GEF that incorporates principles from differential geometry, and facilitates evaluation agnostically across tasks, and interpretability approaches. Through extensive cross-domain benchmarks on natural language processing, vision, and tabular tasks, we provide first-of-its-kind insights into the comparative performance of various interpretable methods. This includes local linear approximators, global feature visualisation methods, large language models as post-hoc explainers, and sparse autoencoders. Our contributions are important to the interpretability and AI safety communities, offering a principled, unified approach for evaluation.

Published in Transactions on Machine Learning Research (02/2025)
Evaluating Interpretable Methods via Geometric Alignment of Functional Distortions
Anna Hedström
bnna.hedstroes @tt--brriin. de philine. Cemmer@@tu-heet de L.f. burns J.rnns@gmaall com sebastion. kapuschkin frouniifrooooffeee de wojeiech. samek-thee froaiiirrraoofeee de mhoehne
Philine Bommer1.8
Thomas F Burns
Sebastian Lapuschkin
Wojciech S.m
Marina M.-.. Höhne
wwwhhhhneatbbbeeee &otb @atb-potsdam. de
Department of Electrical Engineering and Computer Science, Technical University of Berlin BIPOLD Berlin Institate for the Poundations of Learning and Data Institutc for Computational and Esperimental Reseorch in Mathematics, Brown University Scientific Artificial Intelligence Center, Cornell University, USA
Neural Coding and Broin Computing Unit, Okinewa Insitute of Science and Technology Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute
Department Computer Science, University of Potsdam
UMI Lab, Leibniz Institute of Agricultural Engineering and Bioeconomy e. V. (ATB)
corresponding authors
Reviewed on OpenReview: httpps//gpaaarrrieeeee//////ooop
Abstract
Interpretability rescarchers face a universal question: without access to ground truth labels, how can the faithfulness of an explanation to its model determined? Despite immense efforts develop new evaluation methods, current approaches remain in a pre- pre-paradigmatic state: fragmented, difficult to calibrate, and lacking cohesive theoretical grounding. Observ- ing the lack of unifying theory, we propose novel evaluative criterion entitled Generalised Explanation Faithfulness (GEF) which is centered explanation-to-moodel alignment, and integrates existing perturbation-basee evaluations to eliminate the need for singular, task-specific evaluations. Complementing this unifying perspective, from a geometric point view, we reveal a prevalent yet critical oversight in current evaluation practice: the failure to account for the learned geometry, and non-linear mapping present in the model, and explanation spaces. To solve this, we propose general-purpose, threshold-free faithfulness evaluator GEF that incorporates principles from differential geometry, and facilitates evaluation agnostically across tasks, and interprelability approaches. Through extensive cross-domain benchmarks on natural language processing, vision, and tabular lasks, we provide first-ofils-kind insights into the comparative performance of various interpretable methods. This includes local linear approximators, global feature visualisation methods, large language models as post-hoc explainers, and sparse autoencoders. Our contributions are important to the interpretability and Al safety communities, offering principled, unified approach for evaluation.
http:://gstaab.cccmmmmmmaallttes
1 Introduction
Explaining the general behaviour, and predictions of machine learning (ML) models, particularly those func- tioning as black boxes, is critical, especially domains such as healtheare, finance, and law. Driven by the urgency to comply with regulations like the EU AT Act, and GDPR, the interpretability (or eXplainable

---

Published in Transactions on Machine Learning Research (02/2025)
AI (XAI)) research community has produced a plethora of interpretable methods in recent years
Bachrens
et al.
2010 Zeiler & Fergus, 2014a] Lundberg & Lee 2017 Bykov et al.] 2022 Fel et al. 2024 Lieberum
et al. 2024). Simultaneously, the rise of large-scale, multi- tauliitasking large language models (LLMs) (or "foun-
dation models") (OpenAl 2023: Mesnard et al.) 2024) has spurred a significant shift in the interpretability
landscape, with the mechanistic interpretability community producing a new generation of methods specifi-
cally designed to decompose, and reverso-engineer these increasingly black-box models (Elhage et al. 2022:
Conmy al.) 2023 Bykov et al.) 2023; Bills et al. 2023; Templeton et al.] 2024). Despite this immense activity, consensus is lacking whether existing methods are of sufficient quality or trustworthy (Adebayo
ct al. 2018 Ghassemi et al. 2021 Bordt & von Luxburg) 2024 Bhattacharjoe & von Luxburg, 2024). Since
black-box models lack ground truth explanation labels (Bellido & Fiesler [1933; [enitez et al.] [997]] the
universal question: "how faithful is the explanation to the model it seeks Lo explain?" remains difficult to
answer. The prevalence of disagreements within the interpretability community about which methods work,
and under what conditions (Neely et al. 2021] Watson ct al. 2022 Krishna ct al. 2022
Koenen & Wrightt
2024) signals that the challenge of evaluation still unsolved.
To approcimate explanation quality (Agarwal et al.
2022b; Hedström et al.
2023b)
researchers commonly
use perturbation-based evaluations, where robustness (Montavon ct al. 2018
Alvarcz-Melis & Jaakkola
2018b; Ych et al.
2019
Nguyen & Martinez, 2020 Dasgupta et al. 2022).
sensitivity (Adebayo et al.) 2018
Hedström et al.
2024). and faithfulress (Bach et al. 2015
Samek et al.
2017 Ancona et al. 2018 Rieger
Rong et al.) 2022) are well-embraced criteria to examine the relationship between explanation, and model outputs under perturbation, albeit with different emphases. Here, robustness, and sensitivity criteria refer to making small or large perturbations (e-g-, adding noise to the input or randomising model parameters), and then measuring corresponding changes in the explanation oulput. Faithfulness criterion generally measures how much the model's performance degrades when inputs, such as tokens or pixels, are cumulatively perturbed according to the explanation values. Significant changes in model behaviour are interpreted as indicators of explanation faithfulness.
& Haansen 2020
Dasgupta et al. 2022; Bhatt et al.
2020
Lack of Cohesive, Unified Theory. Despite repeated attempts to define, and measure faithfulness Mon-
tavon et al. 2018 Jacovi & Goldberg, 2020 Bhatt et al.] 2020; Turpin et al. 2023 Lanham et al.) 2023 Agar-
wal et al. 2024), fragmented mathematican lerminology makes it an ongoing, and unresolved matter |Bordt
& von Luxburg 2024). What exactly is explanation faithfulness, and how do robustness, and sensitivity evaluations differ from it? From a conceptual standpoint, although these evaluations share common steps such as perturbing the inputs or the model parameters, measuring the effects, and interpreting the functional outcomes- the overwhelming number of evaluation methods under these distinct criteria (Takkaraju et al., 2022), and the absence of cohesive, unified theory makes it difficult to answer such seemingly straightfor- ward questions. To better undersland these evaluations shared attributes, assumptions, and outcomes, a mathematical discussion is required. In Section 2 we propose a unifying perspective that formalises robust- ness, scnsitivity, and faithfulness evaluations, providing a principled Generalised Explanation Faithfulness (GEF) definition in Section 3 to substitute singular evaluations.
Ignoring the Impact of Geometry. Alongside the lack of a cohesive, unified theory, most perturbation- based evaluations (Section 2.1.2 while well-intended, and intuitive- rely on overly simplistic as- sumptions about the underlying geometry of both model, and explanation spaces. When perturbations are introduced, the functional outcomes of models, and explanations are frequently compared using direct dis- tance measures or correlation coefficients (Alvarez- Melis & Jaakkola, 2018b Yeh et al., 2019 Ancona et al,, 2018 Bhatt et al. 2020 Nguyen & Martinez] 2020 Agarwal et al., 2022a). From a geometric perspective, this overlooks a simple yet critical fact: that a uniform perturbation such as input noise or parameter shifts can affect non-linear systems in highly non-uniform ways. Only in a linear system, the perturbation effects would be uniform. By neglecting the geometric differences (e.g., differences in curvatures) between the model, and explanation spaces, current evaluations risk misjudging how faithful the explanation is w.r.t. its underlying model. For fair measurements across non-linear systems, perturbation effects must be measured in the context of the distinct geometric structures of the respective manifolds (Leo 2012). In Section we
I Throughout this work, we use the terms "interpretable methods" and "explanation methods" interchangeably, without implying a difference in their scope or function.
2

---

Published in Transactions on Machine Learning Research (02/2025)
examine these geometric factors, and introduce solution that accounts for the intrinsic geometry of each space, thereby improving current evaluation practice.
To address the research gaps in unified theory (Section 3 and the neglected impact of geometry (Section our work offers a fourfold contribution.
(C1) In the absence of cohesive theory, we systematise common steps in numerous perturbation-basel evaluation algorithms (Section 2), and provide a unifying evaluative criterion for robustness, sensi- tivity, and faithfulness evaluations (Section 3)
(C2) To account for geometric discrepancics many evaluation methods, we propose a solution based on differential geometry that ensures fair measurements across non-linear mappings (Section
(C3) Recognising the need for saneral-puppose threshold-free, task-agnostic faithfulness evaluators, we provide GEF and Fast-GEF methods, serving distinct compute budgets (Section 5).
(C4) Observing the lack of cross-domain insights on the faithfulness across distinct explanation approaches such as local, global, LLM as an explainer, and sparse autoencoders (SAEs), we perform extensive experiments across vision, tabular, and natural language processing (NLP) tasks (Section
Our contributions carry substantial importance to the interpretability (and related) communities. The reliability of individual explanation methods, and XAI as a field is already under hot debate, thus it is not only timely but relevant to provide clarity on the matter of explanation faithfulness. As we enter a new era of interpretability, it of utmost importance to revisit, and revise existing evaluation approaches. We hope this work will clarify how best to approach and perform faithfulness evaluation, ultimately empowering rescarchers confidently select and develop new interpretable methods.
2 Interpretability Evaluation: Where Are We Now?
In this section, we present the scope of this work. We begin by outlining preliminaries to estimate explanation quality, followed by a description of the general workflow of perturbation- evaluation. Finally, we mathematically formalise robustness, sensitivity, and faithfulness evaluation, revealing critical assumptions that are essential for their validity. Complete notation tables are provided in Appendix A.9
2.1 Preliminaries
Let fo : + y be a differentiable neural network (NN) that maps inputs x E IR to predictions y E RC C classes. By functionally mapping x E X' y E y with parameters f such that y = f(x; a trained model So is obtained, which we refer θ E g for fixed architecture in function space fo E F. The model may represent NN architectures ranging from simple feedforward MLPs, CNNs to highly parameterised trausformer-baset models.
J. Here, 0 includes weights, and biases, and exists in parameter space
To interpret a specific model prediction (i.e., logit) : we may employ a local method. Let Φl : F X x y RRV. be a local explanation function that takes an input, and logit pair, and assigns importance scores subset (or all) of its input features such that
Local Explanations.
= Ye of a class c E [1, C).
= é x, y; A)
where e E R the explanation output, paramcterised by X.
A broad variety of local explanation approaches fall within the scope of our work, e.g., gradient-based (Si- monyan & Zisserman] 2015 Smilkov et al.) 2017 Sundararajan et al., 2017 Bykov et al. 2022 Krishna
ct al.] 2023 Sclvaraju ct al. 2020), back-propagation-baadd (Bach et al. 2015 Shrikumar et al. 2017)
model-agnostic ZZeiler & Fergus, 2014a Lundberg & Lee 2017), local surrogate (Ribeiro et al. 2016a
attention- based Chefer et al. 2021 Covert et al.) 2022). ), or prototypical explanation methods (Simonyan &
Zisserman 2015). More recent approaches (Krishna al. 2023 Kroeger et al. 2023) that leverage separate
LLMs the explanation function to interpret local predictions in post-hoc manner, are also within the scope of this work.
3

---

Published in Transactions on Machine Learning Research (02/2025)
Global Explanations. To study the model f from global point of view, explanation is produced independent of a specific instance, i.e., a. Here, global explanation method de F x y - R takes a trained model f, and generates an explanation e E R for specific neural activation associated with a target class c. Here, c us represented by logit y such that
e= dolf y: k))
(2)
where de is parameterised by Here @G may be variants of activation-maxiiisstiin (or "feature visualisa-
tion") which provide either natural, or synthetic data points of maximal activation (Berkes & WWikkott 2006
Erhan et al. 2009 Olah et al. 2017 Nguyen 2020 Fel et al. 2024 or concept- explanations Bykov
et al. (2023). Recently, trained SAEs Bricken et al. (2023): ; Lieberum et al. (0024); Huben et al. (2024)
have emerged as an alternative formulation for ΦG, aiming to produce interpretable "monoscmantic" feature
encodings at the layer level, providing insight into a model's intermediate representations.
For convenience, we let Φ & denote ΦL, and dG although they formally reside in different spaces. To avoid label leakage (Jethani ct al. 2023 explanation
the predicted class (and not the true class) to generate the
2.1.1 Estimate Explanation Quality
Without ground truth explanation labels, the task of estimating the quality of an explanation Φ is non- trivial. To approximate explanation quality, researchers rely metric-based heuristics (or "metrics"). Following Hedström et al. (2023a), we define general evaluation function xxxxxxxxx
q = 20
which returns a quality estimate q E K, indicating the quality of a given explanation, parameterised by т. When global explanations &G are evaluated, is omitted from Equation 3 Unless required, we omit hyperparameters 7, N, for notational convenience.
2.1.2 Related Works
Within approaches that evaluate explanation quality by approximation, we concentrate on those that examine the functional relationship between the explanation, and the model through means of perturbation, í.e., assessing qualities such as robustness, sensitivity, and faithfulness. These are briefly introduced below, and mathematically formalised in Section 2.3
Robustness. Robustness (also referred to as "continuity", and "stability") methods evaluate the explana- tion function's resilience to infinitesimal input noise, and is a widely used evaluation technique (Ych et al.
2019 Montavon et al., 2018 Mlvarez-Melis Jaakkola 2018h Nguyen & Martinez, 2020 Agarwal el al. 2022a Dasgupta al., 2022 Most commonly, robustness is evaluated by first pertur bingurbing bing an input sample, then generating the explanation for the perturbed input, and finally comparing this explanation to the orig- inal explanation. Higher similarity between the original, and perturbed explanation indicates higher quality. Existing robust.ness measures dilfer in how noise is applied to the input (e.g., using a Gaussian (Alvarez-
Melis & Jaakkola, 2018b Yeh et al. 2019) or a uniform distribution AAgawal et al. 2022a)) ), and how explanation similarity is measured (e.g.. Yeh et al.] (2019) computes difference with Monte- Carlo sampling, and Alvarez-Melis Jaakkola (2018b); Agarwal et al. 2022a) rely on variants of a Lipschitz constant).
Sensitivity. Sensitivity (or "randomisation") methods (Adebayo et al. 2018 Hedström et al. 2024 act complementary to robustness, and assesses a critical, and perhaps indisputable evaluative quality: that the explanation function should be sensitive to randomisation of model parameters. Existing sensitivity measures differ in how the change in the explanation outputs is mcasured (e.g., Adebayo et al. (2018) relies on Structural Similarily Inder (SSIM), and Hedstrrm et al. (2024) uses discrete entropy calculations), and how perturbation is applied (e.g., Adebayo et al. (2018) randomises model parameters byyer-byllayer in top-down fashion, and Hodström al. (2024) uses bottom-up up or full parameter randomisation). The sensitivity criterion asks that the explanation should change significantly when the model parameters are randomised, whether layer-by-layer (Adebayo et al. 2018) or entirely (Hedström al.] 2024
4

---

Published in Transactions on Machine Learning Research (02/2025)
Faithfulness. Faithfulness (or "fidelity") methods Bach et al. 2015
Samek et al. 2017 Montavon et al, Ancona et al.] 2018 Rieger & Hansen 2020 Dasgupia al. 2022 Bhatt et al., 2020 Rong et al.
2018
2022
Atanasova et al. 2023 Blücher et al. 2024 Chuang et al. 2024) evaluate explanations by gradually
perturbing the imput based on the importance of pixels or tokens indicated by the explanation, and obscrving the resulting degradation in model performance. Mehthods differ in how model responses are reported (with
logits (Alvarzz-Melis & Jaakkola, 2018a
Yoh et al. 2019: Bhatt al. 2020) or softmax probabilities (Mon-
tavon et al. 2018 Ancona
2018
Rieger & Hansen
2020; Nguyen & Martinez 2020 Dasgupta al.
2022 Rong et al. 2022 ) how perturbations are ordered ascendinn (Arya et al. 2019 Nguyen & Martinez,
2020 or descending (Bach et al. 2015 Samek et al.
2017 Rong et al. 2022 ). and in the general approach to perturbation (whether using single-pixel changes (Bach et al.) 2015). patch-based masking with a con-
stant value (Samee et al.) (017], or linear interpolation (Rong et al. 2022)). Faithfulness methods lypically aggregate model responses into a single quality estimate, such as AUC (Bach et al. 2013 Samek et al, 2017 Rong ct al. 2022). For faithfulness to be considered fulfilled, the model's performance should rapidly decrease as perturbation are applied the steeper the degradation, the higher the explanation quality.
Beyond approximation techniques, interpretability researchers have explored alternative ways evaluate explanation quality, such using human judgment (Zeiler & Fergus 2014b; Ribeiro et al. 2016b). and restricting tasks to synthetic or toy environments [Guidotti] 2021 Carmichael & Scheirer 2023). While such approaches complement evaluation methods that approximate explanation quality, they lack scalability, and generalisability to real-world scenarios, and are not covered in this work.
2.2
Perturbation-ben based Evaluation
A key observation is that robustness, sensitivity, and faithfulness evaluations generally rely on three common steps. First,, a perturbation is applied to either the input (e-g., by adding infinitesimal noise) or the model parameters (e.g., by randomisation). Second, the effect of the perturbation is measured on the output of either the explanation function @ or the model f. Third, an interpretation is made assess whether this change in functional outputs is acceptable given criterion, such requiring the distance in explanation outputs to be small when the perturbation is small. We refer to Figure T for illustration.
Question
Step Perturb on space with some magnitude
Step 2 Measure functional distortion on or 4
Step Interpret outcomes per or
Estimate quality of explanation v(e(f....))
Robustness? Sensitivity? Faithfulness?
Figure 1: An overview of the "perturb, measure, and interpret evaluation methodology (Section 2.3
To facilitate mathematical unification (Section 4 and further insights (Section next formalise the three steps of perturbation- based evaluation. Therefore, some general notation for perturbation (Eqs and measurement (Definition] 1 is introduced. By systematising evaluation, can advance our conceptual understanding, especially in clarifying how existing methods differ, and what attributes are shared.
2.2.1
Step 1.
Perturbation
First, a perturbation is initiated. This is typically done either on the model parameter space in large magnitudes, e.g., by randomising weights, noise. Alternatively, perturbations can be applied cumulatively, such as by masking pixels regions of pixels, or by replacing tokens in textual inputs. To accommodate diverse evaluation methods across different data modalities, we follow Hedström et al. (2023a), and define a general perturbation function that can be applicd on any real-valuod space S C (xY O.Y). Let Ps: S + S be a perturbation function of E with parameters w such that.
the input in small magnitudes, e.g., by adding Gaussian
Ps(a:w) =
where Vs. s E S, and á * s. For brevity, we may omit such that Ps(s) Ps(s:w). With Equation may, e.g-, generate perturbed instance & with input perturbation, i.e., & = =Px(x) model parameter
5

---

Published in Transactions on Machine Learning Research (02/2025)
randomisation, i.e., 0 = (e) Since robustness, sensitivity, and faithfulness evaluations require distinct perturbation udenitudes, we let & denote the difference between 8, and & as follows
o(s, 8) =,
(5)
where o: SSSS - IR is a general discrepancy function, e.g., an ly-norm, cosine distance or Pearson correlation.
2.2.2
Step 2.
Measurement
As second step, the impact of the perturbation measured on relevant functions. Common approaches include measuring the distance between explanation outputs or rocording the change in model responses under random cunulative masking guided by the explanation output. We define a general approach to measure the perturbation impact separate function (e.g., the impact of input perturbation the model function) below.
Definition 1 (Functional Distortion) Let s, E S denote instances in space S S
before, and after perturbation, respectively. Let k S + H denote a seporate function that maps s, 8 to distinct space H C {F,E} from S. Then, perturbation impact in function is measured by functional distortion D& S x + R as follows
(s(s,8) ((()))
(6)
where k(s) = h with h E H. and 8: H H RR
Model, and Explanation Distortion. With Definition I we can flexibly apply perturbation in one space, and then evaluate the effect in different space For example, assume we have applied perturbation on the input space, f.e., a = (x(x) (Equation [). and therefore have two instances æ, and a. Then, to measure the perturbation impact on the model function f, we follow Definition D and set k == where = y. Evaluating (x a) from Equation 0 effectively means that we compare model evaluations on perturbed, and non-perturbed inputs, i.e., 8(y. () with y = fa(t;; 0) for the same class e. Alternatively, measure perturbation impacts on the explanation function Ø, we set k e where h e. Evaluating Dø(æ, it), practically means that we compute 8(e, é) where è o(..... is the explanation w.r.t. perturbed input. è. For comparability, é is generated w.r.l. the same class as its non-perturbed counterpart Similarly, to compute functional distortion after parameter perturbation, i.e., 0 = Po( we compute (0(0,0) 0), and (0,0, using logit f(x; 0), and explanation è (fgg....), respectively. To generalise the notation across different perturbation types, we let Dj, and D denote the model, and explanation distortion quantities, respectively.
2.2.3 Step 3.
Interpretation
In the final step of the evaluation workflow (Figure [ the distortion quantities are examined separately according their evaluative criteria. For example, robustness is evaluated, generally low values for D& are expected, assuming perturbation magnitude & is small. Conversely, sensitivity evaluated, high values for Do are expected, assuming perturbation magnitude & is large. Tf faithfulness evaluated, model distortion Dy is anticipated to increase as perturbation is cunulatively applied according to the explanation function output. Notably, a key limitation of this step is the need for thresholds to be set by researchers in order to distinguish between low-, and high-quality evaluation outcomes, which has shown could be adversarially manipulated [Wickstrom et al. 2024
2.3 Formalising Robustness, Sensitivity, and Faithfulness
Equipped with a general perturbation function Ps (Equation and its magnitude & (Eiquation as well as a mcasure to compute functional distortion of the explanation, and model functions (Definition we can combine a wide variety of existing evaluation techniques into general formalisations of robustness, sensitivity,
While both the perturbation magnitude (Equation 5 and the distortion D; (Equation 6 the discrepancy function -()-),, their outputs differ. Notably, expresses the discrepancy between the original, and perturbed instance, and Dy measures the discrepancy distinct. space from the perturbation space.

---

Published in Transactions on Machine Learning Research (02/2025)
and faithfulness evaluation methods (ef. Section 2.1.1 Based these three main criteria (Definitions
we show that the validity of each explanation criterion critically depends on fulfilling a separate, implicit model assumption (aststtionAAAApppppl (Assumptions 3
We proceed by presenting a definition of explanation robustness, absorbing the spirit of numerous existing
robustness method (Yeh et al. 2019
Montavon et al. 2018: Alvarez-Melis & Jaakkola 2018b Nguyen
Martinez 2020 Agarwal et al. 2022a
Desinition 2 (Explanation Robustness) Let & Px(x) be perturbed input, and t be quality estimalor to gield robustness eslimates RRo E R such that g8o Da(a,, Given thresholds a, D, aitt, an explanation function & is robust if the perturbation magnitude go S a:
q RO <
=0 D2
(7)
For explanation function to be considered robust, the estimator Ro should yield low values, i.e. q qR < reflecting minor differences between the original explanation e, and the perturbed explanation é. Since the stability expectations of the explanation function & are dictated by the robustness of f (Yeh)
let al. 2019 Chalasani et al., 2020 Agarwal et al. 2022a, Tan & Tian 2023), it would be false to expect ø
to exhibit robustness if its underlying model not robust. Consequently, ly, the validity of Equation 7 depends
on the fulfillment of model robustness (Assumption
Assumption 1 (Model Robustness) Given an input perturbation Px of magnitude eoo, and thresholds a, RO a, E Rt ERO S a, the model distortion (Equation 6) is bounded by (x, S <
.RO
In line with works of Adebayo et al. 2018 Hedström et al. 2024), we define explanation sensitivity in the following.
Definition 3 (Explanation Sensitivity) Let Ô = Pe(0) create a model fg with perturbed parameters, and SiSE be a quality estimator that yields sensiliviiy estimates U R. such that qBE = D0 6). Given SE
D, thresholds a, Dg E Rt, an explanation function o is sensitive if the perturbation magnitude ESE > a:
q SE > D
(8)
For o to be considered sensitive to randomness, the differences between explanations should be substantial, meaning ySE yields high estimates, i.e., و.و >, reflecting significant discrepancies between e, and è. This expectation that ς should be large is based on the assumption that the model responded strongly to the perturbation. Similar to how explanation robustncss depends on the stability of f, the emphasis on large S assumes different model response. Therefore, the validity of the sensitivity evaluation (Equation g depends on model sensitivity (Assumption D.
Assumption (Model Sensitivity) Given a parameter perturbation Pe of magnitude EEE and thresh- SE olds a, a. e 1 > a, the model distortion (Equation 6) bounded by (, 0) > D
SE
With various existing interpretations of explanation faithfulness (Section 2.1.2 to combine these interpretations into a single definition below.
focus common criteria
Definition 4 (Explanation Faithfulness) Let -² = (xx(z;2) denote the input after the tht perturbation for = E [1, Z}, where Px progressively masks the top-z features according to the indices given by argmaz(e), with perturbation magnitudes S, satisfying & S E2 < ... < Z2. A quality estimator F yields a vector of faithfulness estimates FA E TR22 with entries q. F.A = f(x=, 0), where the overall faithfulness score o E R is obtained by aggregating these estimates via function v 12 R:
(9)
SSme algorithmic details omitted in the definition. For completeness, mathematical definitions are provided for each evaluation methad in Appendix A.4.5
7

---

Published in Transactions on Machine Learning Research (02/2025)
When is defined using AUC, faithful explanation is expected to produce low aggregated scores q & (Equation 9) The conventional expectation in laithfulness evaluation (Bach et. al. 2015 Samek et al. 2017 Rong et al. 2022) is that significant distortions should occur early, as the "more important features" are removed first To ensure that the faithfulness score is solely driven by the quality of o, and not by other factors, such as out-of-distribution samples (OOD) (Hase al.) 2021 Hesse el al. 2024), non-linear feature effects or artifacts introduced by cumulative perturbations (Hooker et al. 2019 Brunke al., 2020 Hase iet al. 2021 Rong al., 2022 Brocki & Chung) 2022), the model distortion to these perturbations should be monotonically non-decreasing, i.e., satisty model faithfulness (Assumption
Assumption 3 (Model Faithfulness) Given Z cumulaativ perturbations Px of magnitudes & with E S
& S ..... S Ez the corresponding model distortions (Equation 6 are: Dj S D3 S ... D2 with Dj (²)
2.4 Model Assumptions in Practice
Evaluations under Desinitions Lypically assume that model distortions are proportional to perturbation magnitudes, i.e., that larger perturbations lead to greater distortions, and smaller perturbations result in lesser distortions. This naturally raises the question: with commonly used perturbation techniques for evalu- ating robustness (e.g., additive Gaussian noise), sensitivity (e.g., layer-wise randomisation), and laithfulness (e.g., cumulative input masking, does this assumption hold in practice? In Appendix A.5 we extensively analyse the extent to which Assumptions hold versus fail across various explanation methods, and NN models.
Notably, we find that Assumptions 33 are systematically violated practice. While this is expected due to the inherent von-linearity embedded in f, has significant consequences for the validity of existing eval- uations (Definitions 4 Evaluation outcomes may be misleading when explanation robustness is enforced
for models that fundamentally lack it (Chalasani et al. 2020 Tan & Tian 2023 Agarwal et al. 2022a
or when faithfulness scores are attributed to explanation quality without considering OOD scenarios (Hase
et al. 2021
Hesse et al. 2024). In Section 3.3 we propose a mitigation strategy to address this issue.
3 A Unifying Perspective
With clear definitions of robustness, sensitivity, and faithfulness evaluations (Section 2.3 explore their shared attributes, and outcomes. In the following, we discuss the unifying aspects of these eval- uations, and introduce a novel definition to evaluate faithfulness, which integrates these distinct evaluations into a single criterion of explanation quality.
we may now
3.1 Unifying Attributes
Upon formalising the evaluation criteria (Definitions 2 notable observation is that robustness, sensitivity, and faithfulness exhibit common attributes. Each of the evaluative criteria (1) introduces a perturbation of a specific magnitude g, (2) measures the functional effect, and (3) interprets the results, i.e., the quality estimate g. Also, the evaluation performed under distinct model assumptions about its distortion Dy. We refer to Table 1 for summary of these findings.
In Figure (A), we illustrate these theoretical similarities graph, with axes corresponding to the shared attributes &, and g. Here, we observe that robustness evaluation (green) involves minimal perturbation with small difference in expected explanation output (or low q). Sensitivity (red) employs substantial perturbation, expecting a significant difference in explanation output (or high 4). Faithfulness (blue) uses cumulative perturbation of Z steps, evaluating the corresponding variations in model output. By placing the different perspective of explanation quality onto Figure 2 (A), and thereafter examining the positions of the post-pert instances E S, we can advance our understanding of how the criteria relate one another: specifically, that diverse evaluation methods can be unified under shared conceptual framework.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table A concise overview of the attributes of the robustness, sensitivity. and faithfulness evaluations. The last row presents umified evaluation proposal (Def whose theory and practical implementation described in Section and Section respectively.
EVALUATION (W)
Step 1. PERTURBATION:
Step 2.
MEASCREMENT
Step 3. INTERPRETATION
MODEL ASSUMPTIONS
(DEFINITIONS 2
MAGNETUDE (EQUATIONS
DEFINITION
(EQuAtION 7
(ASSUMPTIONS
ROBUSTNESS R
Px(()); <R Sa Po(0); < Px(2;:): & 1 5 $ 5
Do(a, 2))
RO Dr I &D cD. Dj Dj Dj NONK
SENSFTIVITY (Q-58)
Da
8 CD
PAITHFULNESS (QFAA
f(a".0) .0)
p vlq A
UNIVEDD (g)me)
Pa(d, 2); E $ MR
1 ... EERP
D,(0,6), AND Dy(0,0)
rid d.)
3.2 Unifying Outcomes
Another point of unification emerges when considering the outcomes of these evaluation criteria, and how they interact in practice. In Figure 2 (B), similar to the traditional confusion matrix (in ML) or contingency table (in statisties), perturbation. Although model, and explanation outcomes are typically continuous in reality-for conceptual clarily, we classify them into four distinct quadrants: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). A key benefit of discretising evaluation outcomes in this way, is that we can distinguish between aligned, and misaligned explanation behaviour:
provide visual representation of the possible model, and explanation outcomes, post-
Aligned outcomes (TP + TN). The green quadrant represents outcomes where the explanation, and model agree, indicating explanation robustness, i.e., e = é, and y = D, and satisfying Assumption []] Conversely, the red quadrant contains outcomes where both explanation, and model outputs differ, reflecting explanation sensitivity, f.c., e o é, and y # ú, and satisfying Assumption 2 Explanation faithfulness achieved when evaluation outcomes are aligned over Z steps (Assumption 3p.
Misaligned outcomes (FP + FN). The orange quadrants highlight misalignment between s. and f. The lop-left quadrant shows explanation dissimilarity despile prediction stability (f.e. y = 9, and The bottom-right quadrant shows explanation similarity despite a prediction change (i.e., y + y, and e = é), failing Assumption
# é), failing Assumption
Robustness Sersitivity Fasthfulness
Explanation Outcone (Pred)
FN
TN
p(dg.dr)
Quality ostinate (q)
y 9
y = y e = é
Fumctiional distortion
y
df
dg
é
Porturbation Magnitude (E)
Model Outcone (Actual)
Perturbation path
(A) Unifying Attributes
(B) Unifying Outcomes
(C) Unifying Criteria
Figure 2: Intuition behind the relationship between robustness, sensitivity, and faithfulness evaluations. (A) illustrates the shared attributes, i.e., perturbation magnitude &, and quality estimate that unifies robustness (green), sensitivity (red), and faithfulness (blue) evaluations. (B) displays a confusion matrix of discretised model, and explanation outcomes, with green, and red quadrants indicating aligned behaviour, and orange quadrants showing misalignment. (C) shows our proposed GEF criteria (Definition 5 which ieasures explanation to model aligument over diverse evaluation perspectives.
Explanation Faithfulness Alignment. Our analysis reveals that evaluation according to Definitions fundamentally concerns the alignment between the explanation, and the model's behaviour, whether multiple (Definition perturbation steps. A key observation that
across single (Definitions 2

---

Published in Transactions on Machine Learning Research (02/2025)
existing robustness, and sensitivity measures provide limited view of isolated model conditions: robustness evaluates alignment when the model's predictions remain stable (TP quadrant), while sensitivity evaluates alignment when predictions change (T' quadrant). Faithfulness (Definition 4 evaluates alignment over Z steps, assuming non-docrcasing, monotonic model responses under cumulative perturbations (Assumption 3 These singular perspectives require strict adherence to specific model conditions, and consequently fail to evaluate the full behaviour of the explanation function. Next, we propose unifying criterion for faithfulness evaluation. We refer to Figure 2 (C) for an illustration.
3.3 Unifying Criteria
We extend, and generalise the current faithfulness criterion (Definition by integrating the robustness, and sensitivily evaluations into combined criterion, that is free from restrictive model assumplions. Using a series of Z perturbations, we measure explanation alignment across spectrum of model outcomes- from cases where model predictions remain consistent, i.e., y = D, to those where predictions diverge, i.e., # 9. In this way, a generalised definition of explanation faithfulness is obtained.
Definition 5 (Generalised Explanation Faithfulness) Let dj
[D], Di...... Df1 and D Di....,D be the model, and explanation distortion vectors, where Dj, and D are distortion quan- tities of the 2th step along perturbation path [], from robustness at 1 to sensitivity at z Z such that Vy. 0 EY: Y:
(= y and Z= +0),
where g. and y are perturbed versus unperturbed model outputs, respectively. Let &GEF be a quality estimator that yields estimates GGEF R via the correlation coefficient p IR R + such that qGEF p(dd An explanation function & & is fasthful to f E if:
1 GEF 1
(10)
With Equation 10
define quality estimator @GEF that yields values ranging between [-1, 11, with value of implying perfect generalised faithfulness, 0 suggesting an absence of it, and -1 an inverse relationship. GEF estimation is, therefore, threshold-free in the sense that the correlation coefficient directly indicates the quality of the explanation, eliminating the need for arbitrary cut-offis. Note that in Definition 5 we implicitly rely on predicted class c to generate the perturbed logit ý as the target for the explanation, and model distortion. In Appendix [A.12 we discuss a broader application of GEF where the targets f, and y are replaced by any dd neuron within a layer ( [1, L] of feed-forward model. Moreover, Definition 5 applies to a wide range of explanation functions, as discussed in Section 2.1 The choice of p, and perturbation applied to construct the distortion vectors depends the practical implementation (Section 5.2)
Remarks. Our definition shares similarities with faithfulncss estimation (Definition 4 in that it assesses explanation quality along a perturbation path. However, it fundamentally differs by focusing on general alignment rather than a specific scenario of measuring the magnitude of model response to cumulative input perturbation. A key benefit of our proposal that we use the model distortion anchor the expectations of the explanation distortion, and as such, eliminate the need to rely arbitrary thresholds. In this way, the evaluation will be grounded in the exact functional response of the model, and thus resilient to OOD sccnarios: expecting small explanation distortions only when model distortions are small, and vico-versa.
Theoretical Benefits. A good faithfulness measure should assign low scores to unfaithful explanations, and high scores faithful explanations. In Appendix(A.13. [A.1.3] we prove that a linear model f Ox +c where 0 acts as the explanation, attains perfect faithfulness score, i.e., GGFP 1 with GEF. Conversely, unfaithful explanations are penalised by GEF. For instance, constant explanations that generate distortion, i.e., RO e) = 0, pass the conventional robustness test RO < ED (Definition 2), but correctly fails in the GEF formulation. Similarly, random explanations (e.g., generated by uniform sampling, i.e., é, ((, 1)) produce maximal distortion (Binder et al. 2022), and thus generally pass the sensitivity test, i.e., S E (Definition 3) but fails in our defnition.
We provide proofs for both cases in Appendix A.1.1 and outline
empirical evidence in (Section 4.2
10

---

Published in Transactions on Machine Learning Research (02/2025)
4 A Geometric Perspective
With an advanced understanding of explanation faithfulness (Section 3 we can more systematically study the behaviour of the explanation function. Without assuming the restricted model conditions (Assump- tions are met, which are often violated in practice (Appendix A.5 we can more objectively measure the true explanation function behaviour. We can formalise questions such as: do explanation functions real-world evaluation scenarios align or misalign with their model? How do difffernt types of perturbation impact model and explanation functions? In the following, and in Appendix A.6] we empirically exam- ine such questions across common explanation methods (Section 2.1 for different NN models. Guided by differential geometry, we provide thcoretical considerations on the impact of geometry (Section 4.2
4.1 Explanation Alignment Patterns
To empirically analyse whether explanation functions are aligned with their model, we study how the dis- tortions of various local, and global explanation functions, and models change under perturbation. Here, we use additive Gaussian noise, ke., V ~ (0, to generate perturbed inputs â = a + 14, with a standard deviation a increasing until the model behaves randomly (i.e., with an accuracy equal to 1/C) using perturbation steps. We refer Table in Section for details regarding datasets, and models, and Appendix A.6 for extended results.
Avila La-laye-
COMPAS Layer
KNIST (Lelet)
Inagelet (Res/let1e)
10
SPAP - Disiin Lian
5900- Distort.on
geap 5 Distorting Lan
SHAP- - Distortion
400
-90
2
29
-00
2.5
5.0
7.5
16.0
12.5
100
150
288
28
18
28
Yodel Distartion
Podel Disto rtion
Model Distartion
Hodel Distortion
Avila (2- layer NLPI
COMPAS [3 13-layer VLP)
NNIST (LaNet)
Tragolet (ResNet18)
Glob Distor Lion
(1440 Distortion
CRAI Distortion
CR1D Distortion
500
4.
9.5
188
15B
Model Distertion
Distortion
Hodel Distortion
Input 1015e
2+1
2+5
Zw0
Figure 3: Model (x-axiss) and explanation distortions (y-axis) under varying levels of additive Gaussian input noise for vision, and tabular tasks. The scatter points represent individual samples, coloured by perturbation magnitude (x=1, z=5, ==9), with overlapping contours highlighting the relative alignment patterns. The top, and bottom rows represent GradientSHAP (SHAP-G), and Gradient (GRAD) explanations, respectively.
Figure 3 (top, and bottom) presents the results, which can be interpreted as a continuous analogue of the confusion matrix presented in Section 3.2 The scatter points, coloured by perturbation magnitude, reveal that 6, and rarely align fully. Instead, the relative alignment varies with both the model, and the explanation functions. Here, we include GrudienlSHAP (SHAP-G) (Lundberg & Lee 2017). and Gradient
(GRAD) (Mrcch et al. 1995 Baehrens et al. 2010)) on top, and bottom rows in Figure respectively, with more results in Appendix A.6 The overlapping contours (e.g., Avila results in Figure underscore a simple but nonetheless systematically overlooked aspect of perturbation based evaluation (Section 2.3 that uniform perturbation of its inputs may affect highly non-lincar systems in non-uniform way. the effects were uniform, the system would likely be linear.
4.2 The Impact of Geometry
By considering the geometric nature of the spaces these functions inhabit, we can understand the abserved misalignment better. In differential geometry, each space whether it is the model output space y or the
11

---

Published in Transactions on Machine Learning Research (02/2025)
explanation output space C-can be viewed a manifold with its unique geometric characteristics (Lee 2012). When perturbation is applied, a new point on these manifolds may be accessed, and then, when functional distortion (Definition D is computed in each space, we are effectively computing a distance between two points each manifold. For example, with model parameter perturbation, i.e., 0 = Pe (Equation we obtain perturbed model outputs ÿ (or, a logit f) given ğ = fz(æ). From this, model distortion (Definition 1 is calculated using, e.g., Euclidean distance between the original, and the perturbed instance.
A key observation is that, when distances in two different spaces are directly compared, we ignore the fact that manifolds have their own separate geometric characteristics which are distinct, where distances in one space not necessarily reflect equivalent distances in another. In direct comparisons such as correlation (Ancona et al. 2018 Bhatt et al. 2020) or Lipschitz calculations (Alvarez-Melis & Jaakkola 2018a Agarwal et al. 2022a a global flat metric is assumed. We refer to Figure (A), and (B) for an illustration of the problem of ignoring the impact of geometry. As a result, the quality estimation of the explanation may be misleading.
(A) Distinct Manifolds
(B) Global Flat Netric
(C) Applying Pullback
VCB
ECR
Do)
y CR
8 ECR"
8(4)
D
D2
79
Dj
Ki....)
c - olf....)
Figure 4: An illustration of the relationship between the manifolds of the model, and explanation. (A) shows how the explanation function maps between the model, and explanation spaces, y, and E. (B) displays the problem with directly comparing distortions across spaces, assuming global flat metric. (C) illustrates the pullback operation using metric tensor adjust distortions 8 for comparison in V.
4.3 Reconciling Geometric Discrepancies
To enable geometrically sound comparison between explanation, and model distortion, the aim is to recompute D to incorporate the non-linear mappings used in generating explanations. This can be achieved by mapping the distortion from the explanation space & to the model space y, effectively "pulling back" the measured distance into y (see Ch. 11 of Lee (2012) for further details). Guided by differential geometry, we create a metric tensor g that serves as this pullback onto y. This process illustrated as g(é) in Figure 4 (C).
To construct the metric tensor g, consider an infinitesimal neighbourhood around the parameter pertur- bation 0 + du, for a fixed æ, and y. By applying first-order Taylor expansion in this neighbourhood, we obtain
offo+du...)
Jydu,
(11)
where J. E VxC is the Jacobian for fixed input x, with elements = dc We use fj as shorthand Ø7,
for (x).. Effectively, 0 + du yields a new perturbed model Sa which is computed with model parameter perturbation (Section 5.1). With Equation I1 we can compute the elements of the pullback tensor g VxV as the sum of the resulting changes in each explanation element e, w.r.l. the changes in each model element fy
(. Σ dc, de, oh. af,
(12)
Thus, Equation 12 captures the sensitivity of & to model output changes, with g corresponding the squared Jacobian g- Jy. In this way, we can obtain a more reliable measurement of distances in the Sseudo-Riemannn manifold ().g) of space y.
12

---

Published in Transactions on Machine Learning Research (02/2025)
With the pullback metric tensor g in place, measure explanation distortion that is equivalent to computing the path length under the induced parameter changes in the "pulled-back" space
D=: = L<))= [ dr(t) dt 97(1)) dt
dr(t)
dt,
(13)
where (() is a path between endpoints e, E & derived from the original, and perturbed models, respectively. Here, 1. denotes the slep size. Now, we replace our original definition of explanation distortion D, = 8(e, é) (Definition D with the total accumulated distortion along the path, i.e., Equation 13. Here, longer paths correspond to greater distortions. Upon taking this geometric perspective, we can study y using extrinsically- defined geometry, and contrast it with the simpler assumption of flat, intrinsic Euclidean metric. As result, Da: and Dj are more fairly compared in the same space.
5 Method: From Theory to Practice
While our unified theory (Section 3), and solution to reconcile geometric discrepancies in measurement (Section 4 provide first steps lowards resolving issues in perturbation-basel evaluation, many practical concerns remain regarding the choice of perturbation. In this section, describe how to reliably translate our theory (Definition 5) to practice we propose general-purpose, task-agnostic perturbation technique based model parameter scaling (Section 5.1). and introduce the full evaluation algorithms,, i.e., GEF and Past-GEF(Section 5.2
5.1 Selecting Perturbation Strategy
While all pewturbation evaluations inherently require parameterisation, baput-based perturbation (Def-
initions 2 and 4 has proven particularly challenging calibrate (Sturmfels et al. 2020: Haug et al.) 2021]
Without ground truth labels, selecting parameters such patch size, pixel, token replacement strategics
is typically based on researchers' judgment. Small changes to input parameters have been shown to signifi-
cantly impact evaluation outcomes (Brunke et al. 2020 Brocki & Chung 2022 Rong et al. 2022
Blücher
et al. 2024), raising concerns about reliability.
Moreover, perturbing on the input space not only impractical from a practitioner's standpoint but also compromiscs impartiality- parameters must be adjusted for cach model, and datasot, how can task-specific confounds be controlled? In Appendix A.5.1. we provide empirical evidence for the existence of confounds in faithfulness evaluations (Definition
Rescarchers need general-purpose, datasct, and architecturo-agnosti perturbation strategy that facilitates evaluation across distinct explanation approaches (e.g., local, and global methods), and magnitudes, í.e., &. Following Bykov et al. 2022). we propose a simple perturbation strategy in the following.
Model Parameter Scaling. Introduce perturbations Vz E [1, 2] by scaling parameters 0 E R with Gaussian noise ~ (1,,oo)) and o Rt such that 02 = 0 m,, yielding perturbed model Sā.
By systematically perturbing model parameters instead of the input, from low to high magnitudes with incremental increases of o2, ranging from robustness at = 1 to sensitivity at 2 = Z, explanation behaviour is evaluated comprehensively, and agnostically across lasks. With & = ((f(x)), (æ:) (Equation 5) we can mcasure the perturbation impact at cach 2th step so that robustness, i.c., = 0. and scnsitivity, i.e., y A 9 criteria are fulfilled (Definition 5) Our approach contrasts with the model parameter randomisation proce- dure of Adebayo et al. (2018), which proposes layer-wize randomisation in a top-down order, an approach that faces methodological concerns (Sundararajan & Taly 2018 Binder et al.) 2022 Kokhlikyan et al. 2021 Yona & Greenfeld 2021).. For an illustration of how model parameter scaling affects the classifier's decision boundary, we refer to Fig. 1 of Bykov al. 2002
13

---

Published in Transactions on Machine Learning Research (02/2025)
5.2 Introducing GEF Evaluator
From an algorithmic perspective, three steps are necessary to perform the evaluation. First, given model, and a test set of input-output pairs, we generate perturbed models : given Z sets of parameters 01....,0z along a perturbation path (see Algorithm line 6). Then, for each model fø. we compute the model, and explanation distortion quantities, i.e., Dj and Dz using the pullback tensor g (lines 7, 9, and 10). Finally, distortion vectors are constructed, and correlated using p(dj. d&) (lines 14, and 15). Due to the stochastic nature of model perturbation, we repeat this process M times average out the effects. We refer to Figure 5 and Algorithm for overview of the steps involved. An ablation study hyperparamcter choices is provided in Appendix A.7
GEF Evaluator (Algo. 5)
(Step 1) Perturbation
(Step 2) Neasurement
(Step 3) Interpretation
f8z
k())
q Ger p(dd;
of1
n
dy = - D ... Df)
8. = 8
fa.
((()
L(T)
d = (D D
D
for
fo, (a))
(Us.....))
Model Parameter Scaling
Pullback Distortion
Anchor Expectations
Figure The three steps of GEF evalustion (Algorithm) estimate generalised explanation faithifuleess (Definition 5
Practical Benefits. Our proposed evaluation (Algorithm provides several practical benefits. First, anchoring, negates the need to rely on arbitrary thresholds in evaluation, e.g., when determining a permissible value for the evaluations themselves (Equations 7 and 9 or what perturbation magnitude leads to model alignment for particular task. Second, perturbing via model parameter scating, at varying intensities combines distinct criteria of explanation quality into single unified evaluation metric (Section [3.3] that is agnostic to the data, model, and explanation approach. Third, the pullback metric calculation provides geometrically grounded faithfulness measurement, capturing the true functional impacts of the explanation w.r.t. its model.
Algorithm GEF Evaluator
1: Require: Model f. explanation function Φ, input-prediction pairs E X, Y with X C CN,YCy
Parameters: Integers Z, M, K, correlation measure 3: for v, in range(X, Y)) do
- @+o(f.y,. ....
5: for z in range(Z] 2) do
g- --, (x)
7:
D; 8-y Ú) // Equation 6
8:
if Fast-GEF then
9:
D + 8(e, é) with è :(Jo.....) // Equation 6
10:
else
11:
D compute_par length N. Equation
12: end if
end for
14: Construct: 4 (D), DJ. D....., D71, and + [D], D.. D
15: Calculate: GEF + p()).s)
Return: q GEF
16:
17:
end for
Implementation Details. Unless stated otherwise, we use Euclideon distonce for in the functional distortion calculations (Definition D. and define p using Spearman Rank Correlation, assessing the degree of
14

---

Published in Transactions on Machine Learning Research (02/2025)
monotonic relationship between the distortion quantitics. For the experiments, set Z = 5 (see discussion of the influence of Z in Appendix A.1.4p but it is a choice that can be flexibly updated in the open-source implementation. In Appendix A.2 we provide further details on the implementation, including how to generate the perturbation path (line 2), and how to tune parameters (line 2). This also includes information on how we compule the path length (line 11); where we follow an approximation procedure outlined in Equations 18 and 19
5.3 Balancing Computational Constraints
While the pullback operation ensures a fair geometric comparison of distortion quantities, its use of high- dimensional Jacobian calculations, and integral steps (Equation 12 also increases computational demands. To accommodale evaluation contexts involving large model archilectures or high-dimensional explanations, we offer an alternative method. For faster yet naive approximation of explanation quality, we omit the pullback operation, and instead define D3 according to Equation G This approach, entitled Fast-GEF, is less computationally demanding, and complements the ezact approach with pullback, entitled GEF, providing geometrically sound quality estimate.
Choosing between GEF or Fast-GEF. Users can choose between these methods based on their specific computational constraints and demands for accurate quality estimates. We recommend using GEF wherever possible due to its ability to account for manifold-specific distortions. However, Fast-GEF provides a com- putationally efficient alternative that is suitable for large-scale tasks or resouree-constraince environments. Empiriaal results show that while the GEF or Fast-GEF may diverge in individual estimates (Appendix A.6 they often share categorical rankings of explanation methods (Appendix A.8.2).
6 Experiments
Our experiments aim to answer the following questions:
(Q1) Are unified, GEF and Fast-GEF evaluations more empirically reliable than competitive singular ap- proaches?
(Q2) How does generalised faithfulness of local, and global explanation methods compare across distinct data domains?
(Q3) How faithful are LLMs as a top- K token post-hoc explainer for NLP classifications?
(Q4) Are SAEs generally faithful, and does more capacity in their width improve their faithfulness?
To answer these questions, we select a diverse set datasets, model architectures on tabular, vision, and NLP classification tasks. See Table 2 for an overview. Our experiments evaluate the faithfulness of various explanation approaches, as detailed below.
Global, and Local Methods. For global methods, we include feature visualisation techniques with dif- ferent regularisation, and optimization procedures: Deep- Viz (DV) (Yosinski et al., 2015), Magnitude Con- strained Optimization (MACO) (Fel et al.] 2024)), and Fourier preconditioning (FO) dOlah et al. 2017) Optimization steps are set to 50, 100, and 250, otherwise, default values are used as provided in the respec- tive publications ((Fel et al. 2024), and (Nguyen) 20200). For local methods, two variants of Layer-wise Relevance Propagation (LRP), )2 the E-rule (LRP-s) (Bach al. 2015) with 5 -6 and the =--rule (LRP-z*) (Montavon et al. 2017) are employed. Also, include several gradient- rradeett-based approaches such Gradient (GRAD) (Morch et al. 1995 Baehrens et al. 2010). Saliency (SAL) (Simonyan et al.) 2014 Inputx Gradient (IXG) (Shrikumar et al.
2016). GradCAM (G-CAM) (Sclvaraju et al. 2020), Guided Back- 2015), SmoothGrad (SMG) (Smilkov el al. 2017 with 10 noisy
propagation (GBPG) Springenberg et al.
samples, and noise level 0.1/(=max - xmin), Integrated Gradients (INT-G) (Sundararajan et al. 2017) with
itcrations, and zero bascline. For NLP tasks, we evaluate KegerrnnterateeeGraaae
(L-INTG) explana-
tions w.r.t. the first embedding layer. Two Shapley- based algorithms (Lundberg & Lee
2017 are included:
GradienSHHAP (SHAP-G) with 10 samples, and PartitionShap (SHAP-P) for NL. tasks.
15

---

Published in Transactions on Machine Learning Research (02/2025)
LLM-x Methods. An emerging research area in explainability uses separate LLMs to generate post-hoc
attributions for important features of given model (Bills et al. 2023 Kroeger et al. 2023 Krishna et al. 2023 Amara et al. 2024). We create LLM-x explanations by prompting Gemma-2B-B MMesard et al. 2024 to rank the top- most important tokens given a textual input, which is then parsed, decoded, and mapped to input lokens, producing binary altribution vectors. LLM prompls describe the model's classification task, and prediction certainty before, and after model perturbation (Section 5.1]. The temperature is set to 0 for deterministic outputs. Varying synonyms, the order of tokens, and the number of top-K values to {5, 10} contribute to the robustness of our findings. The full explanation methodology is described in Appendix A.4.4 with an illustration in Figure A.2
Sparse Autoencoders. SAEs have lately come forth as an interpretability method for understanding the internal representations of LLMs Templeton et al. 2024 Huben et al., 2024). In our work, we generate SAE explanations using Gemma-Scope (Lieberum et al. 2024). pretrained on the residual block representations of the Gemma-2-22 2-2B model. Explanations are saved for all 26 layers at both 16K, and 65K widths. Given the sparsity of the explanation vectors, we use cosine distance to compute explanation distortion, defined as | as it effectively measures similarity regardless of magnitude. Appendix A.4.4 provides a detailed description of the generation process for each SAE explanation.
u.
Control Variants We also evaluate the faithfulness of two control variants: a random explanation (RAN) sampled from a uniform distribution, é, ),((1,,0 0), and a top- control variant (RAN-K) with K non-zero attributions, cach equal 1. Unless spocified, all experiments evaluate 250 explanations for the logit of the predicted class. For comparability, global, and local explanations are normalised by dividing the attribution map by the square root of its average second-moment estimate (Equation 21 (Binder al. 2022), with further explanation preprocessing details provided in Appendix A.4.4 For metric implementation, and mota-evaluation, we use the Quantus (Hedström et al. 2023b), and MetaQuantus (Hedström et al. 2023a libraries, respectively. Further experimental details for Q1, Q2, and Q3 are provided in Appendix 1.8.1
A.8.4. and A.8.5. respectively.
Table 2: An overview of datasets, and models, with references in Appendix
A.4 A semicolon separates models used per datasct.
MODALITY DATASET (N. CLASSES)
MODEL (size)
Acc. %
SOURCE
EXPL.
DIM
TASK
SMS SPAM (2)
SERT-TINY FT (1-M))
98.0
HF
128
SPAM
TEXT
IMDB (2)
PYTHIA FT (7.6M)); GEMMA-2 (2B)
86.4; 95.6
HF
312
SENTIMENT
SST-2 (2)
BERT-TIYY FT (4.4M)
98.0
59
SENTIMENT
IRAGENET-IK (1000)
RESNET18 (11.7M)
89.1
TORCHVISION
30176
OВJECT
PATH (9)
MEDCNN (235.2K)
84.3
LOCAL
784
PATHOLOGY
VISION
DERMA (7)
MEDCNN (23119
73.2
LOCAL
781
DERMATOLOGY
MNIST (10)
LENET (61.7K)
97.7
LOCAL
781
DIGIT
MNIST (10)
LENET (61.7K)
87.7
LOCAL
784
FASHION
ADULT (2)
3-LAYER MLP (11.7K); LR (28)
81.6; 83.3
OPENXAI
13
INCOME
TABULAR
COMPAS (2)
3-LAYER MI.P (IIIK): LR (16)
85.0; 85.3
OPENN
RECIDIVISM
AVILA (12)
2-LAYER MLP (3.5K)
80.8
LOCAL
10
TER
6.1 Measuring Empirical Reliability
To investigate the empirical reliability of GEF and Fast-GEF evaluations compared to singular approaches, we perform meta-evaluation, which is the practice of evaluating the evaluation method itself. To this end, adopt the meta-evaluation methodology from Hedström et al. (2023a), which bypasses the lack of ground truth labels by focusing on metric consistency ("does this evaluation method produce similar results under consistent conditions?"). For this, two practical meta-cvaluative tests are performed: the Input Perturbation Test (IPT), and the Model Perturbation Test (MPT). Each test returns a mela-consistency (MC) score (see Equation 20). which ranges between [0, 1]. Higher values indicate greater reliability. Full meta-evaluution scoring methodology is provided in Appendix
[ AA. As a sanity check, we also show Appendix A.8.3 that our proposed evaluators assign low scores different random control variants, where other metrics fail to
do so.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Sctup. We benchmark three evaluation methods per criterion. In the robustness category, we include Rela- Live Input Stability (RIS); Relative Representation Slabilily (RRS), Relative Ouput Stability (ROS) (Agarwal et al. 2022a). In the sensitivity category, we include Model Parameter Randomisation Test (MPRT) (Ade-
bayo ct al. 2018), Smooth MPRT (sMPRT), and Efficient MPRT (EMPRT) (Hodström et al. 2024 In the laithfulness category, we include Faithfulness Correlation (FC) (Bhatt el al. 2020), Fizel-Flippinn (PF) (Bach
ctt al. 2015), and Region-Perturbation (Samek et al. 2017). All metrics are mathematically described
Appendix A.4.5 To ensure comparability with the original publication (Hedström et al. 2023a), we run meta-evaluation on the same set of tasks, which includes ImageNet (Russakovsky et al.) 2015), MNIST (Le-
Cun et al. 2010 and IMNIST (Xiao et al., 2017] datasets with architectures such as ResNets (He et al. 2016) and LeNets (LeCun et al. 1998) architectures. Each metric evaluates GRAD, SAL, G-CAM, SHAP-G explanations. Further results, and details are provided in Tables A.1 and A.2 and Appendix A.8.5
Results. Figure G (A) shows that our proposed unified methods (GEF and Fast-GEF) achieve the highest overall MC scores, averaged over both MPT, and IPT tests. Our unified methods significantly outperform the most comparable evaluation approach, the faithfulness metrics, which also use Z perturbation steps, with average MC scores of 0.733 compared to 0.601. Although no evaluation method achieves perfect score (i.e., MC-1), the unified methods still perform comparably to robustness metrics, and surpass sensilivity metries, with average scores of 0.727, and 0.673, respectively. These results are encouraging as they show that unified methods can achieve high reliability, even when explanation behaviour is evaluated under multiple model conditions, unlike robustness, and sensitivity metrics that focus on a single perspective. While the ROS metric has the highest individual score, this not statistically significant, and it only offers a limited view of explanation quality. Figure (B) shows that unified metries excel in MPT, while robustness metrics perform slightly better in IPT. These score differences correspond to robustness metrics using input perturbations, and unified metrics relying on model perturbations. Further details are provided in Appendix A.8.4
(A) Meta- Eata--Evluation
(B) By Dataset
1.0
Category
Mean WC (MPT, IPT)
0.0
8.730
8.736
0.640
0.667
8.711
6.716 T
8.737
8.779
0.9
Unified
Faithfulness
0.560
MC (NPT)
Sensitivity Robustness
0.4
0.7
Dataset
InageMet
THNIST
WNIST
0.8
0.5
GEF Fast-CEF
MPRT SHPRT
eMPRT
RIS
ROS
RRS
0.5
0.6
0.0 MC (IPT)
0.7
0.9
1.9
Figure 6: Meta-evaluation, and comparison to established explanation evaluation methods. (A) shows the mean MC scores across MPT, and IPT, aggregated over all datasets, with the crror bars showing the standard deviation. (B) displays MC scores aggregated by the test type, and datasct, where the size of the scatter point denotes the standard deviation. GEF scores are computed for IMNIST, and MNIST datasets due to computational constraints.
6.2 Gross-Evaluating Local, and Global Methods
While local, and global explanations serve distinct purposes, and provide different insights w.r.t. their model, it is beneficial to compare them side-by-side in a unified view, as they often rely on similar methodological components, such as network gradients (LeCun et. al. 1998 Olah et al., 2017). The absence of general- purpose evaluations has however so far prevented such comparison. GEF and Fast-GEF effectively fill this gap, facilitating a first, cross-domain comparative faithfulness benchmarking between global, and local meth Extended results are provided in Appendix A.8.4
Figures 7 and provide an overview of cross-domain results for tabular, and vision tasks. For all tabular lasks, GEF estimales are computed. For vision tasks, due to the high computational cost of global hods Fast-GEF is used to allow for a fair comparison to local methods. As shown in Figure 7 no explanation mothod perfectly faithful to its model (f.e., no score equals 1) nor consistently outperforms others across tested tasks. This variation aligns with most benchnarking studies of local linear approximation methods, which rarely identify single winning method (Hedström et. al. 2024 Hesse et al. 2024). Among tested
17

---

Published in Transactions on Machine Learning Research (02/2025)
global feature visualisation methods, MACO generally outperforms FO variants, consistent with Fel et al. (2024). Comparing the faithfulness scores of DV, MACO, and FO reveals that more optimisation steps do not necessarily result in higher explanation faithfulness. All tested methods significantly outperform the random baseline (RAN), which serves as the theoretical lower bound. As expected, RAN produces faithfulness scores centered around zero. In Figure (A), and (B), we observe that RAN explanation distortion quantities are flat, i.e., independent of the model distortion. Tables A.4 and A.5
Appendix
A.8 present the result of
Figure 7
Global and Local Nethods
DV-58
FO-58
MACO-50
SMG
RAN
DV-100
FO-100
MMCO-100
LRP-2*
DV-250
FO-258
MACO-250
SHAP-G
INTG
LRP-
Cross-Domain Benchnarking of Global and Local Nethods
1.09
(4)
GEF Score
0.50
0.25
Avila (2-layer MLP)
Adult
Adult (LR)
COMPAS (3-layer MLP)
COMPAS (LR)
(3 La-tayer WIP)
1.00
Fast-GEF Score
(B)
0.50
0.25
IeageNet (ResNet16)
Dema (NedCHN)
Path (WddCNN)
ANIST (LeNet)
THNIST
(LeNet)
Figure GEF and Fast-GEF results on (A) local across tabular, and (B) local versus global methods across vision tasks, The where the standard deviation, and the sample size.
error bar shows the standard error, s.e., VN
InageNet (ResNet18)
Patth (MedCNN)
Global Locall Methods
Vision Tasks
Lanatinn Distortion
388
Expl.ane Lanaaiaion Cistor Lion
48
(D)
Scope
38
Global Rechods
Distortion Deltta
Local
290
29
20
180
58
(N)
(B)
(C)
10
15
58
100
158
268
758
-1.8
-0.5
0.00
0.5
1.9
FO 25e
NM000 250
D4-256 250
LOS 2+
SPC
SAL
600
INTC
GRID
non
Distortion
Distortion
GEF
SHA
LRD
Figure Fast-GEF results for vision tasks. (A), and (B) plot the model, and explanation distortion for ImageNet (ResNet18). and Path (MedCNN) along the perturbation path with perturbation steps. Here, global methods (DV, MACO, FO) selected with 250 optimisation steps. (C) displays the distribution Fast-GEP over all vision tasks. (D) reports the aggregated difference in explanation distortion between start 2 1, end =
local, and global methods, aggregated
Local Methods are Moderately Aligned. Despite local methods showing imperfect, and highly varying scores across tested models, and datasets, most GEF estimates in tabular tasks, and Fast-GEF estimates in vi- sion exceed 0.5, suggesting that the explanation retains some alignment with its model. This is not surprising given that parameter scaling directly effects the model's curvature, to which local gradient- based methods are highly sensitive (Dombrowski et al. 2019), thereby instantancously influencing their responsiveness to perturbation.
Figure 7 (A) shows that some local methods produce distortion outputs nearly monotonically related to its model, particularly at lower magnitudes (i.e., a 2 < 3). This finding nuances studies by [Adebayo et al.) (2018), which provide single-point sensitivity estimates, conclusively reporting low reactivity to parameter randomisation in local methods. Corroborating recent rebuttal works (Yona & Greenfeld] 2021 Sundararajan
18

---

Published in Transactions on Machine Learning Research (02/2025)
& Taly 2018: Binder et al. 2022 that challenges stark claims of method failure (Adebayo et al. 2018 we find that gradient-based methods are moderately faithful.
Global Methods are Constrained by Regulariser. Figure (C) shows aggregate Fast-GEF scores, indicating that global feature visualisation methods typically are less faithful compared to local linear ap- proximation methods. These differences in faithfulness estimates may be attributed to the global mothods' inherent reliance on optimisation procedure (Olah et al. 2017): and NN's ability to retain learned features despite perturbation via parameter perturbation (Binder et al., 2022 For reference, DV applies multiple regularisation techniques directly to the image, such Gaussian blur, and cropping regions based on norm, and pixel contribution, while MACO, and FO regularise the frequency domain representation, with MACO adding an extra layer of regularisation via predefined magnitude template. As observed in Figure g (A); and (B), despite model perturbation, explanation distortions stay relatively flat, with lower distortion deltas compared to most local methods, as displayed in Figure (D). A strongly regularised optimisation procedure may inherently limit the faithfulness of global methods, in favour maximally activated neuron response.
(D).
6.3 Evaluating LLMs as Post-hoc Explainers
While researchers have recently begun exploring the potential of using LLMs post-hoc explainers, there is still limited theoretical understanding, and empirical evidence on the general faithfulness of such approach. Can an LLM which is inherently decoupled from the model it seeks to explain, provide faithful outcomes? In our evaluation, we prompt Gemma-2B-IT for top- token explanation for a given input, and prediction pair for datasets characterisod by short tokenized lengths, f.e., for SST-2, and 128 for SMS Spam. The post-processed binary explanation vectors are then evaluated with GEF and Fast-GEF. See Appendix A.8.5 for further details, and extended results.
Relative GEF Rardom
(6) LLMx Random Local
(C) Tap-5 Tokens
(0) Top-10 Takens
Nethods
LLM-X-5
Pus Score
LLK-x-18
RAR-5
LINTG
0.0
3.0
1.0
INTG
L-INTG
-05
vil2
0.5
sma, spmm
SHAP-P-18
200
Palatiue improoement (5)
11M--:5 (DM-x
10
RAN 10
LINGG-S
LoNY 10
SHAP.P-S
SHAR 10
Figure 9: GEF (with M=3), and Fast-GEF results different top- K explanation NL.P tasks. (A) shows the percentage improve- ment GEF scores relative RAN, aggregated over all tasks, with error bars showing the standard error. (B) shows the results in the form of box plots for the datasets with SST-2 (left), and SMS Spam (right). (C), and (D) show the distribution of Fast-GEF scores for top-5, and top-10 explanations respectively, both aggregated over all tasks.
LLM-x Explanations Comparable to Random. Our GEF and Fast-GEF results in Figure g (A), and (B) show that Gemma-2B-IT as explainer is (i) significantly less faithful than local methods such SHAP-P, and L-INTG, and (ii) similarly unfaithful random explainers RAN-- or RAN-10, both SST- 2, and SMS Spam classification tasks. Figure g (C), and (D) demonstrate that these findings generalise over both top-5, and top-10 tokens tasks, aggregated over both datasets. Our results, showing that LLM-x explanations are not more faithful than random, differ from the encouraging results reported by Kroeger et al. 2023 who found GPT-4 to as faithful as local methods in identifying top-K tokens for tabular tasks. This divergence may naturally stem from variations in the experimental setup, including the specific explanation (ask, LLM used, methodology to evaluate faithfulness, and prompting straeggees, however, it also underscores that the faithfulness of LLM-x is still open research question. To fully understand the potential of LLMs as explainers, further rescarch with additional LLMs would be beneficial.
6.4 Measuring Faithfulness of Sparse Autoencoders
SAEs are gaining attention for their claimed ability construct interpretable "monosemantic" features of given layer of I.I.M. Yet, their general faithfulness remains underexplored Makelov et al. 2024 Mallen
19

---

Published in Transactions on Machine Learning Research (02/2025)
& Belrose 2024 To this end, we evaluate SAE explanations for Gemma-2-2B model on N 250 samples the IMDb dataset (Maas et al. 2017). Here, the Gemma-2-2B model is repurposed as a binary classifier by extracting the logits of the "positive" or "negative" classes at the final token position of the prompt. Appendix A.4.4 provides more details.
High Faithfulness Independent of SAE Width. Figure 10 (A) demonstrate that the SAE explanations generally are faithful w.r.d the model's intermediate representations. Fast-GEF scores are consistently above 0.75 except for fluctuations in layers 1, and 16 - 19. No significant difference is observed between 16K, and 65K widths, suggesting that the width of the encoding, i.e., the capacity of the SAE, does not correlate with explanation faithfulness. Moreover, Figure 10 (B) shows that although sparsity of the latent activations decreases in later layers, it does not. influence its faithfulness (i.e., P 0.023, computed with Spearman Rank correlation). This suggests that higher activation in SAE latents does not necessarily relate to its measured faithfulness. This raises the question of whether the faithfulness of SAE explanations is inherently scalable within different statistical or qualitative contexts where SAEs are studied. Figure 10 (C)-(G) further illustrates how explanation distortions vary across layers, with values (y-azis) increasing in the middle layers.
(A) SAE 16K and 65K Falthfuiness
(B) SAE-16K Falthfuiness vs Sparseness
1.0
1.0
Layer
Score
0.9
0.8
Layer 20
Layer 15
RostGGE
0.6
20 Layer
0.4
Wieth 65K
0.8
0
2
10
12
14
16
10
20
22
24
1000
2000
3000
4000
Layer
L1 Distance
(C) Layer
(D) Layer 10
(E) Layer 15
Layer
(G) Layer
Z=1
Distortonn
0.8
0.0
0.8
2+3
0.6
Zo4
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
10
10
Distortion
Model Distortion
Distortion
Distortion
Distortion
Figure Fast-GEF results for SAE explanations IMDb dataset, using = 5 perturbation levels, and models. (A) shows Fast-GEF faithfulness scores across layers. (B) shows the spansity as defined by T.I distance against the Fast GEF indicating no relationship. (C)-(G) illustrates how SAE distortions develop across model layers, coloured by perturbation level.
7 Discussion: Where Are We Going?
With the evolving landscape of interpretability, redefining both the conceptual framework, and the geometric foundations of explanation faithfulness is important. Our work puts forward long-overdue unification of robustness, sensitivity, and faithfulness evaluations, providing novel, and urgently needed, revised approach (Definition 5p to evaluate the direct alignment between explanation, and model functions (Section B In this work, we address the fundamental flaws of many existing evaluations: a systematic overlook of the intrinsic geometry of non-linear spaces (Section 4j. Our solution offers a threshold-free, fair comparison of functional distortions, making our approach not just another evaluation method but a necessary foundation for future interpretability research (Section
Novel, Empirical Insights. In a first-ever cross-domain faithfulness benchmarking of global, and local explanations on vision, tabular, and NLP tasks (Section 6), we learn that tested local explanation methods generally are moderately faithful. We find that global feature visualisation methods are comparatively less faithful, which is important understanding considering the recent evidence pointing to their general
20

---

Published in Transactions on Machine Learning Research (02/2025)
susceptibility to adversarial manipulation (Geirhos et al.) valuable to compare our findings with existing studies, to our knowledge, there is no direct study on the faithfulness of feature visualisations, Existing evaluations focus on the alignment with human preferences or improvement on a downstream task (Borowski et al. 2021 Zimmermann et al. 2021 Krishna et al. 2023; Bareeva et al. 2024b) or similarity to natural samples of the explained class (Fel et al. 2024]. Our findings
2023 Bareeva et al. 2024a). While it would be
on generalised faithfulness thus provide complementary insights into the quality of feature visualisation as model explainers.
Additionally, due to the recent interest LLMs potential post-hoc explainers Krishna et al 2023 Kroeger ct al. 2023). we study their faithfulness. We find no improved faithfulness compared to random explanations, and encourage more investigation on this question. Finally, we observe that residual stream SAEs on Gemma- 2-2B exhibit generally high faithfulness, with the width having limited inftuence (i.e., 16K or 65K). Further investigation required to fully understand the potential of SAEs, and L.M-x as generally faithful explainers.
7.1 Limitations
While the results in our paper allow us to claim that our proposed method is more sound geometrically (Section 4 more reliable empirically (Section 6.1p. p. and easier to use practically (Section 5), our evaluation alone does not imply that the explanation quality is sufficient. Without ground truth labels, we cannot assess the statistical validity of explanation function. An explanation may be estimated to be generally faithful but still lack intrinsic value (Bhattacharjee & von Luxburg 2024) or interpretable qualities [Bordl & von Luxburg 2024). The need for a thorough, application-groume
assessment of explanation quality that asserts value on a downstream task (Krishna et al. 2023: Lanham et al., 2023) is not eliminated when using GEF. Evaluation using synthetic models with known ground truth (Carmichael & Scheirer complement our proposal.
2023 could
7.2 Future Work
There are several exciting geometric, and empirical questions worth exploring. The geometric considera- tions in GEF suggest a deeper examination of the computational trade-offs of computing accurate pullbacks individual explanation functions, specifically in comparing global versus local methods. In future work, there is opportunity to build on the growing body of research in ML that draws from geometry, and related topics in higher mathema Lies to deepen our understanding of NNs, and problems to which they are ap- plied (Stephenson et al., 2021 Burns & Tang, 2023 Papamarkou et al. 2024). Recent theoretical studies on LLMs, and transformer models (Hoogland et al.] 2024 Burns] 2024] have illustrated how noural activations may arrive at, and utilise "superpositional" encoding stralegies (Elhage et al.. 2022), which prominently feature considerations or findings of a geometric or topological nature. Continued development of general frameworks, and theories that conceptualise NNs in terms of geometry, and topology (Bianchini & Scarselli 2014 Hauser & Ray 2017: Naitzat et al.] 2020 Benfenati & Marta 2023a b Burns & Fukai, 2023) will likely facilitate deeper understanding of both explanations, and evaluations particularly in relation to the underlying mathematical characteristics of data, optimisation processes, and learned functions.
Recent advances in manifold geometry have introduced tools to analyse how input data modulates internal processing through perturbations (Kvinge et al. 2023). Exploring how explanation faithfulness varies with training data, and how intersects with the geometric charactcristics of the model presents an exciting direction. We also expect models optimised with non-Euclidean methods (Fei et al., 2023) to reveal stronger differences between GEF and Fast-GEF, providing new opportunities to study the interplay between geometry, and faithfulness in explainability.
Lastly, we plan to expand our benchmarking scope to include natural aclivalion-maxxinsstiion explana- tions (Borowski et al. 2021), concept- based explanations like INVERT (Bykov et al.) 2023), and non- classification tasks. Given that pullback calculations can be computationally prohibitive for high-dimensional explanations, and highly parameterised models, exploring ways to speed up the Jacobian calculation (Equa- tion 19 and employ adaptive noise schedules would be valuable.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Broader Impact Statement
Interpretability, or XAI, is widely acknowledged as essential for responsible ML. This paper critically ex- amines current evaluation methods from unifying, and geometric perspectives, and proposes improvements. While negative societal impacts are improbable, overreliance on any single evaluation method is not advised.
Acknowledgments
This work was partly funded by the German Ministry for Education and Research (BMBF) through the project Explaining 4.0 (ref. 011S200551). Additionally, this work was supported by the European Union's Horizon Europe rescarch and innovation programme (EU Horizon Europe) as grant TEMA (101093003); the European Union's Horizon 2020 research and innovation programme (EU Horizon 2020) as grant iToBoS (965221); the German Research Foundation (DFG) research unit KI-FOR 5363 (project ID: 459422098); the state of Berlin within the innovation support programme ProFIT (IBB) as grant BerDiBa (10174498); and BIFOLD (refs. 01IS18025A, 01IS18037A); T.B. thanks Vasiliki Liontou for helpful discussions.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NearIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9525-9536, 2018.
Chirag Agarwal, Nari Johnson, Martin Pawelczyk, Satyapriya Krishna, Eshika Saxena, Marinka Zitnik, and Himabindu Lakkaraju. Rethinking stability for attribution-basel explanations. CoRR, abs/2203.06877, 2022a.
Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. OpenXAI: Towards transparent evaluation of model explanations. In Thitty-sirth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022b.
Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models. CoRR, abs abs/2402.04614, 2024.
AlignmentRessarchh robust-lan--ythii-imd----- -am--imdd---------aa-----33- (revision da04413), 2024.
Tiago A. Almeida, Jose Maria Gomez Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: New collection and results. In Proceedings of the 2011 ACM Symposium on Document Engineering (DOCENG'11), 2011.
David Melis and Tommi S. Jaakkola. Towards robust interpretability with explannng neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cisa-Bianchi, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7786-7795, 2018a.
David Alvarez-Meli Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cosa-Bianchi. and Roman Garnett (eds.), Advances in Newral Information Processing Syslems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7786-7795, 2018b.
Kenza Amara, Rita Sevastjanova, and Mennatallah El-Assady. Challenges and opportunitics in text genera- tion explainability. In Luca Longo, Sebastian Lapuschkin, and Christin Seifert (ods.). Explainable Artificial Intelligence, pp. 244-264, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-63787-2.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. In 6th. International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
Christopher J. Anders, David Neumann, Wojciech Samek, Alaus-Robert Müller, and Sebastian Lapuschkin. Software for datasot-wide xai: From local explanations to global insights with Zennit, CoRelAy, and ViRelAy. CoRR, 0bbs/2106.130000 2021.
Vijay Arya, Rachel K. E. Bellamy, Pin-Ya Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffiman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilovic, Sami Mourad, Pablo Pedemonte, Ramya Raghavendra, John T. Richards, Prasanna Sattigeri, Karthikeyan Shanmugam, Moninder Singh, Kush R. Varshney, Dennis Wei, and Yunfeng Zhang. One explanation does not fit all: A toolkit and laxonomy of AT explainability lechniques, 2019.
Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 283 294. Association for Computational Linguistics, 2023.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. J. Mach. Learn. Res., 11:1803-1833 1831, 2010.
Dilyara Bareeva, Marina MC Höhne, Alexander Warnecke, Lukas Pirch, Klaus Robert Muller, Konrad Rieck, and Kirill Bykov. Manipulating feature visualizations with gradient slingshots. In ICML, 2024 Next Generation of AI Safety Workshop, 2024a.
Dilyara Bareeva, Galip Ümit Yolcu, Anna Hedström, Niklas Schmolenski, Thomas Wiegand, Wojciech Samck, and Sebastian Lapuschkin. Quanda: An interpretability toolkit for training data attribution evaluation and beyond, 2024b. URL https://arxxv..rree prg/abs//410.000
Barry Bocker and Ronny Kohavi. hlups://doi.org/ /ttp://o..r./..443222/// 20..24432/555V
Adult. UCI Machine Learning Repository, 1996. DOL:
I. Bellido and E. Fiesler. Do backpropagation trained neural networks have normal weight distributions? In Stan Gielen and Bert Kappen (eds.), ICANN '98, pp. 772-775, London, 1993. Springer London. ISBN
078-1-4471-2063-66
Alessandro Benfenati and Alessio Marta. A singular riemannian geometry approach to deep neural networks theoretical foundations. Neural Networks, 008:331-443 2023a. ISSN 0893-6080. doi: 10.1016/j.neund 2022. 11.022.
A Alessandro Benfenati and Alessio Marta. A singular riemannian geometry approach to deep neural networks ii reconstruction of 1-d oquivalence classes. Neural Networks, 158:344-358, 2023b. ISSN 0893-6080. doi: hupp:://oi..org/00 0011:/0/../....eeee.00000 11.022
Jose Manuel Benitez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes? IEEE Trans. Neural Networks, 8(5):1156 1164, 1997.
Pictro Berkes and Laurenz Wiskott. On the analysis and interpretation of inhomogencous quadratic forms receptive fields. Neural Comput., 18(8):1868 1895, 2006.
Umang Bhatt, Adrian Weller, and José M. Moura. Evaluating and aggregating feature-basod model explanations. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAT 2020, pp. 3016 102022 ijeai.org, 2020.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Robi Bhattacharjee and Ulrike von Luxburg. Auditing local explanations is hard, 2024.
Monica Bianchini and Franco Scarsolli. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Transactions on Neural Nelworks and Learning Systems, (5(8):1553-1566 2014. doi: 1A.1009/NNNS..011.... 20..222936......
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Teike, Jeff Wu, and William Saunders. Language models explain neurons in language models. URL blob. core, windows. 2023), 2, 2023.
hutps://openaiiiiie
patteerrppppr///d/ee html. (Date accessed: 14.05.
Alexander Binder, Leander Weber, Scbastian Lapuschkin, Grégoire Montavon, Klaus-Robent Müller, and Wojeiech Samek. Shortcomings of top-down randomization-basee sanity checks for evaluations of deep neural network explanations. CoRR, abs/22 abs/2211.11465 2022.
Stefan Blücher, Johanna Vielhaben, and Nils StrodthofT. Decoupling pixel flipping and occlusion strategy for consistent xai benchmarks. arXiv preprint arXi::2411.06654 2024.
Douglas G Bonett and Thomas Wright. Sample size requirements for estimating pearson, kendall and spearman correlations. Psychometrika, 65:23 28, 2000.
Sebastian Bordt and Ulrike von Luxburg. Statistics without interpretation: A sober look at explainable
machine learning. CoRR, abs/2402.028870, 2024.
Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain CNN activations better than state-of-the-art feature visualization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee net, 2021.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfiel-Doddss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuils Thread, 2023. https://trarn crreansformee-ciruuus....t... http://eeiii.iit./oooommooooaane mamonosemanticcffaturess iaddx.htmm.
Lennart Brocki and Neo Christopher Chung. Evaluation of interpretability methods and perturbation arti- facts in deep neural networks. CoRR, abs/2203.0200) 02928, 2022.
Lukas Brunke, Prateek Agrawal, and Nikhil George. Evaluating input perturbation methods for interpreting CNNs and saliency map comparison. In Computer Vision ECCV 2020 Workshops, pp. 120 134. Springer International Publishing, 2020.
Thomas F Burns. Semantically memories in dense associative model. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 4936-4970. PMLR, 21-27 Jul 2024.
Thomas F Burns and Tomoki Fukai. Simplicial hopfield networks. In The Eleventh International Conference on Learning Representations, 2023. URL httte:://opearrrriieeeeet/tooooe wett://foorrrriiiessssssttaaaaa _LsHHgatwxxxxx
Thomas F Burns and Robert Tang. Detecting danger in gridworlds using Gromov's Link Condition. Trans- actions on Machine Learning Research, 2023. ISSN 2835-8856.
Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M.C. Höhne. Noisegrad - enhancing ex- planations by introducing stochasticity model weights. In Thirty- Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, February 22-Maan March 1, 2022, pp. 6132 6140. AAAI Press, 2022.
Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina MC Höhne. Labeling neural rep- resentations with inverse recognition. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Zachariah Carmichael and Walter Scheirer. How well do feature-additive explainers explain feature-additive predictors? In XAT in Action: Past, Present, and Fubure Applications, 2023.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In Proceedings of the 97th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1383 1391. PMLR, 2020.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In IEEE Conference on Compuler Vision and Pallern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp- 782-791. Computer Vision Foundation IEEE, 2021.
Yu-Neng Chuang, Guanchu Wang, Chia- Yuan Chang, Ruixiang Tang, Fan Yang, Mengnan Du, Xuanting Cai, and Xia Hu. Large language models as faithful explainers. CoRR, abs/ /1s/2402.04678, 2024.
Arthur Commy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimershcim, and Adrià Garriga-Alons, Towards automated circuit discovery for mechanistic interpretability. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
lan Covert, Chanwoo Kim, and Su-ln Lee. Learning to estimate shapley values with vision transformers. CoRR, 0bs/2200.05222 2022.
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local expla- nalions. In International Conference on Machine Learning, pp. 4794-44444 4815. PMLR, 2022.
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Rober Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'AlchéBBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Voncouver, BC, Canada, pp. 13567-13578, 2019.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dockks Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. Technical Report, Univeristé de Montréal, 01 2009.
Yanhong Fei, Xian Wei, Yingjie Liu, Zhengyu Li, and Mingsong Chen. survey of geometric optimization for deep learning: From cuclidean space to ricmannian manifold. arXiv preprint arXim:2302.082. 2023.
Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousscan, Rémi Cadène, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deeper networks with magnitude constrained optimization. Advonces in Neural Information Processing Systems, 36, 2024.
Robert Geirhos, Roland S. Zimmermann, Blair L. Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un)rcliability of feature visualizations. CoRR, abs//2306004719 2023.
Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L. Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(1))e774 e750, 2021. ISSN 2589-7500. doi: 10.1016 121016//2589-7500 SS////2559970000 (5000)))000)
Riccardo Guidotti. Evaluating local explanation methods on ground truth. Artif. Intell., 291:103428, 2021. doi: 10.1016/J.ATTINT 2020. 103428.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Peter Hase, Harry Xie, and Mohit Bansal. The out-of-distribution problem in explainability and search methods for feature importance explanations. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 94: Annuul Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virlual, pp. 3650 3666, 2021.
Johannes Hang, Stefan Zürn, Peter El-Jiz, and Gjergji Kasneci. On baselines for local feature attributions. CoRR, abs/2101.0000 2021.
Michael Hauser and Asok Ray. Principles of riemannian geometry in neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016.
Anna Hedström, Philine Lou Bommer, Kristoffer Knutsen Wickstrøm, Wojeiech Samek, Sebastian La- puschkin, and Marina M.-. Höhne. The meta-evaluation problem in explainable Al: identifying reliable estimators with metaquantus. Trans. Mach. Learn. Res., 2023, 2023a.
Anna Hedström, Leander Weber, Daniel Krakowezyk, Dilyara Bareeva, Franz Motzkus, Wojeiech Samek, Scbastian Lapuschkin, and Marina M.-. Höhnc. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34):
1-11, 2023b.
Anna Hedström, Leander Weber, Sebastian Lapuschkin, and Marina Höhne. A fresh look at sanity checks for saliency maps. In Explainable Artificial Intelligence, pp. 403 420, Cham, 2024. Springer Nature Switzer- land.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Benchmarking the attribution quality of vision models. CoRR, abs/2407.11910, 2024. doi: 10.48550//22iv.2207..11 11910.
Jesse Hoogland, George Wang, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, and Daniel Murfet. The developmental landscape of in-context learning. arXiv preprint arXiv:2402. 02364, 2024.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9744-9745, 2019.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL http::tt::::/ttt /ttttps:///penneviieee http://......ororr/////////t1.1.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? Tn Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 4198 4205. Association for Computational Linguisties, 2020.
Neil Jethani, Adriel Saporta, and Rajesh Ranganath. Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation. In Francisco J. R. Ruiz, Jennifer G. Dy, and Jan- Willem van de Meent (eds.), International Conference on Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valencia, Spain, volume 206 of Proceedings of Machine Learning Research, pp. 8925 8953. PMLR, 2023.
feature attribution. CoRR, abs/2404.11330/ 2024. Niklas Koenen and Marvin N. Wright Toward understanding the disagreement problem in neural network
26

---

Published in Transactions on Machine Learning Research (02/2025)
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. Captum: A unified and generic model interpretability library for pytorch. CoRf, abs/2009.070 2020.
Narine Kokhlikyan, Vivek Miglani, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson. Investi- gating sanity checks for saliency maps with image and text classification. CoRR, abs abss22106007475, 2021.
Laura Kopf, Philine Lou Bommer, Anna Hedström, Sebastian Lapuschkin, Marina MC Höhne, and Kirill Bykov. Cosy: Evaluating textual explanations of neurons. In ICML 2024 Workshop on Mechanistic Interpretability, 2024.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Takkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. CoRR, abs/2202.01602, 2022.
Satyapriya Krishna, Jiaqi Ma, Dylan Z Slack, Asma Ghandcharioum, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Nicholas Kroeger, Dan Ley, Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju. Are large language models post hoc explainers? CoRR, abs/2310.057997, 2023.
Henry Kvinge, Grayson Jorgenson, Davis Brown, Charles Godfrey, and Tegan Emerson. Internal represen- tations of vision models through the lens of frames on data manifolds. In NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023.
Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. Rethinking explainability as a dialogue: A practitioner's perspective. CoRR, abs/ hbss/2202.0187 2022.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in thain-of-thougeee reasoning. CoRR, abs/2307.1183 2023.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to docu- ment recognition. Proc. IEEE, 86(11):2278-238) 23224, 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database, 2010. URL http:
//yann. wewaaa.eeeuncco//eebbbbbee comm/xdbbmniss/
John M Lee. Smooth manifolds. Springer, 2012.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders every all at once on gemma 2, 2024. URL
Scott M. Lundberg and Su-Tn Lee. A unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4765-4774 2017.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
27

---

Published in Transactions on Machine Learning Research (02/2025)
Aleksandar Makelov, Georg Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In TCLR 2024 Workshop on Secure and Trushworthy Large Longuage Models, 2024. URL http:s/p/eeeeeereoonoooooooooorr net//foruu7/ddMMHHH9HHH
Alex Mallen and Nora Belrose. Balancing label quantity and quality for scalable elicitation, 2024.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Mor- gane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros Rosto Ambrose Slone, Amélie Héliou, Andrea Tac- chetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Crine Cristian Muraru, Grigory Rozhdestvenskiy, Hemryk Michalewski, lan Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/24035 220.22243...9,, 2024.
Grégoire Montavon, Scbastian Lapuschkin, Alexander Binder, Wojciech Samck, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Putlern Recoyvil., 65:211 222, 2017. doi: 10. 10.1016/).patcog.2066 20w...1.1...... 11.008.
Grégoire Montavon, Wojeiech Samek, and Robert Müller. Methods for interpreting and understanding deep neural networks. Digit. Signal Process., 73:1 15, 2018.
Niels J. Morch, Ulrik Kjems, Lars Kai Hansen, Claus Svarer, Ian Law, Benny Lautrup, Stephen C. Strother, and Kelly Rehm. Visualization of neural networks using saliency maps. In Proceedings of International Conference Neural Networks (ICNN'95), Perth, WA, Australia, November 27 December I, 1995, pp. 2085-2090. IEEE, 1995.
Gregory Naitzat, Andrey Zhitnikov, and Lek- lek-Heng Lim. Topology of deep neural networks. Journal of Machine Learning Research, 21(18)):1440, 2020.
Michael Necly, Stefan F. Schouten, Maurits J. R. Bleeker, and Ana Lucic. Order in the court: Explainable Al mothods prone to disagrecment. CoRR, abs/2105.03287 2021.
Anphi Nguyen and Maria Rodriguez Martinez. On quantitative aspects of model interpretability. CoRR, abs/2007.0753 2020.
Hoa Nguyen. Activation maximization. 2020.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
OpenAL. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Theodore Papamarkou, Tolga Birdal, Michael M. Bronstein, Gunnar E. Carlsson, Justin Curry, Yue Gao, Mustafa Hajij, Roland Kwitt, Pictro Lio, Paolo Di Lorenzo, Vasilcios Maroulas, Nina Miolane, Farzana Nasrin, Karthikeyan Natesan Ramamurthy, Bastian Rieck, Simone Scardapane, Michael T Schaub, Petar Veličković, Bei Wang, Yusu Wang, Guowei Wei, and Ghada Zamzmi. Position: Topological deep learning is the new frontier for relational learning. In Forty-firs International Conference on Machine Learning, 2024.
ProPublica. Compas recidivism risk score data and analysis, 2016. URL
compas- compas-analysia analysis
Luyu Qiu, Yi Yang, Caleb Chen Cao, Jing Liu, Yucyuan Zheng, Hilary Hei Ting Ngai, Janet H. Hsiao, and Lei Chen. Resisting out-of-distribution data problem in perturbation of XAL. CoRR, abs/2107.14000 2021.
28

---

Published in Transactions on Machine Learning Research (02/2025)
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should I trust you?": Explaining the pre- dictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Son Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144. ACM, 2016a.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should T trust you?": Explaining the pre- dictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Son Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144 ACM, 2016b.
Laura Rieger and Lars Kai Hanson. IROF: a low resource evaluation metric for explanation methods. CoRR. 2009/2003.08777, 2020.
Manuel Romero. bertttingfiiettuued------p-o---d
(revision 01 012e268), 2024.
Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci. A consistent and efficient evaluation strategy for attribution methods. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, TCML, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 18770 18795. PMLR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., (15(3)2211-2533 2015.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Miller. Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Networks Learn. Syst., 2811)):26602667 2017.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis., 128(3):336 359, 2020.
Avanti Shrikumar, Peylon Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. CoRR, abs/1605.01777 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Tch (cds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3145-3153, PMLR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Represenlations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Karen Simonyan, Andrea. Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 706.03825, 2017.
Irwin Sobel, Gary Feldman, et al. A 3x3 isotropic gradient operator for image processing. a talk at the Slanford Artificial Project in, 1968:271-277 -272, 1968.
29

---

Published in Transactions on Machine Learning Research (02/2025)
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christo- pher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631 1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplic- ity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
Clandio Stefano, Francesco Fontanella, Marilena Maniaci, and Alessandra Freca. Learning Repository, 2018. DOI: 10.24432 10.24432/C5K022 C5K02X
Avila.
UCI Machine
Cory Stephenson, suchismita padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yuo Chung, On the geometry of generalization and memorization in deep neural networks. In International Conference on Learning Representations, 2021.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature altribution baselines. Distill, 2020. doi: 10.23915 /dittil..00000022..
Mukund Sundararajan and Ankur Taly. A note about: Local explanation methods for deep neural networks lack sensitivity to parameter values. CoRR, abs/ 1806.04205, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Tn Doina Precup and Yee Whye Tch (ods.), Proceedings of the 94th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3319 3328. PMLR, 2017.
Zeren Tan and Yang Tian. Robust explanation for free or at the cost of faithfulness. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.); International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 13534-33501 PMLR, 2023.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmannel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum Mc- Dougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Roes, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting inter- pretable features from claude 3 sonnet. Thansformer Circuits Thread, 2024.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
VilyaVitalioa sert-tiny-ss12 (revision 2e14b76), 2023.
Matthew Watson Bashar Awwad Shiekh Hasan, and Noura Al Moubayed. Agree to disagree: When deep learning models with identical architectures produce distinct explanations. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 875 884, January 2022.
Kristoffer K Wickstrom, Marina M. C. Höhne, and Anna Hedström. From flexibility to manipulation: The slippery slope of xai evaluation. CoRR, abs/INSERT, 2024.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987.
30

---

Published in Transactions on Machine Learning Research (02/2025)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Syslem Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Chil-Kuan Yeh, Cheng-Y Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikunar. On the (in)fidelitty and sensitivity of explanations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Pro- cessing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 19965-10976, 2019.
Gal Yona and Daniel Greenfeld. Revisiting sanity checks for saliency maps. CoRR, abs/2110.14229) 2021.
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. CoRR, abs/1500.066777 2015.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet,, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 1, volume 8689 of Lecture Notes in Computer Science, pp. 818-833. Springer, 2014a.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Compuler Vision ECCV 2014- 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, volume 8689 of Lecture Notes in Computer Science, pp. 818-833. Springer, 2014b.
Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of CNN activations? In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 11730-11744, 2021.
31

---

Published in Transactions on Machine Learning Research (02/2025)
Appendix
The Appendix is organised as follows theoretical considerations (Section A.1 implementation notes for GEF
and Fast-GEF (Section A.2), details on the MetaQuantus framework (Section A.3), the general experimen- tal setup (Section A.4), analysis of model assumptions (Section A.5), aligument, patterns exlended results
(Section A.6 ablation experimental results (Section A.7), extended results from individual experiments
(Section A.8 and notation tables (Section A.9
A.1
Theoretical Considerations
The following subscctions provide detailed proofs, extensions, and discussions surrounding the GEF criterion.
A.1.1 GEF: Penalising Random Explanations
Following the discussion in Section a quality estimator should be able identify unfaithful explanations. In the following, show that our proposed GEF criterion (Definition 5) recognise two specific types of unfaithful explanations: constant, and random explanations, which is independent of its model, by design.
Corollary 1 (Penalising Unfaithful Explanations) Let &GEF be a quality estimator that yields esti- mates q GEF E R, with q GEF = 0 indicating lack of generalised faithfulness. To be valid measure of explonation quality, UGEF should assign low scores to both (1) constant, and (II) random explanations
Constant (I): Vé c = =ē GEF =0
Random (I): vē: é ~ ((0, = GEF =0
where é, and are perturbed, and unperturbed explanations, and ((, 1) denotes a uniform distribution. The GEF estimate GFF d()),dg) (Definition assigns low scores in the first, and the second case.
Proof. In case (I), the explanation does not change across perturbations, leading to explanation dis- tortion vector ds that contains only zeros
vé, = E [1, Z) e ē= D =
whereas the model's distortion vector d, will contain non-zero values due to perturbations
V2 [1, Z| Dj +
Consequently, the correlation coefficient p(d,, d3) will be zero with 9 GEF 0.
In case (II), the explanation distortion D will be approximately uniform across all perturbation steps since each perturbation is independently drawn from the same distribution:
vē, [L, Z1, #j j::e: è ((00,) 1) = D D
whereas the model distortion Dj will vary according to the degree of the perturbation
v, E [1, Z), 2j: è u(0, e D
The lack of correlation between d,, and de results in quality measure GEF that is equal to zero. This completes the proof.
A.1.2 GEF: Extension
To extend the applicability of GEF (Definition 5 to global methods that explain any neuron within model, we adopt Kopf et al. (2024). and view the model f as composition of two functions, F F: X + G. and L: G + Y,, such that f = LoF. F. Here G C Rexw" xh* where e e N is the number of neurons in the layer, and w², h² e N represent the width, and height the feature map, respectively. The function F, is referred as the feature extractor. We redefine the model function as a chosen feature extractor, and replace y Definition 5 with the activation of the th neuron such that i.e., y = Fx 6) X Rw'xh" While the model's output space y is replaced by G, similarly define the perturbed instance ÿ-
32

---

Published in Transactions on Machine Learning Research (02/2025)
A.1.3 GEF: Derivation of Linear Case
Our definition of GEF is based on the observation that any distortion present the model output space y; should be mirrored the explanation space &. Since neural networks are non-linear functions, a fair distortion in V, and &, requires the introduction of the pullback (Section
In the case of a linear model, however, the relationship between the distortion quantities Dy. and Dø be derived analytically. Here, the explanation is based on the first-order Taylor term, which is a linear approximation of the model's behaviour, forming the foundation of many established explanation methods (e.g., Montavon et al. (207)). We proceed to derive this relationship explicitly below.
Proof. Consider f to a linear model of the form f(x; 0) = ox + The explanation is the parameter vector 0. We can derive the expected distortion Dj == E8 ([( 0) [(x; 0m)) (see Equation 6 where m E [M] denotes the number of perturbed models for a fixed perturbation magnitude 5, L.e., a step 2.
D = (fx + c)² - 2(0x + cc)
[(0m + cc] + +Bom [Omm + c)²
Dj = 0² +c0x + 26 2x + +c]BBg.. [0m] 22(0x + +
+c)+ BBg.. Omxx² 200m=
Dj 00= 2- 2x (0x + +c)lE8m. [0m] 22xlEE [0m]
+ +x²2gg
02
Dj x² - 20xx2 BBn..
(m/] + 2 Bmn
(0
(14)
For the explanation distortion, a similar decomposition can be performed
D = 8² 26E
[0m] + E
02 m
(15)
By combining Equation 14 and we arrive at
D x² Dj.
(16)
We can construct the distortion vectors dø, and d,, and for each entry Equation 16 holds. When is defined as the Pearson correlation coefficient, we find the distortion of the model d,, and the distortion of the explanation function de to be perfectly correlated
p(dg, = cove (d)
which is equal to
1/2 Vare (dy) 1.
p(d,, d,)
(17)
1/2:2/rr Vara
This proves that in a simplified scenario, the key assumption of correlated distortion quantities holds, i.e., the model parameters 0 provide a perfectly faithful explanation. Since monotonicity is a weaker condition than linearity, Equation 17 also holds when p is defined as the Spearman Rank correlation coefficient.
A.1.4 GEF: Influence of
The parameter Z represents the number of steps in the perturbation path, and consequently dictates how finely the model's response will be captured by the GEF criterion (Definition 5) As such, selecting an appropriate value for Z is critical because it affects the interpretation of the results. A higher Z allows for finer evaluation of how well an explanation aligns with the model's behaviour under varying conditions. When using Spearman's rank correlation coefficient as our measure of p, larger Z generally stabilises the faithfulnes score due to the reduction in confidence intervals with more samples (Le.,, CT 1)) (Bonett &
33

---

Published in Transactions on Machine Learning Research (02/2025)
Influence of Z-2
(B) Influence
CCOPAAS 3-tayer MLPI
Model Distortion (nornalised)
0.6
Model Distortion Lise)
(Path, NesCae))
(Avdla, 2-Layer
ResNotia)
8.4
3.4
(MRIST, Lolet)
8.8
9.3
ACc.
0r1g
Per Purbubstion Levels
Pertubetion levels
Figure A.I: Model distortion (normalised maximum value) with (A) showing 2 perturbation steps, (B) showing Z 20 perturbation steps.
Wright 2000). Nonetheless, this assumes a monotonic response from both the model, and the explanation, which may not be realistic (Section [. If the model itself is not monotonic across perturbations, expecting the explanation to behave monotonically is also unrealistic.
Figure A.1 (A) (Z 2), and (B) (Z = =20) demonstrate the violation of the monotonicity assumption, as we observe large error bars, and divergent behaviour for 20, indicating non-monotonic responses. Accordingly, a moderate value of Z (Zar 2005 is advised for meaningful measurement.
A.2 Notes on GEF and Fast-GEF Implementation
In the following, we provide details on the GEF algorithm.
A.2.1 Generate Perturbation Path
To generate the perturbation path of length Z, satisfying * ÿ, Vy.0 E V, computationally find the minimum noise level of at = Z, such that the perturbed model's accuracy (ACC) approximates T where C is the number of classes, within a threshold, i.e., 6 < 1. Here, ACC N Ei (J(X;; where N is the number of samples in the test set, denoted X. This achieved by progressively increasing o², and applying to the model according to Section [.1] which process concludes when the model's accuracy satisfies the condition |ACC d < e, thereby determining perturbation level for subsequent evaluation.
Compute Path Length. For more faithful estimate of explanation distortion, for each step 2 E [0, Z]. we compute Dø, defined as the path length T<7)) We replace the integral in Equation 13 with sum over T steps:
T L(7) = Σ deT (J t=1
(18)
where dea E VRV denotes the feature-wise difference in explanations ie., (e- with .( é, and (y,) E VVCC is the Jacobian for fixed æ, and fa. To numerically approximate this Jacobian, for each step E T) perturb the neural activations (i.e., logits 9) by adding infinitesimal noise. In practice, sample from Gaussian distribution Uą (O 0.001) such that Os ÿ + uk. k E [I, times. After each perturbation, we recalculate the corresponding explanation ((ūr....) = é. Elements of the Jacobian (ét) are then computed as feature- wise difference between e, and ēμ:
De:
K
lim Σ - 0f; Koo
(19)
where i,j refers to the indices of the Jacobian (ê:).
Unless specified otherwise, we set M, Z, T, and in all experiments. Please find Appendix A.7 for an ablation study motivating these hyperparameters.
34

---

Published in Transactions on Machine Learning Research (02/2025)
A.3 Notes on MetaQuantus framework
For mela-evaluation, two-step process is employed. First, two types of controlled perturbations are in- troduced: minor, and disruptive. These are designed to evaluate the metric's resilience to noise (NR), and its scnsitivity to adversarial conditions (AR), respoctively, Specifically, these perturbations are applied in both the imput, and model spaces, resulting in two distinct tests: the Input Perturbation Test (IPT), and the Model Perturbation Test (MPTT) Second, the effects of the perturbations are measured in two meta-evaluative criteria: intra-consistency (LAC), and inter-consistency (IEC). Here, LAC refers mea- suring the similarity in score distributions cost-perturbation, and IEC refers to the occurrence of categorical ranking changes within set of distinet explanation method Each metric is then assigned summarised meta-consistency score, denoted as MC E [0,
IACNR lACAR IECNR
MC
2 where
(20)
[m*]
IECAR
with m² E R² representing an ideal quality estimator, essentially a vector of ones. A higher MC score, approaching 1, indicates superior reliability according to the defined evaluation criteria. Metrics that demon- strate both resilience to minor perturbations, and reactivity to disruptive changes achieve higher MC scores. We refer to the original publication (Hedström et al.] 2023a) for further details on the elements the meta-evaluation vector (Equation 20 and the framework in general.
A.4 General Experimental Setup
Here, we describe the models, datascts, tooling, hardware, explanation, and evaluation methods in this work.
A.4.1 Models, and Datasets
We employ varions models for vision, text, and tabular tasks in our experiments. See Table
For vision classification, we use ImageNet-1K for object recognition (Russakovsky et al. 2015 with ResNet18 (He et al. |016);; Pathology, and Derma for medical image analysis with proposed Med- CNN architecture (Yang et al. 2023): and MNIST (LeCun et al. 2010), and fMNIST, (Xiao et al.
2017) for digit, and fashion recognition with LeNet LeCCnn et al.
1998).
For text classification, we use SMS Spam Almeida et al.
2011
with a tiny, fine-tuned BERT [Alignment Research 2024 and
model [Romero] 2024g; IMDb (Maas et al. 2011) with Pythia
SST-2
Socher et al. 2013) with a tiny, fine-tuned BERT model
Vitya Vitaliih 2023).
For tabular classification, we use Adult (Becker & Kohavi 1996) and, COMPAS ProPublica 2016 with 3-layer MLP; and Avila (Stefano et al. 2018) with 2-layer MLP.
All models that are not publicly accessible are released at GitHuu repository at https://githun comm annahedstroem/GEr
A.4.2 Tooling
Several libraries, and open-source implementations enabled this work, including transformers (Wolf et al.
2020), OpenXAI (Agaraal et al. 2022b), Captum (Kokhlikyan et al. 2020), Zennit (Anders et al. 202]
Forr the IPT, independent, and identically distributed (f..d.) additive uniform noise is applied, defined as d; + P4. where vi Uía, J). For the MPT, multiplicative Gaussian noise is applied to all network weights, represented 0i 0 vr
with vi N(p, G²). The hyperparaneters a. oa follow the specifications of the original study (Hedström al. 2023a
'IAC provides normalised p-value derived from the nou- parametric Wilcozon signed-rank fest [Wilcoxon 19455) comparing the original, and perturbed score distributions. For NR, similar distributions expected, whereas for Alt, the distributions are anticipated differ. IEC counts ranking changes within explanation methods pettureetur,,tion, with an ideal metric showing consistent. rankings under minor noise (NR), and altered rankings under disruptive noise (AR).
35

---

Published in Transactions on Machine Learning Research (02/2025)
Shap (unddbrgg & Lee 2017), Activationn-Haxiiizatioo (Nguyen 2020), and Horama [eel et al.
2024
For metric implementation, and meta-evaluation, we use the Quantus (Hdssrröm et al. 2023) and
MetaQuantus (Hedström et al. 2023a libraries, respectively.
A.4.3 Hardware
The experiments were conducted using two hardware configurations: a cluster with four Tesla V100S-PCIE- 32GB GPUs, each offering 32 GB memory, and DGX-2 system featuring cight NVIDIA A100-SXM4- 40GB GPUs, each with 40 GB of memory. Both selups support the NVIDIA driver version $35.161.07 and CUDA 12.2.
A.4.4 Explanation Methods
All the hyperparameters of the individual explanations methods, listed in the main manuscript. Concern- ing the preprocessing, the signs of the attributions maintained, unless the method algorithmically relics it such SAL. Note, that not every explanation method is suitable intended to be used for all data modalities, and/ or model architectures. For example, GradCAM explanations are primarily designed for convolutional neural networks (CNN) models, and global feature visualisation methods are generally applied to vision tasks. We only report GEF and Fast-GEF results where appropriate.
Normalisation. We perform normalisation using the square root of the mean of the squared values (as detailed in the Appendix of (Binder et al. 2022)). This approach introduces less variance compared to normalisation techniques like scaling by the maximum value. It defined as follows
norm(e)
Ch
(21)
1/2
En nor e
HW
where H, and W represent the height, and width, respectively, and ên,w denotes the explanation value the pixel location (h, w
LLM-x Methodology. In the following, we describe the methodology used to produce LLM-x explana- tions. An illustration is provided in Figure A.2
Step 1. Ask LLM to explain
Step
Step Make explanation
Fost-process LLN answer
8630 #1 16001 ea
"given an [nnut]], explain top-K tokens that led to perturbed fg. decision [pred] ....."
(1: takent, 2: tokenZ,
Filter
nor mariie
encode
token- Input n satcning
tokenk)
(Geems-28-IT,
Figure A.2: A high-level overview of the three-step LLM-x methodology.
To generate LLM-X explanations, we use Gemma-2B3 2B-IT (Mesnard al., 2024) as the explainer. For each instance, create prompt describing the task, softmax confidence before, and after perturbation, and the class labels. The prompt template introduces the task (e.g., classifying analysis), and uses synonyms for model descriptions (e.g., "AI", "machine learning"), and perturbation types (e.g., "adversarially manipulated", perturbed with noise") to vary language. The softmax change calculated, and addod to the template and is described in the context of the model getting "more"
messages sentiment
6This normalisation method cusures that the squared distance from of each explanation score equals Unlike other normalisation techniques that constrain attribution values predefined range-mon range-making them suitahle for visualisa- tion-this method retains metric useful for comparing the distances across different explanation methods.
36

---

Published in Transactions on Machine Learning Research (02/2025)
"less" certain of a class label. The LLM is asked to return the top-K important tokens in structured JSON format, ranking tokens from 1 to K. The temperature is set to 0 for deterministic outputs.
After prompting, invalid or non-JSON outputs are removed. The LLM-ranked tokens are normalised by lowercasing, removing punctuation. Then, these tokens (or words) are encoded with the original model's tokeniser. Binary explanation vectors are created by matching the I.M-ranked tokens to the original input tokens, with a value of for matching tokens, and otherwise.
For full details, including the code, and prompt template, refer to our GitHub repository at https: //github. com./anaahesttrrmmmcf....
SAE Methodology. SAEs are designed to create sparse, interpretable represenlations of the internal activations a LLM, while preserving their reconstruction. The activations of a given layer I, i.e., fo(æ) are encoded into a sparse latent vector z(x), with latent dimensions larger than the internal representation, Then, these decoded representations are reconstructed such that g(z) fo(æ). This process is defined by the encoder and decoder functions
x(f((x)) == o (W Wen)fo(x) + bene).
(22)
g(2) = W deo + Ddecs
(23)
where a enforces sparsity through activation functions like ReLU or JumpReLU (Lieberum et al. 2024
using L1 LO regularisation during training. The SAE explanations generated by performing forward pass through the SAE encodings, and storing the activated values of z.
A.4.5 Evaluation Methods
Next, we mathematically define the evaluation methods (or "metrics") used in this work (Section 6.1
Faithfulness. Within the faithfulness category, evaluate three metrics, including, Faithfulness Correla- lion (FC) (Bhatt et al. 2020), Pinel- Flipping (PF) (Bach et al. 2015 and Peggion-Perturaatioonn (RP) (Samek
ct al. 2017 FC is defined as follows
(1 - (m))
FF
corr Se SICd
(24)
where [8| S D is a subset of indices of sample x, is the chosen baseline value, and a [,,,= are the masked input, with randomly chosen indices.
PF returns vector of prediction scores ps corresponding to pixel replacements i E n, which are sorted in descending order by the highest relevant pixel in the explanation @(æ, f. 0r A). To return one evaluation score per imput sample, we calculate the area under the curve (AUC) as follows
UP = > (Ús + 0141)
Pit1 P:
(25)
where Pt, and Pi+1 are the prediction values of the ith and (i+ 1)th perturbation step, and Ds. and Øs+1 the corresponding network prediction.
RP follows the most-reelevant-frse perturbation strategy, creating consecutive perturbed samples 0t 0c+1 such that for ği perturbed pixels correspond to larger respective explanation values than the pixel perturbed in fi+1. Across each perturbation curve, the area over the curve (AOC) is calculated, and averaged across multiple masked inputs & as follows
L (Do + 9k)
Σ
VRP = L
E(()
(26)
k=1
where L is the number of perturbed features in the input.
37

---

Published in Transactions on Machine Learning Research (02/2025)
Robustness. Within the robustness category, evaluate three metrics, including, Relative Input Stability (RIS), Relative Representation Slabilily (RRS), Relative Oupul. Slabilily (ROS) (Agarwal et al. 2022a) RIS extends (Alvarez-Mene Melis & Jaakkola 2018b), which is a measure of how much the explanation changes w.r.t. the input under slight perturbation á a u, The change is measured as the lp norm, and the RIS metric only considers perturbations that result in the same model prediction, i.e., f(()) = f()). It. defined as follows
V RIS max
Vn (() f())
(27)
max
Emin
where Cmin ensures non-zero denominator.
In contrast to the RIS metric, RRS considers the internal representation of the model C(-) (e.g., output embedding), while maintaining similar perturbation conditions
f(m.f.j:)) (.t..))) $.:A).
RRS
max max
ve E Ne: f(x) f())
(28)
(1
(1= c
Smin
where Emin ensures non-zero denominator.
ROS makes similar adaptations as the RRS metric, assumes however that the model's internal representations are not accessible. Instead the output logits h(x), and h()) are assessed
((..f.ū:)) ((2 f.ūAA)
ROS max max (11(z)) - h())) Emin
Vā E Ne: = f(i)
(29)
where Emin ensures non-zero denominator.
Sensitivity. Within the sensitivity calegory, we evaluate three metrics, including, Model Parameler Ran- domisation Test (MPRT) (Adebayo et al.) 2018), Smooth Model Parameter Randomisation Test (sMPRT), Efficcentt Model Parameter Randomisation (cMPRT) (Hedström et al. 2024). MPRT measures the simi- larity between the original explanation et, and the explanation == o(x, fi. y) of the perturbed model f randomised top-down fashion up to layer I [L. I,....
GMPRT ê(e,
(30)
with similarity function p: D x D + E.
sMPRT computes quality estimate IR between explanations e, == 6(2,, ys A), and êτ, 6(2,, fi A) averaged over i E [1, N] where é,, corresponds to the perturbed model ft randomisod a bottom-down
fashion up to layer / E [1.2....,
N
GOMPRT
Σ
e
1 N
ē1,6
(31)
N
with i; x + ne, and 17 N(0, with ||ml|p holding with high probability, for o,e R.
eMPRT measures the relative rise in the complexity the explanation from a fully randomised model f such that è dbx; y; A):
CEMPRT -
(32)
<(c)
where D + RR is complexity function, e.g., discrete entropy.
38

---

Published in Transactions on Machine Learning Research (02/2025)
A.5 Analysing Violations of Model Assumptions
To understand whether perturbation techniques commonly employed for robustness, sensitivity, and faithful- ness evaluations gencrally fulfill the critical assumptions of model distortion (Assumptions 3 we performed several experiments. To investigate how often model robustness, sensitivily, and faithfulness (Assumptions 3 hold versus fail in practice, we set up a simple experiment that tracks model, and explanation distortions, I.C., Dj, and De, while applying perturbation commonly used in evaluation such as additive Gaussian noise for robustness evaluation, top-down, and bottom-up layer-by-layer parameter randomisation for sensitivity evaluation, and cumulative masking for faithfulness evaluation.
[
Model Robustness (Ass,
Mocol Robustness (Ass.
Model Robustress MAss. 1) (0) Model Fai:hfulness (Ass.
Fodel Distortion
200
Tragehet
1200)
200
(PleOOWW)
Math
200
Aur
15-
Input Farameter
Wh
Black
250
150
1.0
50-
59
30
Act. Drig
Acc.
2/CAet.
Arc.
1/6
Orig
40
100
Additive Noise Level (2)
Additive Noise Level
(2)
Additive hoise Level
Comulative Perturbat 191
(F) Model. Sonsitivity (Ass. 2!
(F) Model Sonsi tivirt (Ass. 2;
(6)
Model Sensitivity (AsS. 2) (1) Model Foithfulness (Ass, 3)
Ingut Parzeeter
order
Order
pistortion
Bottoe
Botton-up
patch
patci
2x8 putch
Expectation
0-der Buttce-up
2
Top-covn
20
80
Conulative Per turbattoon 18)
Layer Layer Randorisso.ion
Layer-by- Layer Rardonisatior
byyerr Sandanisation
Figure A.3: Impact of model distortion (y-axis) over common perturbation types robustness, sensitivity, and faithfulness evaluations, across different datasets, and NN architectures. (A), (B), and (C) depict the distribution of model distortions across different perturbation magnitudes of additive Gaussian noise for ImageNet (ResNet18), Path (MedCNN), and Avila (2-layer MLP), respectively. (D), (E), and (F) show the average, and standard deviation model distortions over layer-wise top-down, and bottom-up randomisation for the same datasets (as indicated by colour). (G), and (H) display model distortions for randomly chosen MNIST (solid line), and IMNIST (dashed line) samples (LeNet) under cumulative perturbations using different patch sizes (1 x 1, 2 x 2, 4 x 4, 8 x 8), and bascline replacement strategics (block, white, mean).
Model Robustness Under Additive Noise. To understand the extent to which model robustness (Assumption 1 is generally satisfied for robustness evaluation (Definition 2), we examine Figure A.3 (A), (B), and (C), and Figure A.4 (A) and (B). Here, the distribution of D, is visualised over Z = 10 input perturbation steps, showing how model distortion varies with increasing input perturbation magnitude, using additive Gaussian noise, i.e., ; 14 ~ .N(0, to generate perturbed inputs a; = x + Vi, with a increasing until the model behaves randomly (i.e., accuracy = 1/C). While the average trend (blue line) indicates that larger perturbation causes higher model distortion, sample-wise exceptions frequently appear. In Figure [A.5] random sample trajectories reveal both correlated and uncorrelated patterns between perturbation levels and model distortions. This is key observation, as it implies that model robustness cannot be assured by a general threshold without inspecting each evaluation sample individually.
Model Sensitivity Under Layer-by-Layer Randomisation. For sensitivity evaluations (Definition 3) to be mcaningful, the model distortion caused by perturbation must be significant (Assumption 2 To test this practice, we perform consecutive layer-wise model parameter randomisation; in both a lop-down (Ade- bayo et al. 2018), and bottom-up [Hedström et al., 2024) manner. From Figure A.3 (E), (F), and (G), and Figure A.4 (D) and (E), we observe that, although model distortion generally increases with layer-wise randomisation, there are exceptions of mon-ooootyyyyity (see, e.g., Path, and Avila results in Figure A.3 (E), and (F), respectively). The high standard deviation (see the error bars) suggests that layer-wise randomisa-
39

---

Published in Transactions on Machine Learning Research (02/2025)
(A) Mode Robustness
(Ass. 1)
(8)
Model Robustness JAss.
(C) Model
Faithrutness (ASS. 31
Model Distortion
265
(Lokeet)
280
layer MLP)
154
150
Erput Paramcter
Black
100
10
Pean
53
Expectatior
Acc. Orig
Acc.
1/C. Acc. drip
Acc.
1/C
28
48
68
66
188
Acditive Noise Leval (2)
Additive Noise Level (2)
Cunutative Pertarbation (9)
(0) Model Sonsitivity
(E) Model Sensitivity
Model Foithruiness
15.4
Pode: biscortion
12.5
Sot Sotton-ue up "ap-down
15
Irput Paraketor
30.0
30-
potch
464 potch
axd patch
Expectation
Bocton-up up Top-doon
9.8
2
40
100
Curslatiiv Parturbation (%)
Loyer-by bvyr-by-Laye Laye Pandomisation
Layer-by-Layer Randonisation
Figure A.4: Impact of model distortion (y-axis) over common perturbation types robustness, sensitivity, and faithfulness ovaluations, across different datasets, and NN architectures. (A), and (B) depict the distribution of model distortions across different perturhation magnitudes additive Gaussian noise for MNIST (TeNett), and COMPAS (2-layer MLPP), respectively. (D), and (E) show the average, and standard deviation of model distortions over layer-wise top-down, and bottom-up randomi- sation for the same datasets (as indicated by colour). (C), and (F) display model distortions for randomly chosen MNIST (solid line), and IMNIST (dashod line) samples (LeNet) under cumulative perturbations using different patch sizes (Ix1, 2x., 424, 828), and baseline replacement strategies (black, white, meon).
(Avila, 2-layer HLP)
(COMPAS, 3-layer MLP)
(PRIST, LeNet)
Model Distortion 60
Nodel Distortion
125
188
Model Distortion
80
Perturbation
75-
30
20
10-
2
2
6
Nerturbation Level
Perturbation Level
Perturbation Level
(Path, MedCNN)
(InageNett ResNet10)
Nodel 0istortion
380
Nocel Distortion
Perturbo
oblo
100-
38
Level
6
Perturbation Level
Perturbation Level
Figure A.5: Sample-wise trajectories of model distortion (y-ozis) across different perturbation magnitudes of additive Gaussian
noise. Each panel includes line plats across N = 10 samples, showing both correlated and uncorrelated outcomes.
tion fails predictably dictate the degree of model distortion, undermining the assumption that significant model distortions will always occur in sensitivity evaluations.
Model Faithfulness Under Cumulative Input Perturbation. To investigate whether model distor- tion increases monotonically under cumulative input perturbation (Assumption 3), measure D, using standard "pixcl-flipping" faithfulness procedure (Bach et al.) 2015). By randomising the perturbation order, the resulting faithfulness curve should reflect only the model's response; any deviation from a linear trend suggests that Assumption 3 is failed. Observing Figure A.3 (D), and (H), and Figure A.4 (C) and (F), we see that neither patch size (top) nor replacement strategy (bottom) induces monotonic non-decreasing model behaviour, While these results are expected due to the model's inherent nonlinearity, and OOD effects (Hase
et al. 2021
Hesse et al.] 2024). it is not accounted for in the faithfulness evaluation itself (Definition
40

---

Published in Transactions on Machine Learning Research (02/2025)
When genuine signals (i.e., explanation quality) are not decoupled from noise (i.e. non-monotonie model behaviour); interpretations may become biased (Hokeer et al. 2019 Brocki & Chung 2022 Brunke et al. 2020).
Together, these results reveal how easily, and systematically Assumptions 1 are violated by perturbation strategies commonly applied practice (Section 2.1.1 Our findings are consequential as they demon- strate that the validity of existing robustness, sensitivity, and faithfulness evaluations (Definitions 2 are frequently undermined. As displayed in Figure A.3 there many sample-wise exceptions where the per- turbation magnitude, and the model distortion quantity not strictly monotonically related, challenging the assumption that increased perturbations lead to proportionally greater distortions, and vico-versa.
A.5.1 Issues with Cumulative Input Perturbation
If small changes in input parameters, cause large variations in evaluation outcomes, evaluation reliability is compromised. Corroborating previous studies [Brunke et al. 2020; Brocki & Chung, 2022 Rong et al. 2022), the varied faithfulness curves in Figure A.3 (G), and (H), and Figure A.4 (C) and (F), demonstrate how input parameter choices, such as patch size or pixel value, can drastically influence the evaluation outcomes across tasks, i.e., act as evaluation confounds (cf. the same paramcter for MNIST solid line vs. IMNIST dolled line). These variations between lasks expose simple, yel systematically overlooked issue in faithfulness evaluations: that parameter choices to perturb the input inherently introduce task-specific biases to the evaluation. Attempts to mitigate these biascs-using ussng inverse curves (Blücher et al. 2024)
assessing the OOD impact of perturbations (Qiu et al. 2021 Haug et al., 2021] fail to address the core problem: that evaluation methods (Section 2.1.2) that require input parameters to be tuned according to its task, are inherently biased, impeding impartial comparisons across tasks, and explanation approaches.
A.6 Alignment Patterns, and Extended Results
Figure A.6 provides complementary results to Figure g the main manuscript. The scatter points are coloured by perturbation magnitude up to Z 10, using additive Gaussian noise. The varying but consistent overlaps of points of high and low perturbation magnitudes alongside the almost uniform distribution along the y-axis, rllustrate that perturbation elfects are not guaranteed to have a proportional effect on the model and explanation functions. Thus, using the perturbation magnitude as an indicator of the magnitude of
which the explanation should change (as done in cxisting ovaluations, see Definitions 3 is not reliable.
(NedCIN))
(MedCNN)
La-Layer PLP)
(LeNet)
SWAP. .a Distortion
Input noise
Input noise Zet
Ioput noise
Input noise
2e1
Distortin Lion
Distort ion
2x1
Distortion
Z+3
245
2+5
2+5
2+5
2:9
18
:00
200
330
100
105
200
300
<00
2.5
5.0
7.5
10.0
12.3
10
20
30
40
50
Model Distortion
Model Distortion
Model Dis Distortion
Model Distortion
Path (MedCMN)
COMPAS (3 leyer NLP)
Inageliet (ResNet
G980 Distortion
68
Ingut notse
SAl Distortion
2,8
2.5
Toput
SAL Distortion
Input
2+4
1.0
28
Input noise
2+1
Z+5
Z+9
Figure A.6: Model (x-axis), and explanation distortions (y-axis) under varying levels of additive Gaussian input noise for vision and tabular tasks, indicated in the titles. Scatter points represent individual samples, coloured by perturbation magnitude = 10), with overlapping contours highlighting the relative aligmment patterns. The individual plots contain SHAP-G, GRAD, and SAL explanations, as indicated by the y-axis labels.
41

---

Published in Transactions on Machine Learning Research (02/2025)
Next, we show the change in the relationship between model distortion and explanation distortion when comparing the commonly used input perturbations with model perturbation (Fast-GEF) and when mea- suring explanation distortion and model distortion on the same geometry (GEF). Figure A.7 illustrates the relationship of model and explanation distortion across these three cases from left to right. Each contour plot. includes N 10 samples per perturbation magniludes from Z to Z 5. The scalter points are colored to one of three increasing perturbation magnitudes (Z = to Z = 3). We can observe the distortion quantities across tasks with increasing model complexity, from simpler tabular task (first row) to highly parameterised model for vision task (last row).
As expected, the first column (input perturbation) coincides with Figure A.6 and yiclds the same findings. However, in the following two columns, we can observe how Fast-GEF and GEF behave in practice. Most notably we observe that the alignment between model and explanation distortion changes from less complex to more complex tasks. Furthermore, we find that Fast-GEF tends to generate higher coherence compared to imput perturbation, except for ImageNet, and that GEF yields the most coherent distortions. While these findings appear to support both our approaches, it is important to note that without access to ground truth, it is unclear whether the contour plots should necessarily show stronger coherence (i.e., post-perturbed correlation), as it depends on the relationship between the explanation and model functions.
Avila (2- Layer RLP)
Avila (2-layer MLP]
Avila 12-layer MLPI
GRAD distortion
as 3u4.48
GR4D oistortzo
ple.
Distortion
608-
100
209-
20
Model Distortion
Yodel Distortion
Nodel Distorrtion
COMPAS (3-layer MLP)
COMPAS (3-layer PLP)
COMPAS [3-Layer HLP)
2.0
pts St-0.52
GRIO Distortion N
pls, $
GRID Distortion
40
51+8.66
2.5
-8
2.8-
28
0.3
20
18
10 Model
15
20
Model Distortion
Model Distortion
Distortion
(Lelict)
NHIST (LoNet)
KNIST (Lelet)
10
pis 31-8.68
Eistortion
48
Srm8 64
GR1D Distertion
1.5
pts. 0t5.800.50
30
1.0
28
Ghib
38
10
5
10
15
28
25
1000
1593
208
460
668
Model. Distortion
Distortion
Model Distortion
ImageNet
(Resllet1B)
IeageNet (ResMet18)
xie Inagetict (Restie118)
Glib Diztortion
100
dG 3 44
Eistortion
pls, S1e
0M6 Distortion
2.0
300
280
1.5
200
200
1.0
J00-
8.5
18
15
29
Model Distortion
Podel Distortion X18
Model Distortion x182
Input
2-1
Z+1
Figure A.T: Each plot shows the model (x-axis), and explanation distortions (y-axis) under different types of noise for Gradient (GRAD) explanation. The first column shows distortion outcomes after applying additive Gaussian input. noise. The second and third columns show distortion outcomes after applying model parameter scaling (Section 5.1 The second columu computes explanation distortion using Fast-CEF and the third column computes distortion using CEF (i.., with pullback mechanism). Scatter points represent individual samples, coloured by perturbation magnitude (2=1, 2=2, 2=3), with Z = 5 number of steps.
42

---

Published in Transactions on Machine Learning Research (02/2025)
A.7 Ablation Study
To better understand the influence of the hyperparameters, on the proposed GEF evaluation method, we conducted an ablation study. We employed two tasks, i.e., a tabular dataset (Avila) using a 2-layer MLP model with SAL explanations and a vision datasct (MNIST) using a LeNet model with 250 random expla- nations, sampled from a uniform distribution, i.e., é, = ((0, 1). For each hyperparameter, é.e., the number of perturbed models M, the length of the perturbation path Z, the number of summation steps T, and the number of samples K, we enumerated over values from 0 20, while fixing the others at default value of 10. For each configuration, recorded mean (solid line) and standard deviation (shaded area) of the model distortion, explanation distortion, Jacobian quantity, and the mean computation time.
Model Distortios Analysis
Exclaration Distortion Analysis
Medel Distortion Analysis
Explanation Distortion Analysis
17.5
10032
Wein Valuee
5032
Weln
7.3
-5932
35-
Paraneter
Poramcter value
lysis
Jacobian Anelysis
Analysis
Analysis
3959
Yaat Velle
1028
0.52
0.35
2003
2030
Puan
Paan
1043-
-0.23
15 Paramoter Value
Poroecter Value
Porancter Value
Poroecter Value
Figure A.8: Ablation study results across hyperparameterss M, Z, T, and K for two tasks: (left) Saliency explanation on Avila (2-layer MLP), and (right) Random explanation ou MNIST (LeNet). The mean value (solid line) and variance (shaded area) are reported. The time analysis is measured in seconds.
A.8 demonstrates the results for both the tabular (left) and the vision task (right). As can be obscrved by the converging values of the standard deviation and mcans, the hyperparametors are resilient to key parameter changes once parameter values reach 5 or higher. The Jacobian variance reflects the curvature captured in its estimate. While the variance increases for parameter values above 5, the mean stabilises, indicating diminishing returns in capturing additional curvature. At parameter values of 5, the majority of the curvature is already captured, providing a practical trade-off between computational efficiency and quality of approximation. Considering computational time, all parameters lead to linear, non-negligible increase. Among them, Z and M are identified as the primary drivers of time. Based on these experimental findings, setting K = T = Z = M = 5 balances computations efficiency and stability of the quality est estimate.
Figure
A.8 Experiments, and Extended Results
This section provides descriptions of experimental setups, and extended results, including meta-evaluations, and agreement between different scoring methods. Additionally, we present further results for random control variant sanity checks, cross-domain benchmarking, and LLM-X methodology, and extended results.
A.8.1 Meta-tvaluation
To employ the scoring methodology (Section A.3 we used the pre-existing test suite available in the MetaQuantus library[ with their pre-defined hyperparameters.
MetaQuantus Hyperparameters. We applied these metries over K = 5 perturbations, conducting 3 iterations with the test configurations specified in the library for two different sets explanation methods, namely (GRAD, G-CAM), and (SAL, SHAP-G), which were evaluated by cach metric. The explanation
Find the library at http://hhhhcceeemmmmmmattttttttp
43

---

Published in Transactions on Machine Learning Research (02/2025)
method groups were created by randomly selecting methods from the complete set of available methods, ensuring consistency across various experimental setups, such as dataset, and model combinations. In terms of choosing K, and the number of iterations, we followed the recommendations from the original study to keep the standard deviation between different sets relatively low. To ensure a fair comparison across metrics, all shared hyperparameeers were assigned the same values.
Metrics Hyperparamckers. All metrics have been implemented in Quantus (Hedström et al. 2023b Different hyperparameters were chosen for the individual metrics based the dataset. For the robustness metrics, we use 5 noisy samples, and employ additíve Gaussian noise such that v ~ NO 0.001). For the faithfulness metrics, we use 28 features per perturbation step, and patch size of 7 for the MNIST, and IMNIST datasets. For ImageNet, we set the number of features to 896, and the patch size to 28. For FC, similar to the robustness metrics, we let run times. For the sensitivity metries, namely MPRT, and sMPRT, we use a noise magnitude of 0.01 for cach sample, and sMPRT uses 5 samples in its calculation. For all sensitivity metrics, we use the Spearman rank correlation coefficient.
Table A.I: MC scores and standard deviation for unified, and faithfulness methods listed in A.4.5 for ImageNer, MNIST, and IMNIST datasets. The final row shows the mean score for each metric across the datasets. Values range between [0. 11, with higher values indicating better outcomes. Due to computational constraints, GEF scores are only computed for IMNIST, and MNIST datasets.
UNITIED
FAITHPULNESS
GFF
FAST-GBP
PF
FC
RP
IMAGENET
NAN = NAN
0.78 = 0.02
0.63 # 0.01
0.51 :: 0.02
0.63 = 0.06
MNIST
0.75 + 0.07
0.74 + 0.03
0.61 + 0.04
0.63
+ 0.08
0.59
+ 0.03
FMNIST
0.71 = 0.07
0.71
0.03
0.63 4: 0.01
0.50
0.04
0.58
+: 0.09
Mean
0.73 = 0.07 0.74 = 0.03
0.62 + 0.02
0.56 +: 0.03
0.59 = 0.06
Table A.2: MC scores and standard deviation for sensitivity, and robustness methods listed in A.4.5 for ImageNet, MNIST, and IMNIST datasets. The final row shows the mean score for each metric across the datasets. Values range between [o. 1]. with higher values indicating better outcomes.
SENSITIVITY
ROBUSTNESS
MPRT
SMPRT
EMPRT
RI8
ROS
RRS
IMAGENET
0.71 0.02
0.69 # 0.04
0.71
0.02
0.72 # 0.06
0.76
0.07
0.75 # 0.04
MNIST
0.63 0.02
0.66 = 0.04
0.76
0.03
0.73 : 0.02
0.70
0.09
0.74 = 0.09
FMNIST
0.63 0.01
0.67 + 0.05
0.67
0.05
0.70 + 0.02
0.77
0.06
0.70 + 0.03
Mean
0.64 0.02
0.67 = 0.04
0.71
0.03
0.72 = 0.03
0.74 0.07
0.73 = 0.05
Extended Results. In Tables A.1 and A.2
we provide the corresponding results for Figure G
A.8.2 Agreement between GEF, and Fast-GEF
To determine whether the simpler, computationally efficient Fast-GEF method can serve an alternative to the more exact but computationally intensive GEF method, we compare the agreement between their respective faithfulness estimates. For subset of explanation methods, and tasks (sce Table D we thus compute scores, and rank explanation methods from RI to RN. While it is expected that estimates from the two methods differ, a high agreement in a categorical ranking would make Fast-GEF a practical alternative in resource-constriined environments.
Results. Figure A.9 (A) visually compares how GEF and Fast-GEF ranks (x-axis) each explanation method in terms of increases (y-axis), highlighting the relative agreement between them. The explanations in the tabular, and text tasks show perfect ranking agreement. In the MNIST vision task, with minimal nominal differences, GRAD, and SHAP-G methods disagree in their ranking (R1, and R2), but such disagreement can be expected acknowledging the algorithmic similarity between these explanation methods. In the Derma vision task, the same pattern is observed, yet with a slightly larger difference for the global method FO-50. Interestingly, we observe that nominal differences are pronounced for global methods (DV-50, and FO-50), and that Fast-GEF tends to generate slightly lower faithfulness estimates cf. GEF.
44

---

Published in Transactions on Machine Learning Research (02/2025)
(A) Avila (2- Layer PLP)
(B) COMPAS (3 Layer HLP)
(C) MNIST Lellet)
(D) Derna (MedCMN)
(E) INDU (Pythia
8
1.0
1.0
0R31
1.09
8.8
GEF Score
0.75
0.5
9.4
183)
0.5
8.50
0.4
3.2
0.5
0.2-
3.8
1843
(84)
6.2-
(86)
(84)
185)
(R3)
8.3-
-3.2
-0.2
0.25
0.2
Fast GEF
GEF
Fast
Fast GEF
LRP-2
SA.
GRAD
NAN
DV-50
SHAP-6
FO-50
RAN
L-INTG
SHAP-P
RAN
SHAP-G
SMC
GRAD
INTC
RAN
Figure A.9: (A) to (E) illustrates the GEF scores of GEF and Fast-GEF (with M 1) for various explanation methods, and tasks. Explanation methods are ranked between R.l to RN, in descending order.
A.8.3 Scoring Control Variants
Next, we validate that both GEF and Fast-GEF assign low faithfulness scores to different control variant explanations. In our sanity checks, we evaluate explanations generated by uniform sampling, i.c., é u(0, 1), a constant value, i.e., é; 0, and with model-independdnt Sobel filter. For non-random reference, evaluate GRAD explanations for the predicted class of the Derma lask (see Table 2 (Sobel el. al. 1968). For comparability, extend this sanity check exercise to metric per evaluative criteria, i.e., FC (faithfulness), MPRT (scnsitivity), and RIS (robustnoss). Hyperparamoters are provided in Appendix A.8.1
Table A.3: Evaluation scores of Derma (MedCNN) explanations for three random, and one regular (GRAD) explanation. The arrow (f,4) indicates whether higher or lower values are better. A nan value indicates that no score is produced.
EXPLANATION
GEP (t)
PAST-GEP (1)
PC (f)
MPRT (L)
RIS (4)
CONTROL VAR. CONSTANT
NAN + NAN
NAN + NAN
XAN + NAN
NAN + NAN
0.11 + 0.29
CONTROL VAR. RANDOM UNIFORM
-0.01 0.30
-0.01 0.22
-0.00 = 0.51
-0.00 = 0.04
3.21 +: 2.89
CONTROL SOBEL FIITER
1,00 0.00
82197.21 1: 132718.26
GRAD
0.47
0.01 # 0.04
1761.60
10007.26
presents the results. Some metrics produce no values (nan), c.g., when correlating identical vectors, and by that identify the unfaithful explanation. Fast-GEF, and GEF consistently assign low scores to random explanations, and high scores to non-random GRAD explanations, indicating their ability to identify the control explanations. Conversely, other metrics fail at least in one random test, either showing little discrepancy betweon regular, and control variants or even giving higher scores to the control. For instance, MPRT, and RIS score random uniform explanations as good or better than regular ones.
Results. Table A.3
A.8.4 Cross-Domas Benchmarking
We benchmark various local, and global explanation methods with GEF and Fast-GEF. In Figure A.10 we exlend the results in Figure g
Extended Results.
The results presented Figure are provided Tables A.4 and A.5
A.8.5 LLM-x
In the following, we provided extended results of the LLM-x experiments.
Extended Results.
The results presented in Figure are provided in Tables A.6 and A.7
45

---

Published in Transactions on Machine Learning Research (02/2025)
(A) Derna (NedCHN)
AMNIST (LeNet)
(C) MNIST (LeNet)
Expl uplanation Distartion
49
Exp lanatiion Distartion
40
Expl tolaattioon Distortion
30
30
20
10
20
Distortion
X10*
Model Distortion
Model
MACO 258
INTG
LRP.c
SMG
GR4D
DV. 259
SHAP.G G
IX6
Figure A.10: Fast-CEF results for vision tasks. (A), (B), and (C) plot the model and explanation distortion for Derma (MedCNN), and IMNIST (TeNet), and MNTST (TeNet) along the perturbation path with Z 5 perturbation steps. The size of the scatter point represents each perturbation steps, from to 5.
Table A.4: GEF results local methods tabular tasks. Mean faithfulness scores, and standard errors are reported, with higher values indicating better quality.
TASK
ADULT
ADULT LR
AVILA (2-LAYER MLP)
COMPAS
COMPAS
[3-LAYER MLP)
(3-LAYER NLP)
LR
SMG
0.86 1: 0.00
0.69 1. 0.00
0.72 1 0.01
0.81 1: 0.01
0.73
0.01
SHAP-G
0.78 + 0.00
0.84 + 0.01
0.75 + 0.01
0.84 + 0.00
0.66 = 0.01
LOCAL NETWOOS
SAL
0.84 + 0.00
0.76 + 0.00
0.69 + 0.01
0.75 + 0.01
0.70 = 0.01
RAX
-0.00 1: 0.02
0.00 1 0.02
0.00 1: 0.02
0.02 1 0.02
0.02
0.02
LRP-C
+ 0.01
+: 0.66 0.01
LRP-s
0.79 + 0.00
0.75 0.00
0.78 0.00
0.70 0.01
IXG
0.84 1: 0.00
0.77 0.01
0.69 1 0.01
0.74 0.00
0.72 0.00
INTO
0.82 1: 0.00
0.82 4 0.00
0.69 + 0.01
0.81 4 0.00
0.80
0.00
GRAD
0.86 + 0.00
0.74 + 0.00
0.69
- 0.01
0.81 + 0.011
0.67 = 0.01
GBP
0.80 1 0.00
0.60 + 0.01
0.68 1: 0.01
0.78 1 0.01
0.70
0.01
Table A.5: Fast-CEF result local methods for vision tasks. Mean faithfulnness scores, and standard reported, with higher values indicating better quality.
TASK
DERMA MEDCNN
FNNIST LENET
IMAGENET-Ix RESMET18
MNIST LEXET
PATH MEDCUINi
SMG
0.61
0.01
0.69 1: 0.01
0.63 +: 0.01
0.73 1: 0.01
0.63 1: 0.02
SHAP-G
0.19
0.01
0.74 + 0.01
0.69 - 0.01
0.77 - 0.01
0.60 + 0.01
METHODS
SAL
0.67
0.01
0.76
0.01
0.71 = 0.01
0.74 = 0.01
0.65 0.01
LRP-E
0.70
0.35
0.73 0.01
0.66 0.01
LAP
0.74 0.01
0.73 - 0.01
0.91 0.00
0.73 = 0.01
0.71 0.01
LXG
0.73 0.01
0.72 + 0.01
0.64 - 0.01
0.73 - 0.01
0.71 + 0.01
INTG
0.19 = 0.01
0.73 + 0.01
0.78 0.01
0.77 = 0.01
0.71 0.01
GRAD
0.54 0.01
0.76 - 0.01
0.71 0.01
0.79 = 0.01
0.66 0.01
0.77 + 0.01
0.01
0.47
0.29
0.02
0.42
METHOOS
MACO-250
0.30 0.02
0.54 1 0.01
0.31 0.01
0.31 = 0.02
0.40 0.01
M4C0-109
0.24 0.01
0.52 + 0.01
0.31 + 0.01
0.41 + 0.01
0.45 0.01
F0-50
0.17 = 0.02
0.42 + 0.01
0.22 - 0.02
0.26 + 0.02
0.35 0.35 0.01
FO-250
0.01
0.35 + 0.02
0.19 0.02
0.15 = 0.02
0.31
PC-100
+ 0.02
0.54
0.26
0.36 0.02
0.5
DV-250
0.14 0.01
0.50 0.01
0.10 0.02
0.40 = 0.02
DV-100
0.43 0.02
0.49 + 0.02
0.40 + 0.02
0.43 + 0.02
0.51 + 0.01
46

---

Published in Transactions on Machine Learning Research (02/2025)
Table A.6: Fast-GEF results LLM-x, and local methods for top- tasks. Mean laithfulness scores, and standard errors reported, with higher values indicating better quality.
TASK
SMS SPARE
S8T2
BERT-TIXY FT
BERT-TINY FT
SHAP-P-6
0.62 +: 0.01
0.75 # 0.01
LOCAL NETHODS
SHAP-P-10
0.62 + 0.01
0.75 + 0.01
RAN-5
0.08 0.01
-0.08 1: 0.01
L.M-X-8
0.06 0.02
0.05 + 0.02
LIM-X-10
0.05 + 0.02
0.08 + 0.02
L-INIG-6
0.58 1: 0.01
0.77 1 0.01
1-IIIG-10
0.58 0.01
0.77 + 0.01
Table A.7: Fast-CEP results on LLM-x, and local methods for top-K tasks. Mean faithfulness scores, and standard errors are reported, with higher values indicating better quality.
TASK
SMS SPAN
SST2
LLN-X
-4.25
-3.73 + 7.72
L-INIG
195.49 + 2.91
238.15
SHAP-P
185.95 + 3.06
230.27 3.75
A.9
Notation Tables
All notations used in this paper is provided in the following.
Spaces, and Elements
x, x
The input space X C RP with a sample æ
F, 0
The model space F C RD with parameters 0 E F
y, 1
function output space y C RC with logits y E V:y = [y]]".., for C classes ye E ydc [1,C]
8,
The explanation space 8 C RV with an explanation e E
2, 9
The evaluation space Q C RM with a quality estimate g Q A set of spaces (XFF,),&& 2} where S SCR. SCR with 8 A subset spaces H S (F,8) with h E
8, 8
H.
8,
A sample, input, parameters, logit, explanation, post- perturbatio.n
Functions
f
A classifier function f: - y with [w; 0) = (we refer So as s), paramelerised by 0
d2
A local explanation function Φr
y R.V with d1 y: e, parameterised by
X
6G
A global explanation function dG F y RV with @o v; x) e, parameterised x Collectively, denoting ér., and ΦG although they formally reside in different spaces
V
An evaluation function V V x X' F y + I with Vie, x, f.v; q, parameterised by where ((); on space S
Ps
A perturbation function P:
A general discrepancy function 8 S x S + with ō(s, 8) = parameterised by B
A separate mapping function S - H mapping 8, å to a distinct space H.
De
A functional distortion D2 : S x S IR with (s 3) 6(k(s), k($))
A correlation function with p: R
47

---

Published in Transactions on Machine Learning Research (02/2025)
Constants
C
number of classes The dimension of the imput
D
W
The dimension of the parameter vector The dimension of the explanation outputs The number of perturbation steps
V
Z
K
The number of samples to approximate the Jacobian
T
The number of integral steps between two points, and
The number of models average over in GEF, and Fast-CEP
Variables
The pertarbation magnitude defined the discrepancy 6(s, 8) & between â, and The model distortion Dy across parameter- Dy(0, 0), and inpul perturbation (, 2) The explanation distortion D3 across parameter- (6 0), and input perturbation (s 2))
Dy
Ds
The implicit upper boundary value with gRo RR and {á, f} used in robustness The implicit lower boundary value with g E Rt, and & E {6,ʃ} used in sensitivity A boundary value for the perturbation magnitude, with R*
SF
ED
Gaussian noise matrix with 7, ~ -V(0, of)
The covariance scale of Gaussian distribution with of E R+ 2th perturbation The network Jacobian fixed input æ, and model f, with Jy E RVCC and elements J., 8f;
Jy
Pullback metric tensor based the elementwise Jacobian with E RVV Index perturbation steps with = [1. 2]
g
D
The model distortion at perturbation step with Dj (0,
D
The explanation distortion at perturbation step 2 with D = (1 0.) The vector of model distortion with Z sleps, d, [D] D Dj...... Df1
dR
ds
The vector of explanation distortion with Z steps, de [D), D D D
48


Reviews:
[{"review_id": "o1LoaLpP1W", "reviewer": "Reviewer_FKvG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The integration of robustness, sensitivity, and faithfulness under a single evaluative framework addresses fragmentation in the XAI literature.\n-The paper provides extensive experiments that showcase the utility of GEF and Fast-GEF as faithfulness evaluators. By testing these evaluators across diverse datasets (vision, tabular, NLP), models, and explanation methods (including LLM-as-explainer methods). Correlation of distortion vectors is simple and flexible and makes GEF potentially useful.\n-The recognition of the geometric differences between function spaces and the use of Riemannian geometry to account for these differences is innovative.\n\n\nWeaknesses, such as computational complexity due to GEF's reliance on path integrals using the pull-back metric tensor, or the underlying assumptions that perturbations of model parameters effectively simulate realistic changes in model behavior, are partially addressed in the paper. One question to ask is: how much the faithfulness evaluation depends on the perturbation path, and therefore on the particular noise schedule used to construct the distortion vector? This seems like a key choice to make beforehand. Second question is: depending on the nature of the function, e.g., implicit or non-differentiable functions, it can be impossible to compute the Jacobian (and consequently g = J^TJ). To what extend can GEF be applied to popular explanation methods that have been proved useful in the XAI literature? If I understand correctly you approximate the Jacobian numerically as infinitesimal differences between explanations. I suspect this approach to potentially yield very different results to the actual (perhaps intractable) Jacobian. Perhaps investigating how numerical approximations are consistent with the true Jacobian in (big, and highly overparameterized) differentiable models would be interesting. Last, in A.2.1 you outline how the integral (eq. 11) is computed. You say you use only 5 points, I suspect because of the computational complexity. I am worried that the noise schedule used and the small number of points fail to approximate well a smooth path.\n\nOverall, I find that the strengths outweigh the weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of explanation faithfulness. In particular, the absence of a general framework that unifies perturbation-based evaluative criterions. The paper first develops a unifying theory and then, within the theory, proposes a Generalized Explanation Faithfulness criterion that is both threshold-free and task agnostic. The proposed quality estimator is based on the correlation of distortion vectors, as collections of distortions for perturbations of increasing magnitude, to account for both robustness and sensitivity. The paper further develops GEF to account for geometric characteristics of the different functions spaces in which the model output and the explanation output functions live. They equip the model output space with use the pull-back metric tensor, rather than the Euclidean metric,  and compute distances accordingly. Finally, the paper provides extensive experiments in support of the claim that GEF, and Fast-GEF provide a sound faithfulness evaluator and test various explanation approaches such as the recently proposed LLM-as-explainer methods. Overall, the unifying theory seems coherent and helpful and the use of Riemannian geometry to account for different geometric properties of the functions spaces is ingenious. The proposed method based on correlation of distortion vectors can be helpful for the XAI community, even though, this not being my main field of expertise I cannot evaluate its novelty.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As pointed out in the Strengths And Weaknesses section, the proposed approach to the computation of Jacobians can potentially yield very different results to the actual (perhaps intractable) Jacobian. Maybe investigating how numerical approximations are consistent with the true Jacobian in (big, and highly overparameterized) differentiable models, where it can be computed, would be interesting. Second, I am worried that the noise schedule used and the small number of points can fail to approximate well a smooth path. It would be interesting to see if this is the case, perhaps ablating over the number of steps and noise magnitudes.", "broader_impact_concerns": "The manuscript already has a broader impact statement that survives well the purpose.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B5GsQ63A3y", "reviewer": "Reviewer_4myU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is clearly written and presentation is good. \n\n- The figures could have been better and the captions could have been more informative.\n\n+ The experiments are fine and the results with LLMs are also interesting, although not surprising: why should an LLM provide any better an explanation than another black-box model?\n\n+ Overall, this is a very relevant area of research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new evaluation criterion for explainable AI: \"a threshold-free metric based on principles from differential geometry\". The main argument is that XAI relies too much on task-specific evaluations. In the absence of a general theory, and considering the costs of subjective evaluations, this line of research is highly relevant.\n\nWithout ground-truth explanation labels, achieving faithfulness is arguably an impossible task. Early work in the area of neurosymbolic AI has instead focused on achieving fidelity of explanations to the trained model, rather than to the real-world problem being modelled. It could be argued that this is in practice how far one can go.\n\nNow, if explanations can guide alignment and this process can be shown to take the model closer to solving the real-world problem, essentially a proxy to the ground truth explanation labels, then this brings XAI real close to neurosymbolic AI because it requires not only explanations but an ability to intervene in the model (re-training) based on the explanations and to show that the intervention is beneficial. This as far as I am aware was first proposed in https://neurosymbolic-ai-journal.com/system/files/nai-paper-729.pdf", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The claims initially confined to the provision of a better XAI metric supported by a geometric argument, extend without much justification to: unifying criterion for robustness, sensitivity, and faithfulness and fair measurements across non-linear mappings. Some of the claims could be toned down. As far as I can tell, robustness and fairness are not really evaluated empirically in the paper.\n\nFig.2 should be explained in a lot more detail. This is crucial to justify the idea that \"diverse evaluation methods can be unified under a shared conceptual framework\", and to convince the reader that having such a shared framework is desirable.\n\nIt wasn't clear to me what we're expected to conclude from Fig.3. I get it that different geometries will affect XAI results, but just like we don't know the ground truth, the geometry isn't known, or is it? In particular, I'd like to know better what defines an \"explanation function\" and how distortions are calculated in practice.\n\nAlso, if I understood correctly, Fig2(B) is not intended to split the space equally as shown with the yellow, green and red squares, since the goal is to avoid arbitrary thresholds. The figure could have reflected this better.", "broader_impact_concerns": "+ The way that the work is presented can help guide the field towards the key methodological questions that are discussed in the paper rather than to more and more application-specific approaches and algorithms that at the end are not very relevant. \n\n- One limitation of the proposed approach is that, despite the claim of generality, it seems to work only for perturbation-based local approaches such as LIME and image-based aggregations approaches such as GradCAM. There are well-documented problems with both classes of XAI when it comes to fidelity, see e.g. https://arxiv.org/abs/1908.03020. There are other layerwise global explanation approaches that seek to extract high-fidelity logic statements from trained neural networks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5WkwoV5QFz", "reviewer": "Reviewer_H3Wy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper is of great educational value in that it systemises existing approaches for evaluating explanations without access to ground-truth explanation labels.\n2. It proposes a generalised criteria GEF which combines robustness, faithfulness and sensitivity scores into one metric.\n3. It brings up the important fact that model and explanation distortion belong to different spaces and their euclidean distance measurements cannot be directly compared.\n4. It tries to question existing assumptions in the field about how distortions should increase monotonically with perturbation amount.\n\n**Weaknesses**\n1. GEF is expensive, and it is unclear if Fast-GEF does not suffer from the same issue shown in Figure 3. While emphasis is made on the importance of the differential geometry intuition, it does not seem Fast-GEF uses it, so it is unclear how important this perspective is.\n\n2.  I am not so sure I buy the evidence for how model distortions and perturbation don't align in practice. It's unclear what each point in the scatter plot in Fig A.2 A-C represents, making it difficult to assess whether the claim is justified. Are these the distortions on individual samples? if so, its still possible that for a fixed sample, increased perturbation leads to increased distortion.  Similarly, for layer-by-layer randomisation, it's unclear how many seeds the result is computed across, and whether the deviations are high just because of that. What is being varied in the randomisation? How can we be sure the setup is faithful to existing work? Mostly, I see an increasing trend in distortion as the perturbation level increases, with variance across samples as expected. I am not sure how this is evidence for the monotonicity assumption being violated.\n\n3. The GEF metrics by combining three desiderata for explanations (faithfulness, sensitivity, robustness) leads to lesser insight about which of these three desiderata an explanation method underperforms in which can be valuable to both researchers developing new methods and practitioners using them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper first systemises existing explainability evaluations, focusing on the three criterion of robustness, sensitivity and faithfulness. It proposes unifying them by viewing them as the correlation between model distortion and explanation distortion under perturbations. It highlights that directly comparing euclidean-distance measurements of model and explanation distortion is not enough, as they belong to different spaces. The paper proposes GEF, which fixes this by using a jacobian-based pullback operator, though computing this pullback is expensive so it also proposes Fast-GEF for practitioners which still uses euclidean distance. Using GEF, Fast-GEF, the paper evaluates local, global and llm-explanation methods across nlp, vision and tabular tasks. The high-level takeaways are that LLM-explanations are near-random, and local explanation methods are more faithful than global ones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Can you plot Figure 3 but using the GEF, Fast-GEF metrics to see whether they lead to more coherent scatter plots between model and explanation distortion.\n\n2. The code url in the abstract is broken. Please fix it.\n\n3. Can more recent interpretability techniques on LLMs such as attention head ablations, causal patching, sparse auto-encoders etc. be added to the evaluations? This would make the methodology immediately interesting to the audience.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the problem of measuring explanation faithfulness, i.e. whether an interpretability method's explanation for a model's prediction is actually reliable. The paper makes the case that existing evaluation procedures are scattered and insufficient, and therefore proposes a more principled and comprehensive evaluation procedure. A large-ish scale evaluation (across many modalities/settings) of existing explanation faithfulness methods is undertaken using the proposed evaluation, and interesting insights are shared. After some back-and-forth to improve the paper, all reviewers agreed the paper should be accepted.
Decision: Accept (poster)