Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance
Authors: Jonathan Crabbé
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: interpretability
explainability
robustness
invariance
equivariance
geometric deep learning
TL;DR: We assess the robustness of various interpretability methods by measuring how their explanations change when applying symmetries of the model to the input features.

Abstract: Interpretability methods are valuable only if their explanations faithfully describe the explained model. In this work, we consider neural networks whose predictions are invariant under a specific symmetry group. This includes popular architectures, ranging from convolutional to graph neural networks. Any explanation that faithfully explains this type of model needs to be in agreement with this invariance property. We formalize this intuition through the notion of explanation invariance and equivariance by leveraging the formalism from geometric deep learning. Through this rigorous formalism, we derive (1) two metrics to measure the robustness of any interpretability method with respect to the model symmetry group; (2) theoretical robustness guarantees for some popular interpretability methods and (3) a systematic approach to increase the invariance of any interpretability method with respect to a symmetry group. By empirically measuring our metrics for explanations of models associated with various modalities and symmetry groups, we derive a set of 5 guidelines to allow users and developers of interpretability methods to produce robust explanations.

Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance
Jonathan Crabbé DAMTP University of Cambridge jc21330cam. ac uk
Mihaela van der Schaar DAMTP University of Cambridge
mv472@cam. ac. uk
Abstract
Interpretability methods are valuable only if their explanations faithfully describe the explained model. In this work, we consider neural networks whose predictions are invariant under specific symmetry group. This includes popular architectures, ranging from convolutional to graph neural networks. Any explanation that faith- fully explains this type of model needs to be in agreement with this invariance property. We formalize this intuition through the notion of explanation invari- ance and equivariance by leveraging the formalism from geometric deep learning. Through this rigorous formalism, we derive (1) two metrics to measure the robust- ness of any interpretability method with respect to the model symmetry group; (2) theoretical robustness guarantees for some popular interpretability methods and (3) a systematic approach to increase the invariance of any interpretability method with respect to a symmetry group. By empirically measuring our metrics for ex- planations of models associated with various modalities and symmetry groups, we derive set of 5 guidelines to allow users and developers of interpretability methods to produce robust explanations.
Introduction
With their increasing success in various tasks, such computer vision [1], natural language process- ing [2] and scientific discovery [3]. deep neural networks (DNNs) have become widespread. State of the art DNNs typically contain millions to billions parameters and, hence, is unrealistic for human users to precisely understand how these models issue predictions. This opacity increases the difficulty to anticipate how models will perform when deployed [4]; distil knowledge from the model [5] and gain the trust of stakeholders in high-stakes domains [6, 7]. To address these shortcomings, the field of interpretable machine learning has received increasing interest [8, 9J. There exists mainly 2 approaches to increase model interpretability [10, 11]. (1) Restrict the model's architecture to intrinsically interpretable architectures. A notorious example is given by self-explaining models, such as attention models explaining their predictions by highlighting features they pay attention to [12] and prototype-bas based models motivating their predictions by highlighting related examples from their training set [13-15]. (2) Use post-hoc interpretability methods in plug-in in fashion after training the model. The advantage of this approach is that it requires no assumption on the model that need to explain. In this work, focus several post-hoc methods: feature importance methods (also known as feature attribution or saliency) that highlight features the model is sensitive to [16-20]; example importance methods that identify influential training examples [21-23] and concept-based explanations that exhibit how classifiers relate classes to human friendly concepts [24, 25].
With the multiplication of interpretability methods, it has become necessary to evaluate the quality of their explanations [26]. This stems from the fact that interpretability methods need to faithfully describe the model in order to provide actionable insights. Existing approaches to evaluate the quality
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of interpretability methods fall in 2 categories [5, 27]. (1) Human-centree evaluation investigate how the explanations help humans (experts or not) anticipate the model's predictions [28] and whether the model's explanations are in agreement with some notion of ground-truth [29-31]. (2) Functionality-gron grouunde evaluation measure the explanation quality based on some desirable properties and do not require humans to be involved. Most of the existing work in this category measure the robustness of interpretability methods with respect to transformations of the model input that should not impact the explanation [32]. Since our work falls in this category, let us now summarize the relevant literature.
Related Works. [33] showed that feature importance methods are sensitive to constant shifts in the model's imput. This unexpected because these constant shifts do not contribute to the model's 's prediction. Building on this idea of invariance of the explanations with respect to input shifts, [34-36] propose sensitivity metric to measure the robustness of feature importance methods based on their stability with respect to small perturbations of the model input. By optimizing small adversarial perturbations, [37-39] show that imperceptible changes in the input can modify the feature importance arbitrarily by approximatively keeping the model prediction constant. This shows that many interpretability methods, as neural networks, are sensitive to adversarial perturbations. Subsequent works have addressed this pathologic behaviour by fixing the model training dynamic. In particular, they showed that penalizing large eigenvalues of the training loss Hessian with respect to the inputs make the interpretations of this model more robust with respect to adversarial attacks [40, 41]. To the best of our knowledge, the only work that discusses the behaviour of explanations under more general transformations of the input data is [42]. However, the work's ' focus is more on model regularization rather than on the evaluation of post-hoc interpretability robustness.
Motivations. In reviewing the above literature, we notice gaps. (1) The existing studies mostly focus on evaluating feature importance methods. In spite of the predominance of feature importance the literature [43], we note that other types of interpretability methods exist and deserve to be analyzed. (2) The existing studies mostly focus on images. While computer vision is undoubtedly an interesting application of DNNs, it would be interesting to extend the analysis to other modalities, such times series and graph data [44]. (3) The existing studies mostly focus on simple transformation of the model imput, such small shifts. This is motivated by the fact that the predictions of DNNs mostly invariant under these transformations. Again, this is another direction that could be explored more thoroughly numerous DNNs are also invariant more complex transformation of their input data. For instance, graph neural networks are invariant to permutations of the node ordering in their input graph [45]. Our work bridges these gaps in the interpretability robustness literature.
(1) Invariant Model
(2) Invariant Explanation
(3) Equivariant Explanation
Top Examples
Jepataie
Input at
Abmornal Beat
Input z
Input
Explanation el (æ)
p|g] Symmetry [
((g) Symmetry
Model
plg] Symmetry
Explainer
Explainer
Irpurtine
Prediction
fax) = ((lg)a)
Explanation
New Imput
New Explanation
New Inpat pfg]
New Input plg] m
<(plgiz) z)
e|g a
<(elg)a) - (
Figure 1: Illustration of model invariance and explanation invariance/equivariace with the simple
case of an electrocardiogram (ECG) signal. In this case, the heartbeat described by the ECG remains the same if we apply any translation symmetry with periodic boundary conditions. (1) A model is invariant under the symmetry if the model's prediction are not affected by the symmetry we apply to the signal. In this case, the model identifies an abnormal heartbeat before and after applying a translation. Any explanation that faithfully describes the model should reflect this symmetry. (2) For some explanations, the right behaviour is invariance as well. For instance, the most influential examples for the prediction should be the same for the original and the transformed signal, since the model makes no difference between the two signals. (3) For other type of explanations, the right behaviour is equivariance. For instance, the most important part of the signal for the prediction should be the same for the original and the transformed signal, since the model makes no difference between the two signals. Hence, the saliency map undergoes the same translation the signal.
2

---

Predicted Class: T-shirt/top
Original Saliency Map
Predicted Class: T-shirt/top
Transformed Saliency Map
Figure 2: Examples of non-robust explanations obtained with Gradient Shap on the FashionMNIST dataset. From left to right: the original image for which the invariant model predicts t-shirt with given probability, the Gradient Shap saliency map explain the model's prediction for this image, the transformed image for which the model predicts t-shirt with the exact same probability and the Gradient Shap saliency map for this transformed image. Clearly, the image transformation changes the explanation when should not.
Contributions. We propose new framework evaluate the robustness of interpretability methods. We consider setting where the model we wish to interpret is invariant with respect to a group G of symmetry acting on the model input. Any interpretability method that faithfully describes this model should have explanations that are conserved by this group of symmetry G. We illustrate this reasoning in Figure with the simple group G of time translations acting on the input signal. We show examples of interpretability methods failing to conserve the model's 's symmetries, hence leading to inconsistent explanations, in Figure 2 and Appendix With this new framework, we bring several contributions. (1) Rigorous Interpretability Robustness. We define interpretabilityy robustness with respect to group of symmetry through explanation invariance and equivariance. demonstrate in Section 2.2 that our general definitions
In agreement with our motivations, cover different type of interpretability methods, modalities and transformations the input data, (2) Evaluation of Interpretability Methods. Not all interpretability methods equal with respect to our notion robustness. In Section 2.3, we show that some popular interpretability methods naturally endowed with theoretical robustness guarantees. Further, introduce 2 metrics, the invariance and cquivariance scores, to empirically evaluate this robustness. In Section 3.1, we use these metries evaluate the robustness of 3 types of interpretability methods with 5 different model types corresponding to 4 different modalities and symmetry groups. Our empirical results support our theoretical analysis. (3) Insights to Improve Robustness. By combining
theoretical and empirical analysis, we derive set of actionable guidelines to ensure that interpretability methods are used in a way that guarantees robustness with respect to the symmetry group G. In particular, we show in Sections 2.3 and 3.2 that we can improve the invariance score of any interpretabilityy method by aggregating explanations over various symmetries. We summarize the guidelines with flowchart in Figure 6 from Appendix A, that helps users to obtain robust model interpretations.
2 Interpretability Robustness
In this section, we formalize the notion of interpretability robustness through explanation invariance and equivariance. We start with reminder of some useful definitions from geometric deep learning. We then define two metrics to measure the invariance and equivariance interpretability methods. We leverage this formalism derive some theoretical robustness guarantees for popular interpretability methods. Finally, we describe a rigorous approach to improve the invariance of any interpretability method.
2.1 Useful Notions of Geometric Deep Learning
Some basic concepts of group theory are required for our definition of interpretability robustness. To that aim, leverage the formalism of Geometric Deep Learning. Please refer to [46] for more details. To rigorously define explanation equivariance and invariance, we need some form of structure in the data we manipulating. This precludes tabular data but includes graph, time series and image data. In this setting, the data defined on a finite domain set Ω (e.g. grid Ω = Z, x Zn for n n images). On this domain, the data is represented by signals z E X2, (), mapping each point E Ω of the domain to channel vector æ(u) C dd
We note that do N+, corresponds to the

---

number of channels of the signal (e.g. de = 3 for RGB images). The set of signals has the structure of vector space since 21, 22 E (2 + X1 - + X2 I2 E XS2, C) for all X A2 E R.
Symmetries. Informally, symmetries are transformations of the data that leave the information content unchanged (e.g. moving an image one pixel to the right). More formally, symmetries correspond to a set G endowed with a composition operation : G + G. Clearly, this set G includes an identity transformation id that leaves the data untouched. Similarly, if transformation g E G preserves the information, then it could be undone by an inverse transformation g E G such that g-oog id. Those properties give G, o the structure of a group. In this paper, we assume that the symmetry group has finite number of elements.
Group Representation. We have yet to formalize how the above symmetries transform the data. To that aim, we need to link the symmetry group 9 with the signal vector space (S2, C). This connection is achieved by choosing a group representation p 9 Aut[XX(S, C)) that maps each symmetry g E 9 to an automorphism pig] E Aut (St, C)). Formally, the automorphisms ( ()) are defined as bijective linear transformations mapping ( C)) onto itself. In practice, each automorphism plg] is represented by an invertible matrix acting on the vector space (S2, C). For instance, an image translation g can be represented by a permutation matrix p(g]. To qualify as group representation, the map p needs to be compatible with the group composition: plg2 o 91] = pl92) This property guarantees that the composition of two symmetries can be implemented as the multiplication between two matrices.
Invariance. We first consider the case of deep neural network f C.) + V where the output (a) E y is a vector with no underlying structure (e.g. class probabilities for a classifier.. In this case, expect the model's prediction to be unchanged when applying a symmetry g & to the input signal x E (S2 C). For instance, the probability of observing a cat on an image should not change if we move the cat by one pixel to the right. This intuition is formalized by defining the G-invariance property for the model f: [plg]a=) = f(z) for all E G E C),
Equivariance. We now turn to the case of deep neural networks f (2, + V(L'' where the output f(x) E ( C') is also a signal (e.g. segmentation masks for an object detector). We note that the domain O' and the channel space C' are not necessarily the same Q and C. When applying a transformation g E 9 to the input signal x E (2, C), it is legitimate to expect the output signal (a:) to follow similar transformation. For instance, the segmentation of cat on an image should move by one pixel to the right if we move the cat by one pixel to the right. This intuition is formalized by defining the G-equivariance property for the model f: f(lgg] = p' (()) Again, the representation o' : G + (St is not necessarily the same as the representation p since the signal spaces X (2, C) and (I',C") might have different dimensions.
2.2 Explanation Invariance and Equivariance
We will now restrict to models that are G-invariant?" It is legitimate to expect similar invariance properties for the explanations associated to this model. We shall now formalize this idea for generic explanations. We assume that explanations are functions of the form e X(( + E, where C Rd is an explanation space with dæ E N+ dimensions
Invariance and Equivariance. The invariance and equivariance of the explanation e with respect to symmetries G are defined as in the previous section. In this way, say that the explanation is G-invariant (() = e(z) and G-equivariant if <(p[g)z) = p' [gje(z) for all g E G, E X22 C). There is no reason to expect these equalities to hold exactly priori. This motivates the introduction of two metrics that measure the violation of explanation invariance and equivariance by an interpretability method.
Definition 2.1 (Robustness Metrics). Let ; XQQ,C) + y' be neural network that is invariant with respect to the symmetry group G and e: x' + E be an explanation for f. We assume that G acts on (l, C) via the representation p: + AutiX C).. We measure the invariance of e with
NNoe that groups also satisfy associativity: 91 (g2 ga) (0r o 92) o ga for all Ø1, 92, 9s E G.
We also restrict to supervised models, since only early works exist to interpret unsupervised models [47, 48]. Note that the explanation also depends on the model f. Since the model is fixed, make this implicit.
4

---

respect to G for some x E X(2,C) C) with the metric
Invg(e, =
Σ gEG
se le (plg)z),
(I)
[
where 88 : E² + IR is a similarity score on the explanation space E. We use the cos-similarity s b) = / -|21) for real-valued explanations a, b E IRd and the accuracy score sg
de D/ L(a; = 6;) for categorical explanations a, E z where is the indicator function and K N is the number of categories. If assume that G acts on & via the representation p' P - Aut [)) we measure the equivariance of e with respect to & for some x E (22, with
the metric
Squivy(e,,n =
Σ se le (plgl=), . d'(gle(z)).
(2)
9
gEg
A score x)) = or Equiv (equivgee,eeee x) = indicates that the explanation method is G-invariant or equivariant for the example x E C),
Remark 2.2. The metries Invg and Equivg might be prohibitively expensive to evaluate whenever the size []] of the symmetry group G is too big. Note that this is typically the case in our experiments Monte Carlo
as we consider large permutation groups of order G >>>>> 10 In this case, estimators for both metrics by uniformly sampling G ~ U(G) and averaging over a number sample Nsamp G We study the convergence of those Monte Carlo estimators in Appendix E.
The above approach to measure the robustness of interpretability method applies to wide variety of settings. To clarify this, we explain how to adapt the above formalism popular types of interpretability methods: feature importance, example importance and concept-hased explanations.
Feature Importance. Feature importance explanations associate saliency map e(x) E X' C) to each example æ E (N, C) for the model's prediction fxx). In this case, we note that the explanation space corresponds to the model's 's input space & = (S2, C). since the method assigns an importance score to each individual feature. If we apply symmetry to the input, we expect the same symmetry to be applied to the saliency map, as illustrated by the example from Figure 1. Hence, the most relevant metric to record here is the explanation equivariance Equiv Since the input space and the explanation space are identical & X22, C), we work with identical representations p' p. We note that this metric generalizes the self-consistency score introduced by [42] beyond affine transformations.
Example Importance. Example importance explanations associate an importance vector e(x) E Rr-rs to each example x (I( for the model's prediction f(x). Note that Nirain N+ is typically the model's training set size, so that each component of e(x) corresponds to the importance of a training example. If we apply symmetry the input, we expect the relevance of training examples to be conserved, as illustrated by the example from Figure 1. Hence. the most relevant metric to record here is the invariance Inv..
Concept- Loncept-Based Explanations. Concept-based explanations associate a binary concept presence vector e(x) E 10 to each example E X C) for the model's prediction f(x). Note that CE is the number of concepts one considers, so that each component of e(x) corresponds to the presence/absence of concept. If we apply a symmetry the input, there no reason for a concept to appear/vanish, since the information content of the input is untouched by the symmetry. Hence, the most relevant metric record here is again the invariance Invg.
2.3 Theoretical Analysis
Let us now provide a theoretical analysis of robustness in a setting where the model f is G-invariant. We first show that many popular interpretability methods naturally offer some robustness guarantee if we make some assumptions. For methods that are not invariant when they should, we propose an approach to enforce G-invariance.
Robustness Guarantees. Table 1, we summarize the theoretical robustness guarantees that we derive for popular interpretability methods. All of these guarantees are formally stated and proven in Appendix When comes to feature importance methods, there are mainly two assumptions that necessary to guarantee cquivariance. (1) The first assumption restricts the type of baseline input
5

---

Table 1: Theoretical robustmess guarantees that derive for explanations of invariant models. We split the interpretability methods according to their type and according to model information they rely (model gradients, perturbations, loss representations). We consider 3 levels of guarantees: indicates unconditional guarantee, ~ conditional guarantee and X no guarantee.
Type
Computation
Example
Invariant
Equivariant
Details
Feature
Grad. Vxf(()
[18]
Prop. D.6
Importance
Pert. f(x dz)
[49]
Prop. D.8
Example
Loss [.)
[21]
Prup. D.9
Importance
Rep. h(z)
[23]
Prop. D.12
Concept-Band Based
Rep. h(z)
[24]
Prop. D.14
((2, C) which the feature importance methods rely. Typically, these baselines signals used to replace ablated features from the original signal E X(2,, C) (i.e. remove a feature x; by replacing it by T. ). order to guarantee equivariance, we require this baseline signal be invariant to the action of each symmetry E G pig] ε. (2) The second assumption restricts the type of representation p that can be used to describe the action of the symmetry group on the signals. In order to guarantee equivariance, we require this representation to be permutation representation. represented by permutation matrix pig]
which means that the action of each symmetry g acting on the signal space C),
When it comes example importance methods, the assumptions depend how the importance scores are obtained. the importance scores are computed from the model's loss, then the invariance of the explanation immediatly follows from the model's invariance. If the importance scores are computed from the model's internal representations h X (S2, C) + Rdop, then the invariance of the explanation can only be guaranteed if the representation map is itself invariant action of each symmetry: [ = h(r). Similarly, concept-bass explanations are also computed from the model's 's representations h. Again, the invariance of these explanations can only be guaranteed if the representation map h is itself invariant.
construct auxiliary explanation Ginv built upon e that is G-invariant. This permits to improve the robustness of any interpretability method that has invariance guarantee. The idea is simply to aggregate the explanation over several symmetries.
Enforcing Invariance. If the explanation is not G-invariant when it should,
Proposition 2.3. [Enforce Invariance] Consider a neural network f XX(S,,) - y that is invariant with respect the symmetry group G and e (S2,C) - E be explanation for We assume that Aut[AA(S C)). We define the auxiliary explanation
9 acts on (9, C) via the representation Cinv X(2, + E as
Ciny (x)
Σ c(p(g)=)
9
9Eg
for all x E X), The auxiliary explanation einy is invariant under the symmetry group G.
Proof. Please refer to Appendix D.
Remark 2.4. Once again, a Monte Carlo estimation for einv might be required for groups with many elements. This produces explanations that are approximatively invariant.
3 Experiments
In this section, we use our interpretability robustness metrics to draw some insights with real-world models and datasets. We first evaluate the G-invariance and equivariance popular interpretability methods used on top of G-invariant models. With this analysis, we identify interpretability methods that are not robust. We then show that the robustness of these interpretability methods can largely be improved by using their auxiliary version defined in Proposition 2.3. Finally, we study how the G-invariance and equivariance of interpretability methods varies when we decrease the invariance of the underlying model. From these experiments, derive 5 guidelines to ensure that interpretability

---

methods are robust with respect to symmetries from g. We summarize these guidelines with flowchart Figure from Appendix A.
The datasets used in our experiment are presented in Table 2. We explore various modalities and symmetry groups throughout the section, descrihed in Table 3. For each dataset, we fit and study classifier from the literature designed to be invariant with respect to the underlying symmetry group. For each model, we evaluate the robustness of various feature importance, example importance and concept-based explanations. More details on the experiments are available in Appendix F. We also include comparison between our robustness metrics and the sensitivity metrie Appendix G. The code and instructions to replicate all the results reported below are available in the public repositories https: /gtthug. com/ wwwattonnaaaaaaaaooa Row.bbbee//obuuttAt.. and vanders chaaraab/Robuueeeeeaeeea
com/
Table Different datasets used in the experiments.
Dataset
# Classes
Modality
Symmetry Group
Model
Eleetrocandiograns (50, 51]
2
Time Series
Cyclic Trunslations Z/TZ
All-CNN [52]
Mutagenicity [53-55]
2
Graphs
Node Permutations Sv,
GraphCony GNN [56]
ModeINet40 [57-59]
40
3D Point Clouds
Point Permutations Sxz,
Deep Set [58]
IMDb [60]
2
Text
Token Permutation Sy
Bag-of-words MLP
MNSH [61]
10
Images
Cyclic Translations (2/102)²
All-CNN [52]
CIFARI00 [62]
100
Images
Dihedral Group LDs
B(2)-WidcReseet [63, 64]
STL10 [65]
10
Images
Dihedral Group IDs
(2)-WiidRResNe(66.3 64]
Table 3; Various symmetry groups used in the experiments.
Symmetry Group
Acting
Description
Translation 7/NZ
Time series, Images
Shifts signals in time and images borizontally & vertically. Changes the vedering of nodes points lokens festure matrices. Rotate / reflects the images though angles 45" 10P, 135", 18U", 225" 315
Permutation Sy
Graph nodes, Points clouds, Tokens Images
Dihedral D,
3.1 Evaluating Interpretability Methods
The purpose of this experiment is to measure the robustness of various interpretability methods. Since we manipulate models that are invariant with respect to a group G of symmetry, we expect feature importance methods to be G-equivariant (Equiv [e, = 1 for all æ E ( C)). Similarly, we expect example and concept- based methods to be G-invariant (Invgle, æ] = for all C)). We shall now verify this empirically.
Motivation.
Methodology. To measure the robustness of interpretability methods empirically, we use a set Diest of Nust examples (Niest = 433 for Mutagenicity, Nust = 1, 000 for ModelNet40 and Electrocardiograms (ECG) and Niest 500 in the other cases). For each interpretability method e, we evaluate the appropriate robustness metric for each test example E Drest For Mutagenicity and ModelNet40, the large order [G] makes the exact evaluation of the metric unrealistic. We therefore use Monte Carlo approximation with Naump 50. As demonstrated in Appendix E, the Monte Carlo estimators have already converged with this sample size. In all the other cases, these metrics are evaluated exactly since G has tractable order (G). Since the E(2)-WideResh) and STL10 are only approximativerly invariant with respect to Dg. we defer their discussion to Section 3.3. We note that some interpretability methods cannot be used in some settings. Whenever this is the case, we simply omit the interpretability method. Please refer to Appendix F for more details.
for CIFAR 000
Analysis. We report the robustness score for each metric and each dataset on the test set Drest in Figures 3(a) to 3(c). We immediately notice that not all the interpretability methods are robust. We provide some real examples of non-robust explanations in Appendix in order to visualize the failure modes. When looking at feature importance, we observe that equivariance is not guaranteed by methods that rely on baseline that are not invariant. For instance, Gradient Shap and Feature Permutation rely on random bascline, which has reason to be G-invariant. We conclude that the
7

---

0.75
Expleration Equirattaece
0.53
Emplamation Inventannc
0.2
0.25
0.09
0.09
Explanation
Explanation
SimplEx
-0.5
Imagrelted
-0.5
SmalEx-tpul
Gradient
5im
Peature Nemmucaiion
3im Equx
Feature Meetion
Tracin
-1.00
Fealure Ocdludin
-1.00
Infureoa Funchim
(004)) STLLO!")
80001
(a) Feature Importance
(b) Example Importance
0.75
Explassioo Inceranee
Dataset
0.25
CIFARI0O
Score
0.94
STL10
Property
Model Invariance
-0.5
0.92
Gradient Equiveriance
Explanation
0.90
-0.55
0.88
-1.00
100
150
Epach
(c) Concept Importance
(d) Training Dynamics E(n)- Wind-WiieResNett
Figure 3: Explanation robustness of interpretability methods for invariant models. The interpretability methods are grouped by type. Each box-plot is produced by evaluating the robustness metrics Inve or Equivg across several test samples x E DDest-- The asterisk (*) indicates dataset where the model only approximatively invariant. Those models are discussed in Section 3.3. For all other models, any value below for the metrics is unexpected, as the model is G-invariant.
invariance of the baseline 전 is crucial to guarantee the robustness of feature importance methods. When it comes to example importance, we note that loss-based methods consistently invariant, which is in agreement with Proposition D.9. Representation-be based and concept-based methods, on the other hand, are invariant only if used with invariant layers of the model. This shows that the choice of what we call the representation space matters for these methods. We derive set of guidelines from these observations.
Guideline 1. Feature importance methods should be used with group invariant baseline signal (plg]æ = & for all E G) to guarantee explanation equivariance. Only methods that conserve the invariance of the baseline can guarantee equivariance.
Guideline Loss- bass- example importance methods guarantee explanation invariance, unlike representation- based methods. When using the latter, only invariant layers guarantee explanation invariance.
Guideline 3. To guarantee invariance of concept-hased explanations, concept classifiers should be used on invariant layers of the model.
3.2 Improving Robustness
Motivation. In the previous experiment, we noticed that not all the interpretability methods are G-invariant when they should. Consider, for instance, concept-based methods used on equivariant layers. The lack of invariance for these methods implies that they rely concept classifiers that

---

1.00
Explanaiion alion Inveriatce
0.9
Explonatton Invoriance
0.7
0.05
CAV Equiv
0.80
CAV-Eqully
CAR-Fquil
Naw
Myw
(a) ECG
(b) FashionMNIST
Figure 4: Explanation invariance can be increased according to Proposition 2.3. This plot shows the score averaged on a test set Pest together with 95% confidence interval.
not G-invariant. This behaviour is undesirable for two reasons: (1) since any symmetry G preserve the information of a signal æ E (Q,, the signal p[g]æ should contain the same concepts implicitly encodes these symmetries through equivariance of
and (2) the layer that the output representations. Hence, concept classifiers that are not G-invariant fail to generalize by ignoring the symmetries encoded in the structure of the model's representation space. Fortunately, Proposition 2.3 gives us prescription to obtain explanations (here concept classifiers) that are more robust with respect the model's symmetries. We shall now illustrate how this prescription improves the robustness of concept-based methods.
Methodology. In this experiment, restrict analysis to the ECG and FashionMNIST datasets. For each test signal, we sample Nuv symmetries G, E G, E ZN... without replace- ment. As prescribed by Proposition 2.3, we then compute the auxiliary explanation Civ NEw ENar e(p for each concept importance method.
Analysis. We report the average invariance score E (Xxx((D),.))) (nvg X) for several values of Nine in Figure 4. As
see, the invariance of the explanation grows monotonically with the number of samples Niv to achieve perfect invariance for Niw = | Interestingly, the explanation invariance increases more quickly for CAR. This suggests that enforcing explanation invariance is less expensive for certain interpretability methods and motivates the below guideline.
Guideline 4. Any interpretability method can be made invariant through Proposition 2.3. In doing so, one should increase the number of samples Niw until the desired invariance is achieved. In this way. the method is made robust without increasing the number of calls more than necessary. Note that it only makes sense to enforce invariance of the interpretabillty method if the explained model is itself invariant.
3.3 Relaxing Invariance
Motivation. In practice, models are not always perfectly invariant. A first example given by the CIFAR100 and STL10 WideResNet that has strong bias towards being Dy-invariant, altough it can break this invariance at training time (see Appendix H.3 [64]). Another popular example a CNN that flattens the output of convolutional layers, which violates translation invariance [52, 66, 67]. This motivates the study of interpretability methods robustness when models are not perfectly invariant.
Methodology. This experiment studies the two aforementioned settings. First, we replicate the experiment from Section 3.1 with the CIFAR 100 and STL10 WideResNeet Second, we consider CNNs that flatten their last convolutional layer with the ECG and FashionMNIST datascts. In this case, we introduce 2 variants of the AlI-CNN where the global pooling is replaced by a flatten operation: Augmented- CNN trained by augmenting the training set Dtrain with random translations and a Standard- CNN trained without augmentation. We measure the invariance/equivariance of the interpretability methods for each model.
Analysis. The results for the WedeResNets are reported in Figures 3(a) to 3(c). We see that the robustness of various interpretability methods substantially drops with the model invariance. This particularly noticeable for feature importance methods. To illustrate this phenomenon, we plot in Figure 3(d) the evolution during training of the model's prediction f() G-invariance and the

---

G-equivariance of its gradient Vxf(x), on which the attribution methods rely. As we can see, the model remains almost invariant during training. while the gradients equivariance destroyed. Similar observations can be made with the CNNs from Figure 5. In spite of the Augmented-Chn CNN being almost invariant, we notice that the symmetry breaks significantly for feature importance methods. These results suggest that the robustness of interpretability methods can be (but is not necessarily) fragile if model invariance is relaxed, even slightly. This motivates our last guideline, which safeguards against erroneous interpretations of our robustness metrics.
Guideline 5. One should not assume a linear relationship between model invariance and explanation invariance/equivarianee partieular, the robustness an interpretability method for an invariant model does not imply that this method is robust for an approximatively invariant model.
Festure importance Equiverlance
Concept Importance Invarfance
0.8
0.0
0.0
Example importance
0.6
0.6
0.6
0.4
0.0
Feature Importance Equivariance
Invartance
Fa. MNISS
Concept Importance Invariance
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.4
Example
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Model Invariance
Model Invariance
Model Invariance
Type
Explanation
Permutation Gradient Shap
Influence Runctions
AlI-CNN
Deeplift
Rep. SimiliaLinn
CAR-Lin1
gienttd CNN
Feature Ablation
Integrated Gradients
SimplEx-Lin1
CAV-Lin1
Standard-CNN
Feeture Occlusion
Figure 5: Effect of relaxing the model invariance on interpretability methods invarianceeequiivriaeee.. The interpretability methods are grouped by type in each column. The error bars represent a 95% confidence interval around the mean for Inv and Equiv. Linl is to the output of the first dense layer of the CNN, which corresponds the invariant layer used in Section 3.
4 Discussion
Building on recent developments in geometric deep learning, we introduced two metrics (explanation invariance and equivariance) assess the faithfulness of model explanations with respect to model symmetries. In our experiments, we considered a wide range of models whose predictions are invariant with respect to transformations of their input data. By analyzing feature importance, example importance and concept-basee based explanations of these models, we observed that many of these explanations are not invariant/equivariant to these transformations when they should. This led to establish a set of guidelines in Appendix A to help practitioners choose interpretability methods that are consistent with their model symmetries.
Beyond actionable insights, we believe that our work opens up interesting avenues for future research. An important one emerged by studying the equivariance of saliency maps with respect models that are approximately invariant. This analysis showed that state-of-the high equivariance score when the model's invariance is slightly relaxed. This important observation could be the seed of future developments of robust feature importance methods.
saliency methods fail to keep
10

---

Acknowledgements
The authors are grateful to the anonymous NeurIPS reviewers for their useful comments on an earlier version of the manuscript. Jonathan Crabbé is funded by Aviva and Mihaela van der Schaar by the Office of Naval Research (ONR). This work was supported by Azure sponsorship credits granted by Microsoft's ' Al for Good Research Lab.
References
[1] Saining Xic, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[3] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588, 2021.
[4] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretabilityy is both important and slippery. Queue, 16(3):31-57, 2018.
[5] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: / 702.08608 2017.
[6] Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, 15(141):20170377 2018.
[7] Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. What do we want from explainable artificial intelligence (xai)?-a stakeholder perspective on xai and a conceptual model guiding interdisciplinary xai 2021.
research. Artificial Intelligence,
[8] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEFE Access, 6:52138-52160, 2018.
[9] Alejandro Barredo Arrieta, Natalia Diaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-L-opez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, tax- onomies, opportunities and challenges toward responsible ai. Inf. Fusion, 58(C):22- jun 2020.
[10] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-211 2019.
[11] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: Survey on Interpreting the Inner Structures of Deep Neural Networks. 2022.
[12] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart, Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29, 2016.
[13] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. Advances neural information processing systems, 27, 2014.

---

[14] Chaofan Chen, Oscar Li. Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[15] Oscar Li. Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case- case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAl Conference on Artificial Intelligence, volume 32, 2018.
[16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD "16, page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery.
[17] Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[18] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 2319-3322, JMLR.org, 2017.
[19] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference machine learning, pages 3 3145-3153, PMLR, 2017.
[20] Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2166-21777 PMLR, 18-24 Jul 2021.
[21] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1885-1894. PMLR, 2017.
[22] Garima, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[23] Jonathan Crabbe, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining latent representations with corpus of examples. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2214-12166. Curran Associates, Inc., 2021.
[24] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677. PMIR, 2018.
[25] Jonathan Crabbé and Mihacla van der Schaar. Concept activation regions: A generalized framework for concept- based explanations. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2590-2607. Curran Associates, Inc., 2022.
[26] Leif Hancox-Li. Robustness in machine learning explanations: Does it matter? In Proceedings of 2020 Conference on Fairness, Accountability, and Transparency. FAT* '20, page 640-647, New York, NY, USA, 2020. Association for Computing Machinery.
[27] Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. Evaluating the quality of machine learning explanations: A survey methods and metrics. Electronics, 10(5):593, 2021.
12

---

[28]
Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman, and Finale Doshi-Velez. Human evaluation of models built for interpretability. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 59-67, 2019.
[29] Yuansheng Xie, Soroush Vosoughi, and Saeed Hassanpour. Interpretation quality score for measuring the quality of interpretability methods. arXiv preprint vrXiv:2205.1225:: 2022.
[30] Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Ng. et al. Bench- marking saliency methods for chest x-ray interpretation. Nature Machine Intelligence, pages 1-12, 2022.
[31] Jonathan Crabbé, Alicia Curth, Ioana Bica, and Mihaela van der Schaar. Benchmarking heterogeneous treatment effect models through the lens of interpretability. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2295-12309. Curran Associates, Inc., 2022.
[32] Ian E. Nielsen, Dimah Dera, Ghulam Rasool, Ravi P. Ramachandran, and Nidhal Carla Bouay- naya. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine, (94))33848 2022.
[33] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[34] David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv; 1806.08049 2018.
[35] Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32, 2019.
[36] Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. arXiv preprint wrXiv:2005.00633 2020.
[37] Ann-Kathrin Dombrowski. Maximillaan Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in Neural Information Processing Systems, 32, 2019.
[38] Amirata Ghorbani, Abubakar Abid, and James Zou, Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 2019.
[39] Wei Huang, Xingyu Zhao, Gaojie Jin, and Xiaowei Huang. Safari: Versatile and efficient evaluations for robustness of interpretability. arXiv preprint arXiv:2208.09411. 2022.
[40] Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. Advances in Neural Information Processing Systems, 13::13223-13633 2020.
[41] Ahmad Ajallocian, Meyee-Mohsen Moosavi-Dezten Michalis Vlachos, and Pascal Frossard. On smoothed explanations: Quality and robustness. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Management, pages 15-25, 2022.
[42] Yipei Wang and Xiaoqian Wang. Self Seff-interprettable model with transformation equivariant interpretation. Advances in Neural Information Processing Systems, 34:2359-2372, 2021.
[43] Vijay Arya, Rachel KE Bellamy, Pin-Yo Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss Aleksandra Mojsilović, al. One explanation does not all: toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv: ww..//9990... 000090030222.. 2019.
[44] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 22(1):4-22 2020.
13

---

[45] Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances Neural Information Processing Systems, 32, 2019.
[46] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković, Geometric deep learning: Grids, groups, graphs. geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
[47] Jonathan Crabbé and Mihaela van der Schaar. Label-free explainability for unsupervised models. Proceedings of Machine Learning Research, 2022.
[48] Chris Lin, Hugh Chen, Chanwoo Kim, and Su-In Lee. Contrastive corpus attribution for explaining representations. arXiv preprint 22X1v:2200.00107, 2022.
[49] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res., 20 (0(117) 2019.
[50] A. L. Goldberger, L. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23), 2000.
[51] G.B. Moody and R.G. Mark. The impact of the mit-bih arrhythmia database. IEEE Engineering Medicine and Biology Magazine, 20(3):45-50, 2001.
[52] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.688 2014.
[53] Jeroen Kazius, Ross McGuire, and Roberta Bursi. Derivation and validation of toxicophores for mutagenicity prediction. Journal of Medicinal Chemistry, 48(1):3 2005. PMID: 1 15634026.
[54] Kaspar Riesen and Horst Bunke. Iam graph database repository for graph based pattern recognition and machine learning. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 287-297. Springer, 2008.
[55] Christopher Morris, Nils M. Kriege. Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020.
[56] Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 4602-4609, 2019.
[57] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1920-1920, 2015.
[58] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnbbas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[59] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Scungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invarian neural networks. In International conference on machine learning, pages 3744-3753. PMLR, 2019.
[60] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Anmual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 152-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
[61] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
14

---

[62] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[63] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[64] Maurice Weiler and Gabriele Cesa. General e (2)-quiiari steerable cnns, Advances in Neural Information Processing Systems, 32, 2019.
[65] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsu- pervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL, USA, 2011. PMLR.
[66] Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cmns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274-14285, 2020.
[67] Valerio Biscione and Jeffrey Bowers. Learning translation invariance in cnns. arXiv preprint arXiv:2011.11757, 2020.
[68] Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Softw. Eng., 48(1):1-36, 2022.
[69] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07355, 2017.
[70] Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for the l_0 norm. arXiv preprint arXiv:1804, 05805, 2018.
[71] Divya Gopinath, Guy Katz, Corina S Pasareanu, and Clark Barrett. Deepsafe: A data-driven ap- proach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00450 2017.
[72] D. Robinson, F.W. Gehring, and P.R. Halmos. A Course in the Theory of Groups. Graduate Texts in Mathematics. Springer New York, 1996.
[73] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Workshop Track Proceedings, 2014.
[74] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1);e22, 2020.
[75] Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7):620-66 631, 2021.
[76] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, (8(301):13-300 1963.
[77] Klock and H. K. van Dijk. Bayesian estimates of equation system parameters: An application of integration by monte carlo. Econometrica, 46(1):1-19, 1978.
[78] Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[79] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. arXiv preprint, 2017.
15

---

[80] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[81] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[82] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXtv:1412.6980, 2014.
[83] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In European conference on computer vision, pages 818-833. Springer, 2014.
[84] Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadeh. Eeg heartbeat classification: deep transferable representation. In 2018 IEEE international conference on healthcare informatics (ICHI), pages 444.44.. IEFE, 2018.
[85] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Gaäl Varoquaux, Travis Vaught, and Jarrod Millman, editors, Proceedings of the 7th Python in Science Conference, pages 11 15, Pasadena, CA USA, 2008
[86] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020.
[87] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2833
2011.
[88] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014.
[89] Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990-2999, New York, New York, USA, 2016. PMLR.
[90] Chaitanya Joshi. Transformers are graph neural networks. The Gradient, page 5, 2020.
[91] Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv: // 188, 2018.
[92] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical cnns. arXiv preprint 11X1v:1801.101100, 2018.
[93] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or Meshenn: network with edge. ACM Transactions on Graphics (TOG), 38(4): 1-12, 2019.
[94] Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9323 9332. PMLR, 2021.
[95] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess Smidt, and Boris Kozinsky, E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11,000 2022.
[96] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet cifar-10. 10. arXiv preprint 1xXiv:18000003505,000 2018.
16

---

Appendices
How to Use our Framework in Practice
18
B Contribution and Hope for Impact
19
Limitations
19
D Theoretical Results
20
D.1 Feature Importance Guarantees
20
D.2 Example Importance Guarantecs
23
D.3 Concept-Based Explanations Guarantees
24
D.4 Enforcing Invariance
25
Convergence of the Monte Carlo Estimators
26
F Experiment Details
27
Comparison with Sensitivity
33
H Other Invariant Architectures
34
I Examples of Non-robust Explanations
35
Effect of Distribution Shift
36
17

---

A
How to Use our Framework in Practice
Start with a neural network f (2 C) + y' that is invariant under symmetry group ş with representation
p: 9 Aut, an interpretability method e (9). - an example x'2 C) to explain.
Type of e?
Feature Importance
Concept- Based
Example Importance
Evaluate Equivele. defined in (2).
Evaluate Inve (e, a] defined in (1).
Evaluate Invgle,
defined in (1).
Equivgle, a| = 1?
Invole,æ) 1?
Invele,a]
Yes
Yes
Yes
No
can either
(a) Use your method with another
Restrict to gradient perturbation-buse bused with invariant bascline plala per Prop. D.6 and D.8.
representation
representation bused?
invariant per Prop. D.12 and D.14
Yes
(b) Enforce robustness replacing by as per Prop. 2.3.
Replace by per Prop. 2.3, For large [9], use Monte Carlo sampling an increase New until the desired Invg (eine, 2] is achieved.
You should have robust explanation by now. you wish relax model invariance, remember to re-evaluate metrics since the robustness of interpretahility method an invariant model does imply robustness approximately invariant model.
Figure 6: Our guideline to improve the robustness of interpretability methods with respect to model symmetries.
To show how these guidelines can be used in practice, we illustrate two possible categories based on our experiments from Section 3. Let us start with Influence Functions and the ECG dataset. This is an example importance method, hence we choose the central branch from Figure 6. Mcasuring the invariance in Figure 3(b) shows that Invo [e, æc] & 1, hence we may jump to the terminal node of the flowchart. This is correct since Influence Functions are G-invariant as demonstrated in Proposition D.9.
Now let us take the example of CAV used with the equivariant layer from the Deep Set for the ModelNet40 dataset. This a concept importance method, hence we choose the right branch of Figure 6. Measuring the invariance in Figure 3(c) shows that Invgle, < 1, hence we go down. As the flowchart recommends, we may decide to use the explainability method on an invariant layer instead. Doing this yields Invgle, æ) as shown in Figure 3(c). Hence may jump to the
18

---

terminal node of the flowchart. This is correct since concept important methods used with invariant layers are also G-invariant, as demonstrated in Proposition D.14.
B Contribution and Hope for Impact
In this appendix, we discuss the significance of our work. We should firstly emphasize the importance of the problem we are tackling in this work. In order to provide valuable explanations of a model, crucial to ensure that interpretability methods faithfully describe the explained model. Indeed, failing in this basic criterion implies that the explanations could be inconsistent with the true model behaviour, hence leading to false insights about the model. For this reason, believe that guaranteeing an alignment between interpretability methods and the model is a problem the utmost importance. Beyond the significance of this problem, we believe that we bring substantial contribution to address it. Below, we enumerate the novelties we claim.
Connecting interpretability with geometric deep learning. In Sections. 2, we show that the formalism of geometric deep learning naturally extends to the description of interpretability methods. We believe that this bridge can be valuable for at least two communities. For the interpretability community, geometric deep learning provides rigorous framework to ensure that interpretations are consistent with strong inductive biases that underpin cutting edge deep neural networks, such as GNNs, CNNs, Deep Sets, Transformers and the other examples cited in Appendix H. For the geometric deep learning community, interpretability provides way to extract actionable insights from increasingly sophisticated architectures. We hope that our paper promotes collaborations between these two communities.
Proving that explanation robustness imposes restriction on interpretability methods. By us- ing our formalism derived from geometric deep learning, we show in Appendix D that enforcing equivariance or invariance for interpretability methods imposes strong restrictions. For instance, gradient-based attribution methods that rely on a baseline signal x (S2, C) are equivariant if this baseline signal is invariant: p[g]= = H. We provide a similar analysis for 3 popular types of interpretability methods (feature importance, example importance and concept-based explanations). We hope that these insights will guide the future development of interpretability methods tailored for cutting edge deep learning models, such as GNNs.
Introducing two well-defined and sensible robustness metrics. Beyond adapting invariance and equivariance to interpretability methods, we provide principled way to evaluate those properties in Section 2 with two robustness metrics. Since these metries cannot always be computed exactly (e.g. for groups G with large cardinalities), we show in Appendix E that Monte Carlo approximations pro- vide sensible approximations, both theoretically by adapting Hoeffding's inequality and empirically in the setup from Section 3.
Demonstrating empirically that not all interpretability methods are created equal. We have performed extensive experiments on 6 different datasets, corresponding to 4 distinct modalities and model architectures, with 12 different interpretability methods. These experiments show that some interpretability methods (such as GradientShap and Feature Permutation) consistently fail to be robust with respect to symmetries of the model. We believe that these results are not trivial and should impact the way we use these methods to explain deep neural networks.
Through these contributions, we hope to reinforce the 3 below statements.
1. Interpretability should be used with skepticism.
2. Not all interpretability methods are created equal.
3. Robustness can guide the design of interpretability methods.
C
Limitations
In this appendix, we discuss the main limitations of this work. As first limitation, we would like to acknowledge the fact that our robustness metrics have no pretention to provide a holistic evaluation approach for interpretability methods. We believe that evaluating the robustness of interpretability methods requires a combination of several evaluation criteria. To make this point more clear, let us take inspiration from the Model Robustness literature. By looking at reviews on this subject (see e.g.
19

---

Section 6.3 of [68]), we notice that various notions of robustness exist for machine learning models. To name just few, existing works have investigated the robustness of neural networks with respect to noise [69]], adversarial perturbations [70] and label consistency across clusters [71]. All of these notions of robustness come with their own metrics and assessment criteria. In a similar way, we believe that the robustness of interpretability methods should be assessed on several complementary dimensions. Our work contributes to this initiative by adding one such dimension.
As second limitation, our robustness metrics are mostly useful for models that are (approximately) invariant with respect to a given symmetry group. That being said, we would like to emphasize that our robustness metrics can still be used to characterize the gquivariance/Invarian even if the model is not perfectly invariant. The metries would still record the same information, even if the invariance of the model is relaxed. The main thing to keep in mind when using these metrics with models that are not perfectly invariant is that the explanations themselves should not be exactly invariant/equivariant That said, for model that is approximately invariant, we expect a faithful explanation to keep a high invariance / equivariance score. This is precisely the object of Section 3.3.
of explanations
D Theoretical Results
In this appendix, we prove all the theoretical results mentioned in the main paper. We start by deriving robustness guarantees mentioned in Table 1. We then prove that any method can be made G-invariant by aggregating the explanation over several symmetries.
D.1 Feature Importance Guarantees
Let us start by feature importance methods. As we are going see, the robustness guarantees in this case typically require us to restrict the type of group representation p we use to encode the action of the symmetry group G on the signal space X() C). This motivates the two following types of representations that can be found in group theory textbooks (see e.g. [72]..
Definition D.1 (Orthogonal Representation). Let p G - A C)] be representation of the group G. and let d E N+ denote the dimension of the signal space (S2, We say that the representation is an orthogonal representation if its image is a subset on the orthogonal matrices acting on (2, C): p(g) C O(d), where (dd) denotes the set of d d orthogonal real matrices. This is equivalent to p ['g)pT[ [g] p [g]eاو] La for all g G. where Ia denotes the d x d identity matrix. Definition D.2 (Permutation Representation). Let p : 9 + Aut (12 C)] be representation of the group G, and let d E N+ denote the dimension of the signal space (S, We say that the representation is a permutation representation if its image a subset on the permutation matrices acting on ( C): p()) C P(d), where P(d) denotes the set of d x d permutation matrices.
This means that for all g E G. there exists some permutation π E (dd) such that we can write (plgiz); (() for all E Zg. x E C)..
Remark D.3. One can easily check that all permutation representations are also orthogonal represen- tations. The opposite is not true.
D.1.1 Gradient-Based
We shall now begin with gradient-based feature importance methods. These methods compute the model's gradient with respect to the input features to compute the importance scores. Hence, it is useful characterize how this gradient transforms under the action of the symmetry group
Lemma D.4 (Gradient Transformation). Consider a differentiable function f X(2 C) + y that is invariant with respect to the symmetry group G. We assume that acts on X(2 C) via the representation p: g > Aut for some g E G we define x' = plg)a, pr then we have the following identity:
Ve = (c
(3)
where p -11.T [g] denotes the matrix obtained by applying an inversion followed by transposition to the matrix pig).
Proof. We start by noting that the G-invariance of ʃ implies that f(x') (xx) and, hence:
Vz = f
(4)
20

---

It remains to establish the ling between V and Z.. To this end, we simply note that p '99 Hence, for all E Z with d (S C)], we have that
2, Σ Péy L0|z'--
Hence, by using the chain rule, we deduce that for all h E Zd:
0
Oz
Dxy
Oz's dr:
0
Pek [91
0z: 1=1
0
Pki
1.T
[9]
Oz;
The above identity implies F = =p-1.T V By injecting this to the hight-hand side of (4), we obtain (3).
The simplest gradient-basee baseed attribution is simply given by the gradient itself [73]. We refer to it as the vanilla saliency feature importance. Although this attribution method is bit naive, we may still deduce an equivariance guarantee from the previous proposition.
Corollary D.5 (Equivariance of Vanilla Saliency). Consider a differentiable neural network f X(S, C) + y that is invariant with respect to the symmetry group G. We assume that G acts on X(),C) via the representation importance explanation e(xx) = V (x)). If the represenlation p is orthogonal, then the explanation e is G-equivariant.
G - Aut[XX(2, We consider a vanilla saliency feature
Proof. From Lemma D.4, we have that <(p[g)=) = = (g)ee))) for all E G. Now if p orthogonal, we note that p [g] plg], which proves the proposition.
We now turn to more general family of gradient- based feature importance methods. These methods attribute importance to each feature by aggregating gradients over a line the input space X2, C) connecting baseline example I E X C) with the example X C) we wish to explain. It was shown that the choice of this baseline signal has a significant impact on the resulting explanation [74]. show that enforcing equivariance imposes restriction the type
In the following proposition, of baseline signal E that can be used in practice.
Proposition D.6 (Gradient- Based Equivariance). Consider a differentiable neural network f X(Ω,C) + y that is invariant with respect the symmetry group G. We assume that G acts X((, C) via the representation G -> Aut[X((2, C)j. We consider a gradient-bant based explanation built upon baseline signal & E (2,,C) and of the form
e(x) (x x) o J.
v(1) V 1(x =))) dl,
where @) denotes the Hadamard product and is a functional defined on the Hilbert space L², 1]). If p is permutation representation and the baseline signal is G-invariant, i.e. pig] ž for all g E G, then the explanation is is--eee G-equivariant.
Remark D.7. Note that we have introduced the functional yp to make the class of explanation as general possible. For instance, we obtain exact Integrated Gradients for (tt) 1 and Input*Gradinn [19] for <p(t)) = 5(t - 1), where is Dirac delta distribution. Similarly, this includes discrete approxima- tions of Integrated Gradients by taking e.g. (p() > 5(l (n), with l,, I + (a: .:) for all n E Zn. Finally, we note that the equivariance of Integrated Gradients also implies the equivariance of Expected Gradients [75].
21

---

Proof. For all 9 E G. we have that:
<(plg)=) (olo)> 8 I ((t) V= x)) de -lol(x - 8) [ ((1) V.S(pig + (1x -))) di o p(g) I. 4(1) v.f(Ø t/x x)) de -lg](x -)
(Invariance of a)
(Corollary D.5).
Since P a permutation representation, there exists permutation π dim[X'(2, C)). such that for all a, b X(2, C) and E Zx::
S(d), where d
(pígla p(g]b); (plgla); @ (plglb);
(Definition of Hadamard product)
ar(1) @ bo())
(p is a permutation representation)
(a ④ (2)())
(Definition of Hadamard product)
(plg)(a 3 6));
(p a permutation representation).
We deduce that plg]a @ p(g]b pla] (a @ b). By applying this to the above equation for [())g)
get:
<(p(gl=) plg F) 2 plg] / ((t) Vef + t/r - F)) dt
plgl (x- I ((t) V f( +|+ =)) dt
= plg)e(x),
which proves the equivariance property.
D.1.2 Perturbation-Batt
The second type of feature importance methods we consider are perturbation- bbased methods. These methods attribute importance to each feature by measuring the impact of replacing some features of the example x E X(2, C) with features from a baseline example ã E X',, C) on the model's prediction. Again, enforcing equivariance imposes restriction the type of baseline that can be manipulated.
Proposition D.8 cetturbation-Basee Equivariance). Consider neural network ((,,C + + that invariant with respect to the symmetry group G. We assume that acts on X(2, C) via the representation p G - (S, C)). We consider a perturbation- based explanation built upon baseline signal E X22 C) and of the form
[e(x)), f(x) f(r:(x)),
for all i E Lgg, where d = (S,, (L, The perturbation operator T; replaces feature I;, i E Ed with the baseline feature 3t. is defined as follows: [r:(x)]; x, & x;), Kronecker delta symbol, for all j E Za and x E X',) If is permutation representation and the baseline signal G-invariant, i.e. pigl E for all E then the explanation G-equivariant.
where ö denotes the
Proof. For all g E G and i,j E Ed, there exists permutation क E S(d) such that:
[r<(pg]=))1
Tn(i) + dty(Tz za(i)) In(i)) (zy((x(1) xx(1)) -[=())())<())
(p is permutation representation)
(Ø is invariant)
=(0o)r=(0)())
(p is permutation representation).
22

---

We deduce that r:(p(g)a) = pig) for all i E Zd. We are now ready to conclude as:
[e(plg]=)], f(x) f(r:(pl))z)
(Definition of e)
f(z) ((p)))r=())))))
(Above identity)
f(x) - (x)r
(Invariance of f)
= [[(()))<()
(Definition of e)
= [o(gle(x)]
(o is a permutation representation),
which proves the equivariance property.
D.2 Example Importance Guarantees
We proceed with example importance methods.
D.2.1 Loss-Based
We start with loss-based methods. These methods attribute importance to each training example of (x", y²) E Durain by comparing the loss C(f(="), (f(="))""" y") with the loss C(f(z), y) of the example E X(2,,) C) wish to explain. We show that these methods naturally invariant without imposing any restriction on the representation p.
Proposition D.9 (Loss-Basse Invariance). Consider differentiable neural network fo X(2,C) + D parametrized by P E N+ parameters 0 IR,, that invariant with respect the symmetry group g. We assume that G acts on XS, C) via the representation p p:g - Aut (S,, C)). We consider an example importance explanation based on the loss (22, x R and of the form
c(x, v) F(C(S v))
where F maps any function 1 corresponding to the number of training examples for which we evaluate the importance. The explanation e is G-invariant.
R.P B++ to a vector in F(I] E TRNNiii with Nurain E N+
Remark D.10. We note that F typically contains differential operators. For instance, Influence Functions are obtained by taking
(F(I) VT H Vol(O),
for n E EN...n. where (x", "") E Dirsin is a training example and Hg E PxP the Hessian of the training loss with respect the model's parameters.
Remark D.11. We note that the dependency of the explanation e with respect the label y E y is omitted in the main paper. The reason for this is that the symmetry group G only acts on the input signal x.
Proof. The proposition can directly be deduced from the G-invariance of the model. For any g E G. we have:
e(piglx,)) = F[C(folplg))) y)]
=F[C(fo(=)..) v) =e(x,y),
(Invariance of f)
which proves the desired property.
D.2.2 Representation-Ban
We proceed with representation- methods. These methods attribute importance to each training example of (x".y") E Dtrain by comparing the model's representation hi(a") (typically the output of model's layer) with the representation h(()) of the example x E X C) we wish to explain. We show that these methods are invariant if we restrict to representations ht that are invariant.
23

---

Proposition D.12 (Representation Based Invariance). Consider differentiable neural network f: ((),C) + y that is invariant with respect to the symmetry group G. We assume that G acts on x(2,C) C) via the representation explanation based on a representation h (S, > of the neural network) and of the form
+ Aut ( C)}. We consider an example importance
H extracted from f (e.g. an intermediate layer
e(x) =[bx))]
where F H - RRNuoin maps any representation: E H to a vector in F(r) E TR. TRNNuii, with Nirain N+ corresponding to the number of training examples for which we evaluate the importance. If the representation h is G-invariant, then the explanation e is G-invariant.
Remark D.13. We note that F can be adapted to the method we want to describe. For instance, SimplEx obtained by taking
Nirain
Fir] arg
min
(2")
w Nizoin
n=1
Nerain
s.t.
Un
where xn E Derain a training example for n E ZN Similarly, Representation Similarity is obtained with
Firl rTh(="),
for n E Neraiu"
Proof. The proposition can directly be deduced from the G-invariance of the representation. For any E G, we have:
e(plg|x) F[h(pl91=)
= =[](())]
(Invariance of h)
=((x)
which proves the desired property.
D.3 Concept-Based Explanations Guarantees
We turn concept-based methods. These methods attribute importance specified by the user for the model to predict a certain class. Although these explanations are typically global (i.e. at the dataset level), they are based on concept classifiers that attempt to detect the presence/absence of the concept on each individual example x E ( C') based on its representation h(x)). We show that these classifiers are G-invariant if restrict to representations h that G G-invariant.
of concept
Proposition D.14 (Concept-Bate Bassdd Invariance). Consider differentiable neural network f X(2,. C) y that invariant with respect the symmetry group G. We assume that G acts on X(2,C) C) via the representation p:: + Aut[X((,,C), We consider a concept based explanation based on a representation h X), - H extracted from f (e.g. an intermediate layer of the neural nelwork) and of the form
e(x) c[h(x)).
where c: - 13c maps any representation E H. to a binary vector in c/r) E {0, 1pc indicating the presencelabsence C E N+ selected concepts. If the representation G-invariant, then the explanation is G-invariant.
Remark D.15. We note that concepts activation vectors (CAVs) are obtained by fiting a linear classifier Concept activations regions (CARs), on the other hand, are obtained by fitting a kernel-based concept classifier.
24

---

Proof. The proposition can directly be deduced from the G-invariance of the representation. For any g E G, have:
<(plg)a) c[h(plg]=)]
= c[h(x))
(Invariance of h)
e(z).
which proves the desired property.
D.4 Enforcing Invariance
Finally, we prove Proposition 2.3 that allows us to turn any interpretability method into G-invariant method.
Proposition D.16. [Enforce Invariance] Consider a neural network f X C) - y that is invariant with respect to the symmetry group 9 and (S),C) & be explanation for f. We assume that G acts on X (S), C) via the representation p:: g - (1t(X((),C)) We define the auxiliary explanation Biav 2(2, C) & as
1 Σ <plgg\>)
Cinv (x) =
gEg
for all x. E C), The auxiliary explanation Ciny is invariant under the symmetry group g.
Proof. For any E 9, we have that
Cinv (plī)x)
gE0
cppl g/æ),
DEG
where we have used the fact that the representation p compatible with the group composition. We
now define the map lg 9 + 9 as [g(g) = ÿ ogog for all G. We note that lg is bijection from g to itself, since it admits an inverse 1- 13-1. Indeed, for all g E G:
(1g o (g) = g ogog=
(gg o 1--1)))1 (g) = o gog--o--- oo-
Hence, we have that (g(g) G. By denoting g' (g(g) = gog. we can therefore write
Cinv (p(ī)=)) Σ e(plg'}=)
8'EG
Caav (x).
This proves the G-invariance of the explanation Einv.
The interpretability method einv has to be understood as way to assign a unified explanation to each class of equivalent signals rather than a patched version of To make this argument more rigorous, we define the equivalence relation on the set of signals X'2,, C) as follows: two signals I, X(22 C') are equivalent iff there exists symmetry g E G relating the two signals x' p|g]æ. This equivalence relation is the one that underpins a G-invariant neural network as x' and x are assigned the same prediction f(x') = f(x). If we denote by S. {x' X( x' x} the class of signals equivalent to the signal H, we may write Cinv (z) = [S..1 E c(x'). This reformulation gives nice interpretation to Einv: the explanation Cinv (x) is simply given by averaging the explanation e over the class S. of signals equivalent to x. If is an example importance method, then eiv allows us to identify the examples that are the most related the equivalence class S,- If, on the other hand, e is a concept classifier, then Cinv measures the fraction of examples in the equivalence class S₂ where the concept of interest is detected. We believe that using Car with this interpretation in mind should help to avoid unwarranted trust in the resulting explanations.
Our opinion is that explanations that are robust by design are preferable over auxiliary explanations Cinv- In this way, a possible hierarchy between interpretability methods based on our robustness criterion for G-invariant models could be as follows:
25

---

Method 1 An interpretability method e that has G-invariance by design.
Method 2 An auxiliary interpretability method eiv obtained from e with Proposition 2.3.
Method 3 An interpretability method e that is not G-invariant.
In this way, Method 2 should be avoided whenever Method is available. This is the case, for instance, of example-based interpretability methods where loss-based methods are naturally invariant. Hence, in this case, the benefit of patching representation- based methods is limited. However, in the case of concept-based based explanations, we note that neither CAVs nor CARs grant invariance. Method I is unavailable, Method 2 might be the best we can do. In the specific case of concept-based interpretability, we note that Proposition 2.3 implements a sensible fix to the lack of invariance. Indeed, applying Proposition 2.3 is equivalent to making the concept-classifeer invariant by applying a G-invariant group aggregation, which is standard way implement classifier invariance (examples include GNNs and Deep Sets). In this way, the usefulness of Proposition 2.3 is context- context-depeneet and we believe that good usage should always be informed by domain knowledge.
this case, since
E Convergence of the Monte Carlo Estimators
In this appendix, we discuss the Monte Carlo estimators used approximate the invariance and cquivariance metrics defined in Definition 2.1. We first note that our experiments typically aggregate the metries over set Drest of examples. Hence, we are interested in the metries
I (e) = ExwU(D [&g (e (p[G) Equiv (e) = Ex G-UU(()) [sg
[']]](
where U' denotes a uniform distribution. Clearly, whenever the order g of the symmetry group is large. these metries might become prohibitively expensive compute. In this setting, we simply build Monte Carlo estimator for the above metrics by sampling Nsmmp symmetries GI...., G Neamp with Namp E N+ and samp << [g). estimators can be written as
If we have Niest
(Dessl test examples, the Monte Carlo
Nosst Neamp
(nvg(e) =
s8 s [=ee(pGmlxx"))((x))
n=l m==
Niest Neamp
1
Equiv Neest Nsamp
88 [e (p(Gm)))) ,/ (mGm]eeX"")
Those are the estimators that we use in our experiments. Let us first discuss the convergence of these estimators theoretically. Since (a, b) [-1, for all a, E E, Hoeffding's inequality [76] guarantees that for all t R
Nieet Nsampt" t2
PP ([ivv((e) Invele) 2t) s 2exp
2
Niest Nsampt t2
IP [quivy(e) (e) 2 21) < 2 exp
2
Let us now plug-in some numbers see how these inequalities translate our experiments. In Section 3, we typically use Nest = 1, 000 and Naamp 50. Hence, the probability of making an error larger than = 2% our experiments is smaller than 10 This guarantees that all the metrics reported in the main paper are precisely evaluated.
We shall now verify this theoretical analysis with the experimental setup described in Section 3. Since do not resort any Monte Carlo approximation for the Electrocardiogram dataset, exclude
26

---

it from analysis. Similarly, robustness scores (e) = 1 or Equiv as these can only be produced by having Inv(e: Gm] x or Equiv(e, Gm R for all m E & Naamp' which guarantees that the estimators have already converged. By applying these filters with the help of Figure 3, we restrict our analysis to Gradient Shap for the Mutagenicity dataset and to Gradient Shap, Feature Permutation, SimplEx-Equiv, CAV-Equiv and CAR-Equiv for the ModelNet4O dataset.
can be excluded
We plot the Monte Carlo estimators ( and Equiv as a function of Naamp for various interpretability methods in Figure 7. As we can see, all the Monte Carlo estimators have already converged for Naamp = 50 used in the experiment. This is due to the fact that we use relatively large test set in each experiment, with Nuest 433 for the Mutagenicity dataset and Ntest 1, 000 for the ModelNet40 experiment.
1.00
100
0.50
0.50
Nerte Care Deraer
Neyter Cara mor
-3.25
-3.2
-0.50
-350
Stadee:: Sha Eq. variance
-0.75
-2.75
CAVEtshi noar arce CAN Tour Invelance
seretce Casi traarnnc
25
02
120
28
60
82
100
(a) Mutagenicity Dataset
(h) ModelNet40 Dataset
Figure 7: Convergence of the Monte Carlo estimators. Each curve represents the value of the estimator (nvg(e) Equiv as function of Nsamp- In each case, build 95% confidence interval around this estimator.
We finish this appendix by mentioning that all the groups manipulated in this paper are finite groups It goes without saying that an extension of our analysis to infinite, or even uncountable groups would probably require more sophisticated sampling technique, such as e.g. importance sampling [77].
F Experiment Details
In this appendix, provide all the details for the experiments conducted in Section 3.
Computing Resources. Almost all the empirical evaluations were run on single machine equipped with 64-Core AMD Ryzen Threadripper PRO 3995WX CPU and a NVIDIA RTX A4000 GPU. The only exceptions are the CIFAR 100 and STL10 experiments, for which we used a Microsoft Azure virtual machine equipped with single Tesla V100 GPU. All the machines run on Python 3.10 [78] and Pytorch 1.1 [79].
The MIT-BIH Electrocardiogram (ECG) dataset [50, 51] consists of univariate time series x E X(ET, R) with T 187 time steps, each representing heartbeat cycle. Each time series comes with a binary label indicating whether the heartbeat is normal or not. We train I-dimensional convolutional neural network (CNN) to predict this label. This CNN is made invariant under the action of the cyclic translation group G = Z/TZ on the time series by using only circular padding and global pooling.
Electrocardiograms.
Mutagenicity. The Mutagenicity dataset [53-55] consists of graphs x E X [x], IN.,. represent- ing organic molecules. In a graph, each node u e Vg is assigned an atom indexed by æ(u) E EN.,. where Nap = 14 is the number of atom species. We ignore the attributes for the edges E, C V Each graph comes with binary label indicating whether the molecule is mutagen not. We train a graph neural network (GNN) to predict this label. This GNN made invariant under the action of the permutation group G Sv, on the node ordering by using global pooling.
ModelNet40. The Ned4 [57] dataset consists of CAD representations the same process as [58, 59] to convert each CAD representation into cloud x E XZ 1R3) of
objects. We use
27

---

Npt = 1, .000 points embedded in R². Each cloud of point comes with label y E Zn indicating the class of object represented by the cloud of points among the 40 different classes of objects present in the dataset. We train a Deep Set [58] model to predict this label. Thanks to its architecture, this model is naturally invariant under the action of the permutation group 9 SN on the points in the cloud. Each model is trained on a training set D Drrain.
IMDb. The IMDb dataset [60] This dataset contains 50k text movie reviews. Each review comes with a binary label E {0, J.. We represent each review as a sequence of tokens x E X (22, RR RR)), , where we cap the sequence length to T 128 and set the vocabulary size to = 1, 000. We perform train- suiin-validationtess split of this dataset randomly (90%-5%-5% and fit 2-layers bag-of-wont MLP on the training dataset for 20 epochs with Adam and a cosine annealing learning rate. The best model (according to validation accuracy) achieves a reasonable 86% accuracy on the test set D Let us justify the bag-of-wort classifier invariance with respect to the token pertmutation group ST. A bag-of- words classifier f receives sequence of tokens z E X R and outputs class logits fxxx E R.2. By definition, the bag-of-word classifier can be written as a function f() = 9 1 onehot( In this form, the invariance of the classifier with respect to tokens permutation is manifest. Let E ST be a permutation of the token indices. Applying this permutation to the token sequence does not change the classifier's 's output: f(z)) = g [Σi= onehot(z = g (ET= onehot(z1)) = f(x). We conclude that bag-of-words classifiers are Sy-invariant.
T
FNSHIonMNIST. The MaST dataset [61] consists of 28 x 28 grayscale images x E (2w X RH with W = H = 28, each representing a fashion object (e.g. dress) from 10 categories. Each image comes with a label E Z10 indicating object's s category. We train a 2- 2-dimensional CNN to predict this label. We pad each image by adding 10 black pixels in each direction, so that the image information content is conserved by applying translations from the group G = (Z/ The CNN made invariant under the action of this group by using global pooling.
CIFAR100. The CIFAR 100 dataset [63] consists of 32 x 32 RGB images x E X' x Σn R3) with W- H 32, each representing an object (e.g. truck) from 100 categories. Each image comes with a label E 100 indicating object's category. We train the Dg - D4 - D1 28/10 WideResNet from [64] to predict this label. The design of this model imposes strong bias toward Dş invariance. To avoid artifacts created by rotating an image by 45", we restrict all our evaluations to the subgroup DA C Dg.
STL10. The STL10 dataset [65] consists of 96 x 96 RGB images I E XZZw X R2) with
W = H = 96, each representing an object (e.g. truck) from 10 categories. Each image comes with label y E Z.o indicating object's 's category. We train the D3g - IDd D1 16/8 WideResNet from [64] to predict this label. The design of this model imposes strong bias toward D3 invariance. To avoid artifacts created by rotating an image by 45,, we restrict all our evaluations to the subgroup ID C Dg.
Data Split. All the datasets are endowed with natural train-test split. In the ECG dataset, the different types of abnormal heartbeats are imbalanced (e.g. fusion beats only amount for .7% of the training set). We use SMOTE [80] to augment the proportion of each type of abnormal heartbeat in the training set.
Symmetry Groups. Each dataset in the experiment is associated to a specific group of symmetry and group representation. We detail those in Table 4. We note that all these groups and group representa- tions are easily implemented as tensor operations on the dimensions of the tensor corresponding to the domain 12.
Models. We provide a detailed architecture for each model in Tables 5 to 11. All the models are implemented with Pytorch [79] and PyG [81]. All the models except the WideResNets are trained using Adam [82]. The CNNs are trained to minimize the cross entropy loss for 200 epochs with early stopping and patience 10 with a learning rate of 10 3 and a weight decay of 10 5, The GNN is trained to minimize the negative log likelihood for 200 epochs with carly stopping and patience 20 with a learning rate of 10 3 and a weight decay of 10 The Deep Set is trained to minimize the cross entropy loss for 1,000 epochs with early stopping and patience 20 with learning rate of 10-3 a weight decay of 1 and multi step learning rate scheduler with ? = 0.1. The WideResNets are trained with Stochastic Gradient Descent to minimize the negative cross entropy loss for 200 epochs (1,000 for ITL with an initial learning rate of 0.1, a weight decay of 5 10-5, momentum 0.9 and an exponential learning rate scheduler with 7 = 0.2 applied each 60 epochs (300 for STL10). The test set used as a validation set in some cases, as the model generalization is never used as
28

---

Table 4: Different groups and representations appearing in Section 3. Since Mutagenicity has heterogeneous graphs, V2 C N+ denotes the vertices specific the graph data æ. We use the notation a(u, u) to denote the elements of the edges data matrix.
Dataset
Modality
Input Signal
Symmetry
Representation
Electrocardiogram
Time Series
[n(ti]iezp
g E//T
plgla(t) x(t 1it--g00 9 mod
pla x [(m)]
Mulagenicity
Graphs
aa(u), vllswev
g
Plgla(u,r) = aly '[u).9 (())
ModelNet40
Tabular Set
y Sx,-
Plyix(n) = -(x]]
PashionMNIST
Image
[n(u, e), (u.))e xZa
(Nt (22)
(2/ [2/102]
((g.x(t) z(" -9 mod W,e D2 mod H)
CIFARI00
Image
Infu, e). (u.s)eRm xZa
E Ds
p() =(g-)( pla zig [(a,e))
STL10
Image
(atu, e) (mus) x2,
9 Dy
an evaluation criterion. All the parameters hyperparameters that are not specified are chosen to Pytorch and PyG default value. Note that for each architecture, we have highlighted the layer that we call lnv and Equiv Section 3. The representation- interpretability methods rely on the output of these layers.
Table 5: ECG All-CNN Architecture.
Layer Type
Parameters
Activation
Notes
Convld
in chanels=1, out cot_chanels=16, kernel_size=3,
ReLU
stride=1, padding=1, pacplinggg_iaadiggg____oneong coaaddinn_mod==cciiiioo
Conv ld
in m__chanels=16, out_ out_chanels=64, kernel_sizess 1 in chancls=64, out_ cott_chanels=128, kernel size=3,
ReLU
Convld Ld
ReLU
Equiv layer
stride=1, padding=., padding Soaadding_mod==22 mode= circular'
Pooling
Global Average Pooling
Linear
in mn_chanels=128, out out_chanels=32
LeakyReLU
Inv layer
Linear
in chancls=32, out chanels=32 in chanels=32, out chanels==
LeakyReLU
Linear
Table 6: ECG Augmented-CNN and Standard-CNN Architecture.
Layer Type
Parameters
Activation
Notes
Convid ld
in ei_chheee==1,, out out_chanels=16. kernel_sizess stride=1, padding=l, paddinng boreddiin_od===/ circullar' kernel_size-s
MaxPoolld Convld
in chanels=16, out chanels=64, kernel kernel_size=3, stride=1, padding=1, moaddding_ooe=='ciie kernel_size=2
ReLU
MaxPoolld Convid
in chanels=64, out_ aut_ccanels=128. kernel kernel_size=3,,
ReLU
Equiv layer
stride=l, padding=1, kernel_size=2
MaxPoolld Flatten
Collapse all the dimensions together except the batch
dimension
Linear
in chancls=2944. out ootttouttteeeee channel=aa222
LeakyReLU LeakyReLU
Inv layer
Linear
chancls=32, out chancls=32
Linear
in chanhccs=s32, out chancls=2
29

---

Table Mutagenicity GNN, The GraphConv layers correspond to the graph operator introduced in
[56]
Layer Type
Parameters
Activation
Notes
GraphConv
in en_chanels=14, oat_ootttteeoooo cut_chaelss33
ReLU
GraphConv
in chanels=32,, chanell===3
ReLU
GraphConv
in_chanels=32, out_chanels=32
ReLU
GraphConv
in in_chanels=32, out_chaneis=32
ReLU
GraphConv
in chanels=32, out chanels=32
ReLU
Pooling
Global additive pooling on the graph
Linear
in chancls=32, out chancelsss=
ReLU
Inv layer
Dropout
p=0.5
Linear
in chchneel==32, out out_chanels=2
Log Softmax
Table 8: ModelNet40 Deep Set adapted from [58]. The Sub. Max layers correspond to the operation Tbs + Tb,a,1 m EBy Tb,s', i for each batch index N, set index ZNp. and feature index i Zz.
Layer Type
Parameters
Activation
Notes
Sub. Max
Linear
in chancls=3, out, chanels=256
Tanh
Sub. Max
Linear
in chanell==266, out ch5
Tanh
Equiv layer
Sub. Max
Linear
in chanels=256, out_chanels=256
Tanh
Max Pooling
Takes the maximum along the set dimension
Dropout
p=0.5
Linear
in chanels=256, out out_ chancelsls=25
Tanh
Inv Layer
Dropout
p=0.5
Linear
in chanels=256, out channllsss
Tanh
Table 9: FashionMNIST AII-CNN Architecture.
Layer Type
Parameters
Activation
Notes
Conv2d
in chanelansls1 out_chanels=16. kernel_size=3,
ReLU
stride=1, padding=1, padding paaddingmmode=2222 wwt__/ood=eeee circular''
Conv2d
kernel_size=3, in chancls=16, out out_ooottaaaa chamels=64, kernel size=3,
ReLU
stride=1, padding=l, padding pwwddig_moodeeeeoo modeem'circllarr
Conv2d
in chanels=64, out_ mat__chanels=122. kernel size=3,
ReLU
Equiv layer
stride=1, padding=-, pawidinnngi..pdiinn.g.. Boaaddinggmodee circular'
Pooling
Global Average Pooling
Linear
in_chanels=128, out_ out_chanels=32
LeakyReLU LeakyReLU
Inv layer
Linear
in mo_chanels=32, out cuL_chanels=32
Linear
in in_chanels=32, out out_chanels=10
30

---

Table 10: FashionMNIST Augmented-CNN and Standard-CNN Architecture.
Layer Type
Parameters
Activation
Notes
Conv2d
in chanels=1, out cht_cchnels=16, hernel_sizeeii
stride=1, padding=1, padddpaadddeeeeeeeo modee='cirulaar" circular' kernel_size=2
padding
MaxPool2d Conv2d
in m1_chanels=16, out_chanels=64, kernel_size=3,
ReLU
stride=1, padding=1, maddiing_
MaxPool2d
kernel_size-2
Conv2d
in chanels=64, out chuncclanl=128,, kernel_szze=3,
ReLU
Equiv layer
stride=1, padding=1, pawding_moodeeeeco 'cinull' kernel_size=2
MaxPool2d
Flatten
Collapse all the dimensions together except the batch
dimension
Linear
in chanels==94912. out chanels=32
LeakyReLU LeakyReLU
Inv layer
Linear
in in_chanels=32, out_ po__chaels=32
Linear
in in_chanels=32, out _u__chaell=2
31

---

Table 11: WideResNet from [64]
Layer Type
Parameters
Activation
Notes
Conv Residual Layer
Equiv Layer
Residual Layer
Residual Layer
in ch_chhnels=32, chanelssss3
BatchNorm
ReLU
Inv Layer
Pooling
Global Average Pooling
Flatten
Linear
out chancls= or 10
Model Invariance. We note that Figure 5 includes model invariance on the x-avis. The metric to evaluate this invariance is simply adapted from Definition 2.1: Inv x) = {yy](((gg))) f fx)), where sy 2)² - R is defined as (y(11, y2) vî 02/Ilvil Illvall for all y1, 1/2 E Y. We note that this cos-similarity similarity metric is sensible in a classification setting since (y
1 implies
y1 = a y/2 for some a R+. Since |/ II ||
this is equivalent to 3/1 1/2-
Feature Importance. We study gradient- -ased methods with Integrated Gradients [18] and Gradient Shap [17] as well as perturbation- based methods with Feature Ablation, Permutation [49] and Occlusion [83]. We also use DeepLift [19]. Whenever a baseline is required, we consider the trivial signal ē 0 as baseline. We note that this signal is trivially G-invariant as p[g] 0 for all g E G.
Example Importance. We study loss-based methods with Influence Functions [21] and TracIn [22] as well as representation-base loss-based methods are expensive to compute, we differentiate the loss only with respect to the last layer of each model and for a subset of Nirain = 100 training examples from Dirain For representation-batt baaed methods, we use the output of both invariant and equivariant layers of the model as representation spaces. We denote e.g. SimplEx-Ing to indicate that SimplEx was used by using the output of a G-invariant layer of the model as a representation space. Similarly, SimplEx-Eon Equiv corresponds to SimplEx used with the output of G-equivariant layer as representation space.
methods with SimplEx [23] and Representation Similarity. Since
Concept- Explanations. To the best of our knowledge, the only 2 post-hoc concept expla- nation methods in the literature are CAV [24] and CAR [25]. We use these methods to probe the representations of our models through a set of C = 4 concepts specific to each dataset. Again, we use both invariant and equivariant layers of the model for each method.
Concepts. We use of C concepts for ECG, Mutagenicity and ModelNet40: ; C 2 for FashionMNIST and STL10; C = 3 for CIFAR100. For the ECG dataset, we use concepts defined by cardiologists to identify abnormal heartbeats: Premature Ventricular, Supraventricnlar Fusion Beats and Unknown. We note that these concepts labels are directly available in the ECG dataset [84]. For the Mutagenicity dataset, we use the presence of known toxicophores [53] in the molccule of interest: Nitroso (presence of a N in the molecule), Aliphatic Halide (presence of a CI, in the molecule), Azo-type (presence of N-N in the molecule) and Nitroso-type (presence of a O N-O in the molecule). To detect the presence concepts, each molecule in the dataset is checked by using NetworkX [85]. For the ModelNet40 dataset, we use visual concepts whose presence can immediately be inferred from the class label: Foot (whether the represented object has a foot), Container (whether the represented object has the shape of a container). Purallelepiped (whether the represented object has parallelepiped shape) and Elongated (whether the represented object is stretched in one direction). For all the image datasets, we use concepts that can be directly inferred from the class of each example. We use the Top and Shoe concepts for FashionMNIST. the Aquatic, People and Vehicle concepts for CIFARI00; the Vehicle and Wings concept for STL10.
Interpretability Methods Implementation. For the feature importance methods, we use their Captunn [86] implementation. All the other explanations methods are reimplemented based on their official repository and adapted to graph data. All the concept classifiers are implemented with scikit- learn [87]. We use training sets of size 200 for the ECG, Fashion CIFAR 100, STL10 datasets and 500 for the Mutagenicity and ModelNet40 datasets. CAR classifiers are chosen to be SVCs with Gaussian RBF kernel and default hyperparameters. CAV classifiers are linear classifiers trained with stochastic gradient descent optimizer with learning rate of 10-2 and tolerance of 10-3 3 for 1, 000
32

---

epochs. In the case of CAR classifiers, we found it useful to apply PCA with 10 principal components reduce the dimension of the latent representation before applying the classification step. Indeed, this does not significantly reduce the accuracy of the resulting classifiers while significantly reducing the runtime.
Impossible Configurations. We note that some combinations of interpretability methods and models impossible. We summarize and explain the incompatibilities in Table 12. Further, do not use perturbation-baset feature importance methods with the CIFARIO 100 and STLIO ResNets, since those methods require (ldim(XX C))] model calls per example, which is prohibitively expensive for large vision models.
Table 12: Not all interpretability method can be used in all settings. We detail the exceptions here.
Method
Incompatible with
Reason
Feature Permutation
Mutagenicity
Heterogeneous graphs have different num- ber of features.
Feature Occlusion SimplEx-Equiv
Mutagenicity, ModelNet40 Mutagenicity
Specific to CNNs.
Heterogeneous graphs lead equivariant representations with different dimensions. Heterogeneous graphs lead equivariant representations with different dimensions. Heterogeneous graphs lead equivariant representations with different dimensions. Heterogeneous graphs lead equivariant representations with different dimensions.
Rep. Similar-Equit
Mutagenicity
CAV-Equiv
Mutagenicity
CAR-Equiv
Mutagenicity
Explanation Similarity. In Definition 2.1, we also chose the cos-similarity metric 88 [a, bj for two real-valued explanations a, b E Rd This choice might seem arbitrary at first glance. By looking more closely, notice that se [a, bj 1 implies explanation that we consider in this paper, a and will typically be vectors that aggregate various important scores (e.g. for each feature training example). In practice, the scale of these importance scores does not matter. Indeed, if a = a b, then the most important components will be the same for both a and b. Furthermore, the relative importance between any pair of component will be identical for both a and Therefore, we can consider that 8g (a, b] = implies that both explanations are equivalent to each other ~ b.
b for some
R+ For all type of
G Comparison with Sensitivity
In this appendix, we compare robustness metric with the sensitivity metric introduced by [35]. This metric studies the robustness of an explanation with respect to a small perturbation in the input features. It is defined as follows:
Sens(e, x)
max [le(x)) e(x')l1
x'EX((1,))
s.t. || 21100 S.
where | oloo denotes the loo norm, ||- denotes the le norm and e E RR a small positive constant that is fixed to e .02 in the default Captum implementation of the metric. The rationale behind this metric is the following: a small perturbation of the input should have little impact on the model's prediction and, hence, little impact the resulting explanation well. For this reason, one typically expects low sensitivity Scns(e, x>) for an explanation that is robust to small input shifts. We simply note that the previous reasoning is debatable due to the existence of adversarial perturbations that small norm but that have large effect on the model's prediction [88].
Like our invariance and equivariance robustness metrics, Sens measures how the explanation is affected by transformation of the input. Therefore, a natural question arises: is there a difference in practice between our robustness metries and the sensitivity metric? To answer this question,
33

---

consider the setup of Section 3 with the ECG dataset and the Augmented-CNe CNN. We study feature importance methods e, for which we evaluate Sens(e, x) and Equivz(e, x) for Neett 000 test examples x E Destt We report the results in Figure 8.
1.0
Integrated Gradients
Gradient Shap
Feature Permutation
Festure Ablation
Feature Occlusion
Explamation Equivariance
0.6
0.4
0.0
-0.2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Explanation Sensitivity
Figure 8: Comparison between the sensitivity metric Sens(e, x) and our equivariance metric Equiv z). Each point represents test example x Diest extracted from the ECG dataset explained by feature importance method e. The downward trend between the two metrics weak.
As we can observe Figure 8, there exists weak downward trend between the two metries. By evaluating the Pearson correlation coefficient between the two metrics for each method, we found r -62 for Integrated Gradients, r -448 for Gradient Shap, -.38 for Feature Permutation, r" -.23 for Feature Ablation and r = -33 for Feature Occlusion. This indicates that examples with higher sensitivity tend to be associated with lower equivariance. That being said, the correlation between the two metrics is too weak to suggest any redundancy between them. We deduce that our robustness metries should be use to complement rather than replace the sensitivity metric. In practice, we believe that many forms of explanation robustness deserve to be investigated.
H Other Invariant Architectures
In Section 3, we have illustrated the flexibility of our robustness formalism by applying it to various architecture. However, geometric deep learning extends well-beyond the examples covered this paper and an exhaustive treatment of the field is beyond our scope. To suggest future extensions of our work beyond the architectures examined this paper, we provide in Table 13 other architectures that are invariant or equivariant with respect to other symmetry groups. Our formalism straightforwardly applies to most of these examples. For some others (like Spherical CNNs), an extension to infinite groups would be required. We leave this extension for future work.
Table 13: Other invariant architectures. Table partially adapted from [46].
Architecture
Symmetry Group
Reference(s)
G-CNN
Any finite group
[89]
Transformer
Permutation S(n)
[90]
LSTM
Time Warping
[91]
Spherical CNN
Rotation SO(3)
[92]
Mesh CNN
Gauge Symmetry S0(2)
[93]
E(n)-GNN
Euclidean Group E(n)
[94, 95]
34

---

I Examples of Non-robust Explanations
In this appendix, we illustrate the importance of our notion of robustness by presenting failure modes. Since images are the easiest way to visualize, we focus our attention to the FashionMNIST and STLI0 datasets. In Figures 9 and 10, show non-robust saliency maps obtained with Gradient Shap and DeepLift. As we can observe, the model prediction is largely unaffected by applying symmetry (sg |f (plg)z> æ²) (z)] in all cases), while the saliency map look qualitatively different. We have included a Jupyter Notebook in our code to easily visualize other failure modes.
Predicted Class:
Original Sallency Map
Predicted
Transformed Saliency Map
Predicted Class: T-shirt/top
Original Saliency Map
Predicted Class: T-shirt/top
Transformed Saliency Map
Predicted Class: Dress
Original Saliency Map
Predicted Class: Dress
Transformed Saliency Map
Predicted Class: Sneaker
Original Saliency Map
Predicted Class: Sneaker
Transformed Saliency Map
Predicted Class: Shirt
Original Saliency Map
Predicted Class: Shirt
Transformed Saliency Map
Figure 9: Examples of non-robust explanations obtained with Gradient Shap on the FNSH dataset. The corresponding cosine similarity ss [e (plg)æ) p(g] [())) between the saliency maps are respectively 0.06, 0.08, 0.18, 0.15 1.18 and 0.21
35

---

Predicted Class: Car
Original Saliency Map
Predicted Class: Car
Transformed Saliency Map
Predicted Class: Monkey
Original Sallency Map
Predicted Class: Monkey
Transformed Sallency Map
Predicted Class: Truck
Original Saliency Map
Predicted Class: Truck
Transformed Saliency Map
Predicted Class: Ship
Original Saliency Map
Class: Ship
Transformed Saliency Map
Predicted
Original Saliency Map
Predicted
Transformed Saliency Map
Figure 10: Examples of non-robust explanations obtained with DeepLift on the STL10 dataset. The corresponding cosine similarity [e (ellol)),pgge((()) between the saliency maps are respectively 0.66, 0.70, 0.70, 0.71 and 0.72
Effect of Distribution Shift
In this appendix, study the impact of evaluating out robustness metrics samples from distribution that differs from model's training distribution. To that end, we shall use CINIC-10 dataset [96], which contains ImageNet images that have the same classes as CIFAR-10 classes (and, hence, the same classes the STL-10 dataset). In this way, we are able to reuse the ResNets trained on STL-10 for evaluation on the CINIC-10 dataset. The only requirements is resize the CINIC-10
36

---

images so that they match the resolution of the STL-10 images ( (96 x 96 pixels). We evaluate the robustness metrics with respect to the dihedral group D4 for few feature importance and example importance methods in Tables 14 and 15. In both cases, we find that these metrics are consistent with their 10 counterpart in Figures 3(a) and 3(b).
Table 14: Equivariance of feature importance methods computed CINIC-10.
Method
Equivariance Deo-Dennioo (e,
Integrated Gradients
.87
DeepLift
.85
Gradient Shap
.81
Table 15: Invariance of example importance methods computed on CINIC-10.
Method
Invariance Ex-Deuxccu Inve, [e,
TracIN
Influence Functions
.99
SimplEx Equiv
SimplEx Inv
.87
Representation Similarity Equiv
Representation Similarity Inv
.98
37


Reviews:
[{"review_id": "I5ledHfmWW", "reviewer": "Reviewer_7REw", "summary": "In this paper, the authors propose a set of desiderata for explanation methods for neural networks ranging from CNNs to GNNs. They postulate that any explanation that is able to faithfully explain the model should be in agreement with the invariance properties exhibited by the underlying model. They formalize this idea by introducing explanation invariance and equivariance with reference to specific symmetry groups. Using this formulation, they derive metrics that measure robustness of several interpretability methods and some theoretical guarantees. Their experiments verify explanation robustness for models trained on diverse modalities such as images, time series and tabular data. They provide guidelines for developers to develop robust model explanations using their metrics.", "questions": "1. Can the metrics be extended to approximately-invariant/equivariant networks? What are the pros and cons of such an extension?\n2. Are invariance and equivariance the only properties that need to be considered for explanations from a geometric perspective?", "limitations": "Please refer to weaknesses above. There is not much potential for a negative societal impact - evaluating explanations would only serve to impact society positively.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Evaluating explanation approaches is an important area of research not only for evaluating existing approaches but also to aid practitioners in developing new explanations that are robust. \n\nThe paper is written very clearly albeit a bit dense to parse – this does not take away from the general reader’s experience of the paper, but it could use a more exemplar way of introducing concepts (Specific comments below)\n\nThe explanation methods and the models tested encompass several modalities. This is a real strength of the paper as explanation evaluations are usually limited to analysis of salience maps.", "weaknesses": "1. Dense nature of the writing: While I appreciate the authors’ efforts in introducing the concepts of geometric priors and group symmetry to the readers, the paper could be more clear in explaining what transformations are being considered in different symmetry groups. For instance, what transformations are contained in the D8 symmetry group for the CNN based examples? A table to this effect can help the readers understand what transformations are being evaluated.\n2. Example results (positive and negative): The discussion of the results is rather pedantic, examples of why improving invariance and equivariance would make sense given an invariant model would help the reader understand the importance of the metrics. Without this context, it is hard to differentiate, say, an invariance score of 0.9 to a 0.5.\n3. Why only exact invariance and equivariance? I wonder if in its current form the metrics are too rigid. As the authors mention in the paper, most networks for real world problems are only approximately invariant or equivariant. In fact, characterizing such property is a hard problem in itself. In these cases, it’s hard to see how the current approach can help. \n4. This approach is limited in that the only transformations that are considered are geometric in nature. While this is alluded to in the appendix, more real world examples of invariance are from non-trivial corruptions such as measurement error or signal degradation. \n5. The paper presents no conclusion and more discussion + visualization of the results is needed, especially in cases where the metrics fail to capture the expected behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer 7REw\n\nWe would like to thank the reviewer for taking the time to make encouraging comments and constructive criticisms. By following the reviewer's suggestions, we were able to:\n\n* Clarify the various symmetry groups appearing in the experiments.\n* Better contextualize the raw values of our robustness metrics.\n* Emphasize that our metrics can be used with models that are not perfectly invariant.\n* Stretch the applicability of our framework even further with NLP applications.\n* Add a conclusion to the manuscript.\n\nWe believe that all of these points make a great addition to the manuscript. We hope that they will also address any residual doubt the reviewer had about the paper. If that is not the case, we are happy to engage during the discussion phase. \n\n## Clear explanations for symmetry groups\n\nPlease refer to the same section in the global rebuttal.\n\n## More context to the results\n\nWe agree with the reviewer that raw invariant equivariance/invariance scores tell a limited part of the story. In order to gain a better intuition of the effect of increasing those scores, we propose to analyze the examples included in *Appendix I*. \n\nBy looking at *Figure 8* from *Appendix I*, we observe that the saliency maps between the original and transformed images highlight completely different pixels. For instance, let us look at the last row of *Figure 8*. The saliency map highlights the bottom of the shirt on the original image.  As we can see, moving this shirt to the upper-left corner shifts the saliency map, which now focuses the attention on the left part of the shirt. In all the images from *Figure 8*, the equivariance score of the saliency map is particularly low (ranging from .06 to .21).\n\nTo contrast with *Figure 8*, let us now consider explanations with higher equivariance scores. In *Figure 9*, the equivariance scores for the saliency map are substantially higher (ranging from .66 to .72). By comparing the saliency maps for the original and transformed images, it is noticeably more difficult to spot their differences. We do see some differences (e.g. the saliency map of row 4 highlights the boat cockpit more after flipping the boat upside down), but those are more subtle than in *Figure 8*. \n\nIn conclusion, this qualitative analysis supports the fact that our robustness metrics are a good proxy to measure how the symmetry changes the explanation. In particular, increasing the equivariance/invariance metric makes it more difficult to distinguish between the explanations for the original and transformed inputs.  \n\n## Beyond exact invariance and equivariance\n\nPlease refer to the same section in the global rebuttal.\n\n## Limited geometric nature of the transformations\n \nWhile our work indeed focuses on a geometric characterization of interpretability robustness, we do not believe that this limitation is very restrictive. In fact, all the models that fit in the geometric deep learning framework can benefit from our robustness metrics. As suggested in *Appendix H* of our paper, the equivariance property extends well beyond the CNNs, GNNs and Deep Sets described in our experiments. We have collected some examples in the below table.  For detailed references, please refer to *Appendix H* from our paper. \n\n | **Architecture**   | **Symmetry Group**  | \n|---|---|\n| G-CNN  | Any Finite Group  |  \n| Transformer  | Permutation S(n)  |\n| LSTM  | Time Warping  | \n| Spherical CNN  |  Rotations SO(3) | \n| Mesh CNN  | Gauge Symmetry SO(2)  |  \n| E(n)- GNN  | Euclidean Group E(n)  | \n\nAll of these examples demonstrate that group invariance/equivariance is a valuable inductive bias that permits to achieve state-of-the-art performances in various tasks. Furthermore, this property seems to be the main focus of the geometric deep learning literature. We do not rule out other geometric properties that might be interesting for interepretability, however none has received as much attention as invariance/equivariance. \n\nTo show the wide applicability of our framework, we have conducted additional experiments with models processing text data. Please refer to the *NLP applications* part of the global rebuttal.\n\n## No conclusion\n\nWe agree with the reviewer that a conclusion is missing from the main paper. We have summarized the main takeaways of our work in *Appendix A*, where we present all the guidelines that we have distilled from our experiments in a unified flowchart. In addition to this, we have added the following two conclusion paragraphs to the main paper.\n\nBuilding on recent developments in geometric deep learning, we introduced two metrics (explanation invariance and equivariance) to assess the faithfulness of model explanations with respect to model symmetries. In our experiments, we considered a wide range of models whose predictions are invariant with respect to transformations of their input data. By analyzing feature importance, example importance and concept-based explanations of these models, we observed that many of these explanations are not invariant/equivariant to these transformations when they should. This led us to establish a set of guidelines to help practitioners choose interpretability methods that are consistent with their model symmetries.  \n\nBeyond actionable insights, we believe that our work opens up interesting avenues for future research. An important one emerged by studying the equivariance of saliency maps with respect to models that are approximately invariant. This analysis showed that state-of-the-art saliency methods fail to keep a high equivariance score when the model's invariance is slightly relaxed. This important observation could be the seed of future developments of robust feature importance methods. "}, {"review_id": "04JeLGUrij", "reviewer": "Reviewer_NZiw", "summary": "The core contribution of this work is the direction of measuring explanation robustness through a more broader set of data perturbations/transformations for different data modalities which have not been discussed in literature thus far. For instance shift transformations in images, cyclic translations in time series, etc. The authors define concepts of explanation invariance \nand equivariance and expose their theoretical properties. They conduct experiments to measure these types of invariances for 3 different types of explanations and several datasets and also present guidelines on which invariances to measure in which use cases. ", "questions": "- Is there any reason why text models were not evaluated? It could make sense in tasks for e.g. summarization, etc. if certain facts are shifted from the beginning to the end of a paragraph and if the importance score on words still come up right. \n\n- In your experiments are you assuming that the models you pick are already robust to the type of perturbations you add to the data to measure explanation robustness(invariance/equivarance)? Or is explanation invariance being measured irrespective of whether the model is robust or not.\n\n", "limitations": "See above. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Overall, making explanations more trustworthy is an important area to investigate. Measuring robustness of explainability methods is valuable for their use in practical deployed systems. \n\n- Authors define several types of perturbations for different data modalities. Authors define the concepts of invariance and equivariance for different data modalities and show their theoretical properties. \n\n- Authors have presented a broad collection of experiments with several data modalities and explainability types to show that many existing explainability methods are not robust as measured via invariance & equivariance. ", "weaknesses": "There seem to be links between model robustness and explanation robustness which are not explored in the draft. \nFor example, if the model is not robust to certain types of suggested data perturbations (when we might expect it to), then would it not make sense to measure if the explanations are not in fact the same? Because the same top reason/explanation why a data sample was classified in class A cannot also be the reason why its not classified in class A. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer NZiw\n\nWe would like to thank the reviewer for taking the time to make encouraging comments and constructive criticisms. By following the reviewer's suggestions, we were able to:\n\n* Show that our metrics can also be used to characterize explanations of models that are not invariant to a group of symmetry.\n* Stretch the applicability of our framework even further with NLP applications.\n* Clarify the robustness assumptions that are made with the models appearing in our experiments.\n\nWe believe that all of these points make a great addition to the manuscript. We hope that they will also address any residual doubt the reviewer had about the paper. If that is not the case, we are happy to engage during the discussion phase. \n\n## (Anti)-robustness for non invariant models\n\nWe thank the reviewer for bringing this point. It is indeed true that if a model prediction changes substantially by applying a transformation to the input, one should expect the same for an explanation of this model. While this analysis is not explicitly mentioned in the paper, it can be deduced from *Figure 4*.\n\nBy looking at the Standard CNNs (*green icons* in *Figure 4*), we notice that these models have low invariance for both the Electrocardiogram and FashionMNIST datasets. This implies that the model predictions change substantially when we apply a translation to their input data. Therefore, it is legitimate to expect the same for their explanation. \nThis is indeed what we observe for all the feature importance methods (*left column* of *Figure 4*), whose equivariance is also close to zero. \n\nSome of the example and concept-based methods, on the other hand, keep a high invariance even if the model is not invariant (see *central* and *right columns* of *Figure 4*). This is the case of TraceIN, Representation Similarity and CAR, for instance. This implies that the explanations from these methods don't change substantially when we apply a translation to the input data. This is in contradiction with the fact that the model prediction does change substantially when we apply this translation. Therefore, in spite of the apparent robustness of these methods when used with an invariant model, we observe that these methods fail to track the model's behavior when invariance is destroyed.\n\n## NLP applications\n\nPlease refer to the same section in the global rebuttal.\n\n## Robustness assumptions\n\nAll the models manipulated in *Sections 3.1* and *3.2* are perfectly invariant to their respective symmetry group. This is guaranteed theoretically by their architecture (e.g. purely convolutional CNN classifiers are translation-invariant, GNN classifiers are permutation-invariant). \n\nFurthermore, we have verified empirically the invariance of each classifier $f$ by measuring the  the invariance score $\\mathrm{Inv}\\_{\\mathcal{G}}(f, x) = \\frac{1}{|\\mathcal{G}|}\\sum\\_{g \\in \\mathcal{G}} \\cos[f(x), f(\\rho[g]x)]$ defined in *Appendix F* of our paper. Note that $f(x)$ denotes the logits predicted by the classifier $f$ and *not* the hard-label prediction of $f$ (i.e. the $\\mathrm{argmax}$ of $f(x)$). For all models in *Sections 3.1* and *3.2*, we have observed that $\\mathrm{Inv}\\_{\\mathcal{G}}(f, x) = 1$ for all $x$ in the test set.\n\nWe note that the only non-invariant models manipulated in our papers are described in *Section 3.3*. In each case, the average invariance $\\mathbb{E}\\_{x \\sim \\mathcal{D}\\_{\\mathrm{test}}}\\mathrm{Inv}\\_{\\mathcal{G}}(f, x)$ is reported on the x-axis of *Figure 4*."}, {"review_id": "5v5so652Ic", "reviewer": "Reviewer_u2n8", "summary": "This paper proposes the definition of the robustness of explanations with respect to the model symmetry group. For models invariant to some symmetry group, the explanation should also be invariant or equivariant to it. The paper derives two metrics to measure the invariance and equivalence of explanations and analyzes the robustness requirement for three explanation methods (e.g., feature importance needs to be equivariant, and example importance and concept-based explanations need to be invariant). It also theoretically analyzes the guarantees of different methods in robustness. In the end, the paper proposes to improve the robustness by aggregating the explanations over several symmetries. Experiments show that different explanation methods have different invariance/equivariance properties.", "questions": "1. The paper only considers the post-hoc interpretability. Is it possible to extend the concept of explanation invariance/equivariance to other methods? For example, the attention should be equivariant under input translation?\n3. When the explanation space is not identical to the input space, what is group representation $\\rho'$?\n4. Is there any trade-off between the robustness and the utility of the explanation?\n\n\nMinor:\n1. The colors in Figure 2 are hard to distinguish.\n2. What is \"Dihedral Group\" in Table 2?", "limitations": "The proposed definitions and evaluations heavily rely on the assumption that the model is perfectly invariant. I appreciate the experiments and discussions in sec 3.3 which suggest that there's no linear relationship between model invariance and explanation invariance/equivariance. Therefore, models not designed but trained for invariance can not use the proposed methods. However, this assumption limits the model architectures and group transformations considered and thus limits the application of the proposed methods.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is very well-written and easy to follow.\n2. The relevant literature is summarized comprehensively. The considered problem, evaluating the robustness of explanation from a geometry perspective, is not well-studied before.\n3. The concepts of explanation invariance and equivaraince are novel. The evaluation metrics are sound.\n4. Multiple explanation methods are evaluated in the experiments.", "weaknesses": "1. The biggest weakness is the limited scenarios that can use the proposed evaluation methods. The paper only considers models that are perfectly invariant. Please see Limitations for details.\n2. The requirement of the model's invariance is not well defined. Under some group transformations, while the hard-label prediction of the model (i.e. after argmax) is unchanged, the soft-label prediction (i.e. after softmax but before argmax) might be changed. The model is invariant from the first perspective. However, from the second perspective, the model is non-invariant and the explanation should also be non-invariant/non-equivariant. I guess that is why the saliency map is not equivariant while the model is considered invariant in Appendix I.\n3. The symmetry groups considered in experiments are limited.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer u2n8\n\nWe would like to thank the reviewer for taking the time to make encouraging comments and constructive criticisms. By following the reviewer's suggestions, we were able to:\n\n* Stretch the applicability of our framework with models that are not perfectly invariant and NLP applications.\n* Discuss how our formalism could be applied beyond post-hoc interpretability.\n* Illustrate how to choose appropriate representations when input and explanation spaces don't match.\n* Discuss trade-offs between robustness and usefulness of explanations.\n\nWe believe that all of these points make a great addition to the manuscript. We hope that they will also address any residual doubt the reviewer had about the paper. If that is not the case, we are happy to engage during the discussion phase. \n\n## Limited use cases\n\nWe address the reviewer's main concern on the limitations of our work in the global rebuttal in two ways.\n\n1. In section *Beyond exact invariance and equivariance*, we explain how to use our metrics for models that are not perfectly invariant. \n2. In section *NLP applications*, we show theoretically and empirically how our framework can be used with text data.\n\n## Model invariance requirement\n\nAll the models manipulated in *Sections 3.1* and *3.2* are perfectly invariant to their respective symmetry group and in the first perspective described by the reviewer. This is guaranteed theoretically by their architecture (e.g. purely convolutional CNN classifiers are translation-invariant, GNN classifiers are permutation-invariant).\n\nFurthermore, we have verified empirically the invariance of each classifier $f$ by measuring the  the invariance score $\\mathrm{Inv}\\_{\\mathcal{G}}(f, x) = \\frac{1}{|\\mathcal{G}|} \\sum\\_{g \\in \\mathcal{G}} \\cos[f(x), f(\\rho[g]x)]$ defined in *Appendix F* of our paper. Note that $f(x)$ denotes the logits predicted by the classifier $f$ and *not* the hard-label prediction of $f$ (i.e. the $\\mathrm{argmax}$ of $f(x)$). For all models in *Sections 3.1* and *3.2*, we have observed that $\\mathrm{Inv}\\_{\\mathcal{G}}(f, x) = 1$ for all $x$ in the test set.\n\nWe conclude that the differences between the original and transformed saliency maps in e.g. *Figure 8* of *Appendix I* can only be explained by the lack of robustness of the underlying feature importance method (GradientShap in this case).\n\n## Limited symmetry groups\n\nOur paper includes a total of 4 symmetry groups acting on 6 different datasets. Those are summarized in the section *Clear explanations for symmetry groups* of the global rebuttal.\n\nWe believe that this is a reasonable number of groups to demonstrate the wide applicability of our framework. That said, if the reviewer has noticed an important symmetry group that is missing in our analysis, we will happily include it in the manuscript.\n\n## Extension beyond post-hoc interpretability\n\nIt is indeed possible to use our framework to characterize the invariance/equivariance of methods beyond post-hoc interpretability.\n\nIn particular, everything that is stated in our theoretical formulation  in *Section 2.2* remains true if we set the explanation method $e$ to be the attention scores outputted by an attention head $e(x) = \\mathrm{softmax}(x^\\intercal \\cdot W_E^\\intercal W_Q^\\intercal W_K W_E \\cdot x)$, where $W_E, W_Q, W_K$ are respectively the token embedding, query and key matrices. Note that one should *not* expect the attention scores to be permutation-equivariant if the embedding encodes positional information. \n\n## Representation for non-identical input spaces\n\nWhen the explanation space $\\mathcal{E}$ is different from the input space $\\mathcal{X}(\\Omega, \\mathcal{C})$, there is no general strategy to pick a group representation $\\rho'$ acting on $\\mathcal{E}$. This group representation should therefore be selected on a case-by-case basis by leveraging domain knowledge.\n\nLet us give a concrete example of such scenario to illustrate a typical representation selection process. Let us consider RGB images, where the input space is $\\mathcal{X}(\\Omega, \\mathcal{C})$ with a grid domain $\\Omega = \\mathbb{Z}_W \\times \\mathbb{Z}_H$ and the chanel space is $\\mathcal{C} = \\mathbb{R}^3$. We consider feature importance explanations that outputs a segmentation mask, hence corresponding to an explanation space $\\mathcal{E}  =\\mathcal{X}'(\\Omega, \\mathcal{C}')$ with $\\mathcal{C}' = \\mathbb{R}$. As we can see, the input and explanation spaces are distinct in this case since $\\mathcal{C} \\neq \\mathcal{C}'$. \n\nLet us consider the translation group as the symmetry group $\\mathcal{G} = \\mathcal{G}\\_{\\mathrm{transl}}$. Typically, the translation group will act independently on each of the 3 chanels of the input image. Therefore, we the original group representation $\\rho$ acting on $\\mathcal{X}(\\Omega, \\mathcal{C})$ contains 3 copies of a representation $\\rho\\_{\\mathrm{transl}}$, each acting on a single chanel. This is formally written as $\\rho = \\rho\\_{\\mathrm{transl}} \\oplus \\rho\\_{\\mathrm{transl}} \\oplus \\rho\\_{\\mathrm{transl}}$, where $\\oplus$ denotes the direct sum of representations. \n\nNow let us explain how to choose an appropriate representation $\\rho'$ for the explanation space $\\mathcal{E}$. As we have explained above, the segmentation mask contains only one chanel as $\\dim(\\mathcal{C}') = 1$. In this case, it is therefore natural to directly use the representation $\\rho\\_{\\mathrm{transl}}$ acting on a single chanel. Hence, we simply set $\\rho' = \\rho\\_{\\mathrm{transl}}$.\n\n## Trade-off between robustness and utility of the explanation\n\nRather than a trade-off, we believe that robustness is a necessary condition for the utility of an explanation. When using an invariant model, any explanation that is not invariant/equivariant should be treated with skepticism, as it does not faithfully capture the model symmetries. Believing such explanations could lead to the conclusion that the model is affected by applying the symmetry to the image, which is incorrect."}, {"review_id": "r2sCV4w0Bi", "reviewer": "Reviewer_LQbZ", "summary": "This paper study the robustness of several post-hoc interpretability methods against the transformation of input data. The robustness is measured by invariance and equivariance metrics. Theoretical robustness guarantees and a systematic approach to increase the invariance are derived. Finally, the authors conduct extensive experiments to validate the theoretical analysis of robustness guarantees, using the proposed evaluation metrics.  5 actionable guidelines are derived to improve the robustness.", "questions": "1. Please provide more details of the symmetry group. How does the change of symmetry group affect the robustness evaluation results?\n\n2. What's challenges when applying proposed evaluation method to real world dataset and interpreter?", "limitations": "Please refer to the weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem studied in this paper is interesting and novel. Robustness of interpretability against more general input transformation is few studied before.\n\n2.  This paper extends the robustness evaluation to other interpretability methods, like example importance and concept-based explanations. These are missing from the current literature.\n\n3.  Based on the evaluation, the theoretical robustness guarantees are derived for the popular interpretability methods.\n\n4. The experiment is comprehensive and insightful that 5 practical guidelines are derived for the robustness improvement.", "weaknesses": "1. The details about the symmetries are vague and even missing. Which kind of input transformation are considered, rotation, crop, or translation?\n\n2. Since the invariance metric is in conflict with the equivariance metric, it may be confusing for users to decide which metric is more suitable, given the symmetries. For example, if the small perturbations are added into the input data, the invariance of feature explanation is expected.\n \n3. The simple Monte Carlo Sampling is inefficient for the evaluation of two robustness metrics, especially for the rare events.\n\n4. The ImageNet dataset should be considered for the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer LQbZ\n\nWe would like to thank the reviewer for taking the time to make encouraging comments and constructive criticisms. By following the reviewer's suggestions, we were able to:\n\n* Discuss how to choose between equivariance and equivariance when measuring robustness.\n* Emphasize our evaluation of the quality of Monte Carlo estimators.\n* Extend our evaluation to a subset of ImageNet images.\n\nWe believe that all of these points make a great addition to the manuscript. We hope that they will also address any residual doubt the reviewer had about the paper. If that is not the case, we are happy to engage during the discussion phase. \n\n## Details about symmetries \n\nPlease refer to the section *Clear explanations for symmetry groups* of the global rebuttal for details on each symmetry group.\n\nBy looking at *Figures 2.a,b,c* from the manuscript, we can observe the effect of changing the symmetry group on the robustness evaluation results. Indeed, different datasets are associated with different symmetry groups. We observe small oscillations in the robustness metrics from one dataset to the other. However, our observations apply to all symmetry groups. \n\n\n## Invariance vs equivariance\n\nChoosing which metric, between invariance and equivariance, \nto record is context dependant and requires domain knowledge. A good rule of thumb is the following: whenever the explanation space $\\mathcal{E}$ is equal to the input space $\\mathcal{X}(\\Omega, \\mathcal{C})$, we choose the equivariance metric. This is naturally the case for feature importance methods, but also for counterfactual explanations. On the other hand, whenever the explanation space $\\mathcal{E}$ is a generic vector space without a signal interpretation (i.e. the vectors of $\\mathcal{E}$ cannot be interpreted as signals mapping a domain $\\Omega$ to a chanel space $\\mathcal{C}$), we choose invariance. This is the case for concept-based and example-based explanations. \n\nLet us now discuss the small additive perturbations mentioned by the reviewer. Adding a small perturbation to the input data is not a symmetry corresponding to a linear action $\\rho$ of a symmetry group $\\mathcal{G}$. For this reason, the equivariance/invariance characterization deployed in our work is not the suitable framework to describe this type of transformation. We note that previous works have studied the robustness of explanations with respect to these small perturbation of the input data (see e.g. the sensitivity metric described in *Appendix G*). \n\nThe geometric robustness studied in our work has to be considered as orthogonal and complementary with respect to this perturbation robustness. In particular, a quantitative study in *Appendix G* shows that our equivariance metric has only weak correlations with respect to the sensitivity metric. We conclude that the two forms of robustness can exist independently in practice. Any feature importance explanation that is faithful to the model should be robust with respect to these two criteria (i.e. low sensitivity to small additive perturbations, high equivariance to symmetries). \n\n## Monte Carlo sampling\n\nWe would like to emphasize that the convergence of Monte Carlo estimators has been studied throroughly in *Appendix E*. In particular, we show that our estimators have converged with the sample sizes considered in the experiments from *Section 3*.\n\nFurthermore, we are not entirely sure what the reviewer refers to when they mention *rare events*. We believe that the reviewer might refer to sampling from the symmetry group $\\mathcal{G}$. If this is indeed the case, this sampling process does not admit rare events by definition. Indeed, as explained in *Section 2.2*, the Monte Carlo estimators are built by sampling *uniformly* elements from the symmetry group $g \\sim U(\\mathcal{G})$. \n\nWe hope that the above discussion brings more clarity to the Monte Carlo sampling. That said, it is perfectly possible that we misunderstood the reviewer's remark. In this case, we would be happy to extend the discussion on this subject.\n\n##  ImageNet experiment \n\nGiven the limited amount of time, retraining a model on ImageNet is infeasible. As a compromise, we have reproduced the experiments from *Section 3.1* with the CINIC-10 dataset. \n\nThe CINIC-10 dataset contains ImageNet images that have the same classes as CIFAR-10 classes (and, hence, the same classes as the STL-10 dataset). In this way, we are able to reuse the ResNet trained on STL-10 for evaluation on the CINIC-10 dataset. The only requirements is to resize the CINIC-10 images so that they match the resolution of the STL-10 images ($96 \\times 96$ pixels). We first evaluate the equivariance score $\\mathrm{Equiv}\\_{\\mathbb{D}_4}$ with respect to the dihedral group $\\mathbb{D}_4$ for a few feature importance methods and find the following averages over the test set $\\mathcal{D}\\_{\\mathrm{CINIC10}}$: \n\n| Method $e$ | $\\mathbb{E}\\_{x \\sim \\mathcal{D}\\_{\\mathrm{CINIC10}}}\\mathrm{Equiv}\\_{\\mathbb{D}_4}[e, x]$ |\n|---|---|\n| Integrated Gradients  | .87  |   \n| DeepLift  | .85  |   \n| Gradient-Shap  |  .81 |   \n\n This is consistent with the results reported in the main paper for the STL-10 dataset (*Figure 2.a*). We perform a similar analysis for example importance methods:\n\n| Method $e$ | $\\mathbb{E}\\_{x \\sim \\mathcal{D}\\_{\\mathrm{CINIC10}}}\\mathrm{Inv}\\_{\\mathbb{D}_4}[e, x]$ |\n|---|---|\n| TracIN  | .99  |   \n| Influence Functions  | .99  |   \n| SimplEx Equiv |  .83 | \n| SimplEx Equiv |  .87 | \n| Representation Similarity Equiv  |  .64 | \n| Representation Similarity Inv  |  .98 | \n\nAgain, this is consistent with the results reported for the STL-10 dataset (*Figure 2.b*).\n\n## Challenges in applying the evaluation\n\nWe did not encounter any significant challenge in the evaluation of our metrics with real-world datasets and interpreters. This is because the computation of our metrics is easy to implement once the interpreters and models are accessible."}, {"review_id": "N7hb8CQFe7", "reviewer": "Reviewer_wVxe", "summary": "The paper focuses on the robustness of explanations. It begins by defining explanation invariance and equivariance concepts using geometric deep learning formalism and demonstrates that certain popular interpretability methods inherently possess theoretical robustness guarantees. Two metrics, invariance and equivariance scores, are introduced for empirically assessing explanation robustness. These metrics are applied to evaluate various interpretability methods across different modalities. Finally, the paper provides a set of five actionable guidelines to ensure that interpretability methods are employed in a manner that guarantees robustness.", "questions": "1. Could the authors provide further discussion and explanations regarding the results in Table 1, especially for those that are conditionally guaranteed?\n\n2. In the experimental section, what does the Dihedral Group represent for CIFAR10 and STL10? Could the authors share an example of the specific transformations applied to images in this scenario?\n\n3. While there are numerous publications on the explanations for NLP, the paper does not mention the robustness of explanations for NLP. Is it possible to apply the proposed framework to evaluate the robustness of these methods within the NLP domain?", "limitations": "See weaknesses.\n\nIn general, I think the paper tackles an important research question in XAI and proposes a valuable framework to address the issue. However, the concerns mentioned in the \"weaknesses\" section, such as missing or unclear information, may lead to confusion and make it difficult for readers to fully understand the paper.\n\nIf these related questions can be well answered, I may consider raising my rating. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a high-level framework for evaluating the robustness of explanations and introduces two corresponding metrics. Unlike previous work that mostly focuses on saliency-based explanations for image classification, this framework can be applied to various explanation methods (feature-based, concept-based, and example-based) and modalities (images, graphs, and time series);\n\n2. In addition to offering an evaluation framework, the authors also provide guidelines for generating robust explanations. These insights can assist the community in developing improved explanation methods.", "weaknesses": "1. The paper's organization could be better aligned with the summaries of contributions provided in the abstract and on page 3. The structure in later sections does not closely follow these summaries, which may make it difficult for readers to follow the narrative.\n\n\n2. The paper appears to cover many points, potentially leading to the omission of important details. \n\nI am interested in understanding how different explanation methods relate to theoretical robustness guarantees (invariant, equivariant), but the paper only presents the results (Table 1) without discussing them. Although the mathematical proofs are available in Appendix D, the main paper lacks an explanation and discussion of these results. For instance, while Table 1 indicates that gradient-based methods have conditional equivariance guarantees, the necessary conditions or assumptions are not explicitly stated. Including explanations or discussions in the main paper would significantly improve clarity.\n\n3. The use of the Dihedral Group for CIFAR10 and STL10 in the experimental section is unclear. It would be helpful if the authors provided an example of the transformations applied to the images in this context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal Reviewer wVxe\n\nWe would like to thank the reviewer for taking the time to make encouraging comments and constructive criticisms. By following the reviewer's suggestions, we were able to:\n\n* Emphasize the theoretical robustness guarantees that we derived in the appendix.\n* Clarify the various symmetry groups appearing in the experiments.\n* Stretch the applicability of our framework even further with NLP applications.\n\nWe believe that all of these points make a great addition to the manuscript. We hope that they will also address any residual doubt the reviewer had about the paper. If that is not the case, we are happy to engage during the discussion phase.\n\n\n## Theoretical robustness guarantees\n\nAs mentioned by the reviewer, theoretical results are indeed detailed in *Appendix D*. This is because a rigorous treatment of these theoretical results requires further definitions and lemmas, which did not fit in the page constraint. We will now give a summary of these results with the right level of mathematical precision. All the details that are not covered here are available in *Appendix D*.\n\nWhen it comes to feature importance methods, there are mainly two assumptions that are necessary to guarantee $\\mathcal{G}$-equivariance. **(1)** The first assumption restricts the type of baseline input $\\bar{x} \\in \\mathcal{X}(\\Omega, \\mathcal{C})$ on which the feature importance methods rely. Typically, these baselines signals are used to replace ablated features from the original signal $x \\in \\mathcal{X}(\\Omega, \\mathcal{C})$ (i.e. remove a feature $x_i$ by replacing it by $\\bar{x}_i$). In order to guarantee equivariance, we require this baseline signal to be invariant to the action of each symmetry $g \\in \\mathcal{G}$: $\\rho[g] \\bar{x} = \\bar{x}$. **(2)** The second assumption restricts the type of representation $\\rho$ that can be used to describe the action of the symmetry group $\\mathcal{G}$ on the signals $\\mathcal{X}(\\Omega, \\mathcal{C})$. In order to guarantee equivariance, we require this representation to be a *permutation representation*, which means that the action of each symmetry $g \\in \\mathcal{G}$ is represented by a permutation matrix $\\rho[g]$ acting on the signal space $\\mathcal{X}(\\Omega, \\mathcal{C})$.\n\nWhen it comes to example importance methods, the assumptions depend on how the importance scores are obtained. If the importance scores are computed from the model's loss $\\mathcal{L}$, then the $\\mathcal{G}$-invariance of the explanation immediatly follows from the model's invariance. If the importance scores are computed from the model's internal representations $h: \\mathcal{X}(\\Omega, \\mathcal{C}) \\rightarrow \\mathbb{R}^{d_{\\mathrm{rep}}}$, then the invariance of the explanation can only be guaranteed if the representation map $h$ is itself invariant to action of each symmetry: $h(\\rho[g]x) = h(x)$.\n\nFinally, concept-based explanations are also computed from the model's representations $h$. Again, $\\mathcal{G}$-invariance of the explanations can only be guaranteed if the representation map $h$ is itself $\\mathcal{G}$-invariant. \n\n\n## Dihedral group\n\nPLease refer to the section *Clear explanations for symmetry groups* from the global rebuttal for a definition of the dihedral symmetry group.\n\n## NLP application \n\nPlease refer to the same section in the global rebuttal."}]

Author rebuttal:
# Global Rebuttal

The below contains a rebuttal for remarks that are common to most reviewers.

## Clear explanations for symmetry groups

In *Table 3* from *Appendix F*, we have included all the details necessary to understand the action of each symmetry group. Following the reviewer's suggestion, we will add the more intuitive table below to the manuscript:

| **Dataset**  | **Symmetry group**  | **Acting on** | **Description**   |
|---|---|---|---|
| Electrocardiograms   | 1D Translations  | Time Series  | Each translation shifts the signal in time.|
|  Mutagenicity | Permutation  | Graphs  | Each permutation changes the ordering of nodes in the graph's feature matrix and adjacency matrix. |
| ModelNet40  | Permutation  | 3D Point Cloud  | Each permutations exchanges the positions of several points in the cloud. Note that the cloud itself remains the same.  |
| FashionMNIST  | 2D Translations   | Images  | Each translation shifts the image content (i.e. the fashion item) horizontally and vertically. Thanks to the padding, the image content never touches the edges of the image. |
| CIFAR100  | Dihedral  | Images  | Each element of the dihedral group either rotates the image or takes a reflection of the image. The dihedral group $\mathbb{D}_8$ includes rotations by angles 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°. It also includes reflections with respect to axes tilted by all these angles from the horizontal.  |
| STL10  | Dihedral  | Images  | Same as above. |

In the case of images, we have included visualizations of real transformed images and explanations in *Figures 8* and *9* in *Appendix I*. We will move some of those figures to the main paper in order to facilitate the visualization of these groups (especially the dihedral group).

## Beyond exact invariance and equivariance

We would like to emphasize that our robustness metrics can still be used to characterize the equivariance/invariance of explanations *even* if the model is not perfectly invariant. The metrics would still record the same information, even if the invariance of the model is relaxed. 

The main thing to keep in mind when using these metrics with models that are not perfectly invariant is that the explanations themselves should not be exactly invariant/equivariant. That said, for a model that is approximately invariant, we expect a faithful explanation to keep a high invariance / equivariance score. This is precisely the object of *Section 3.3*.

In *Section 3.3*, we analyze the equivariance / invariance of several interpretability methods on models that are approximately invariant (two approximately $\mathbb{D}_8$-invariant ResNets for the STL10 and CIFAR100 datasets as well as two Augmented CNNs for the Electrocardiograms and FashionMNIST datasets). *Figure 4* shows that some explainability methods fail to keep a high invariance/equivariance when the invariance of the model is slightly relaxed (e.g. DeepLift, CAV). Further, we notice that no feature importance method manages to keep a high equivariance when the model's invariance is slightly relaxed. This observation could be the seed of future developments in feature importance methods. 


## NLP applications 

Our invariance and equivariance metrics can naturally be used with text models with symmetries. To demonstrate this, we consider a bag-of-word classifier. 


Let us first show that bag-of-words classifiers are invariant to the permutation group. A bag-of-words classifier $f$ receives a sequence $x = (x_t)\_{t =1}^T$ of tokens and outputs class probabilities $f(x) \in \Delta^C$, where $C$ is the number of classes and $\Delta^C$ is the $C$-probability simplex. By definition, the bag-of-word classifier can be written as a function $f(x) = g \left( \sum\_{t=1}^T \mathrm{onehot(x_t)} \right)$. In this form, the invariance of the classifier with respect to tokens permutation is manifest. Let $\pi \in S_T$ be a permutation of the token indices. Applying this permutation to the token sequence does not change the classifier's output: $f(x\_{\pi}) = g\left( \sum\_{t=1}^T \mathrm{onehot}(x\_{\pi(t)}) \right) = g \left( \sum\_{t=1}^T \mathrm{onehot}(x_t) \right) = f(x)$. We conclude that bag-of-words classifiers are $S_T$-invariant. 

Let us now show empirically that our framework extends to these models with the IMDB movie reviews dataset. This dataset contains 50k text movie reviews. Each review comes with a binary label $y \in \{0,1\}$, hence $C=2$. We represent each review as a sequence of tokens $x=(x_t)\_{t=1}^T$, where we cap the sequence length to $T=128$ and set the vocabulary size to $V = 1,000$. We perform a train-validation-test split of this dataset randomly (90%-5%-5%) and fit a 2-layers bag-of-word MLP on the training dataset for 20 epochs with Adam and a cosine annealing learning rate. The best model (according to validation accuracy) achieves a reasonable 86% accuracy on the test set $\mathcal{D}\_{\mathrm{test}}$. 

We use the test set to verify that the resulting model $f$ is *perfectly* invariant to token permutations. We then evaluate the equivariance score $\mathrm{Equiv}$ for a few feature importance methods and find the following averages over the test set:

| Method $e$ | $\mathbb{E}\_{x \sim \mathcal{D\}_{\mathrm{test}}}\mathrm{Equiv}\_{S_T}[e, x]$ |
|---|---|
| Integrated Gradients  | 1.0  |   
| DeepLift  | 1.0  |   
| Gradient-Shap  |  .85 |   

This is perfectly consistent with the results reported for other models in the paper. We perform a similar analysis for example importance methods:

| Method $e$ | $\mathbb{E}\_{x \sim \mathcal{D}\_{\mathrm{test}}}\mathrm{Inv}\_{S_T}[e, x]$ |
|---|---|
| TracIN  | 1.0  |   
| Influence Functions  | 1.0  |   
| SimplEx  |  1.0 | 
| Representation Similarity  |  1.0 | 

We note that even representation-based methods are perfectly invariant in this case. This indeed makes sense, as the representation of bag-of-words models are themselves perfectly invariant (and not equivariant).

Meta-review: This paper proposes a way of stress-testing explanation methods for models that have invariant behavior. All reviewers agree that the evaluation of explanation methods is a useful direction, and that the paper is sound and readable. The authors also adequately responded to specific technical concerns raised by reviewers. Other concerns from the reviewers centered on the dense writing and the limited applicability. In my opinion, neither of these undermines the core and novel idea of the paper. Having said that, the authors should build on the reviewers' questions and put more effort into trying to smooth out the presentation.
Decision: Accept (poster)