Title: Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance
Venue: NeurIPS 2023
Accepted: yes
Keywords: interpretability

explainability

robustness

invariance

equivariance

geometric deep learning
Abstract: Interpretability methods are valuable only if their explanations faithfully describe the explained model. In this work, we consider neural networks whose predictions are invariant under a specific symmetry group. This includes popular architectures, ranging from convolutional to graph neural networks. Any explanation that faithfully explains this type of model needs to be in agreement with this invariance property. We formalize this intuition through the notion of explanation invariance and equivariance by leveraging the formalism from geometric deep learning. Through this rigorous formalism, we derive (1) two metrics to measure the robustness of any interpretability method with respect to the model symmetry group; (2) theoretical robustness guarantees for some popular interpretability methods and (3) a systematic approach to increase the invariance of any interpretability method with respect to a symmetry group. By empirically measuring our metrics for explanations of models associated with various modalities and symmetry groups, we derive a set of 5 guidelines to allow users and developers of interpretability methods to produce robust explanations.

Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariance

Abstract

Interpretability methods are valuable only if their explanations faithfully describe the explained model. In this work, we consider neural networks whose predictions are invariant under a specific symmetry group. This includes popular architectures, ranging from convolutional to graph neural networks. Any explanation that faithfully explains this type of model needs to be in agreement with this invariance property. We formalize this intuition through the notion of explanation invariance and equivariance by leveraging the formalism from geometric deep learning. Through this rigorous formalism, we derive (1) two metrics to measure the robustness of any interpretability method with respect to the model symmetry group; (2) theoretical robustness guarantees for some popular interpretability methods and (3) a systematic approach to increase the invariance of any interpretability method with respect to a symmetry group. By empirically measuring our metrics for explanations of models associated with various modalities and symmetry groups, we derive a set of 5 guidelines to allow users and developers of interpretability methods to produce robust explanations.

1 Introduction

With their increasing success in various tasks, such as computer vision [1], natural language processing [2] and scientific discovery [3], deep neural networks (DNNs) have become widespread. State of the art DNNs typically contain millions to billions parameters and, hence, it is unrealistic for human users to precisely understand how these models issue predictions. This opacity increases the difficulty to anticipate how models will perform when deployed [4]; distil knowledge from the model [5] and gain the trust of stakeholders in high-stakes domains [6, 7]. To address these shortcomings, the field of interpretable machine learning has received increasing interest [8, 9]. There exists mainly 2 approaches to increase model interpretability [10, 11]. (1) Restrict the model's architecture to intrinsically interpretable architectures . A notorious example is given by self-explaining models , such as attention models explaining their predictions by highlighting features they pay attention to [12] and prototype-based models motivating their predictions by highlighting related examples from their training set [13-15]. (2) Use post-hoc interpretability methods in a plug-in fashion after training the model. The advantage of this approach is that it requires no assumption on the model that we need to explain. In this work, we focus on several post-hoc methods: feature importance methods (also known as feature attribution or saliency) that highlight features the model is sensitive to [16-20]; example importance methods that identify influential training examples [21-23] and concept-based explanations that exhibit how classifiers relate classes to human friendly concepts [24, 25].

With the multiplication of interpretability methods, it has become necessary to evaluate the quality of their explanations [26]. This stems from the fact that interpretability methods need to faithfully describe the model in order to provide actionable insights. Existing approaches to evaluate the quality

of interpretability methods fall in 2 categories [5, 27]. (1) Human-centred evaluation investigate how the explanations help humans (experts or not) to anticipate the model's predictions [28] and whether the model's explanations are in agreement with some notion of ground-truth [29-31]. (2) Functionality-grounded evaluation measure the explanation quality based on some desirable properties and do not require humans to be involved. Most of the existing work in this category measure the robustness of interpretability methods with respect to transformations of the model input that should not impact the explanation [32]. Since our work falls in this category, let us now summarize the relevant literature.

Related Works. [33] showed that feature importance methods are sensitive to constant shifts in the model's input. This is unexpected because these constant shifts do not contribute to the model's prediction. Building on this idea of invariance of the explanations with respect to input shifts, [34-36] propose a sensitivity metric to measure the robustness of feature importance methods based on their stability with respect to small perturbations of the model input. By optimizing small adversarial perturbations, [37-39] show that imperceptible changes in the input can modify the feature importance arbitrarily by approximatively keeping the model prediction constant. This shows that many interpretability methods, as neural networks, are sensitive to adversarial perturbations. Subsequent works have addressed this pathologic behaviour by fixing the model training dynamic. In particular, they showed that penalizing large eigenvalues of the training loss Hessian with respect to the inputs make the interpretations of this model more robust with respect to adversarial attacks [40, 41]. To the best of our knowledge, the only work that discusses the behaviour of explanations under more general transformations of the input data is [42]. However, the work's focus is more on model regularization rather than on the evaluation of post-hoc interpretability robustness.

Motivations. In reviewing the above literature, we notice 3 gaps. (1) The existing studies mostly focus on evaluating feature importance methods. In spite of the predominance of feature importance in the literature [43], we note that other types of interpretability methods exist and deserve to be analyzed. (2) The existing studies mostly focus on images. While computer vision is undoubtedly an interesting application of DNNs, it would be interesting to extend the analysis to other modalities, such as times series and graph data [44]. (3) The existing studies mostly focus on simple transformation of the model input, such as small shifts. This is motivated by the fact that the predictions of DNNs are mostly invariant under these transformations. Again, this is another direction that could be explored more thoroughly as numerous DNNs are also invariant to more complex transformation of their input data. For instance, graph neural networks are invariant to permutations of the node ordering in their input graph [45]. Our work bridges these gaps in the interpretability robustness literature.

(2) Invariant Explanation

(3) Equivariant Explanation

/warning_sign

Prediction

f

(

x

) =

f

(

ρ

[

g

]

x

)

Input

x

New Input

ρ

[

g

]

x

ρ

[

g

]

Symmetry

f

Model

Top Examples

Explanation

e

(

x

) =

e

(

ρ

[

g

]

x

)

Input

x

New Input

ρ

[

g

]

x

ρ

[

g

]

Symmetry

e

Input

x

New Input

ρ

[

g

]

x

Explanation

e

(

x

)

New Explanation

e

(

ρ

[

g

]

x

) =

ρ

[

g

]

e

(

x

)

Feature

Importance

Feature

Importance

ρ

[

g

]

Symmetry

e

PredictedClass:T-shirt/top

PredictedClass:T-shirt/top

TransformedSaliencyMap

Contributions. We propose a new framework to evaluate the robustness of interpretability methods. We consider a setting where the model we wish to interpret is invariant with respect to a group G of symmetry acting on the model input. Any interpretability method that faithfully describes this model should have explanations that are conserved by this group of symmetry G . We illustrate this reasoning in Figure 1 with the simple group G of time translations acting on the input signal. We show examples of interpretability methods failing to conserve the model's symmetries, hence leading to inconsistent explanations, in Figure 2 and Appendix I. With this new framework, we bring several contributions. (1) Rigorous Interpretability Robustness. We define interpretability robustness with respect to a group G of symmetry through explanation invariance and equivariance. In agreement with our motivations, we demonstrate in Section 2.2 that our general definitions cover different type of interpretability methods, modalities and transformations of the input data. (2) Evaluation of Interpretability Methods. Not all interpretability methods are equal with respect to our notion of robustness. In Section 2.3, we show that some popular interpretability methods are naturally endowed with theoretical robustness guarantees. Further, we introduce 2 metrics, the invariance and equivariance scores, to empirically evaluate this robustness. In Section 3.1, we use these metrics to evaluate the robustness of 3 types of interpretability methods with 5 different model types corresponding to 4 different modalities and symmetry groups. Our empirical results support our theoretical analysis. (3) Insights to Improve Robustness. By combining our theoretical and empirical analysis, we derive a set of 5 actionable guidelines to ensure that interpretability methods are used in a way that guarantees robustness with respect to the symmetry group G . In particular, we show in Sections 2.3 and 3.2 that we can improve the invariance score of any interpretability method by aggregating explanations over various symmetries. We summarize the guidelines with a flowchart in Figure 6 from Appendix A, that helps users to obtain robust model interpretations.

2 Interpretability Robustness

In this section, we formalize the notion of interpretability robustness through explanation invariance and equivariance. We start with a reminder of some useful definitions from geometric deep learning. Wethen define two metrics to measure the invariance and equivariance of interpretability methods. We leverage this formalism to derive some theoretical robustness guarantees for popular interpretability methods. Finally, we describe a rigorous approach to improve the invariance of any interpretability method.

2.1 Useful Notions of Geometric Deep Learning

Some basic concepts of group theory are required for our definition of interpretability robustness. To that aim, we leverage the formalism of Geometric Deep Learning . Please refer to [46] for more details. To rigorously define explanation equivariance and invariance, we need some form of structure in the data we are manipulating. This precludes tabular data but includes graph, time series and image data. In this setting, the data is defined on a finite domain set Ω (e.g. a grid Ω = Z n × Z n for n × n images). On this domain, the data is represented by signals x ∈ X (Ω , C ) , mapping each point u ∈ Ω of the domain to a channel vector x ( u ) ∈ C = R d C . We note that d C ∈ N + , corresponds to the

number of channels of the signal (e.g. d C = 3 for RGB images). The set of signals has the structure of a vector space since x 1 , x 2 ∈ X (Ω , C ) ⇒ λ 1 · x 1 + λ 2 · x 2 ∈ X (Ω , C ) for all λ 1 , λ 2 ∈ R .

Symmetries. Informally, symmetries are transformations of the data that leave the information content unchanged (e.g. moving an image one pixel to the right). More formally, symmetries correspond to a set G endowed with a composition operation ◦ : G 2 →G . Clearly, this set G includes an identity transformation id that leaves the data untouched. Similarly, if a transformation g ∈ G preserves the information, then it could be undone by an inverse transformation g -1 ∈ G such that g -1 ◦ g = id . Those properties 1 give G , ◦ the structure of a group . In this paper, we assume that the symmetry group has a finite number of elements.

Group Representation. We have yet to formalize how the above symmetries transform the data. To that aim, we need to link the symmetry group G with the signal vector space X (Ω , C ) . This connection is achieved by choosing a group representation ρ : G → Aut[ X (Ω , C )] that maps each symmetry g ∈ G to an automorphism ρ [ g ] ∈ Aut[ X (Ω , C )] . Formally, the automorphisms Aut[ X (Ω , C )] are defined as bijective linear transformations mapping X (Ω , C ) onto itself. In practice, each automorphism ρ [ g ] is represented by an invertible matrix acting on the vector space X (Ω , C ) . For instance, an image translation g can be represented by a permutation matrix ρ [ g ] . To qualify as a group representation, the map ρ needs to be compatible with the group composition: ρ [ g 2 ◦ g 1 ] = ρ [ g 2 ] ρ [ g 1 ] . This property guarantees that the composition of two symmetries can be implemented as the multiplication between two matrices.

Invariance. We first consider the case of a deep neural network f : X (Ω , C ) →Y , where the output f ( x ) ∈ Y is a vector with no underlying structure (e.g. class probabilities for a classifier). In this case, we expect the model's prediction to be unchanged when applying a symmetry g ∈ G to the input signal x ∈ X (Ω , C ) . For instance, the probability of observing a cat on an image should not change if we move the cat by one pixel to the right. This intuition is formalized by defining the G -invariance property for the model f : f ( ρ [ g ] x ) = f ( x ) for all g ∈ G , x ∈ X (Ω , C ) .

Equivariance. We now turn to the case of deep neural networks f : X (Ω , C ) →Y (Ω ′ , C ′ ) , where the output f ( x ) ∈ Y (Ω ′ , C ′ ) is also a signal (e.g. segmentation masks for an object detector). We note that the domain Ω ′ and the channel space C ′ are not necessarily the same as Ω and C . When applying a transformation g ∈ G to the input signal x ∈ X (Ω , C ) , it is legitimate to expect the output signal f ( x ) to follow a similar transformation. For instance, the segmentation of a cat on an image should move by one pixel to the right if we move the cat by one pixel to the right. This intuition is formalized by defining the G -equivariance property for the model f : f ( ρ [ g ] x ) = ρ ′ [ g ] f ( x ) . Again, the representation ρ ′ : G → Aut[ Y (Ω ′ , C ′ )] is not necessarily the same as the representation ρ since the signal spaces X (Ω , C ) and Y (Ω ′ , C ′ ) might have different dimensions.

2.2 Explanation Invariance and Equivariance

We will now restrict to models that are G -invariant 2 . It is legitimate to expect similar invariance properties for the explanations associated to this model. We shall now formalize this idea for generic explanations. We assume that explanations are functions of the form e : X (Ω , C ) → E , where E ⊆ R d E is an explanation space with d E ∈ N + dimensions 3 .

Invariance and Equivariance. The invariance and equivariance of the explanation e with respect to symmetries G are defined as in the previous section. In this way, we say that the explanation e is G -invariant if e ( ρ [ g ] x ) = e ( x ) and G -equivariant if e ( ρ [ g ] x ) = ρ ′ [ g ] e ( x ) for all g ∈ G , x ∈ X (Ω , C ) . There is no reason to expect these equalities to hold exactly a priori. This motivates the introduction of two metrics that measure the violation of explanation invariance and equivariance by an interpretability method.

Definition 2.1 (Robustness Metrics) . Let f : X (Ω , C ) →Y be a neural network that is invariant with respect to the symmetry group G and e : X (Ω , C ) →E be an explanation for f . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We measure the invariance of e with

respect to G for some x ∈ X (Ω , C ) with the metric

where s E : E 2 → R is a similarity score on the explanation space E . We use the cos-similarity s E ( a, b ) = a ⊺ b / ∥ a ∥ 2 ·∥ b ∥ 2 for real-valued explanations a, b ∈ R d E and the accuracy score s E ( a, b ) = d -1 E ∑ d E i =1 ✶ ( a i = b i ) for categorical explanations a, b ∈ Z d E K , where ✶ is the indicator function and K ∈ N + is the number of categories. If we assume that G acts on E via the representation ρ ′ : G → Aut[ E ] , we measure the equivariance of e with respect to G for some x ∈ X (Ω , C ) with the metric

A score Inv G ( e, x ) = 1 or Equiv G ( e, x ) = 1 indicates that the explanation method e is G -invariant or equivariant for the example x ∈ X (Ω , C ) .

Remark 2.2 . The metrics Inv G and Equiv G might be prohibitively expensive to evaluate whenever the size |G| of the symmetry group G is too big. Note that this is typically the case in our experiments as we consider large permutation groups of order |G| ≫ 10 32 . In this case, we use Monte Carlo estimators for both metrics by uniformly sampling G ∼ U ( G ) and averaging over a number of sample N samp ≪|G| . We study the convergence of those Monte Carlo estimators in Appendix E.

The above approach to measure the robustness of interpretability method applies to a wide variety of settings. To clarify this, we explain how to adapt the above formalism to 3 popular types of interpretability methods: feature importance , example importance and concept-based explanations .

Feature Importance. Feature importance explanations associate a saliency map e ( x ) ∈ X (Ω , C ) to each example x ∈ X (Ω , C ) for the model's prediction f ( x ) . In this case, we note that the explanation space corresponds to the model's input space E = X (Ω , C ) , since the method assigns an importance score to each individual feature. If we apply a symmetry to the input, we expect the same symmetry to be applied to the saliency map, as illustrated by the example from Figure 1. Hence, the most relevant metric to record here is the explanation equivariance Equiv G . Since the input space and the explanation space are identical E = X (Ω , C ) , we work with identical representations ρ ′ = ρ . We note that this metric generalizes the self-consistency score introduced by [42] beyond affine transformations.

Example Importance. Example importance explanations associate an importance vector e ( x ) ∈ R N train to each example x ∈ X (Ω , C ) for the model's prediction f ( x ) . Note that N train ∈ N + is typically the model's training set size, so that each component of e ( x ) corresponds to the importance of a training example. If we apply a symmetry to the input, we expect the relevance of training examples to be conserved, as illustrated by the example from Figure 1. Hence, the most relevant metric to record here is the invariance Inv G .

Concept-Based Explanations. Concept-based explanations associate a binary concept presence vector e ( x ) ∈ { 0 , 1 } C to each example x ∈ X (Ω , C ) for the model's prediction f ( x ) . Note that C ∈ N + is the number of concepts one considers, so that each component of e ( x ) corresponds to the presence/absence of a concept. If we apply a symmetry to the input, there is no reason for a concept to appear/vanish, since the information content of the input is untouched by the symmetry. Hence, the most relevant metric to record here is again the invariance Inv G .

2.3 Theoretical Analysis

Let us now provide a theoretical analysis of robustness in a setting where the model f is G -invariant. We first show that many popular interpretability methods naturally offer some robustness guarantee if we make some assumptions. For methods that are not invariant when they should, we propose an approach to enforce G -invariance.

Robustness Guarantees. In Table 1, we summarize the theoretical robustness guarantees that we derive for popular interpretability methods. All of these guarantees are formally stated and proven in Appendix D. When it comes to feature importance methods, there are mainly two assumptions that are necessary to guarantee equivariance. (1) The first assumption restricts the type of baseline input

¯ x ∈ X (Ω , C ) on which the feature importance methods rely. Typically, these baselines signals are used to replace ablated features from the original signal x ∈ X (Ω , C ) (i.e. remove a feature x i by replacing it by ¯ x i ). In order to guarantee equivariance, we require this baseline signal to be invariant to the action of each symmetry g ∈ G : ρ [ g ]¯ x = ¯ x . (2) The second assumption restricts the type of representation ρ that can be used to describe the action of the symmetry group on the signals. In order to guarantee equivariance, we require this representation to be a permutation representation , which means that the action of each symmetry g ∈ G is represented by a permutation matrix ρ [ g ] acting on the signal space X (Ω , C ) .

When it comes to example importance methods, the assumptions depend on how the importance scores are obtained. If the importance scores are computed from the model's loss, then the invariance of the explanation immediatly follows from the model's invariance. If the importance scores are computed from the model's internal representations h : X (Ω , C ) → R d rep , then the invariance of the explanation can only be guaranteed if the representation map h is itself invariant to action of each symmetry: h ( ρ [ g ] x ) = h ( x ) . Similarly, concept-based explanations are also computed from the model's representations h . Again, the invariance of these explanations can only be guaranteed if the representation map h is itself invariant.

Enforcing Invariance. If the explanation e is not G -invariant when it should, we can construct an auxiliary explanation e inv built upon e that is G -invariant. This permits to improve the robustness of any interpretability method that has no invariance guarantee. The idea is simply to aggregate the explanation over several symmetries.

Proposition 2.3. [Enforce Invariance] Consider a neural network f : X (Ω , C ) →Y that is invariant with respect to the symmetry group G and e : X (Ω , C ) →E be an explanation for f . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We define the auxiliary explanation e inv : X (Ω , C ) →E as

for all x ∈ X (Ω , C ) . The auxiliary explanation e inv is invariant under the symmetry group G .

Remark 2.4 . Once again, a Monte Carlo estimation for e inv might be required for groups G with many elements. This produces explanations that are approximatively invariant.

3 Experiments

In this section, we use our interpretability robustness metrics to draw some insights with real-world models and datasets. We first evaluate the G -invariance and equivariance of popular interpretability methods used on top of G -invariant models. With this analysis, we identify interpretability methods that are not robust. We then show that the robustness of these interpretability methods can largely be improved by using their auxiliary version defined in Proposition 2.3. Finally, we study how the G -invariance and equivariance of interpretability methods varies when we decrease the invariance of the underlying model. From these experiments, we derive 5 guidelines to ensure that interpretability

methods are robust with respect to symmetries from G . We summarize these guidelines with a flowchart in Figure 6 from Appendix A.

The datasets used in our experiment are presented in Table 2. We explore various modalities and symmetry groups throughout the section, as described in Table 3. For each dataset, we fit and study a classifier from the literature designed to be invariant with respect to the underlying symmetry group. For each model, we evaluate the robustness of various feature importance, example importance and concept-based explanations. More details on the experiments are available in Appendix F. We also include a comparison between our robustness metrics and the sensitivity metric in Appendix G. The code and instructions to replicate all the results reported below are available in the public repositories https://github.com/JonathanCrabbe/RobustXAI and https://github.com/ vanderschaarlab/RobustXAI .

3.1 Evaluating Interpretability Methods

Motivation. The purpose of this experiment is to measure the robustness of various interpretability methods. Since we manipulate models that are invariant with respect to a group G of symmetry, we expect feature importance methods to be G -equivariant ( Equiv G [ e, x ] = 1 for all x ∈ X (Ω , C ) ). Similarly, we expect example and concept-based methods to be G -invariant ( Inv G [ e, x ] = 1 for all x ∈ X (Ω , C ) ). We shall now verify this empirically.

Methodology. To measure the robustness of interpretability methods empirically, we use a set D test of N test examples ( N test = 433 for Mutagenicity, N test = 1 , 000 for ModelNet40 and Electrocardiograms (ECG) and N test = 500 in the other cases). For each interpretability method e , we evaluate the appropriate robustness metric for each test example x ∈ D test . For Mutagenicity and ModelNet40, the large order |G| makes the exact evaluation of the metric unrealistic. We therefore use a Monte Carlo approximation with N samp = 50 . As demonstrated in Appendix E, the Monte Carlo estimators have already converged with this sample size. In all the other cases, these metrics are evaluated exactly since G has a tractable order |G| . Since the E(2)-WideResNets for CIFAR100 and STL10 are only approximativerly invariant with respect to D 8 , we defer their discussion to Section 3.3. We note that some interpretability methods cannot be used in some settings. Whenever this is the case, we simply omit the interpretability method. Please refer to Appendix F for more details.

Analysis. We report the robustness score for each metric and each dataset on the test set D test in Figures 3(a) to 3(c). We immediately notice that not all the interpretability methods are robust. We provide some real examples of non-robust explanations in Appendix I in order to visualize the failure modes. When looking at feature importance, we observe that equivariance is not guaranteed by methods that rely on baseline that are not invariant. For instance, Gradient Shap and Feature Permutation rely on a random baseline, which has no reason to be G -invariant. We conclude that the

ECG

M.Net40

IMDb

Fa.MNIST CIFAR100(*) STL10(*)

Dataset

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Explanation Equivariance

Explanation

DeepLift

Integrated Gradients

Gradient Shap

Feature Permutation

Feature Ablation

Feature Occlusion

(a) Feature Importance

ECG

M.Net40

IMDb

Fa.MNIST CIFAR100(*) STL10(*)

Dataset

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Explanation Invariance

Explanation

CAV-Inv

CAR-Inv

CAV-Equiv

CAR-Equiv

(c) Concept Importance

ECG

M.Net40

IMDb

Fa.MNIST CIFAR100(*) STL10(*)

Dataset

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Explanation Invariance

Explanation

SimplEx-Inv

SimplEx-Equiv

TracIn

Influence Functions

0

50

100

150

200

250

Epoch

0.88

0.90

0.92

0.94

0.96

0.98

1.00

Score

Dataset

CIFAR100

STL10

Property

Model Invariance

Gradient Equivariance

invariance of the baseline ¯ x is crucial to guarantee the robustness of feature importance methods. When it comes to example importance, we note that loss-based methods are consistently invariant, which is in agreement with Proposition D.9. Representation-based and concept-based methods, on the other hand, are invariant only if used with invariant layers of the model. This shows that the choice of what we call the representation space matters for these methods. We derive a set of guidelines from these observations.

Guideline 1. Feature importance methods should be used with group invariant baseline signal ( ρ [ g ]¯ x = ¯ x for all g ∈ G ) to guarantee explanation equivariance. Only methods that conserve the invariance of the baseline can guarantee equivariance.

Guideline 2. Loss-based example importance methods guarantee explanation invariance, unlike representation-based methods. When using the latter, only invariant layers guarantee explanation invariance.

Guideline 3. To guarantee invariance of concept-based explanations, concept classifiers should be used on invariant layers of the model.

3.2 Improving Robustness

Motivation. In the previous experiment, we noticed that not all the interpretability methods are G -invariant when they should. Consider, for instance, concept-based methods used on equivariant layers. The lack of invariance for these methods implies that they rely on concept classifiers that

0

50

100

150

N

inv

0.5

0.6

0.7

0.8

0.9

1.0

Explanation Invariance

CAV-Equiv

CAR-Equiv

(a) ECG

0

20

40

60

80

100

120

N

inv

0.80

0.85

0.90

0.95

1.00

Explanation Invariance

CAV-Equiv

CAR-Equiv

(b) FashionMNIST

are not G -invariant. This behaviour is undesirable for two reasons: (1) since any symmetry g ∈ G preserve the information of a signal x ∈ X (Ω , C ) , the signal ρ [ g ] x should contain the same concepts as x and (2) the layer that we use implicitly encodes these symmetries through equivariance of the output representations. Hence, concept classifiers that are not G -invariant fail to generalize by ignoring the symmetries encoded in the structure of the model's representation space. Fortunately, Proposition 2.3 gives us a prescription to obtain explanations (here concept classifiers) that are more robust with respect to the model's symmetries. We shall now illustrate how this prescription improves the robustness of concept-based methods.

Methodology. In this experiment, we restrict our analysis to the ECG and FashionMNIST datasets. For each test signal, we sample N inv symmetries G i ∈ G , i ∈ Z N inv without replacement. As prescribed by Proposition 2.3, we then compute the auxiliary explanation e inv ( x ) = N -1 inv ∑ N inv i =1 e ( ρ [ G i ] x ) for each concept importance method.

Analysis. We report the average invariance score E X ∼ U ( D test ) Inv G ( e inv , X ) for several values of N inv in Figure 4. As we can see, the invariance of the explanation grows monotonically with the number of samples N inv to achieve a perfect invariance for N inv = |G| . Interestingly, the explanation invariance increases more quickly for CAR. This suggests that enforcing explanation invariance is less expensive for certain interpretability methods and motivates the below guideline.

Guideline 4. Any interpretability method can be made invariant through Proposition 2.3. In doing so, one should increase the number of samples N inv until the desired invariance is achieved. In this way, the method is made robust without increasing the number of calls more than necessary. Note that it only makes sense to enforce invariance of the interpretability method if the explained model is itself invariant.

3.3 Relaxing Invariance

Motivation. In practice, models are not always perfectly invariant. A first example is given by the CIFAR100 and STL10 WideResNet that has a strong bias towards being D 8 -invariant, altough it can break this invariance at training time (see Appendix H.3 of [64]). Another popular example is a CNN that flattens the output of convolutional layers, which violates translation invariance [52, 66, 67]. This motivates the study of interpretability methods robustness when models are not perfectly invariant.

Methodology. This experiment studies the two aforementioned settings. First, we replicate the experiment from Section 3.1 with the CIFAR100 and STL10 WideResNet. Second, we consider CNNs that flatten their last convolutional layer with the ECG and FashionMNIST datasets. In this case, we introduce 2 variants of the All-CNN where the global pooling is replaced by a flatten operation: an Augmented-CNN trained by augmenting the training set D train with random translations and a Standard-CNN trained without augmentation. We measure the invariance/equivariance of the interpretability methods for each model.

Analysis. The results for the WideResNets are reported in Figures 3(a) to 3(c). We see that the robustness of various interpretability methods substantially drops with the model invariance. This is particularly noticeable for feature importance methods. To illustrate this phenomenon, we plot in Figure 3(d) the evolution during training of the model's prediction f ( x ) G -invariance and the

G -equivariance of its gradient ∇ x f ( x ) , on which the attribution methods rely. As we can see, the model remains almost invariant during training, while the gradients equivariance is destroyed. Similar observations can be made with the CNNs from Figure 5. In spite of the Augmented-CNN being almost invariant, we notice that the symmetry breaks significantly for feature importance methods. These results suggest that the robustness of interpretability methods can be (but is not necessarily) fragile if model invariance is relaxed, even slightly. This motivates our last guideline, which safeguards against erroneous interpretations of our robustness metrics.

ECG

0.0

0.2

0.4

0.6

0.8

1.0

Feature Importance Equivariance

0.0

0.2

0.4

0.6

0.8

1.0

Example Importance Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Concept Importance Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Model Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Feature Importance Equivariance

0.0

0.2

0.4

0.6

0.8

1.0

Model Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Example Importance Invariance

Fa.MNIST

0.0

0.2

0.4

0.6

0.8

1.0

Model Invariance

0.0

0.2

0.4

0.6

0.8

1.0

Concept Importance Invariance

Model Type

All-CNN

Augmented-CNN

Standard-CNN

Explanation

DeepLift

Feature Ablation

Feature Occlusion

Feature Permutation

Gradient Shap

Integrated Gradients

Influence Functions

SimplEx-Lin1

TracIn

CAR-Lin1

CAV-Lin1

4 Discussion

Building on recent developments in geometric deep learning, we introduced two metrics (explanation invariance and equivariance) to assess the faithfulness of model explanations with respect to model symmetries. In our experiments, we considered a wide range of models whose predictions are invariant with respect to transformations of their input data. By analyzing feature importance, example importance and concept-based explanations of these models, we observed that many of these explanations are not invariant/equivariant to these transformations when they should. This led us to establish a set of guidelines in Appendix A to help practitioners choose interpretability methods that are consistent with their model symmetries.

Beyond actionable insights, we believe that our work opens up interesting avenues for future research. An important one emerged by studying the equivariance of saliency maps with respect to models that are approximately invariant. This analysis showed that state-of-the-art saliency methods fail to keep a high equivariance score when the model's invariance is slightly relaxed. This important observation could be the seed of future developments of robust feature importance methods.

References

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018.

Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 , 2017.

Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface , 15(141):20170387, 2018.

Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. What do we want from explainable artificial intelligence (xai)?-a stakeholder perspective on xai and a conceptual model guiding interdisciplinary xai research. Artificial Intelligence , 296:103473, 2021.

Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access , 6:52138-52160, 2018.

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Inf. Fusion , 58(C):82-115, jun 2020.

Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence , 1(5):206-215, 2019.

Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks. 2022.

Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems , 29, 2016.

Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. Advances in neural information processing systems , 27, 2014.

Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '16, page 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 3319-3328. JMLR.org, 2017.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning , pages 3145-3153. PMLR, 2017.

Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2166-2177. PMLR, 18-24 Jul 2021.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1885-1894. PMLR, 2017.

Garima, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.

Jonathan Crabbe, Zhaozhi Qian, Fergus Imrie, and Mihaela van der Schaar. Explaining latent representations with a corpus of examples. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12154-12166. Curran Associates, Inc., 2021.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Jonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A generalized framework for concept-based explanations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 2590-2607. Curran Associates, Inc., 2022.

Leif Hancox-Li. Robustness in machine learning explanations: Does it matter? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , FAT* '20, page 640-647, New York, NY, USA, 2020. Association for Computing Machinery.

Jianlong Zhou, Amir H Gandomi, Fang Chen, and Andreas Holzinger. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics , 10(5):593, 2021.

Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel J Gershman, and Finale Doshi-Velez. Human evaluation of models built for interpretability. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 7, pages 59-67, 2019.

Yuansheng Xie, Soroush Vosoughi, and Saeed Hassanpour. Interpretation quality score for measuring the quality of interpretability methods. arXiv preprint arXiv:2205.12254 , 2022.

Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, et al. Benchmarking saliency methods for chest x-ray interpretation. Nature Machine Intelligence , pages 1-12, 2022.

Jonathan Crabbé, Alicia Curth, Ioana Bica, and Mihaela van der Schaar. Benchmarking heterogeneous treatment effect models through the lens of interpretability. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 12295-12309. Curran Associates, Inc., 2022.

Ian E. Nielsen, Dimah Dera, Ghulam Rasool, Ravi P. Ramachandran, and Nidhal Carla Bouaynaya. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine , 39(4):73-84, 2022.

Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , pages 267-280. Springer, 2019.

David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049 , 2018.

Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems , 32, 2019.

Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. arXiv preprint arXiv:2005.00631 , 2020.

Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, KlausRobert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in Neural Information Processing Systems , 32, 2019.

Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence , volume 33, 2019.

Wei Huang, Xingyu Zhao, Gaojie Jin, and Xiaowei Huang. Safari: Versatile and efficient evaluations for robustness of interpretability. arXiv preprint arXiv:2208.09418 , 2022.

Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. Advances in Neural Information Processing Systems , 33:13623-13634, 2020.

Ahmad Ajalloeian, Seyed-Mohsen Moosavi-Dezfooli, Michalis Vlachos, and Pascal Frossard. On smoothed explanations: Quality and robustness. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 15-25, 2022.

Yipei Wang and Xiaoqian Wang. Self-interpretable model with transformation equivariant interpretation. Advances in Neural Information Processing Systems , 34:2359-2372, 2021.

Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilovi´ c, et al. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012 , 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Jonathan Crabbé and Mihaela van der Schaar. Label-free explainability for unsupervised models. Proceedings of Machine Learning Research , 2022.

Chris Lin, Hugh Chen, Chanwoo Kim, and Su-In Lee. Contrastive corpus attribution for explaining representations. arXiv preprint arXiv:2210.00107 , 2022.

Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res. , 20(177):1-81, 2019.

A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation , 101(23), 2000.

G.B. Moody and R.G. Mark. The impact of the mit-bih arrhythmia database. IEEE Engineering in Medicine and Biology Magazine , 20(3):45-50, 2001.

Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 , 2014.

Jeroen Kazius, Ross McGuire, and Roberta Bursi. Derivation and validation of toxicophores for mutagenicity prediction. Journal of Medicinal Chemistry , 48(1):312-320, 2005. PMID: 15634026.

Kaspar Riesen and Horst Bunke. Iam graph database repository for graph based pattern recognition and machine learning. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR) , pages 287-297. Springer, 2008.

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4602-4609, 2019.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1912-1920, 2015.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning , pages 3744-3753. PMLR, 2019.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems , 32, 2019.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 215-223, Fort Lauderdale, FL, USA, 2011. PMLR.

Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14274-14285, 2020.

Valerio Biscione and Jeffrey Bowers. Learning translation invariance in cnns. arXiv preprint arXiv:2011.11757 , 2020.

Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Softw. Eng. , 48(1):1-36, 2022.

Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356 , 2017.

Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for the l _ 0 norm. arXiv preprint arXiv:1804.05805 , 2018.

Divya Gopinath, Guy Katz, Corina S Pasareanu, and Clark Barrett. Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 , 2017.

D. Robinson, F.W. Gehring, and P.R. Halmos. A Course in the Theory of Groups . Graduate Texts in Mathematics. Springer New York, 1996.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Workshop Track Proceedings , 2014.

Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill , 5(1):e22, 2020.

Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence , 3(7):620-631, 2021.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association , 58(301):13-30, 1963.

T. Kloek and H. K. van Dijk. Bayesian estimates of equation system parameters: An application of integration by monte carlo. Econometrica , 46(1):1-19, 1978.

Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual . CreateSpace, Scotts Valley, CA, 2009.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. arXiv preprint , 2017.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321357, 2002.

Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision , pages 818-833. Springer, 2014.

Mohammad Kachuee, Shayan Fazeli, and Majid Sarrafzadeh. Ecg heartbeat classification: A deep transferable representation. In 2018 IEEE international conference on healthcare informatics (ICHI) , pages 443-444. IEEE, 2018.

Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Gaël Varoquaux, Travis Vaught, and Jarrod Millman, editors, Proceedings of the 7th Python in Science Conference , pages 11 - 15, Pasadena, CA USA, 2008.

Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion ReblitzRichardson. Captum: A unified and generic model interpretability library for pytorch, 2020.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2990-2999, New York, New York, USA, 2016. PMLR.

Chaitanya Joshi. Transformers are graph neural networks. The Gradient , page 5, 2020.

Corentin Tallec and Yann Ollivier. Can recurrent neural networks warp time? arXiv preprint arXiv:1804.11188 , 2018.

Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801.10130 , 2018.

Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshcnn: a network with an edge. ACM Transactions on Graphics (TOG) , 38(4):1-12, 2019.

Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9323-9332. PMLR, 2021.

Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications , 13(1):1-11, 2022.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Appendices

A How to Use our Framework in Practice

Start with a neural network

f

:

X

(Ω

,

C

)

→Y

that is invariant under a symmetry group

G

with representation

ρ

:

G →

Aut[

X

(Ω

,

C

)]

, an interpretability method

e

:

X

(Ω

,

C

)

→E

and an example

x

∈ X

(Ω

,

C

)

to explain.

Type of

e

?

Evaluate

Inv

G

[

e, x

]

defined in (1).

Evaluate

G

[

e, x

]

defined in (2).

Evaluate

Inv

G

[

e, x

]

defined in (1).

G

[

e, x

]

≈

1

?

Inv

G

[

e, x

]

≈

1

?

Inv

G

[

e, x

]

≈

1

?

e

representation

based?

Restrict

e

to

be gradient or

perturbation-based

with invariant

baseline

¯

x

=

ρ

[

g

]¯

x

as per

Prop. D.6 and D.8.

You can either

(a) Use your

method with another

representation that

is invariant as per

Prop. D.12 and D.14

(b) Enforce

robustness by

replacing

e

by

e

inv

as per Prop. 2.3.

Replace

e

by

e

inv

as per Prop. 2.3.

For large

|G|

, use Monte Carlo sampling an increase

N

inv

until the desired

Inv

G

[

e

inv

, x

]

is achieved.

You should have a robust explanation by now.

If you wish to relax model invariance, remember to re-evaluate all metrics since the robustness of

interpretability method for an invariant model

does not

imply its robustness of an approximately invariant model.

Feature Importance

Example Importance

Concept-Based

No

No

No

Yes

No

Yes

Yes

Yes

To show how these guidelines can be used in practice, we illustrate two possible categories based on our experiments from Section 3. Let us start with Influence Functions and the ECG dataset. This is an example importance method, hence we choose the central branch from Figure 6. Measuring the invariance in Figure 3(b) shows that Inv G [ e, x ] ≈ 1 , hence we may jump to the terminal node of the flowchart. This is correct since Influence Functions are G -invariant as demonstrated in Proposition D.9.

Now let us take the example of CAV used with the equivariant layer from the Deep Set for the ModelNet40 dataset. This is a concept importance method, hence we choose the right branch of Figure 6. Measuring the invariance in Figure 3(c) shows that Inv G [ e, x ] < 1 , hence we go down. As the flowchart recommends, we may decide to use the explainability method on an invariant layer instead. Doing this yields Inv G [ e, x ] ≈ 1 , as shown in Figure 3(c). Hence we may jump to the

terminal node of the flowchart. This is correct since concept important methods used with invariant layers are also G -invariant, as demonstrated in Proposition D.14.

B Contribution and Hope for Impact

In this appendix, we discuss the significance of our work. We should firstly emphasize the importance of the problem we are tackling in this work. In order to provide valuable explanations of a model, it is crucial to ensure that interpretability methods faithfully describe the explained model. Indeed, failing in this basic criterion implies that the explanations could be inconsistent with the true model behaviour, hence leading to false insights about the model. For this reason, we believe that guaranteeing an alignment between interpretability methods and the model is a problem of the utmost importance. Beyond the significance of this problem, we believe that we bring a substantial contribution to address it. Below, we enumerate the novelties we claim.

Connecting interpretability with geometric deep learning. In Sections. 2, we show that the formalism of geometric deep learning naturally extends to the description of interpretability methods. We believe that this bridge can be valuable for at least two communities. For the interpretability community, geometric deep learning provides a rigorous framework to ensure that interpretations are consistent with strong inductive biases that underpin cutting edge deep neural networks, such as GNNs, CNNs, Deep Sets, Transformers and the other examples cited in Appendix H. For the geometric deep learning community, interpretability provides a way to extract actionable insights from increasingly sophisticated architectures. We hope that our paper promotes collaborations between these two communities.

Proving that explanation robustness imposes restriction on interpretability methods. By using our formalism derived from geometric deep learning, we show in Appendix D that enforcing equivariance or invariance for interpretability methods imposes strong restrictions. For instance, gradient-based attribution methods that rely on a baseline signal ¯ x ∈ X (Ω , C ) are equivariant if this baseline signal is invariant: ρ [ g ]¯ x = ¯ x . We provide a similar analysis for 3 popular types of interpretability methods (feature importance, example importance and concept-based explanations). We hope that these insights will guide the future development of interpretability methods tailored for cutting edge deep learning models, such as GNNs.

Introducing two well-defined and sensible robustness metrics. Beyond adapting invariance and equivariance to interpretability methods, we provide a principled way to evaluate those properties in Section 2 with two robustness metrics. Since these metrics cannot always be computed exactly (e.g. for groups G with large cardinalities), we show in Appendix E that Monte Carlo approximations provide sensible approximations, both theoretically by adapting Hoeffding's inequality and empirically in the setup from Section 3.

Demonstrating empirically that not all interpretability methods are created equal. We have performed extensive experiments on 6 different datasets, corresponding to 4 distinct modalities and model architectures, with 12 different interpretability methods. These experiments show that some interpretability methods (such as GradientShap and Feature Permutation) consistently fail to be robust with respect to symmetries of the model. We believe that these results are not trivial and should impact the way we use these methods to explain deep neural networks.

Through these contributions, we hope to reinforce the 3 below statements.

Interpretability should be used with skepticism.

Not all interpretability methods are created equal.

Robustness can guide the design of interpretability methods.

C Limitations

In this appendix, we discuss the main limitations of this work. As a first limitation, we would like to acknowledge the fact that our robustness metrics have no pretention to provide a holistic evaluation approach for interpretability methods. We believe that evaluating the robustness of interpretability methods requires a combination of several evaluation criteria. To make this point more clear, let us take inspiration from the Model Robustness literature. By looking at reviews on this subject (see e.g.

Section 6.3 of [68]), we notice that various notions of robustness exist for machine learning models. To name just a few, existing works have investigated the robustness of neural networks with respect to noise [69]], adversarial perturbations [70] and label consistency across clusters [71]. All of these notions of robustness come with their own metrics and assessment criteria. In a similar way, we believe that the robustness of interpretability methods should be assessed on several complementary dimensions. Our work contributes to this initiative by adding one such dimension.

As a second limitation, our robustness metrics are mostly useful for models that are (approximately) invariant with respect to a given symmetry group. That being said, we would like to emphasize that our robustness metrics can still be used to characterize the equivariance/invariance of explanations even if the model is not perfectly invariant. The metrics would still record the same information, even if the invariance of the model is relaxed. The main thing to keep in mind when using these metrics with models that are not perfectly invariant is that the explanations themselves should not be exactly invariant/equivariant. That said, for a model that is approximately invariant, we expect a faithful explanation to keep a high invariance / equivariance score. This is precisely the object of Section 3.3.

D Theoretical Results

In this appendix, we prove all the theoretical results mentioned in the main paper. We start by deriving robustness guarantees mentioned in Table 1. We then prove that any method can be made G -invariant by aggregating the explanation over several symmetries.

D.1 Feature Importance Guarantees

Let us start by feature importance methods. As we are going to see, the robustness guarantees in this case typically require us to restrict the type of group representation ρ we use to encode the action of the symmetry group G on the signal space X (Ω , C ) . This motivates the two following types of representations that can be found in group theory textbooks (see e.g. [72]).

Definition D.1 (Orthogonal Representation) . Let ρ : G → Aut[ X (Ω , C )] be a representation of the group G , and let d ∈ N + denote the dimension of the signal space X (Ω , C ) . We say that the representation is an orthogonal representation if its image is a subset on the orthogonal matrices acting on X (Ω , C ) : ρ ( G ) ⊆ O ( d ) , where O ( d ) denotes the set of d × d orthogonal real matrices. This is equivalent to ρ [ g ] ρ ⊺ [ g ] = ρ ⊺ [ g ] ρ [ g ] = I d for all g ∈ G , where I d denotes the d × d identity matrix.

Definition D.2 (Permutation Representation) . Let ρ : G → Aut[ X (Ω , C )] be a representation of the group G , and let d ∈ N + denote the dimension of the signal space X (Ω , C ) . We say that the representation is a permutation representation if its image is a subset on the permutation matrices acting on X (Ω , C ) : ρ ( G ) ⊆ P ( d ) , where P ( d ) denotes the set of d × d permutation matrices. This means that for all g ∈ G , there exists some permutation π ∈ S ( d ) such that we can write ( ρ [ g ] x ) i = x π ( i ) for all i ∈ Z d , x ∈ X (Ω , C ) .

Remark D.3 . One can easily check that all permutation representations are also orthogonal representations. The opposite is not true.

D.1.1 Gradient-Based

We shall now begin with gradient-based feature importance methods. These methods compute the model's gradient with respect to the input features to compute the importance scores. Hence, it is useful to characterize how this gradient transforms under the action of the symmetry group G .

Lemma D.4 (Gradient Transformation) . Consider a differentiable function f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . If for some g ∈ G we define x ′ = ρ [ g ] x , then we have the following identity:

where ρ -1 , ⊺ [ g ] denotes the matrix obtained by applying an inversion followed by a transposition to the matrix ρ [ g ] .

Proof. We start by noting that the G -invariance of f implies that f ( x ′ ) = f ( x ) and, hence:

It remains to establish the ling between ∇ x ′ and ∇ x . To this end, we simply note that x = ρ -1 [ g ] x ′ . Hence, for all i ∈ Z d with d = dim[ X (Ω , C )] , we have that

Hence, by using the chain rule, we deduce that for all k ∈ Z d :

The above identity implies ∇ x ′ = ρ -1 , ⊺ [ g ] ∇ x . By injecting this to the right-hand side of (4), we obtain (3).

The simplest gradient-based attribution is simply given by the gradient itself [73]. We refer to it as the vanilla saliency feature importance. Although this attribution method is a bit naive, we may still deduce an equivariance guarantee from the previous proposition.

Corollary D.5 (Equivariance of Vanilla Saliency) . Consider a differentiable neural network f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider a vanilla saliency feature importance explanation e ( x ) = ∇ x f ( x ) . If the representation ρ is orthogonal, then the explanation e is G -equivariant.

Proof. From Lemma D.4, we have that e ( ρ [ g ] x ) = ρ -1 , ⊺ [ g ] e ( x ) for all g ∈ G . Nowif ρ is orthogonal, we note that ρ -1 , ⊺ [ g ] = ρ [ g ] , which proves the proposition.

We now turn to a more general family of gradient-based feature importance methods. These methods attribute importance to each feature by aggregating gradients over a line in the input space X (Ω , C ) connecting a baseline example ¯ x ∈ X (Ω , C ) with the example x ∈ X (Ω , C ) we wish to explain. It was shown that the choice of this baseline signal has a significant impact on the resulting explanation [74]. In the following proposition, we show that enforcing equivariance imposes a restriction on the type of baseline signal ¯ x that can be used in practice.

Proposition D.6 (Gradient-Based Equivariance) . Consider a differentiable neural network f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider a gradient-based explanation built upon a baseline signal ¯ x ∈ X (Ω , C ) and of the form

where ⊙ denotes the Hadamard product and φ is a functional defined on the Hilbert space L 2 ([0 , 1]) . If ρ is a permutation representation and the baseline signal is G -invariant, i.e. ρ [ g ]¯ x = ¯ x for all g ∈ G , then the explanation e is G -equivariant.

Remark D.7 . Note that we have introduced the functional φ to make the class of explanation as general as possible. For instance, we obtain exact Integrated Gradients for φ ( t ) = 1 and Input*Gradient [19] for φ ( t ) = δ ( t -1) , where δ is a Dirac delta distribution. Similarly, this includes discrete approximations of Integrated Gradients by taking e.g. φ ( t ) = ∑ N n =1 δ ( t -t n ) , with t n = ¯ x + n N ( x -¯ x ) for all n ∈ Z n . Finally, we note that the equivariance of Integrated Gradients also implies the equivariance of Expected Gradients [75].

Proof. For all g ∈ G , we have that:

Since ρ is a permutation representation, there exists a permutation π ∈ S ( d ) , where d = dim[ X (Ω , C )] , such that for all a, b ∈ X (Ω , C ) and i ∈ Z d :

We deduce that ρ [ g ] a ⊙ ρ [ g ] b = ρ [ g ]( a ⊙ b ) . By applying this to the above equation for e ( ρ [ g ] x ) , we get:

which proves the equivariance property.

D.1.2 Perturbation-Based

The second type of feature importance methods we consider are perturbation-based methods. These methods attribute importance to each feature by measuring the impact of replacing some features of the example x ∈ X (Ω , C ) with features from a baseline example ¯ x ∈ X (Ω , C ) on the model's prediction. Again, enforcing equivariance imposes a restriction on the type of baseline that can be manipulated.

Proposition D.8 (Perturbation-Based Equivariance) . Consider a neural network f : X (Ω , C ) →Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider a perturbation-based explanation built upon a baseline signal ¯ x ∈ X (Ω , C ) and of the form

for all i ∈ Z d , where d = dim[ X (Ω , C )] . The perturbation operator r i replaces feature x i , i ∈ Z d with the baseline feature ¯ x i . It is defined as follows: [ r i ( x )] j = x j + δ ij (¯ x i -x i ) , where δ denotes the Kronecker delta symbol, for all j ∈ Z d and x ∈ X (Ω , C ) . If ρ is a permutation representation and the baseline signal is G -invariant, i.e. ρ [ g ]¯ x = ¯ x for all g ∈ G , then the explanation e is G -equivariant.

Proof. For all g ∈ G and i, j ∈ Z d , there exists a permutation π ∈ S ( d ) such that:

We deduce that r i ( ρ [ g ] x ) = ρ [ g ] r π ( i ) ( x ) for all i ∈ Z d . We are now ready to conclude as:

which proves the equivariance property.

D.2 Example Importance Guarantees

We proceed with example importance methods.

D.2.1 Loss-Based

We start with loss-based methods. These methods attribute importance to each training example of ( x n , y n ) ∈ D train by comparing the loss L ( f ( x n ) , y n ) with the loss L ( f ( x ) , y ) of the example x ∈ X (Ω , C ) we wish to explain. We show that these methods are naturally invariant without imposing any restriction on the representation ρ .

Proposition D.9 (Loss-Based Invariance) . Consider a differentiable neural network f θ : X (Ω , C ) → Y , parametrized by P ∈ N + parameters θ ∈ R P , that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider an example importance explanation based on the loss L : X (Ω , C ) ×Y → R + and of the form

where F maps any function l : R P → R + to a vector in F [ l ] ∈ R N train , with N train ∈ N + corresponding to the number of training examples for which we evaluate the importance. The explanation e is G -invariant.

Remark D.10 . We note that F typically contains differential operators. For instance, Influence Functions are obtained by taking

for n ∈ Z N train , where ( x n , y n ) ∈ D train is a training example and H θ ∈ R P × P is the Hessian of the training loss with respect to the model's parameters.

Remark D.11 . We note that the dependency of the explanation e with respect to the label y ∈ Y is omitted in the main paper. The reason for this is that the symmetry group G only acts on the input signal x .

Proof. The proposition can directly be deduced from the G -invariance of the model. For any g ∈ G , we have:

which proves the desired property.

D.2.2 Representation-Based

We proceed with representation-based methods. These methods attribute importance to each training example of ( x n , y n ) ∈ D train by comparing the model's representation h ( x n ) (typically the output of a model's layer) with the representation h ( x ) of the example x ∈ X (Ω , C ) we wish to explain. We show that these methods are invariant if we restrict to representations h that are invariant.

Proposition D.12 (Representation-Based Invariance) . Consider a differentiable neural network f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider an example importance explanation based on a representation h : X (Ω , C ) →H extracted from f (e.g. an intermediate layer of the neural network) and of the form

where F : H → R N train maps any representation r ∈ H to a vector in F [ r ] ∈ R N train , with N train ∈ N + corresponding to the number of training examples for which we evaluate the importance. If the representation h is G -invariant, then the explanation e is G -invariant.

Remark D.13 . We note that F can be adapted to the method we want to describe. For instance, SimplEx is obtained by taking

where x n ∈ D train is a training example for n ∈ Z N train . Similarly, Representation Similarity is obtained with

for all n ∈ Z N train .

Proof. The proposition can directly be deduced from the G -invariance of the representation. For any g ∈ G , we have:

which proves the desired property.

D.3 Concept-Based Explanations Guarantees

We now turn to concept-based methods. These methods attribute importance to a set of concept specified by the user for the model to predict a certain class. Although these explanations are typically global (i.e. at the dataset level), they are based on concept classifiers that attempt to detect the presence/absence of the concept on each individual example x ∈ X (Ω , C ) based on its representation h ( x ) . We show that these classifiers are G -invariant if we restrict to representations h that are G -invariant.

Proposition D.14 (Concept-Based Invariance) . Consider a differentiable neural network f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We consider a concept-based explanation based on a representation h : X (Ω , C ) → H extracted from f (e.g. an intermediate layer of the neural network) and of the form

where c : H → { 0 , 1 } C maps any representation r ∈ H to a binary vector in c [ r ] ∈ { 0 , 1 } C indicating the presence/absence of C ∈ N + selected concepts. If the representation h is G -invariant, then the explanation e is G -invariant.

Remark D.15 . Wenote that concepts activation vectors (CA Vs) are obtained by fiting a linear classifier c . Concept activations regions (CARs), on the other hand, are obtained by fitting a kernel-based concept classifier.

Proof. The proposition can directly be deduced from the G -invariance of the representation. For any g ∈ G , we have:

which proves the desired property.

D.4 Enforcing Invariance

Finally, we prove Proposition 2.3 that allows us to turn any interpretability method into a G -invariant method.

Proposition D.16. [Enforce Invariance] Consider a neural network f : X (Ω , C ) → Y that is invariant with respect to the symmetry group G and e : X (Ω , C ) →E be an explanation for f . We assume that G acts on X (Ω , C ) via the representation ρ : G → Aut[ X (Ω , C )] . We define the auxiliary explanation e inv : X (Ω , C ) →E as

for all x ∈ X (Ω , C ) . The auxiliary explanation e inv is invariant under the symmetry group G .

Proof. For any ˜ g ∈ G , we have that

where we have used the fact that the representation ρ is compatible with the group composition. We now define the map l ˜ g : G → G as l ˜ g ( g ) = ˜ g ◦ g for all g ∈ G . We note that l ˜ g is a bijection from G to itself, since it admits an inverse l -1 ˜ g = l ˜ g -1 . Indeed, for all g ∈ G :

Hence, we have that l ˜ g ( G ) = G . By denoting g ′ = l ˜ g ( g ) = ˜ g ◦ g , we can therefore write

This proves the G -invariance of the explanation e inv .

The interpretability method e inv has to be understood as a way to assign a unified explanation to each class of equivalent signals rather than a patched version of e . To make this argument more rigorous, we define the equivalence relation ∼ on the set of signals X (Ω , C ) as follows: two signals x, x ′ ∈ X (Ω , C ) are equivalent iff there exists a symmetry g ∈ G relating the two signals x ′ = ρ [ g ] x . This equivalence relation is the one that underpins a G -invariant neural network as x ′ and x are assigned the same prediction f ( x ′ ) = f ( x ) . If we denote by S x = { x ′ ∈ X (Ω , C ) | x ′ ∼ x } the class of signals equivalent to the signal x , we may write e inv ( x ) = 1 |S x | ∑ x ′ ∈S x e ( x ′ ) . This reformulation gives a nice interpretation to e inv : the explanation e inv ( x ) is simply given by averaging the explanation e over the class S x of signals equivalent to x . If e is an example importance method, then e inv allows us to identify the examples that are the most related to the equivalence class S x . If, on the other hand, e is a concept classifier, then e inv measures the fraction of examples in the equivalence class S x where the concept of interest is detected. We believe that using e inv with this interpretation in mind should help to avoid unwarranted trust in the resulting explanations.

Our opinion is that explanations e that are robust by design are preferable over auxiliary explanations e inv . In this way, a possible hierarchy between interpretability methods based on our robustness criterion for G -invariant models could be as follows:

Method 1 An interpretability method e that has G -invariance by design.

Method 2 An auxiliary interpretability method e inv obtained from e with Proposition 2.3.

Method 3 An interpretability method e that is not G -invariant.

In this way, Method 2 should be avoided whenever Method 1 is available. This is the case, for instance, of example-based interpretability methods where loss-based methods are naturally invariant. Hence, in this case, the benefit of patching representation-based methods is limited. However, in the case of concept-based explanations, we note that neither CA Vs nor CARs grant invariance. In this case, since Method 1 is unavailable, Method 2 might be the best we can do. In the specific case of concept-based interpretability, we note that Proposition 2.3 implements a sensible fix to the lack of invariance. Indeed, applying Proposition 2.3 is equivalent to making the concept-classifiers invariant by applying a G -invariant group aggregation, which is a standard way to implement classifier invariance (examples include GNNs and Deep Sets). In this way, the usefulness of Proposition 2.3 is context-dependent and we believe that a good usage should always be informed by domain knowledge.

E Convergence of the Monte Carlo Estimators

In this appendix, we discuss the Monte Carlo estimators used to approximate the invariance and equivariance metrics defined in Definition 2.1. We first note that our experiments typically aggregate the metrics over a test set D test of examples. Hence, we are interested in the metrics

where U denotes a uniform distribution. Clearly, whenever the order |G| of the symmetry group is large, these metrics might become prohibitively expensive to compute. In this setting, we simply build a Monte Carlo estimator for the above metrics by sampling N samp symmetries G 1 , . . . , G N samp with N samp ∈ N + and N samp ≪ |G| . If we have N test = |D test | test examples, the Monte Carlo estimators can be written as

Those are the estimators that we use in our experiments. Let us first discuss the convergence of these estimators theoretically. Since s E ( a, b ) ∈ [ -1 , 1] for all a, b ∈ E , Hoeffding's inequality [76] guarantees that for all t ∈ R + :

Let us now plug-in some numbers to see how these inequalities translate in our experiments. In Section 3, we typically use N test = 1 , 000 and N samp = 50 . Hence, the probability of making an error larger than t = 2% in our experiments is smaller than 10 -4 . This guarantees that all the metrics reported in the main paper are precisely evaluated.

We shall now verify this theoretical analysis with the experimental setup described in Section 3. Since we do not resort to any Monte Carlo approximation for the Electrocardiogram dataset, we exclude

it from our analysis. Similarly, robustness scores ' Inv G ( e ) ≈ 1 or ÷ Equiv G ( e ) ≈ 1 can be excluded as these can only be produced by having Inv[ e, G m ] ≈ 1 or Equiv[ e, G m ] ≈ 1 for all m ∈ Z N samp , which guarantees that the estimators have already converged. By applying these filters with the help of Figure 3, we restrict our analysis to Gradient Shap for the Mutagenicity dataset and to Gradient Shap, Feature Permutation, SimplEx-Equiv, CAV-Equiv and CAR-Equiv for the ModelNet40 dataset. We plot the Monte Carlo estimators ' Inv G ( e ) and ÷ Equiv G ( e ) as a function of N samp for various interpretability methods in Figure 7. As we can see, all the Monte Carlo estimators have already converged for N samp = 50 used in the experiment. This is due to the fact that we use a relatively large test set in each experiment, with N test = 433 for the Mutagenicity dataset and N test = 1 , 000 for the ModelNet40 experiment.

0

20

40

60

80

100

N

samp

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Gradient Shap Equivariance

(a) Mutagenicity Dataset

0

20

40

60

80

100

N

samp

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Feature Permutation Equivariance

Gradient Shap Equivariance

SimplEx-Equiv Invariance

CAV-Equiv Invariance

CAR-Equiv Invariance

(b) ModelNet40 Dataset

We finish this appendix by mentioning that all the groups manipulated in this paper are finite groups. It goes without saying that an extension of our analysis to infinite, or even uncountable groups would probably require a more sophisticated sampling technique, such as e.g. importance sampling [77].

F Experiment Details

In this appendix, we provide all the details for the experiments conducted in Section 3.

Computing Resources. Almost all the empirical evaluations were run on a single machine equipped with a 64-Core AMD Ryzen Threadripper PRO 3995WX CPU and a NVIDIA RTX A4000 GPU. The only exceptions are the CIFAR100 and STL10 experiments, for which we used a Microsoft Azure virtual machine equipped with a single Tesla V100 GPU. All the machines run on Python 3.10 [78] and Pytorch 1.13.1 [79].

Electrocardiograms. The MIT-BIH Electrocardiogram (ECG) dataset [50, 51] consists of univariate time series x ∈ X ( Z T , R ) with T = 187 time steps, each representing a heartbeat cycle. Each time series comes with a binary label indicating whether the heartbeat is normal or not. We train a 1-dimensional convolutional neural network (CNN) to predict this label. This CNN is made invariant under the action of the cyclic translation group G = Z /T Z on the time series by using only circular padding and global pooling.

Mutagenicity. The Mutagenicity dataset [53-55] consists of graphs x ∈ X ([ V x , E x ] , Z N sp ) representing organic molecules. In a graph, each node u ∈ V x is assigned an atom indexed by x ( u ) ∈ Z N sp , where N sp = 14 is the number of atom species. We ignore the attributes for the edges E x ⊆ V 2 x . Each graph comes with a binary label indicating whether the molecule is a mutagen or not. We train a graph neural network (GNN) to predict this label. This GNN is made invariant under the action of the permutation group G = S V x on the node ordering by using global pooling.

ModelNet40. The ModelNet40 [57] dataset consists of CAD representations of 3D objects. We use the same process as [58, 59] to convert each CAD representation into a cloud x ∈ X ( Z N pt , R 3 ) of

N pt = 1 , 000 points embedded in R 3 . Each cloud of point comes with a label y ∈ Z 40 indicating the class of object represented by the cloud of points among the 40 different classes of objects present in the dataset. We train a Deep Set [58] model to predict this label. Thanks to its architecture, this model is naturally invariant under the action of the permutation group G = S N pt on the points in the cloud. Each model is trained on a training set D train .

IMDb. The IMDb dataset [60] This dataset contains 50k text movie reviews. Each review comes with a binary label y ∈ { 0 , 1 } . We represent each review as a sequence of tokens x ∈ X ( Z T , R V ) , where we cap the sequence length to T = 128 and set the vocabulary size to V = 1 , 000 . We perform a train-validation-test split of this dataset randomly (90%-5%-5%) and fit a 2-layers bag-of-word MLP on the training dataset for 20 epochs with Adam and a cosine annealing learning rate. The best model (according to validation accuracy) achieves a reasonable 86% accuracy on the test set D test . Let us justify the bag-of-word classifier invariance with respect to the token pertmutation group S T . Abag-ofwords classifier f receives a sequence of tokens x ∈ X ( Z T , R V ) and outputs class logits f ( x ) ∈ R 2 . By definition, the bag-of-word classifier can be written as a function f ( x ) = g Ä ∑ T t =1 onehot( x t ) ä . In this form, the invariance of the classifier with respect to tokens permutation is manifest. Let π ∈ S T be a permutation of the token indices. Applying this permutation to the token sequence does not change the classifier's output: f ( x π ) = g Ä ∑ T t =1 onehot( x π ( t ) ) ä = g Ä ∑ T t =1 onehot( x t ) ä = f ( x ) . We conclude that bag-of-words classifiers are S T -invariant.

FashionMNIST. The FashionMNIST dataset [61] consists of 28 × 28 grayscale images x ∈ X ( Z W × Z H , R ) with W = H = 28 , each representing a fashion object (e.g. dress) from 10 categories. Each image comes with a label y ∈ Z 10 indicating object's category. We train a 2-dimensional CNN to predict this label. We pad each image by adding 10 black pixels in each direction, so that the image information content is conserved by applying translations from the group G = ( Z / 10 Z ) 2 . The CNN is made invariant under the action of this group by using global pooling.

CIFAR100. The CIFAR100 dataset [63] consists of 32 × 32 RGB images x ∈ X ( Z W × Z H , R 3 ) with W = H = 32 , each representing an object (e.g. truck) from 100 categories. Each image comes with a label y ∈ Z 100 indicating object's category. We train the D 8 -D 4 -D 1 28/10 WideResNet from [64] to predict this label. The design of this model imposes a strong bias toward D 8 invariance. To avoid artifacts created by rotating an image by 45°, we restrict all our evaluations to the subgroup D 4 ⊂ D 8 .

STL10. The STL10 dataset [65] consists of 96 × 96 RGB images x ∈ X ( Z W × Z H , R 3 ) with W = H = 96 , each representing an object (e.g. truck) from 10 categories. Each image comes with a label y ∈ Z 10 indicating object's category. We train the D 8 -D 4 -D 1 16/8 WideResNet from [64] to predict this label. The design of this model imposes a strong bias toward D 8 invariance. To avoid artifacts created by rotating an image by 45°, we restrict all our evaluations to the subgroup D 4 ⊂ D 8 .

Data Split. All the datasets are endowed with a natural train-test split. In the ECG dataset, the different types of abnormal heartbeats are imbalanced (e.g. fusion beats only amount for . 7% of the training set). We use SMOTE [80] to augment the proportion of each type of abnormal heartbeat in the training set.

Symmetry Groups. Each dataset in the experiment is associated to a specific group of symmetry and group representation. We detail those in Table 4. We note that all these groups and group representations are easily implemented as tensor operations on the dimensions of the tensor corresponding to the domain Ω .

Models. We provide a detailed architecture for each model in Tables 5 to 11. All the models are implemented with Pytorch [79] and PyG [81]. All the models except the WideResNets are trained using Adam [82]. The CNNs are trained to minimize the cross entropy loss for 200 epochs with early stopping and patience 10 with a learning rate of 10 -3 and a weight decay of 10 -5 . The GNN is trained to minimize the negative log likelihood for 200 epochs with early stopping and patience 20 with a learning rate of 10 -3 and a weight decay of 10 -5 . The Deep Set is trained to minimize the cross entropy loss for 1,000 epochs with early stopping and patience 20 with a learning rate of 10 -3 , a weight decay of 10 -7 and a multi step learning rate scheduler with γ = 0 . 1 . The WideResNets are trained with Stochastic Gradient Descent to minimize the negative cross entropy loss for 200 epochs (1,000 for STL10) with an initial learning rate of 0 . 1 , a weight decay of 5 · 10 -5 , momentum 0 . 9 and an exponential learning rate scheduler with γ = 0 . 2 applied each 60 epochs (300 for STL10). The test set is used as a validation set in some cases, as the model generalization is never used as

an evaluation criterion. All the parameters hyperparameters that are not specified are chosen to the Pytorch and PyG default value. Note that for each architecture, we have highlighted the layer that we call Inv and Equiv in Section 3. The representation-based interpretability methods rely on the output of these layers.

Model Invariance. We note that Figure 5 includes model invariance on the x-axis . The metric we use to evaluate this invariance is simply adapted from Definition 2.1: Inv G ( f, x ) ≡ s Y [ f ( ρ [ g ] x ) , f ( x )] , where s Y : Y 2 → R is defined as s Y ( y 1 , y 2 ) = y ⊺ 1 y 2 / ∥ y 1 ∥·∥ y 2 ∥ for all y 1 , y 2 ∈ Y . We note that this cos-similarity similarity metric is sensible in a classification setting since s Y ( y 1 , y 2 ) = 1 implies y 1 = α · y 2 for some α ∈ R + . Since ∥ y 1 ∥ 1 = ∥ y 2 ∥ 1 = 1 , this is equivalent to y 1 = y 2 .

Feature Importance. We study gradient-based methods with Integrated Gradients [18] and Gradient Shap [17] as well as perturbation-based methods with Feature Ablation, Permutation [49] and Occlusion [83]. We also use DeepLift [19]. Whenever a baseline is required, we consider the trivial signal ¯ x = 0 as a baseline. We note that this signal is trivially G -invariant as ρ [ g ]¯ x = 0 for all g ∈ G .

Example Importance. We study loss-based methods with Influence Functions [21] and TracIn [22] as well as representation-based methods with SimplEx [23] and Representation Similarity. Since loss-based methods are expensive to compute, we differentiate the loss only with respect to the last layer of each model and for a subset of N train = 100 training examples from D train . For representation-based methods, we use the output of both invariant and equivariant layers of the model as representation spaces. We denote e.g. SimplEx-Inv to indicate that SimplEx was used by using the output of a G -invariant layer of the model as a representation space. Similarly, SimplEx-Equiv corresponds to SimplEx used with the output of a G -equivariant layer as a representation space.

Concept-Based Explanations. To the best of our knowledge, the only 2 post-hoc concept explanation methods in the literature are CAV [24] and CAR [25]. We use these methods to probe the representations of our models through a set of C = 4 concepts specific to each dataset. Again, we use both invariant and equivariant layers of the model for each method.

Concepts. We use a set of C = 4 concepts for ECG, Mutagenicity and ModelNet40 ; C = 2 for FashionMNIST and STL10; C = 3 for CIFAR100. For the ECG dataset, we use concepts defined by cardiologists to identify abnormal heartbeats: Premature Ventricular , Supraventricular , Fusion Beats and Unknown . We note that these concepts labels are directly available in the ECG dataset [84]. For the Mutagenicity dataset, we use the presence of known toxicophores [53] in the molecule of interest: Nitroso (presence of a N O in the molecule), Aliphatic Halide (presence of a Cl,Br,I in the molecule), Azo-type (presence of a N N in the molecule) and Nitroso-type (presence of a O + N O in the molecule). To detect the presence concepts, each molecule in the dataset is checked by using NetworkX [85]. For the ModelNet40 dataset, we use visual concepts whose presence can immediately be inferred from the class label: Foot (whether the represented object has a foot), Container (whether the represented object has the shape of a container), Parallelepiped (whether the represented object has a parallelepiped shape) and Elongated (whether the represented object is stretched in one direction). For all the image datasets, we use concepts that can be directly inferred from the class of each example. We use the Top and Shoe concepts for FashionMNIST; the Aquatic , People and Vehicle concepts for CIFAR100; the Vehicle and Wings concept for STL10.

Interpretability Methods Implementation. For the feature importance methods, we use their Captum [86] implementation. All the other explanations methods are reimplemented based on their official repository and adapted to graph data. All the concept classifiers are implemented with scikitlearn [87]. We use training sets of size 200 for the ECG, FashionMNIST, CIFAR100, STL10 datasets and 500 for the Mutagenicity and ModelNet40 datasets. CAR classifiers are chosen to be SVCs with Gaussian RBF kernel and default hyperparameters. CAV classifiers are linear classifiers trained with stochastic gradient descent optimizer with a learning rate of 10 -2 and a tolerance of 10 -3 for 1 , 000

epochs. In the case of CAR classifiers, we found it useful to apply PCA with 10 principal components to reduce the dimension of the latent representation before applying the classification step. Indeed, this does not significantly reduce the accuracy of the resulting classifiers while significantly reducing the runtime.

Impossible Configurations. We note that some combinations of interpretability methods and models are impossible. We summarize and explain the incompatibilities in Table 12. Further, we do not use perturbation-based feature importance methods with the CIFAR100 and STL10 ResNets, since those methods require O [dim( X (Ω , C ))] model calls per example, which is prohibitively expensive for large vision models.

Explanation Similarity. In Definition 2.1, we also chose the cos-similarity metric s E [ a, b ] for two real-valued explanations a, b ∈ R d E . This choice might seem arbitrary at first glance. By looking more closely, we notice that s E [ a, b ] = 1 implies a = α · b for some α ∈ R + . For all type of explanation that we consider in this paper, a and b will typically be vectors that aggregate various important scores (e.g. for each feature or training example). In practice, the scale of these importance scores does not matter. Indeed, if a = α · b , then the most important components will be the same for both a and b . Furthermore, the relative importance between any pair of component will be identical for both a and b . Therefore, we can consider that s E [ a, b ] = 1 implies that both explanations are equivalent to each other a ∼ b .

G Comparison with Sensitivity

In this appendix, we compare our robustness metric with the sensitivity metric introduced by [35]. This metric studies the robustness of an explanation with respect to a small perturbation in the input features. It is defined as follows:

where ∥ · ∥ ∞ denotes the l ∞ norm, ∥ · ∥ 2 denotes the l 2 norm and ϵ ∈ R + is a small positive constant that is fixed to ϵ = . 02 in the default Captum implementation of the metric. The rationale behind this metric is the following: a small perturbation of the input should have little impact on the model's prediction and, hence, little impact on the resulting explanation as well. For this reason, one typically expects a low sensitivity Sens( e, x ) for an explanation that is robust to small input shifts. We simply note that the previous reasoning is debatable due to the existence of adversarial perturbations that are small in norm but that have large effect on the model's prediction [88].

Like our invariance and equivariance robustness metrics, Sens measures how the explanation is affected by a transformation of the input. Therefore, a natural question arises: is there a difference in practice between our robustness metrics and the sensitivity metric? To answer this question, we

consider the setup of Section 3 with the ECG dataset and the Augmented-CNN. We study feature importance methods e , for which we evaluate Sens( e, x ) and Equiv G ( e, x ) for N test = 1 , 000 test examples x ∈ D test . We report the results in Figure 8.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Explanation Sensitivity

0.2

0.0

0.2

0.4

0.6

0.8

1.0

Explanation Equivariance

Integrated Gradients

Gradient Shap

Feature Permutation

Feature Ablation

Feature Occlusion

As we can observe on Figure 8, there exists a weak downward trend between the two metrics. By evaluating the Pearson correlation coefficient between the two metrics for each method, we found r = -. 62 for Integrated Gradients, r = -. 48 for Gradient Shap, r = -. 38 for Feature Permutation, r = -. 23 for Feature Ablation and r = -. 31 for Feature Occlusion. This indicates that examples with higher sensitivity tend to be associated with lower equivariance. That being said, the correlation between the two metrics is too weak to suggest any redundancy between them. We deduce that our robustness metrics should be use to complement rather than replace the sensitivity metric. In practice, we believe that many forms of explanation robustness deserve to be investigated.

H Other Invariant Architectures

In Section 3, we have illustrated the flexibility of our robustness formalism by applying it to various architecture. However, geometric deep learning extends well-beyond the examples covered in this paper and an exhaustive treatment of the field is beyond our scope. To suggest future extensions of our work beyond the architectures examined in this paper, we provide in Table 13 other architectures that are invariant or equivariant with respect to other symmetry groups. Our formalism straightforwardly applies to most of these examples. For some others (like Spherical CNNs), an extension to infinite groups would be required. We leave this extension for future work.

I Examples of Non-robust Explanations

In this appendix, we illustrate the importance of our notion of robustness by presenting failure modes. Since images are the easiest way to visualize, we focus our attention to the FashionMNIST and STL10 datasets. In Figures 9 and 10, we show non-robust saliency maps obtained with Gradient Shap and DeepLift. As we can observe, the model prediction is largely unaffected by applying a symmetry ( s E [ f ( ρ [ g ] x ) , f ( x )] ≈ 1 in all cases), while the saliency map look qualitatively different. We have included a Jupyter Notebook in our code to easily visualize other failure modes.

Predicted Class:Shirt

OriginalSaliencyMap

PredictedClass:Shirt

TransformedSaliencyMap

Predicted Class:T-shirt/top

Original SaliencyMap

PredictedClass:T-shirt/top

TransformedSaliencyMap

Predicted Class:Dress

Original Saliency Map

Predicted Class:Dress

TransformedSaliencyMap

PredictedClass:Sneaker

Original Saliency Map

PredictedClass:Sneaker

TransformedSaliencyMap

Predicted Class:Shirt

Original Saliency Map

Predicted Class:Shirt

TransformedSaliencyMap

Predicted Class:Car

OriginalSaliencyMap

PredictedClass:Car

TransformedSaliencyMap

Predicted Class:Monkey

Original SaliencyMap

PredictedClass:Monkey

TransformedSaliencyMap

Predicted Class:Truck

Original SaliencyMap

Predicted Class:Truck

TransformedSaliencyMap

Predicted Class:Ship

OriginalSaliencyMap

Predicted Class:Ship

TransformedSaliencyMap

PredictedClass:Car

Original SaliencyMap

PredictedClass:Car

TransformedSaliencyMap

J Effect of Distribution Shift

In this appendix, we study the impact of evaluating out robustness metrics on samples from a distribution that differs from model's training distribution. To that end, we shall use the CINIC-10 dataset [96], which contains ImageNet images that have the same classes as CIFAR-10 classes (and, hence, the same classes as the STL-10 dataset). In this way, we are able to reuse the ResNets trained on STL-10 for evaluation on the CINIC-10 dataset. The only requirements is to resize the CINIC-10

images so that they match the resolution of the STL-10 images ( 96 × 96 pixels). We evaluate the robustness metrics with respect to the dihedral group D 4 for a few feature importance and example importance methods in Tables 14 and 15. In both cases, we find that these metrics are consistent with their STL-10 counterpart in Figures 3(a) and 3(b).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the robustness of explanations. It begins by defining explanation invariance and equivariance concepts using geometric deep learning formalism and demonstrates that certain popular interpretability methods inherently possess theoretical robustness guarantees. Two metrics, invariance and equivariance scores, are introduced for empirically assessing explanation robustness. These metrics are applied to evaluate various interpretability methods across different modalities. Finally, the paper provides a set of five actionable guidelines to ensure that interpretability methods are employed in a manner that guarantees robustness.\n\nSTRENGTHS:\n1. The paper presents a high-level framework for evaluating the robustness of explanations and introduces two corresponding metrics. Unlike previous work that mostly focuses on saliency-based explanations for image classification, this framework can be applied to various explanation methods (feature-based, concept-based, and example-based) and modalities (images, graphs, and time series);\n\n2. In addition to offering an evaluation framework, the authors also provide guidelines for generating robust explanations. These insights can assist the community in developing improved explanation methods.\n\nWEAKNESSES:\n1. The paper's organization could be better aligned with the summaries of contributions provided in the abstract and on page 3. The structure in later sections does not closely follow these summaries, which may make it difficult for readers to follow the narrative.\n\n\n2. The paper appears to cover many points, potentially leading to the omission of important details. \n\nI am interested in understanding how different explanation methods relate to theoretical robustness guarantees (invariant, equivariant), but the paper only presents the results (Table 1) without discussing them. Although the mathematical proofs are available in Appendix D, the main paper lacks an explanation and discussion of these results. For instance, while Table 1 indicates that gradient-based methods have conditional equivariance guarantees, the necessary conditions or assumptions are not explicitly stated. Including explanations or discussions in the main paper would significantly improve clarity.\n\n3. The use of the Dihedral Group for CIFAR10 and STL10 in the experimental section is unclear. It would be helpful if the authors provided an example of the transformations applied to the images in this context.\n\nQUESTIONS:\n1. Could the authors provide further discussion and explanations regarding the results in Table 1, especially for those that are conditionally guaranteed?\n\n2. In the experimental section, what does the Dihedral Group represent for CIFAR10 and STL10? Could the authors share an example of the specific transformations applied to images in this scenario?\n\n3. While there are numerous publications on the explanations for NLP, the paper does not mention the robustness of explanations for NLP. Is it possible to apply the proposed framework to evaluate the robustness of these methods within the NLP domain?\n\nLIMITATIONS:\nSee weaknesses.\n\nIn general, I think the paper tackles an important research question in XAI and proposes a valuable framework to address the issue. However, the concerns mentioned in the \"weaknesses\" section, such as missing or unclear information, may lead to confusion and make it difficult for readers to fully understand the paper.\n\nIf these related questions can be well answered, I may consider raising my rating.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper study the robustness of several post-hoc interpretability methods against the transformation of input data. The robustness is measured by invariance and equivariance metrics. Theoretical robustness guarantees and a systematic approach to increase the invariance are derived. Finally, the authors conduct extensive experiments to validate the theoretical analysis of robustness guarantees, using the proposed evaluation metrics.  5 actionable guidelines are derived to improve the robustness.\n\nSTRENGTHS:\n1. The research problem studied in this paper is interesting and novel. Robustness of interpretability against more general input transformation is few studied before.\n\n2.  This paper extends the robustness evaluation to other interpretability methods, like example importance and concept-based explanations. These are missing from the current literature.\n\n3.  Based on the evaluation, the theoretical robustness guarantees are derived for the popular interpretability methods.\n\n4. The experiment is comprehensive and insightful that 5 practical guidelines are derived for the robustness improvement.\n\nWEAKNESSES:\n1. The details about the symmetries are vague and even missing. Which kind of input transformation are considered, rotation, crop, or translation?\n\n2. Since the invariance metric is in conflict with the equivariance metric, it may be confusing for users to decide which metric is more suitable, given the symmetries. For example, if the small perturbations are added into the input data, the invariance of feature explanation is expected.\n \n3. The simple Monte Carlo Sampling is inefficient for the evaluation of two robustness metrics, especially for the rare events.\n\n4. The ImageNet dataset should be considered for the experiments.\n\nQUESTIONS:\n1. Please provide more details of the symmetry group. How does the change of symmetry group affect the robustness evaluation results?\n\n2. What's challenges when applying proposed evaluation method to real world dataset and interpreter?\n\nLIMITATIONS:\nPlease refer to the weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the definition of the robustness of explanations with respect to the model symmetry group. For models invariant to some symmetry group, the explanation should also be invariant or equivariant to it. The paper derives two metrics to measure the invariance and equivalence of explanations and analyzes the robustness requirement for three explanation methods (e.g., feature importance needs to be equivariant, and example importance and concept-based explanations need to be invariant). It also theoretically analyzes the guarantees of different methods in robustness. In the end, the paper proposes to improve the robustness by aggregating the explanations over several symmetries. Experiments show that different explanation methods have different invariance/equivariance properties.\n\nSTRENGTHS:\n1. The paper is very well-written and easy to follow.\n2. The relevant literature is summarized comprehensively. The considered problem, evaluating the robustness of explanation from a geometry perspective, is not well-studied before.\n3. The concepts of explanation invariance and equivaraince are novel. The evaluation metrics are sound.\n4. Multiple explanation methods are evaluated in the experiments.\n\nWEAKNESSES:\n1. The biggest weakness is the limited scenarios that can use the proposed evaluation methods. The paper only considers models that are perfectly invariant. Please see Limitations for details.\n2. The requirement of the model's invariance is not well defined. Under some group transformations, while the hard-label prediction of the model (i.e. after argmax) is unchanged, the soft-label prediction (i.e. after softmax but before argmax) might be changed. The model is invariant from the first perspective. However, from the second perspective, the model is non-invariant and the explanation should also be non-invariant/non-equivariant. I guess that is why the saliency map is not equivariant while the model is considered invariant in Appendix I.\n3. The symmetry groups considered in experiments are limited.\n\nQUESTIONS:\n1. The paper only considers the post-hoc interpretability. Is it possible to extend the concept of explanation invariance/equivariance to other methods? For example, the attention should be equivariant under input translation?\n3. When the explanation space is not identical to the input space, what is group representation $\\rho'$?\n4. Is there any trade-off between the robustness and the utility of the explanation?\n\n\nMinor:\n1. The colors in Figure 2 are hard to distinguish.\n2. What is \"Dihedral Group\" in Table 2?\n\nLIMITATIONS:\nThe proposed definitions and evaluations heavily rely on the assumption that the model is perfectly invariant. I appreciate the experiments and discussions in sec 3.3 which suggest that there's no linear relationship between model invariance and explanation invariance/equivariance. Therefore, models not designed but trained for invariance can not use the proposed methods. However, this assumption limits the model architectures and group transformations considered and thus limits the application of the proposed methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe core contribution of this work is the direction of measuring explanation robustness through a more broader set of data perturbations/transformations for different data modalities which have not been discussed in literature thus far. For instance shift transformations in images, cyclic translations in time series, etc. The authors define concepts of explanation invariance \nand equivariance and expose their theoretical properties. They conduct experiments to measure these types of invariances for 3 different types of explanations and several datasets and also present guidelines on which invariances to measure in which use cases.\n\nSTRENGTHS:\nOverall, making explanations more trustworthy is an important area to investigate. Measuring robustness of explainability methods is valuable for their use in practical deployed systems. \n\n- Authors define several types of perturbations for different data modalities. Authors define the concepts of invariance and equivariance for different data modalities and show their theoretical properties. \n\n- Authors have presented a broad collection of experiments with several data modalities and explainability types to show that many existing explainability methods are not robust as measured via invariance & equivariance.\n\nWEAKNESSES:\nThere seem to be links between model robustness and explanation robustness which are not explored in the draft. \nFor example, if the model is not robust to certain types of suggested data perturbations (when we might expect it to), then would it not make sense to measure if the explanations are not in fact the same? Because the same top reason/explanation why a data sample was classified in class A cannot also be the reason why its not classified in class A.\n\nQUESTIONS:\n- Is there any reason why text models were not evaluated? It could make sense in tasks for e.g. summarization, etc. if certain facts are shifted from the beginning to the end of a paragraph and if the importance score on words still come up right. \n\n- In your experiments are you assuming that the models you pick are already robust to the type of perturbations you add to the data to measure explanation robustness(invariance/equivarance)? Or is explanation invariance being measured irrespective of whether the model is robust or not.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a set of desiderata for explanation methods for neural networks ranging from CNNs to GNNs. They postulate that any explanation that is able to faithfully explain the model should be in agreement with the invariance properties exhibited by the underlying model. They formalize this idea by introducing explanation invariance and equivariance with reference to specific symmetry groups. Using this formulation, they derive metrics that measure robustness of several interpretability methods and some theoretical guarantees. Their experiments verify explanation robustness for models trained on diverse modalities such as images, time series and tabular data. They provide guidelines for developers to develop robust model explanations using their metrics.\n\nSTRENGTHS:\nEvaluating explanation approaches is an important area of research not only for evaluating existing approaches but also to aid practitioners in developing new explanations that are robust. \n\nThe paper is written very clearly albeit a bit dense to parse – this does not take away from the general reader’s experience of the paper, but it could use a more exemplar way of introducing concepts (Specific comments below)\n\nThe explanation methods and the models tested encompass several modalities. This is a real strength of the paper as explanation evaluations are usually limited to analysis of salience maps.\n\nWEAKNESSES:\n1. Dense nature of the writing: While I appreciate the authors’ efforts in introducing the concepts of geometric priors and group symmetry to the readers, the paper could be more clear in explaining what transformations are being considered in different symmetry groups. For instance, what transformations are contained in the D8 symmetry group for the CNN based examples? A table to this effect can help the readers understand what transformations are being evaluated.\n2. Example results (positive and negative): The discussion of the results is rather pedantic, examples of why improving invariance and equivariance would make sense given an invariant model would help the reader understand the importance of the metrics. Without this context, it is hard to differentiate, say, an invariance score of 0.9 to a 0.5.\n3. Why only exact invariance and equivariance? I wonder if in its current form the metrics are too rigid. As the authors mention in the paper, most networks for real world problems are only approximately invariant or equivariant. In fact, characterizing such property is a hard problem in itself. In these cases, it’s hard to see how the current approach can help. \n4. This approach is limited in that the only transformations that are considered are geometric in nature. While this is alluded to in the appendix, more real world examples of invariance are from non-trivial corruptions such as measurement error or signal degradation. \n5. The paper presents no conclusion and more discussion + visualization of the results is needed, especially in cases where the metrics fail to capture the expected behavior.\n\nQUESTIONS:\n1. Can the metrics be extended to approximately-invariant/equivariant networks? What are the pros and cons of such an extension?\n2. Are invariance and equivariance the only properties that need to be considered for explanations from a geometric perspective?\n\nLIMITATIONS:\nPlease refer to weaknesses above. There is not much potential for a negative societal impact - evaluating explanations would only serve to impact society positively.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}