SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  on which the feature importance methods rely. Typically, these baselines signals are used to replace ablated features from the original signal  $x \in \mathcal{X}(\Omega, \mathcal{C})$  (i.e. remove a feature  $x_i$  by replacing it by  $\bar{x}_i$ ). In order to guarantee equivariance, we require this baseline signal to be invariant to the action of each symmetry  $g \in \mathcal{G}: \rho[g] \bar{x} = \bar{x}$ . (2) The second assumption restricts the type of representation  $\rho$  that can be used to describe the action of the symmetry group on the signals. In order to guarantee equivariance, we require this representation to be a permutation representation, which means that the action of each symmetry  $g \in \mathcal{G}$  is represented by a permutation matrix  $\rho[g]$  acting on the signal space  $\mathcal{X}(\Omega, \mathcal{C})$ .
When it comes to example importance methods, the assumptions depend on how the importance scores are obtained. If the importance scores are computed from the model's loss, then the invariance of the explanation immediately follows from the model's invariance. If the importance scores are computed from the model's internal representations  $h: \mathcal{X}(\Omega, \mathcal{C}) \to \mathbb{R}^{d_{\mathrm{rep}}}$ , then the invariance of the explanation can only be guaranteed if the representation map  $h$  is itself invariant to action of each symmetry:  $h(\rho[g]x) = h(x)$ . Similarly, concept-based explanations are also computed from the model's representations  $h$ . Again, the invariance of these explanations can only be guaranteed if the representation map  $h$  is itself invariant.
Enforcing Invariance. If the explanation  $e$  is not  $\mathcal{G}$ -invariant when it should, we can construct an auxiliary explanation  $e_{\mathrm{inv}}$  built upon  $e$  that is  $\mathcal{G}$ -invariant. This permits to improve the robustness of any interpretability method that has no invariance guarantee. The idea is simply to aggregate the explanation over several symmetries.
Proposition 2.3. [Enforce Invariance] Consider a neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$  and  $e: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{E}$  be an explanation for  $f$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \mathrm{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We define the auxiliary explanation  $e_{\mathrm{inv}}: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{E}$  as
$$
e _ {\text {i n v}} (x) \equiv \frac {1}{| \mathcal {G} |} \sum_ {g \in \mathcal {G}} e (\rho [ g ] x)
$$
for all  $x\in \mathcal{X}(\Omega ,\mathcal{C})$  . The auxiliary explanation  $e_{\mathrm{inv}}$  is invariant under the symmetry group  $\mathcal{G}$
Proof. Please refer to Appendix D.
Remark 2.4. Once again, a Monte Carlo estimation for  $e_{\mathrm{inv}}$  might be required for groups  $\mathcal{G}$  with many elements. This produces explanations that are approximatively invariant.
In this section, we use our interpretability robustness metrics to draw some insights with real-world models and datasets. We first evaluate the  $\mathcal{G}$ -invariance and equivariance of popular interpretability methods used on top of  $\mathcal{G}$ -invariant models. With this analysis, we identify interpretability methods that are not robust. We then show that the robustness of these interpretability methods can largely be improved by using their auxiliary version defined in Proposition 2.3. Finally, we study how the  $\mathcal{G}$ -invariance and equivariance of interpretability methods varies when we decrease the invariance of the underlying model. From these experiments, we derive 5 guidelines to ensure that interpretability
methods are robust with respect to symmetries from  $\mathcal{G}$ . We summarize these guidelines with a flowchart in Figure 6 from Appendix A.
The datasets used in our experiment are presented in Table 2. We explore various modalities and symmetry groups throughout the section, as described in Table 3. For each dataset, we fit and study a classifier from the literature designed to be invariant with respect to the underlying symmetry group. For each model, we evaluate the robustness of various feature importance, example importance and concept-based explanations. More details on the experiments are available in Appendix F. We also include a comparison between our robustness metrics and the sensitivity metric in Appendix G. The code and instructions to replicate all the results reported below are available in the public repositories https://github.com/JonathanCrabbe/RobustXAI and https://github.com/vanderschaarlab/RobustXAI.
Table 2: Different datasets used in the experiments.
Dataset	# Classes	Modality	Symmetry Group	Model
Electrocardiograms [50, 51]	2	Time Series	Cyclic Translations Z/TZ	All-CNN [52]
Mutagenicity [53-55]	2	Graphs	Node Permutations SVx	GraphConv GNN [56]
ModelNet40 [57-59]	40	3D Point Clouds	Point Permutations SNpt	Deep Set [58]
IMDb [60]	2	Text	Token Permutation ST	Bag-of-words MLP
FashionMNIST [61]	10	Images	Cyclic Translations (Z/10Z)2	All-CNN [52]
CIFAR100 [62]	100	Images	Dihedral Group D8	E(2)-WideResNet [63, 64]
STL10 [65]	10	Images	Dihedral Group D8	E(2)-WideResNet [63, 64]
Table 3: Various symmetry groups used in the experiments.
Symmetry Group	Acting on	Description
Translation Z/NZ	Time series, Images	Shifts signals in time and images horizontally & vertically.
Permutation SN	Graph nodes, Points in clouds, Tokens	Changes the ordering of nodes / points / tokens in feature matrices.
Dihedral D8	Images	Rotate / reflects the images though angles 45°, 90°, 135°, 180°, 225°, 315°
3.1 Evaluating Interpretability Methods
Motivation. The purpose of this experiment is to measure the robustness of various interpretability methods. Since we manipulate models that are invariant with respect to a group  $\mathcal{G}$  of symmetry, we expect feature importance methods to be  $\mathcal{G}$ -equivariant (Equiv $_{\mathcal{G}}[e,x] = 1$  for all  $x \in \mathcal{X}(\Omega,\mathcal{C})$ ). Similarly, we expect example and concept-based methods to be  $\mathcal{G}$ -invariant (Inv $_{\mathcal{G}}[e,x] = 1$  for all  $x \in \mathcal{X}(\Omega,\mathcal{C})$ ). We shall now verify this empirically.
Methodology. To measure the robustness of interpretability methods empirically, we use a set  $\mathcal{D}_{\mathrm{test}}$  of  $N_{\mathrm{test}}$  examples ( $N_{\mathrm{test}} = 433$  for Mutagenicity,  $N_{\mathrm{test}} = 1,000$  for ModelNet40 and Electrocardiograms (ECG) and  $N_{\mathrm{test}} = 500$  in the other cases). For each interpretability method  $e$ , we evaluate the appropriate robustness metric for each test example  $x \in \mathcal{D}_{\mathrm{test}}$ . For Mutagenicity and ModelNet40, the large order  $|\mathcal{G}|$  makes the exact evaluation of the metric unrealistic. We therefore use a Monte Carlo approximation with  $N_{\mathrm{samp}} = 50$ . As demonstrated in Appendix E, the Monte Carlo estimators have already converged with this sample size. In all the other cases, these metrics are evaluated exactly since  $\mathcal{G}$  has a tractable order  $|\mathcal{G}|$ . Since the E(2)-WideResNets for CIFAR100 and STL10 are only approximatively invariant with respect to  $\mathbb{D}_8$ , we defer their discussion to Section 3.3. We note that some interpretability methods cannot be used in some settings. Whenever this is the case, we simply omit the interpretability method. Please refer to Appendix F for more details.
Analysis. We report the robustness score for each metric and each dataset on the test set  $\mathcal{D}_{\mathrm{test}}$  in Figures 3(a) to 3(c). We immediately notice that not all the interpretability methods are robust. We provide some real examples of non-robust explanations in Appendix I in order to visualize the failure modes. When looking at feature importance, we observe that equivariance is not guaranteed by methods that rely on baseline that are not invariant. For instance, Gradient Shap and Feature Permutation rely on a random baseline, which has no reason to be  $\mathcal{G}$ -invariant. We conclude that the
(a) Feature Importance
(b) Example Importance
(c) Concept Importance
(d) Training Dynamics for E(n)-WideResNet
Figure 3: Explanation robustness of interpretability methods for invariant models. The interpretability methods are grouped by type. Each box-plot is produced by evaluating the robustness metrics  $\mathrm{Inv}_{\mathcal{G}}$  or  $\mathrm{Equiv}_{\mathcal{G}}$  across several test samples  $x \in \mathcal{D}_{\mathrm{test}}$ . The asterisk (*) indicates a dataset where the model is only approximatively invariant. Those models are discussed in Section 3.3. For all other models, any value below 1 for the metrics is unexpected, as the model is  $\mathcal{G}$ -invariant.
invariance of the baseline  $\bar{x}$  is crucial to guarantee the robustness of feature importance methods. When it comes to example importance, we note that loss-based methods are consistently invariant, which is in agreement with Proposition D.9. Representation-based and concept-based methods, on the other hand, are invariant only if used with invariant layers of the model. This shows that the choice of what we call the representation space matters for these methods. We derive a set of guidelines from these observations.
Guideline 1. Feature importance methods should be used with group invariant baseline signal  $(\rho [g]\bar{x} = \bar{x}$  for all  $g\in \mathcal{G})$  to guarantee explanation equivariance. Only methods that conserve the invariance of the baseline can guarantee equivariance.
Guideline 2. Loss-based example importance methods guarantee explanation invariance, unlike representation-based methods. When using the latter, only invariant layers guarantee explanation invariance.
Guideline 3. To guarantee invariance of concept-based explanations, concept classifiers should be used on invariant layers of the model.
3.2 Improving Robustness
Motivation. In the previous experiment, we noticed that not all the interpretability methods are  $\mathcal{G}$ -invariant when they should. Consider, for instance, concept-based methods used on equivariant layers. The lack of invariance for these methods implies that they rely on concept classifiers that
(a) ECG
(b) FashionMNIST
Figure 4: Explanation invariance can be increased according to Proposition 2.3. This plot shows the score averaged on a test set  $\mathcal{D}_{\mathrm{test}}$  together with a  $95\%$  confidence interval.
are not  $\mathcal{G}$ -invariant. This behaviour is undesirable for two reasons: (1) since any symmetry  $g\in \mathcal{G}$  preserve the information of a signal  $x\in \mathcal{X}(\Omega ,\mathcal{C})$ , the signal  $\rho [g]x$  should contain the same concepts as  $x$  and (2) the layer that we use implicitly encodes these symmetries through equivariance of the output representations. Hence, concept classifiers that are not  $\mathcal{G}$ -invariant fail to generalize by ignoring the symmetries encoded in the structure of the model's representation space. Fortunately, Proposition 2.3 gives us a prescription to obtain explanations (here concept classifiers) that are more robust with respect to the model's symmetries. We shall now illustrate how this prescription improves the robustness of concept-based methods.
Methodology. In this experiment, we restrict our analysis to the ECG and FashionMNIST datasets. For each test signal, we sample  $N_{\mathrm{inv}}$  symmetries  $G_i \in \mathcal{G}, i \in \mathbb{Z}_{N_{\mathrm{inv}}}$  without replacement. As prescribed by Proposition 2.3, we then compute the auxiliary explanation  $e_{\mathrm{inv}}(x) = N_{\mathrm{inv}}^{-1} \sum_{i=1}^{N_{\mathrm{inv}}} e(\rho[G_i]x)$  for each concept importance method.
Analysis. We report the average invariance score  $\mathbb{E}_{X\sim U(\mathcal{D}_{\mathrm{test}})}\mathrm{Inv}_{\mathcal{G}}(e_{\mathrm{inv}},X)$  for several values of  $N_{\mathrm{inv}}$  in Figure 4. As we can see, the invariance of the explanation grows monotonically with the number of samples  $N_{\mathrm{inv}}$  to achieve a perfect invariance for  $N_{\mathrm{inv}} = |\mathcal{G}|$ . Interestingly, the explanation invariance increases more quickly for CAR. This suggests that enforcing explanation invariance is less expensive for certain interpretability methods and motivates the below guideline.
Guideline 4. Any interpretability method can be made invariant through Proposition 2.3. In doing so, one should increase the number of samples  $N_{\mathrm{inv}}$  until the desired invariance is achieved. In this way, the method is made robust without increasing the number of calls more than necessary. Note that it only makes sense to enforce invariance of the interpretability method if the explained model is itself invariant.
3.3 Relaxing Invariance
Motivation. In practice, models are not always perfectly invariant. A first example is given by the CIFAR100 and STL10 WideResNet that has a strong bias towards being  $\mathbb{D}_8$ -invariant, although it can break this invariance at training time (see Appendix H.3 of [64]). Another popular example is a CNN that flattens the output of convolutional layers, which violates translation invariance [52, 66, 67]. This motivates the study of interpretability methods robustness when models are not perfectly invariant.
Methodology. This experiment studies the two aforementioned settings. First, we replicate the experiment from Section 3.1 with the CIFAR100 and STL10 WideResNet. Second, we consider CNNs that flatten their last convolutional layer with the ECG and FashionMNIST datasets. In this case, we introduce 2 variants of the All-CNN where the global pooling is replaced by a flatten operation: an Augmented-CNN trained by augmenting the training set  $\mathcal{D}_{\mathrm{train}}$  with random translations and a Standard-CNN trained without augmentation. We measure the invariance/equivalence of the interpretability methods for each model.
Analysis. The results for the WideResNets are reported in Figures 3(a) to 3(c). We see that the robustness of various interpretability methods substantially drops with the model invariance. This is particularly noticeable for feature importance methods. To illustrate this phenomenon, we plot in Figure 3(d) the evolution during training of the model's prediction  $f(x) \mathcal{G}$ -invariance and the
$\mathcal{G}$ -equivariance of its gradient  $\nabla_x f(x)$ , on which the attribution methods rely. As we can see, the model remains almost invariant during training, while the gradients equivariance is destroyed. Similar observations can be made with the CNNs from Figure 5. In spite of the Augmented-CNN being almost invariant, we notice that the symmetry breaks significantly for feature importance methods. These results suggest that the robustness of interpretability methods can be (but is not necessarily) fragile if model invariance is relaxed, even slightly. This motivates our last guideline, which safeguards against erroneous interpretations of our robustness metrics.
Guideline 5. One should not assume a linear relationship between model invariance and explanation invariance/equivalence. In particular, the robustness of an interpretability method for an invariant model does not imply that this method is robust for an approximatively invariant model.
Figure 5: Effect of relaxing the model invariance on interpretability methods invariance/equivariance. The interpretability methods are grouped by type in each column. The error bars represent a  $95\%$  confidence interval around the mean for Inv and Equiv. Lin1 is to the output of the first dense layer of the CNN, which corresponds to the invariant layer used in Section 3.1.
Model Type	Explanation	Feature Permutation	▲ Influence Functions	▶ TracIn
All-CNN	DeepLift	Gradient Shap	Rep. Similar-Lin1	CAR-Lin1
Augmented-CNN	Feature Ablation	Integrated Gradients	Simplex-Lin1	CAV-Lin1
Standard-CNN	Feature Occlusion
Building on recent developments in geometric deep learning, we introduced two metrics (explanation invariance and equivariance) to assess the faithfulness of model explanations with respect to model symmetries. In our experiments, we considered a wide range of models whose predictions are invariant with respect to transformations of their input data. By analyzing feature importance, example importance and concept-based explanations of these models, we observed that many of these explanations are not invariant/equivariant to these transformations when they should. This led us to establish a set of guidelines in Appendix A to help practitioners choose interpretability methods that are consistent with their model symmetries.
Beyond actionable insights, we believe that our work opens up interesting avenues for future research. An important one emerged by studying the equivariance of saliency maps with respect to models that are approximately invariant. This analysis showed that state-of-the-art saliency methods fail to keep a high equivariance score when the model's invariance is slightly relaxed. This important observation could be the seed of future developments of robust feature importance methods.
The authors are grateful to the 5 anonymous NeurIPS reviewers for their useful comments on an earlier version of the manuscript. Jonathan Crabbé is funded by Aviva and Mihaela van der Schaar by the Office of Naval Research (ONR). This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
[1] Saining Xie, Ross Girshick, Piotr Dólar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[3] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021.
[4] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, 2018.
[5] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
[6] Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, 15(141):20170387, 2018.
[7] Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. What do we want from explainable artificial intelligence (xai)?-a stakeholder perspective on xai and a conceptual model guiding interdisciplinary xai research. Artificial Intelligence, 296:103473, 2021.
[8] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access, 6:52138-52160, 2018.
[9] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Inf. Fusion, 58(C):82-115, jun 2020.
[10] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
[11] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks. 2022.
[12] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems, 29, 2016.
[13] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. Advances in neural information processing systems, 27, 2014.
A How to Use our Framework in Practice 18
B Contribution and Hope for Impact 19
C Limitations 19
D Theoretical Results 20
D.1 Feature Importance Guarantees 20
D.2 Example Importance Guarantees 23
D.3 Concept-Based Explanations Guarantees 24
D.4Enforcing Invariance 25
E Convergence of the Monte Carlo Estimators 26
FExperiment Details 27
G Comparison with Sensitivity 33
H Other Invariant Architectures 34
I Examples of Non-robust Explanations 35
J Effect of Distribution Shift 36
A How to Use our Framework in Practice
Figure 6: Our guideline to improve the robustness of interpretability methods with respect to model symmetries.
To show how these guidelines can be used in practice, we illustrate two possible categories based on our experiments from Section 3. Let us start with Influence Functions and the ECG dataset. This is an example importance method, hence we choose the central branch from Figure 6. Measuring the invariance in Figure 3(b) shows that  $\mathrm{Inv}_{\mathcal{G}}[e,x]\approx 1$ , hence we may jump to the terminal node of the flowchart. This is correct since Influence Functions are  $\mathcal{G}$ -invariant as demonstrated in Proposition D.9.
Now let us take the example of CAV used with the equivariant layer from the Deep Set for the ModelNet40 dataset. This is a concept importance method, hence we choose the right branch of Figure 6. Measuring the invariance in Figure 3(c) shows that  $\mathrm{Inv}_{\mathcal{G}}[e,x] < 1$ , hence we go down. As the flowchart recommends, we may decide to use the explainability method on an invariant layer instead. Doing this yields  $\mathrm{Inv}_{\mathcal{G}}[e,x] \approx 1$ , as shown in Figure 3(c). Hence we may jump to the
terminal node of the flowchart. This is correct since concept important methods used with invariant layers are also  $\mathcal{G}$ -invariant, as demonstrated in Proposition D.14.
B Contribution and Hope for Impact
In this appendix, we discuss the significance of our work. We should firstly emphasize the importance of the problem we are tackling in this work. In order to provide valuable explanations of a model, it is crucial to ensure that interpretability methods faithfully describe the explained model. Indeed, failing in this basic criterion implies that the explanations could be inconsistent with the true model behaviour, hence leading to false insights about the model. For this reason, we believe that guaranteeing an alignment between interpretability methods and the model is a problem of the utmost importance. Beyond the significance of this problem, we believe that we bring a substantial contribution to address it. Below, we enumerate the novelties we claim.
Connecting interpretability with geometric deep learning. In Sections. 2, we show that the formalism of geometric deep learning naturally extends to the description of interpretability methods. We believe that this bridge can be valuable for at least two communities. For the interpretability community, geometric deep learning provides a rigorous framework to ensure that interpretations are consistent with strong inductive biases that underpin cutting edge deep neural networks, such as GNNs, CNNs, Deep Sets, Transformers and the other examples cited in Appendix H. For the geometric deep learning community, interpretability provides a way to extract actionable insights from increasingly sophisticated architectures. We hope that our paper promotes collaborations between these two communities.