SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feature Importance. We study gradient-based methods with Integrated Gradients [18] and Gradient Shap [17] as well as perturbation-based methods with Feature Ablation, Permutation [49] and Occlusion [83]. We also use DeepLift [19]. Whenever a baseline is required, we consider the trivial signal  $\bar{x} = 0$  as a baseline. We note that this signal is trivially  $\mathcal{G}$ -invariant as  $\rho[g]\bar{x} = 0$  for all  $g \in \mathcal{G}$ .
Example Importance. We study loss-based methods with Influence Functions [21] and TracIn [22] as well as representation-based methods with SimplEx [23] and Representation Similarity. Since loss-based methods are expensive to compute, we differentiate the loss only with respect to the last layer of each model and for a subset of  $N_{\mathrm{train}} = 100$  training examples from  $\mathcal{D}_{\mathrm{train}}$ . For representation-based methods, we use the output of both invariant and equivariant layers of the model as representation spaces. We denote e.g. SimplEx-Inv to indicate that SimplEx was used by using the output of a  $\mathcal{G}$ -invariant layer of the model as a representation space. Similarly, SimplEx-Equiv corresponds to SimplEx used with the output of a  $\mathcal{G}$ -equivariant layer as a representation space.
Concept-Based Explanations. To the best of our knowledge, the only 2 post-hoc concept explanation methods in the literature are CAV [24] and CAR [25]. We use these methods to probe the representations of our models through a set of  $C = 4$  concepts specific to each dataset. Again, we use both invariant and equivariant layers of the model for each method.
Concepts. We use a set of  $C = 4$  concepts for ECG, Mutagenicity and ModelNet40;  $C = 2$  for FashionMNIST and STL10;  $C = 3$  for CIFAR100. For the ECG dataset, we use concepts defined by cardiologists to identify abnormal heartbeats: Premature Ventricular, Supraventricular, Fusion Beats and Unknown. We note that these concepts labels are directly available in the ECG dataset [84]. For the Mutagenicity dataset, we use the presence of known toxicophores [53] in the molecule of interest: Nitroso (presence of a N=O in the molecule), Aliphatic Halide (presence of a Cl, Br, I in the molecule), Azo-type (presence of a N=N in the molecule) and Nitroso-type (presence of a O+-N=O in the molecule). To detect the presence concepts, each molecule in the dataset is checked by using NetworkX [85]. For the ModelNet40 dataset, we use visual concepts whose presence can immediately be inferred from the class label: Foot (whether the represented object has a foot), Container (whether the represented object has the shape of a container), Parallelepiped (whether the represented object has a parallelepiped shape) and Elongated (whether the represented object is stretched in one direction). For all the image datasets, we use concepts that can be directly inferred from the class of each example. We use the Top and Shoe concepts for FashionMNIST; the Aquatic, People and Vehicle concepts for CIFAR100; the Vehicle and Wings concept for STL10.
Interpretability Methods Implementation. For the feature importance methods, we use their Captum [86] implementation. All the other explanations methods are reimplemented based on their official repository and adapted to graph data. All the concept classifiers are implemented with scikit-learn [87]. We use training sets of size 200 for the ECG, FashionMNIST, CIFAR100, STL10 datasets and 500 for the Mutagenicity and ModelNet40 datasets. CAR classifiers are chosen to be SVCs with Gaussian RBF kernel and default hyperparameters. CAV classifiers are linear classifiers trained with stochastic gradient descent optimizer with a learning rate of  $10^{-2}$  and a tolerance of  $10^{-3}$  for 1,000
epochs. In the case of CAR classifiers, we found it useful to apply PCA with 10 principal components to reduce the dimension of the latent representation before applying the classification step. Indeed, this does not significantly reduce the accuracy of the resulting classifiers while significantly reducing the runtime.
Impossible Configurations. We note that some combinations of interpretability methods and models are impossible. We summarize and explain the incompatibilities in Table 12. Further, we do not use perturbation-based feature importance methods with the CIFAR100 and STL10 ResNets, since those methods require  $\mathcal{O}[\dim(\mathcal{X}(\Omega, \mathcal{C}))]$  model calls per example, which is prohibitively expensive for large vision models.
Table 12: Not all interpretability method can be used in all settings. We detail the exceptions here.
Method	Incompatible with	Reason
Feature Permutation	Mutagenicity	Heterogeneous graphs have different num-
ber of features.
Feature Occlusion	Mutagenicity, ModelNet40	Specific to CNNs.
SimplEx-Equiv	Mutagenicity	Heterogeneous graphs lead to equivariant
representations with different dimensions.
Rep. Similar-Equiv	Mutagenicity	Heterogeneous graphs lead to equivariant
representations with different dimensions.
CAV-Equiv	Mutagenicity	Heterogeneous graphs lead to equivariant
representations with different dimensions.
CAR-Equiv	Mutagenicity	Heterogeneous graphs lead to equivariant
representations with different dimensions.
Explanation Similarity. In Definition 2.1, we also chose the cos-similarity metric  $s_{\mathcal{E}}[a,b]$  for two real-valued explanations  $a,b\in \mathbb{R}^{d_E}$ . This choice might seem arbitrary at first glance. By looking more closely, we notice that  $s_{\mathcal{E}}[a,b] = 1$  implies  $a = \alpha \cdot b$  for some  $\alpha \in \mathbb{R}^+$ . For all type of explanation that we consider in this paper,  $a$  and  $b$  will typically be vectors that aggregate various important scores (e.g. for each feature or training example). In practice, the scale of these importance scores does not matter. Indeed, if  $a = \alpha \cdot b$ , then the most important components will be the same for both  $a$  and  $b$ . Furthermore, the relative importance between any pair of components will be identical for both  $a$  and  $b$ . Therefore, we can consider that  $s_{\mathcal{E}}[a,b] = 1$  implies that both explanations are equivalent to each other  $a\sim b$ .
G Comparison with Sensitivity
In this appendix, we compare our robustness metric with the sensitivity metric introduced by [35]. This metric studies the robustness of an explanation with respect to a small perturbation in the input features. It is defined as follows:
$$
\operatorname {Sens}(e,x) = \max_{x^{\prime}\in \mathcal{X}(\Omega ,\mathcal{C})}\| e(x) - e(x^{\prime})\|_{2}
$$
$$
\begin{array}{l} \text {s . t .} \| x ^ {\prime} - x \| _ {\infty} \leq \epsilon , \end{array}
$$
where  $\| \cdot \|_{\infty}$  denotes the  $l_{\infty}$  norm,  $\| \cdot \|_2$  denotes the  $l_2$  norm and  $\epsilon \in \mathbb{R}^+$  is a small positive constant that is fixed to  $\epsilon = .02$  in the default Captum implementation of the metric. The rationale behind this metric is the following: a small perturbation of the input should have little impact on the model's prediction and, hence, little impact on the resulting explanation as well. For this reason, one typically expects a low sensitivity  $\mathrm{Sens}(e,x)$  for an explanation that is robust to small input shifts. We simply note that the previous reasoning is debatable due to the existence of adversarial perturbations that are small in norm but that have large effect on the model's prediction [88].
Like our invariance and equivariance robustness metrics, Sens measures how the explanation is affected by a transformation of the input. Therefore, a natural question arises: is there a difference in practice between our robustness metrics and the sensitivity metric? To answer this question, we
consider the setup of Section 3 with the ECG dataset and the Augmented-CNN. We study feature importance methods  $e$ , for which we evaluate  $\mathrm{Sens}(e,x)$  and  $\mathrm{Equiv}_{\mathcal{G}}(e,x)$  for  $N_{\mathrm{test}} = 1,000$  test examples  $x \in \mathcal{D}_{\mathrm{test}}$ . We report the results in Figure 8.
Figure 8: Comparison between the sensitivity metric  $\mathrm{Sens}(e,x)$  and our equivariance metric  $\mathrm{Equiv}_{\mathcal{G}}(e,x)$ . Each point represents a test example  $x \in \mathcal{D}_{\mathrm{test}}$  extracted from the ECG dataset explained by a feature importance method  $e$ . The downward trend between the two metrics is weak.
As we can observe on Figure 8, there exists a weak downward trend between the two metrics. By evaluating the Pearson correlation coefficient between the two metrics for each method, we found  $r = -.62$  for Integrated Gradients,  $r = -.48$  for Gradient Shap,  $r = -.38$  for Feature Permutation,  $r = -.23$  for Feature Ablation and  $r = -.31$  for Feature Occlusion. This indicates that examples with higher sensitivity tend to be associated with lower equivariance. That being said, the correlation between the two metrics is too weak to suggest any redundancy between them. We deduce that our robustness metrics should be use to complement rather than replace the sensitivity metric. In practice, we believe that many forms of explanation robustness deserve to be investigated.
H Other Invariant Architectures
In Section 3, we have illustrated the flexibility of our robustness formalism by applying it to various architecture. However, geometric deep learning extends well-beyond the examples covered in this paper and an exhaustive treatment of the field is beyond our scope. To suggest future extensions of our work beyond the architectures examined in this paper, we provide in Table 13 other architectures that are invariant or equivariant with respect to other symmetry groups. Our formalism straightforwardly applies to most of these examples. For some others (like Spherical CNNs), an extension to infinite groups would be required. We leave this extension for future work.
Table 13: Other invariant architectures. Table partially adapted from [46].
Architecture	Symmetry Group G	Reference(s)
G-CNN	Any finite group	[89]
Transformer	Permutation S(n)	[90]
LSTM	Time Warping	[91]
Spherical CNN	Rotation SO(3)	[92]
Mesh CNN	Gauge Symmetry SO(2)	[93]
E(n)-GNN	Euclidean Group E(n)	[94, 95]
I Examples of Non-robust Explanations
In this appendix, we illustrate the importance of our notion of robustness by presenting failure modes. Since images are the easiest way to visualize, we focus our attention to the FashionMNIST and STL10 datasets. In Figures 9 and 10, we show non-robust saliency maps obtained with Gradient Shap and DeepLift. As we can observe, the model prediction is largely unaffected by applying a symmetry  $(s_{\mathcal{E}}[f(\rho [g]x),f(x)]\approx 1$  in all cases), while the saliency map looks qualitatively different. We have included a Jupyter Notebook in our code to easily visualize other failure modes.
Predicted Class: Shirt
Predicted Class: Dress
Predicted Class: Sneaker
Predicted Class: Shirt
Figure 9: Examples of non-robust explanations obtained with Gradient Shap on the FashionMNIST dataset. The corresponding cosine similarity  $s_{\varepsilon} \left[ e(\rho[g]x), \rho[g]e(x) \right]$  between the saliency maps are respectively 0.06, 0.08, 0.18, 0.18 and 0.21
Predicted Class: Car
Predicted Class: Monkey
Predicted Class: Truck
Predicted Class: Ship
Transformed Saliency Map
Figure 10: Examples of non-robust explanations obtained with DeepLift on the STL10 dataset. The corresponding cosine similarity  $s_{\mathcal{E}} \left[ e(\rho[g]x), \rho[g]e(x) \right]$  between the saliency maps are respectively 0.66, 0.70, 0.70, 0.71 and 0.72
J Effect of Distribution Shift
In this appendix, we study the impact of evaluating out robustness metrics on samples from a distribution that differs from model's training distribution. To that end, we shall use the CINIC-10 dataset [96], which contains ImageNet images that have the same classes as CIFAR-10 classes (and, hence, the same classes as the STL-10 dataset). In this way, we are able to reuse the ResNets trained on STL-10 for evaluation on the CINIC-10 dataset. The only requirements is to resize the CINIC-10
images so that they match the resolution of the STL-10 images ( $96 \times 96$  pixels). We evaluate the robustness metrics with respect to the dihedral group  $\mathbb{D}_4$  for a few feature importance and example importance methods in Tables 14 and 15. In both cases, we find that these metrics are consistent with their STL-10 counterpart in Figures 3(a) and 3(b).
Table 14: Equivariance of feature importance methods computed on CINIC-10.
Method e	Equivariance Ex~DCINIC10 EquivD4 [e, x])
Integrated Gradients	.87
DeepLift	.85
Gradient Shap	.81
Table 15: Invariance of example importance methods computed on CINIC-10.
Method e	Invariance E×∼DCINIC10InvD4[e,x])
TracIN	.99
Influence Functions	.99
SimplEx Equiv	.83
SimplEx Inv	.87
Representation Similarity Equiv	.64
Representation Similarity Inv	.98