SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jonathan Crabbé
DAMTP
jc2133@cam.ac.uk
Interpretability methods are valuable only if their explanations faithfully describe the explained model. In this work, we consider neural networks whose predictions are invariant under a specific symmetry group. This includes popular architectures, ranging from convolutional to graph neural networks. Any explanation that faithfully explains this type of model needs to be in agreement with this invariance property. We formalize this intuition through the notion of explanation invariance and equivariance by leveraging the formalism from geometric deep learning. Through this rigorous formalism, we derive (1) two metrics to measure the robustness of any interpretability method with respect to the model symmetry group; (2) theoretical robustness guarantees for some popular interpretability methods and (3) a systematic approach to increase the invariance of any interpretability method with respect to a symmetry group. By empirically measuring our metrics for explanations of models associated with various modalities and symmetry groups, we derive a set of 5 guidelines to allow users and developers of interpretability methods to produce robust explanations.
With their increasing success in various tasks, such as computer vision [1], natural language processing [2] and scientific discovery [3], deep neural networks (DNNs) have become widespread. State of the art DNNs typically contain millions to billions parameters and, hence, it is unrealistic for human users to precisely understand how these models issue predictions. This opacity increases the difficulty to anticipate how models will perform when deployed [4]; distil knowledge from the model [5] and gain the trust of stakeholders in high-stakes domains [6, 7]. To address these shortcomings, the field of interpretable machine learning has received increasing interest [8, 9]. There exists mainly 2 approaches to increase model interpretability [10, 11]. (1) Restrict the model's architecture to intrinsically interpretable architectures. A notorious example is given by self-explaining models, such as attention models explaining their predictions by highlighting features they pay attention to [12] and prototype-based models motivating their predictions by highlighting related examples from their training set [13-15]. (2) Use post-hoc interpretability methods in a plug-in fashion after training the model. The advantage of this approach is that it requires no assumption on the model that we need to explain. In this work, we focus on several post-hoc methods: feature importance methods (also known as feature attribution or saliency) that highlight features the model is sensitive to [16-20]; example importance methods that identify influential training examples [21-23] and concept-based explanations that exhibit how classifiers relate classes to human friendly concepts [24, 25].
With the multiplication of interpretability methods, it has become necessary to evaluate the quality of their explanations [26]. This stems from the fact that interpretability methods need to faithfully describe the model in order to provide actionable insights. Existing approaches to evaluate the quality
of interpretability methods fall in 2 categories [5, 27]. (1) Human-centred evaluation investigate how the explanations help humans (experts or not) to anticipate the model's predictions [28] and whether the model's explanations are in agreement with some notion of ground-truth [29-31]. (2) Functionality-grounded evaluation measure the explanation quality based on some desirable properties and do not require humans to be involved. Most of the existing work in this category measure the robustness of interpretability methods with respect to transformations of the model input that should not impact the explanation [32]. Since our work falls in this category, let us now summarize the relevant literature.
Related Works. [33] showed that feature importance methods are sensitive to constant shifts in the model's input. This is unexpected because these constant shifts do not contribute to the model's prediction. Building on this idea of invariance of the explanations with respect to input shifts, [34-36] propose a sensitivity metric to measure the robustness of feature importance methods based on their stability with respect to small perturbations of the model input. By optimizing small adversarial perturbations, [37-39] show that imperceptible changes in the input can modify the feature importance arbitrarily by approximatively keeping the model prediction constant. This shows that many interpretability methods, as neural networks, are sensitive to adversarial perturbations. Subsequent works have addressed this pathologic behaviour by fixing the model training dynamic. In particular, they showed that penalizing large eigenvalues of the training loss Hessian with respect to the inputs make the interpretations of this model more robust with respect to adversarial attacks [40, 41]. To the best of our knowledge, the only work that discusses the behaviour of explanations under more general transformations of the input data is [42]. However, the work's focus is more on model regularization rather than on the evaluation of post-hoc interpretability robustness.
Motivations. In reviewing the above literature, we notice 3 gaps. (1) The existing studies mostly focus on evaluating feature importance methods. In spite of the predominance of feature importance in the literature [43], we note that other types of interpretability methods exist and deserve to be analyzed. (2) The existing studies mostly focus on images. While computer vision is undoubtedly an interesting application of DNNs, it would be interesting to extend the analysis to other modalities, such as times series and graph data [44]. (3) The existing studies mostly focus on simple transformation of the model input, such as small shifts. This is motivated by the fact that the predictions of DNNs are mostly invariant under these transformations. Again, this is another direction that could be explored more thoroughly as numerous DNNs are also invariant to more complex transformation of their input data. For instance, graph neural networks are invariant to permutations of the node ordering in their input graph [45]. Our work bridges these gaps in the interpretability robustness literature.
Figure 1: Illustration of model invariance and explanation invariance/equivalence with the simple case of an electrocardiogram (ECG) signal. In this case, the heartbeat described by the ECG remains the same if we apply any translation symmetry with periodic boundary conditions. (1) A model is invariant under the symmetry if the model's prediction are not affected by the symmetry we apply to the signal. In this case, the model identifies an abnormal heartbeat before and after applying a translation. Any explanation that faithfully describes the model should reflect this symmetry. (2) For some explanations, the right behaviour is invariance as well. For instance, the most influential examples for the prediction should be the same for the original and the transformed signal, since the model makes no difference between the two signals. (3) For other types of explanations, the right behaviour is equivariance. For instance, the most important part of the signal for the prediction should be the same for the original and the transformed signal, since the model makes no difference between the two signals. Hence, the saliency map undergoes the same translation as the signal.
Figure 2: Examples of non-robust explanations obtained with Gradient Shap on the FashionMNIST dataset. From left to right: the original image for which the invariant model predicts t-shirt with a given probability, the Gradient Shap saliency map to explain the model's prediction for this image, the transformed image for which the model predicts t-shirt with the exact same probability and the Gradient Shap saliency map for this transformed image. Clearly, the image transformation changes the explanation when it should not.
Contributions. We propose a new framework to evaluate the robustness of interpretability methods. We consider a setting where the model we wish to interpret is invariant with respect to a group  $\mathcal{G}$  of symmetry acting on the model input. Any interpretability method that faithfully describes this model should have explanations that are conserved by this group of symmetry  $\mathcal{G}$ . We illustrate this reasoning in Figure 1 with the simple group  $\mathcal{G}$  of time translations acting on the input signal. We show examples of interpretability methods failing to conserve the model's symmetries, hence leading to inconsistent explanations, in Figure 2 and Appendix I. With this new framework, we bring several contributions. (1) Rigorous Interpretability Robustness. We define interpretability robustness with respect to a group  $\mathcal{G}$  of symmetry through explanation invariance and equivariance. In agreement with our motivations, we demonstrate in Section 2.2 that our general definitions cover different types of interpretability methods, modalities and transformations of the input data. (2) Evaluation of Interpretability Methods. Not all interpretability methods are equal with respect to our notion of robustness. In Section 2.3, we show that some popular interpretability methods are naturally endowed with theoretical robustness guarantees. Further, we introduce 2 metrics, the invariance and equivariance scores, to empirically evaluate this robustness. In Section 3.1, we use these metrics to evaluate the robustness of 3 types of interpretability methods with 5 different model types corresponding to 4 different modalities and symmetry groups. Our empirical results support our theoretical analysis. (3) Insights to Improve Robustness. By combining our theoretical and empirical analysis, we derive a set of 5 actionable guidelines to ensure that interpretability methods are used in a way that guarantees robustness with respect to the symmetry group  $\mathcal{G}$ . In particular, we show in Sections 2.3 and 3.2 that we can improve the invariance score of any interpretability method by aggregating explanations over various symmetries. We summarize the guidelines with a flowchart in Figure 6 from Appendix A, that helps users to obtain robust model interpretations.
2 Interpretability Robustness
In this section, we formalize the notion of interpretability robustness through explanation invariance and equivariance. We start with a reminder of some useful definitions from geometric deep learning. We then define two metrics to measure the invariance and equivariance of interpretability methods. We leverage this formalism to derive some theoretical robustness guarantees for popular interpretability methods. Finally, we describe a rigorous approach to improve the invariance of any interpretability method.
2.1 Useful Notions of Geometric Deep Learning
Some basic concepts of group theory are required for our definition of interpretability robustness. To that aim, we leverage the formalism of Geometric Deep Learning. Please refer to [46] for more details. To rigorously define explanation equivariance and invariance, we need some form of structure in the data we are manipulating. This precludes tabular data but includes graph, time series and image data. In this setting, the data is defined on a finite domain set  $\Omega$  (e.g. a grid  $\Omega = \mathbb{Z}_n\times \mathbb{Z}_n$  for  $n\times n$  images). On this domain, the data is represented by signals  $x\in \mathcal{X}(\Omega ,\mathcal{C})$ , mapping each point  $u\in \Omega$  of the domain to a channel vector  $x(u)\in \mathcal{C} = \mathbb{R}^{d_C}$ . We note that  $d_{C}\in \mathbb{N}^{+}$ , corresponds to the
Predicted Class: T-shirt/top
$\therefore m : x = 1$  或  ${3x} + {4y} + 1 = 0$
Original Saliency Map
$\therefore m : x = 1$  或  ${3x} + {4y} + 1 : 0$
$= \frac{5}{2} - \frac{11}{8} = \frac{7}{8}$
Transformed Saliency Map
number of channels of the signal (e.g.  $d_{C} = 3$  for RGB images). The set of signals has the structure of a vector space since  $x_{1}, x_{2} \in \mathcal{X}(\Omega, \mathcal{C}) \Rightarrow \lambda_{1} \cdot x_{1} + \lambda_{2} \cdot x_{2} \in \mathcal{X}(\Omega, \mathcal{C})$  for all  $\lambda_{1}, \lambda_{2} \in \mathbb{R}$ .
Symmetries. Informally, symmetries are transformations of the data that leave the information content unchanged (e.g. moving an image one pixel to the right). More formally, symmetries correspond to a set  $\mathcal{G}$  endowed with a composition operation  $\circ : \mathcal{G}^2 \to \mathcal{G}$ . Clearly, this set  $\mathcal{G}$  includes an identity transformation  $id$  that leaves the data untouched. Similarly, if a transformation  $g \in \mathcal{G}$  preserves the information, then it could be undone by an inverse transformation  $g^{-1} \in \mathcal{G}$  such that  $g^{-1} \circ g = id$ . Those properties give  $\mathcal{G}, \circ$  the structure of a group. In this paper, we assume that the symmetry group has a finite number of elements.
Group Representation. We have yet to formalize how the above symmetries transform the data. To that aim, we need to link the symmetry group  $\mathcal{G}$  with the signal vector space  $\mathcal{X}(\Omega, \mathcal{C})$ . This connection is achieved by choosing a group representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$  that maps each symmetry  $g \in \mathcal{G}$  to an automorphism  $\rho[g] \in \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . Formally, the automorphisms  $\operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$  are defined as bijective linear transformations mapping  $\mathcal{X}(\Omega, \mathcal{C})$  onto itself. In practice, each automorphism  $\rho[g]$  is represented by an invertible matrix acting on the vector space  $\mathcal{X}(\Omega, \mathcal{C})$ . For instance, an image translation  $g$  can be represented by a permutation matrix  $\rho[g]$ . To qualify as a group representation, the map  $\rho$  needs to be compatible with the group composition:  $\rho[g_2 \circ g_1] = \rho[g_2] \rho[g_1]$ . This property guarantees that the composition of two symmetries can be implemented as the multiplication between two matrices.
Invariance. We first consider the case of a deep neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$ , where the output  $f(x) \in \mathcal{Y}$  is a vector with no underlying structure (e.g. class probabilities for a classifier). In this case, we expect the model's prediction to be unchanged when applying a symmetry  $g \in \mathcal{G}$  to the input signal  $x \in \mathcal{X}(\Omega, \mathcal{C})$ . For instance, the probability of observing a cat on an image should not change if we move the cat by one pixel to the right. This intuition is formalized by defining the  $\mathcal{G}$ -invariance property for the model  $f: f(\rho[g]x) = f(x)$  for all  $g \in \mathcal{G}, x \in \mathcal{X}(\Omega, \mathcal{C})$ .
Equivariance. We now turn to the case of deep neural networks  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}(\Omega', \mathcal{C}')$ , where the output  $f(x) \in \mathcal{Y}(\Omega', \mathcal{C}')$  is also a signal (e.g. segmentation masks for an object detector). We note that the domain  $\Omega'$  and the channel space  $\mathcal{C}'$  are not necessarily the same as  $\Omega$  and  $\mathcal{C}$ . When applying a transformation  $g \in \mathcal{G}$  to the input signal  $x \in \mathcal{X}(\Omega, \mathcal{C})$ , it is legitimate to expect the output signal  $f(x)$  to follow a similar transformation. For instance, the segmentation of a cat on an image should move by one pixel to the right if we move the cat by one pixel to the right. This intuition is formalized by defining the  $\mathcal{G}$ -equivariance property for the model  $f: f(\rho[g]x) = \rho'[g]f(x)$ . Again, the representation  $\rho': \mathcal{G} \to \mathrm{Aut}[\mathcal{Y}(\Omega', \mathcal{C}')]$  is not necessarily the same as the representation  $\rho$  since the signal spaces  $\mathcal{X}(\Omega, \mathcal{C})$  and  $\mathcal{Y}(\Omega', \mathcal{C}')$  might have different dimensions.
2.2 Explanation Invariance and Equivariance
We will now restrict to models that are  $\mathcal{G}$ -invariant $^2$ . It is legitimate to expect similar invariance properties for the explanations associated to this model. We shall now formalize this idea for generic explanations. We assume that explanations are functions of the form  $e: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{E}$ , where  $\mathcal{E} \subseteq \mathbb{R}^{d_E}$  is an explanation space with  $d_E \in \mathbb{N}^+$  dimensions $^3$ .
Invariance and Equivariance. The invariance and equivariance of the explanation  $e$  with respect to symmetries  $\mathcal{G}$  are defined as in the previous section. In this way, we say that the explanation  $e$  is  $\mathcal{G}$ -invariant if  $e(\rho[g]x) = e(x)$  and  $\mathcal{G}$ -equivariant if  $e(\rho[g]x) = \rho'[g]e(x)$  for all  $g \in \mathcal{G}, x \in \mathcal{X}(\Omega, \mathcal{C})$ . There is no reason to expect these equalities to hold exactly a priori. This motivates the introduction of two metrics that measure the violation of explanation invariance and equivariance by an interpretability method.
Definition 2.1 (Robustness Metrics). Let  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  be a neural network that is invariant with respect to the symmetry group  $\mathcal{G}$  and  $e: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{E}$  be an explanation for  $f$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \mathrm{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We measure the invariance of  $e$  with
<sup>1</sup> Note that groups also satisfy associativity:  $g_{1} \circ (g_{2} \circ g_{3}) = (g_{1} \circ g_{2}) \circ g_{3}$  for all  $g_{1}, g_{2}, g_{3} \in \mathcal{G}$ .
2We also restrict to supervised models, since only early works exist to interpret unsupervised models [47, 48].
Note that the explanation  $e$  also depends on the model  $f$ . Since the model is fixed, we make this implicit.
respect to  $\mathcal{G}$  for some  $x\in \mathcal{X}(\Omega ,\mathcal{C})$  with the metric
$$
\operatorname {I n v} _ {\mathcal {G}} (e, x) \equiv \frac {1}{| \mathcal {G} |} \sum_ {g \in \mathcal {G}} s _ {\mathcal {E}} [ e (\rho [ g ] x), e (x) ], \tag {1}
$$
where  $s_{\mathcal{E}}: \mathcal{E}^2 \to \mathbb{R}$  is a similarity score on the explanation space  $\mathcal{E}$ . We use the cos-similarity  $s_{\mathcal{E}}(a,b) = a^{\top}b / \| a\|_2 \cdot \| b\|_2$  for real-valued explanations  $a, b \in \mathbb{R}^{d_E}$  and the accuracy score  $s_{\mathcal{E}}(a,b) = d_E^{-1}\sum_{i=1}^{d_E}\mathbb{1}(a_i = b_i)$  for categorical explanations  $a, b \in \mathbb{Z}_K^{d_E}$ , where  $\mathbb{1}$  is the indicator function and  $K \in \mathbb{N}^+$  is the number of categories. If we assume that  $\mathcal{G}$  acts on  $\mathcal{E}$  via the representation  $\rho': \mathcal{G} \to \operatorname{Aut}[\mathcal{E}]$ , we measure the equivariance of  $e$  with respect to  $\mathcal{G}$  for some  $x \in \mathcal{X}(\Omega, \mathcal{C})$  with the metric
$$
\operatorname {E q u i v} _ {\mathcal {G}} (e, x) \equiv \frac {1}{| \mathcal {G} |} \sum_ {g \in \mathcal {G}} s _ {\mathcal {E}} [ e (\rho [ g ] x), \rho^ {\prime} [ g ] e (x) ]. \tag {2}
$$
A score  $\mathrm{Inv}_{\mathcal{G}}(e,x) = 1$  or  $\mathrm{Equiv}_{\mathcal{G}}(e,x) = 1$  indicates that the explanation method  $e$  is  $\mathcal{G}$ -invariant or equivariant for the example  $x \in \mathcal{X}(\Omega, \mathcal{C})$ .
Remark 2.2. The metrics  $\mathrm{Inv}_{\mathcal{G}}$  and  $\mathrm{Equiv}_{\mathcal{G}}$  might be prohibitively expensive to evaluate whenever the size  $|\mathcal{G}|$  of the symmetry group  $\mathcal{G}$  is too big. Note that this is typically the case in our experiments as we consider large permutation groups of order  $|\mathcal{G}| \gg 10^{32}$ . In this case, we use Monte Carlo estimators for both metrics by uniformly sampling  $G \sim U(\mathcal{G})$  and averaging over a number of sample  $N_{\mathrm{samp}} \ll |\mathcal{G}|$ . We study the convergence of those Monte Carlo estimators in Appendix E.
The above approach to measure the robustness of interpretability method applies to a wide variety of settings. To clarify this, we explain how to adapt the above formalism to 3 popular types of interpretability methods: feature importance, example importance and concept-based explanations.
Feature Importance. Feature importance explanations associate a saliency map  $e(x) \in \mathcal{X}(\Omega, \mathcal{C})$  to each example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  for the model's prediction  $f(x)$ . In this case, we note that the explanation space corresponds to the model's input space  $\mathcal{E} = \mathcal{X}(\Omega, \mathcal{C})$ , since the method assigns an importance score to each individual feature. If we apply a symmetry to the input, we expect the same symmetry to be applied to the saliency map, as illustrated by the example from Figure 1. Hence, the most relevant metric to record here is the explanation equivariance  $\mathrm{Equiv}_{\mathcal{G}}$ . Since the input space and the explanation space are identical  $\mathcal{E} = \mathcal{X}(\Omega, \mathcal{C})$ , we work with identical representations  $\rho' = \rho$ . We note that this metric generalizes the self-consistency score introduced by [42] beyond affine transformations.
Example Importance. Example importance explanations associate an importance vector  $e(x) \in \mathbb{R}^{N_{\mathrm{train}}}$  to each example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  for the model's prediction  $f(x)$ . Note that  $N_{\mathrm{train}} \in \mathbb{N}^+$  is typically the model's training set size, so that each component of  $e(x)$  corresponds to the importance of a training example. If we apply a symmetry to the input, we expect the relevance of training examples to be conserved, as illustrated by the example from Figure 1. Hence, the most relevant metric to record here is the invariance  $\mathrm{Inv}_{\mathcal{G}}$ .
Concept-Based Explanations. Concept-based explanations associate a binary concept presence vector  $e(x) \in \{0,1\}^C$  to each example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  for the model's prediction  $f(x)$ . Note that  $C \in \mathbb{N}^{+}$  is the number of concepts one considers, so that each component of  $e(x)$  corresponds to the presence/absence of a concept. If we apply a symmetry to the input, there is no reason for a concept to appear/vanish, since the information content of the input is untouched by the symmetry. Hence, the most relevant metric to record here is again the invariance  $\mathrm{Inv}_{\mathcal{G}}$ .
2.3 Theoretical Analysis
Let us now provide a theoretical analysis of robustness in a setting where the model  $f$  is  $\mathcal{G}$ -invariant. We first show that many popular interpretability methods naturally offer some robustness guarantee if we make some assumptions. For methods that are not invariant when they should, we propose an approach to enforce  $\mathcal{G}$ -invariance.
Robustness Guarantees. In Table 1, we summarize the theoretical robustness guarantees that we derive for popular interpretability methods. All of these guarantees are formally stated and proven in Appendix D. When it comes to feature importance methods, there are mainly two assumptions that are necessary to guarantee equivariance. (1) The first assumption restricts the type of baseline input
Table 1: Theoretical robustness guarantees that we derive for explanations of invariant models. We split the interpretability methods according to their type and according to model information they rely on (model gradients, perturbations, loss or representations). We consider 3 levels of guarantees:  $\checkmark$  indicates unconditional guarantee,  $\sim$  conditional guarantee and  $\times$  no guarantee.
Type	Computation	Example	Invariant	Equivariant	Details
Feature	Grad. ∇xf(x)	[18]	X	~	Prop. D.6
Importance	Pert. f(x + δx)	[49]	X	~	Prop. D.8
Example	Loss L[f(x),y]	[21]	✓	X	Prop. D.9
Importance	Rep. h(x)	[23]	~	X	Prop. D.12
Concept-Based	Rep. h(x)	[24]	~	X	Prop. D.14