SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proving that explanation robustness imposes restriction on interpretability methods. By using our formalism derived from geometric deep learning, we show in Appendix D that enforcing equivariance or invariance for interpretability methods imposes strong restrictions. For instance, gradient-based attribution methods that rely on a baseline signal  $\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  are equivariant if this baseline signal is invariant:  $\rho[g] \bar{x} = \bar{x}$ . We provide a similar analysis for 3 popular types of interpretability methods (feature importance, example importance and concept-based explanations). We hope that these insights will guide the future development of interpretability methods tailored for cutting edge deep learning models, such as GNNs.
Introducing two well-defined and sensible robustness metrics. Beyond adapting invariance and equivariance to interpretability methods, we provide a principled way to evaluate those properties in Section 2 with two robustness metrics. Since these metrics cannot always be computed exactly (e.g. for groups  $\mathcal{G}$  with large cardinalities), we show in Appendix E that Monte Carlo approximations provide sensible approximations, both theoretically by adapting Hoeffding's inequality and empirically in the setup from Section 3.
Demonstrating empirically that not all interpretability methods are created equal. We have performed extensive experiments on 6 different datasets, corresponding to 4 distinct modalities and model architectures, with 12 different interpretability methods. These experiments show that some interpretability methods (such as GradientShap and Feature Permutation) consistently fail to be robust with respect to symmetries of the model. We believe that these results are not trivial and should impact the way we use these methods to explain deep neural networks.
Through these contributions, we hope to reinforce the 3 below statements.
1. Interpretability should be used with skepticism.
2. Not all interpretability methods are created equal.
3. Robustness can guide the design of interpretability methods.
In this appendix, we discuss the main limitations of this work. As a first limitation, we would like to acknowledge the fact that our robustness metrics have no pretention to provide a holistic evaluation approach for interpretability methods. We believe that evaluating the robustness of interpretability methods requires a combination of several evaluation criteria. To make this point more clear, let us take inspiration from the Model Robustness literature. By looking at reviews on this subject (see e.g.
Section 6.3 of [68]), we notice that various notions of robustness exist for machine learning models. To name just a few, existing works have investigated the robustness of neural networks with respect to noise [69]], adversarial perturbations [70] and label consistency across clusters [71]. All of these notions of robustness come with their own metrics and assessment criteria. In a similar way, we believe that the robustness of interpretability methods should be assessed on several complementary dimensions. Our work contributes to this initiative by adding one such dimension.
As a second limitation, our robustness metrics are mostly useful for models that are (approximately) invariant with respect to a given symmetry group. That being said, we would like to emphasize that our robustness metrics can still be used to characterize the equivariance/invariance of explanations even if the model is not perfectly invariant. The metrics would still record the same information, even if the invariance of the model is relaxed. The main thing to keep in mind when using these metrics with models that are not perfectly invariant is that the explanations themselves should not be exactly invariant/equivariant. That said, for a model that is approximately invariant, we expect a faithful explanation to keep a high invariance / equivariance score. This is precisely the object of Section 3.3.
D Theoretical Results
In this appendix, we prove all the theoretical results mentioned in the main paper. We start by deriving robustness guarantees mentioned in Table 1. We then prove that any method can be made  $\mathcal{G}$ -invariant by aggregating the explanation over several symmetries.
D.1 Feature Importance Guarantees
Let us start by feature importance methods. As we are going to see, the robustness guarantees in this case typically require us to restrict the type of group representation  $\rho$  we use to encode the action of the symmetry group  $\mathcal{G}$  on the signal space  $\mathcal{X}(\Omega, \mathcal{C})$ . This motivates the two following types of representations that can be found in group theory textbooks (see e.g. [72]).
Definition D.1 (Orthogonal Representation). Let  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$  be a representation of the group  $\mathcal{G}$ , and let  $d \in \mathbb{N}^+$  denote the dimension of the signal space  $\mathcal{X}(\Omega, \mathcal{C})$ . We say that the representation is an orthogonal representation if its image is a subset on the orthogonal matrices acting on  $\mathcal{X}(\Omega, \mathcal{C})$ :  $\rho(\mathcal{G}) \subseteq O(d)$ , where  $O(d)$  denotes the set of  $d \times d$  orthogonal real matrices. This is equivalent to  $\rho[g] \rho^\top[g] = \rho^\top[g] \rho[g] = I_d$  for all  $g \in \mathcal{G}$ , where  $I_d$  denotes the  $d \times d$  identity matrix.
Definition D.2 (Permutation Representation). Let  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$  be a representation of the group  $\mathcal{G}$ , and let  $d \in \mathbb{N}^+$  denote the dimension of the signal space  $\mathcal{X}(\Omega, \mathcal{C})$ . We say that the representation is a permutation representation if its image is a subset on the permutation matrices acting on  $\mathcal{X}(\Omega, \mathcal{C})$ :  $\rho(\mathcal{G}) \subseteq P(d)$ , where  $P(d)$  denotes the set of  $d \times d$  permutation matrices. This means that for all  $g \in \mathcal{G}$ , there exists some permutation  $\pi \in S(d)$  such that we can write  $(\rho[g]x)_i = x_{\pi(i)}$  for all  $i \in \mathbb{Z}_d$ ,  $x \in \mathcal{X}(\Omega, \mathcal{C})$ .
Remark D.3. One can easily check that all permutation representations are also orthogonal representations. The opposite is not true.
D.1.1 Gradient-Based
We shall now begin with gradient-based feature importance methods. These methods compute the model's gradient with respect to the input features to compute the importance scores. Hence, it is useful to characterize how this gradient transforms under the action of the symmetry group  $\mathcal{G}$ .
Lemma D.4 (Gradient Transformation). Consider a differentiable function  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . If for some  $g \in \mathcal{G}$  we define  $x' = \rho[g]x$ , then we have the following identity:
$$
\nabla_ {x ^ {\prime}} f \left(x ^ {\prime}\right) = \rho^ {- 1, \tau} [ g ] \nabla_ {x} f (x), \tag {3}
$$
where  $\rho^{-1,\tau}[g]$  denotes the matrix obtained by applying an inversion followed by a transposition to the matrix  $\rho[g]$ .
Proof. We start by noting that the  $\mathcal{G}$ -invariance of  $f$  implies that  $f(x') = f(x)$  and, hence:
$$
\nabla_ {x ^ {\prime}} f \left(x ^ {\prime}\right) = \nabla_ {x ^ {\prime}} f (x) \tag {4}
$$
It remains to establish the ling between  $\nabla_{x'}$  and  $\nabla_x$ . To this end, we simply note that  $x = \rho^{-1}[g]x'$ . Hence, for all  $i \in \mathbb{Z}_d$  with  $d = \dim[\mathcal{X}(\Omega, \mathcal{C})]$ , we have that
$$
x _ {i} = \sum_ {j = 1} ^ {d} \rho_ {i j} ^ {- 1} [ g ] x _ {j} ^ {\prime}.
$$
Hence, by using the chain rule, we deduce that for all  $k\in \mathbb{Z}_d$
$$
\begin{array}{l} \frac {\partial}{\partial x _ {k} ^ {\prime}} = \sum_ {i = 1} ^ {d} \frac {\partial x _ {i}}{\partial x _ {k} ^ {\prime}} \frac {\partial}{\partial x _ {i}} \\ = \sum_ {i = 1} ^ {d} \rho_ {i k} ^ {- 1} [ g ] \frac {\partial}{\partial x _ {i}} \\ = \sum_ {i = 1} ^ {d} \rho_ {k i} ^ {- 1, \mathsf {T}} [ g ] \frac {\partial}{\partial x _ {i}}. \\ \end{array}
$$
The above identity implies  $\nabla_{x^{\prime}} = \rho^{-1,\tau}[g]\nabla_{x}$ . By injecting this to the right-hand side of (4), we obtain (3).
The simplest gradient-based attribution is simply given by the gradient itself [73]. We refer to it as the vanilla saliency feature importance. Although this attribution method is a bit naive, we may still deduce an equivariance guarantee from the previous proposition.
Corollary D.5 (Equivalence of Vanilla Saliency). Consider a differentiable neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We consider a vanilla saliency feature importance explanation  $e(x) = \nabla_x f(x)$ . If the representation  $\rho$  is orthogonal, then the explanation  $e$  is  $\mathcal{G}$ -equivariant.
Proof. From Lemma D.4, we have that  $e(\rho[g]x) = \rho^{-1,\mathsf{T}}[g]e(x)$  for all  $g \in \mathcal{G}$ . Now if  $\rho$  is orthogonal, we note that  $\rho^{-1,\mathsf{T}}[g] = \rho[g]$ , which proves the proposition.
We now turn to a more general family of gradient-based feature importance methods. These methods attribute importance to each feature by aggregating gradients over a line in the input space  $\mathcal{X}(\Omega, \mathcal{C})$  connecting a baseline example  $\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  with the example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  we wish to explain. It was shown that the choice of this baseline signal has a significant impact on the resulting explanation [74]. In the following proposition, we show that enforcing equivariance imposes a restriction on the type of baseline signal  $\bar{x}$  that can be used in practice.
Proposition D.6 (Gradient-Based Equivalence). Consider a differentiable neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We consider a gradient-based explanation built upon a baseline signal  $\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  and of the form
$$
e (x) = (x - \bar {x}) \odot \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f ([ \bar {x} + t (x - \bar {x}) ]) d t,
$$
where  $\odot$  denotes the Hadamard product and  $\varphi$  is a functional defined on the Hilbert space  $L^2([0,1])$ . If  $\rho$  is a permutation representation and the baseline signal is  $\mathcal{G}$ -invariant, i.e.  $\rho[g]\bar{x} = \bar{x}$  for all  $g \in \mathcal{G}$ , then the explanation  $e$  is  $\mathcal{G}$ -equivariant.
Remark D.7. Note that we have introduced the functional  $\varphi$  to make the class of explanation as general as possible. For instance, we obtain exact Integrated Gradients for  $\varphi(t) = 1$  and Input*Gradient [19] for  $\varphi(t) = \delta(t - 1)$ , where  $\delta$  is a Dirac delta distribution. Similarly, this includes discrete approximations of Integrated Gradients by taking e.g.  $\varphi(t) = \sum_{n=1}^{N} \delta(t - t_n)$ , with  $t_n = \bar{x} + \frac{n}{N}(x - \bar{x})$  for all  $n \in \mathbb{Z}_n$ . Finally, we note that the equivariance of Integrated Gradients also implies the equivariance of Expected Gradients [75].
Proof. For all  $g \in \mathcal{G}$ , we have that:
$$
\begin{array}{l} e (\rho [ g ] x) = (\rho [ g ] x - \bar {x}) \odot \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f (\bar {x} + t [ \rho [ g ] x - \bar {x} ]) d t \\ = \rho [ g ] (x - \bar {x}) \odot \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f (\rho [ g ] [ \bar {x} + t (x - \bar {x}) ]) d t \quad \left(\text {I n v a r i a n c e o f} \bar {x}\right) \\ = \rho [ g ] (x - \bar {x}) \odot \rho [ g ] \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f (\bar {x} + t [ x - \bar {x} ]) d t \quad \tag {Corollary D.5}. \\ \end{array}
$$
Since  $\rho$  is a permutation representation, there exists a permutation  $\pi \in S(d)$ , where  $d = \dim[\mathcal{X}(\Omega, \mathcal{C})]$ , such that for all  $a, b \in \mathcal{X}(\Omega, \mathcal{C})$  and  $i \in \mathbb{Z}_d$ :
$$
\begin{array}{l} (\rho [ g ] a \odot \rho [ g ] b) _ {i} = (\rho [ g ] a) _ {i} \odot (\rho [ g ] b) _ {i} \quad \text {(D e f i n i t i o n o f H a d a m a r d p r o d u c t)} \\ = a _ {\pi (i)} \odot b _ {\pi (i)} \quad (\rho \text {i s a p e r m u t a t i o n r e p r e s e n t a t i o n}) \\ = (a \odot b) _ {\pi (i)} \quad (\text {D e f i n i t i o n} \\ = (\rho [ g ] (a \odot b)) _ {i} \quad (\rho \text {i s a p e r m u t a t i o n r e p r e s e n t a t i o n}). \\ \end{array}
$$
We deduce that  $\rho [g]a\odot \rho [g]b = \rho [g](a\odot b)$ . By applying this to the above equation for  $e(\rho [g]x)$ , we get:
$$
\begin{array}{l} e (\rho [ g ] x) = \rho [ g ] (x - \bar {x}) \odot \rho [ g ] \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f (\bar {x} + t [ x - \bar {x} ]) d t \\ = \rho [ g ] \left((x - \bar {x}) \odot \int_ {0} ^ {1} \varphi (t) \nabla_ {x} f (\bar {x} + t [ x - \bar {x} ]) d t\right) \\ = \rho [ g ] e (x), \\ \end{array}
$$
which proves the equivariance property.
D.1.2 Perturbation-Based
The second type of feature importance methods we consider are perturbation-based methods. These methods attribute importance to each feature by measuring the impact of replacing some features of the example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  with features from a baseline example  $\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  on the model's prediction. Again, enforcing equivariance imposes a restriction on the type of baseline that can be manipulated.
Proposition D.8 (Perturbation-Based Equivariance). Consider a neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We consider a perturbation-based explanation built upon a baseline signal  $\bar{x} \in \mathcal{X}(\Omega, \mathcal{C})$  and of the form
$$
[ e (x) ] _ {i} = f (x) - f \left(r _ {i} (x)\right),
$$
for all  $i \in \mathbb{Z}_d$ , where  $d = \dim[\mathcal{X}(\Omega, \mathcal{C})]$ . The perturbation operator  $r_i$  replaces feature  $x_i$ ,  $i \in \mathbb{Z}_d$  with the baseline feature  $\bar{x}_i$ . It is defined as follows:  $[r_i(x)]_j = x_j + \delta_{ij}(\bar{x}_i - x_i)$ , where  $\delta$  denotes the Kronecker delta symbol, for all  $j \in \mathbb{Z}_d$  and  $x \in \mathcal{X}(\Omega, \mathcal{C})$ . If  $\rho$  is a permutation representation and the baseline signal is  $\mathcal{G}$ -invariant, i.e.  $\rho[g] \bar{x} = \bar{x}$  for all  $g \in \mathcal{G}$ , then the explanation  $e$  is  $\mathcal{G}$ -equivariant.
Proof. For all  $g \in \mathcal{G}$  and  $i, j \in \mathbb{Z}_d$ , there exists a permutation  $\pi \in S(d)$  such that:
$$
\begin{array}{l} \left[ r _ {i} (\rho [ g ] x) \right] _ {j} = x _ {\pi (j)} + \delta_ {i j} \left(\bar {x} _ {i} - x _ {\pi (i)}\right) \quad (\rho \text {i s a p e r m u t a t i o n r e p r e s e n t a t i o n}) \\ = x _ {\pi (j)} + \delta_ {i j} \left(\bar {x} _ {\pi (i)} - x _ {\pi (i)}\right) \quad (\bar {x} \text {i s i n v a r i a n t}) \\ \begin{array}{l} = [ r _ {\pi (i)} (x) ] _ {\pi (j)} \\ = [ \rho [ g ] r _ {\pi (i)} (x) ] _ {j} \end{array} \qquad (\rho \text {i s a p e r m u t a t i o n r e p r e s e n t a t i o n}). \\ \end{array}
$$
We deduce that  $r_i(\rho [g]x) = \rho [g]r_{\pi (i)}(x)$  for all  $i\in \mathbb{Z}_d$ . We are now ready to conclude as:
$$
\begin{array}{l} [ e (\rho [ g ] x) ] _ {i} = f (x) - f \left(r _ {i} (\rho [ g ] x)\right) \quad \text {(D e f i n i t i o n o f} e) \\ = f (x) - f \left(\rho [ g ] r _ {\pi (i)} (x)\right) \quad (\text {A b o v e i d e n t i t y}) \\ = f (x) - f \left(r _ {\pi (i)} (x)\right) \quad (\text {I n v a r i a n c e} f) \\ = [ e (x) ] _ {\pi (i)} \quad (\text {D e f i n i t i o n} e) \\ = [ \rho [ g ] e (x) ] _ {i} \quad (\rho \text {i s a p e r m u t a t i o n r e p r e s e n t a t i o n}), \\ \end{array}
$$
D.2 Example Importance Guarantees
We proceed with example importance methods.
D.2.1 Loss-Based
We start with loss-based methods. These methods attribute importance to each training example of  $(x^n, y^n) \in \mathcal{D}_{\mathrm{train}}$  by comparing the loss  $\mathcal{L}(f(x^n), y^n)$  with the loss  $\mathcal{L}(f(x), y)$  of the example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  we wish to explain. We show that these methods are naturally invariant without imposing any restriction on the representation  $\rho$ .
Proposition D.9 (Loss-Based Invariance). Consider a differentiable neural network  $f_{\theta}:\mathcal{X}(\Omega ,\mathcal{C})\to \mathcal{V}$ , parametrized by  $P\in \mathbb{N}^{+}$  parameters  $\theta \in \mathbb{R}^P$ , that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega ,\mathcal{C})$  via the representation  $\rho :\mathcal{G}\rightarrow \mathrm{Aut}[\mathcal{X}(\Omega ,\mathcal{C})]$ . We consider an example importance explanation based on the loss  $\mathcal{L}:\mathcal{X}(\Omega ,\mathcal{C})\times \mathcal{V}\to \mathbb{R}^{+}$  and of the form
$$
e (x, y) = \mathcal {F} [ \mathcal {L} (f _ {\theta} (x), y) ],
$$
where  $\mathcal{F}$  maps any function  $l: \mathbb{R}^P \to \mathbb{R}^+$  to a vector in  $\mathcal{F}[l] \in \mathbb{R}^{N_{\mathrm{train}}}$ , with  $N_{\mathrm{train}} \in \mathbb{N}^+$  corresponding to the number of training examples for which we evaluate the importance. The explanation  $e$  is  $\mathcal{G}$ -invariant.
Remark D.10. We note that  $\mathcal{F}$  typically contains differential operators. For instance, Influence Functions are obtained by taking
$$
\left(\mathcal {F} [ l ]\right) _ {n} = \nabla_ {\theta} ^ {\intercal} \mathcal {L} \left(f _ {\theta} \left(x ^ {n}\right), y ^ {n}\right) H _ {\theta} ^ {- 1} \nabla_ {\theta} l (\theta),
$$
for  $n \in \mathbb{Z}_{N_{\mathrm{train}}}$ , where  $(x^n, y^n) \in \mathcal{D}_{\mathrm{train}}$  is a training example and  $H_{\theta} \in \mathbb{R}^{P \times P}$  is the Hessian of the training loss with respect to the model's parameters.
Remark D.11. We note that the dependency of the explanation  $e$  with respect to the label  $y \in \mathcal{V}$  is omitted in the main paper. The reason for this is that the symmetry group  $\mathcal{G}$  only acts on the input signal  $x$ .
Proof. The proposition can directly be deduced from the  $\mathcal{G}$ -invariance of the model. For any  $g\in \mathcal{G}$  we have:
$$
\begin{array}{l} e (\rho [ g ] x, y) = \mathcal {F} [ \mathcal {L} (f _ {\theta} (\rho [ g ] x), y) ] \\ = \mathcal {F} [ \mathcal {L} (f _ {\theta} (x), y) ] \quad (\text {I n v a r i a n c e o f} f) \\ = e (x, y), \\ \end{array}
$$
which proves the desired property.
D.2.2 Representation-Based
We proceed with representation-based methods. These methods attribute importance to each training example of  $(x^n, y^n) \in \mathcal{D}_{\mathrm{train}}$  by comparing the model's representation  $h(x^n)$  (typically the output of a model's layer) with the representation  $h(x)$  of the example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  we wish to explain. We show that these methods are invariant if we restrict to representations  $h$  that are invariant.
Proposition D.12 (Representation-Based Invariance). Consider a differentiable neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We consider an example importance explanation based on a representation  $h: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{H}$  extracted from  $f$  (e.g. an intermediate layer of the neural network) and of the form
$$
e (x) = \mathcal {F} [ h (x) ],
$$
where  $\mathcal{F}:\mathcal{H}\to \mathbb{R}^{N_{\mathrm{train}}}$  maps any representation  $r\in \mathcal{H}$  to a vector in  $\mathcal{F}[r]\in \mathbb{R}^{N_{\mathrm{train}}}$ , with  $N_{\mathrm{train}}\in \mathbb{N}^+$  corresponding to the number of training examples for which we evaluate the importance. If the representation  $h$  is  $\mathcal{G}$ -invariant, then the explanation  $e$  is  $\mathcal{G}$ -invariant.
Remark D.13. We note that  $\mathcal{F}$  can be adapted to the method we want to describe. For instance, SimplEx is obtained by taking
$$
\mathcal {F} [ r ] = \arg \min  _ {w \in [ 0, 1 ] ^ {N _ {\mathrm {t r a i n}}}} \left[ r - \sum_ {n = 1} ^ {N _ {\mathrm {t r a i n}}} w _ {n} h (x ^ {n}) \right]
$$
$$
\text {s . t .} \sum_ {n = 1} ^ {N _ {\text {t r a i n}}} w _ {n} = 1
$$
where  $x^n \in \mathcal{D}_{\mathrm{train}}$  is a training example for  $n \in \mathbb{Z}_{N_{\mathrm{train}}}$ . Similarly, Representation Similarity is obtained with
$$
\mathcal {F} [ r ] _ {n} = r ^ {\intercal} h (x ^ {n}),
$$
for all  $n\in \mathbb{Z}_{N_{\mathrm{train}}}$
Proof. The proposition can directly be deduced from the  $\mathcal{G}$ -invariance of the representation. For any  $g\in \mathcal{G}$ , we have:
$$
\begin{array}{l} e (\rho [ g ] x) = \mathcal {F} [ h (\rho [ g ] x) ] \\ = \mathcal {F} [ h (x) ] \quad (\text {I n v a r i a n c e o f} h) \\ = e (x), \\ \end{array}
$$
D.3 Concept-Based Explanations Guarantees
We now turn to concept-based methods. These methods attribute importance to a set of concept specified by the user for the model to predict a certain class. Although these explanations are typically global (i.e. at the dataset level), they are based on concept classifiers that attempt to detect the presence/absence of the concept on each individual example  $x \in \mathcal{X}(\Omega, \mathcal{C})$  based on its representation  $h(x)$ . We show that these classifiers are  $\mathcal{G}$ -invariant if we restrict to representations  $h$  that are  $\mathcal{G}$ -invariant.
Proposition D.14 (Concept-Based Invariance). Consider a differentiable neural network  $f: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{Y}$  that is invariant with respect to the symmetry group  $\mathcal{G}$ . We assume that  $\mathcal{G}$  acts on  $\mathcal{X}(\Omega, \mathcal{C})$  via the representation  $\rho: \mathcal{G} \to \operatorname{Aut}[\mathcal{X}(\Omega, \mathcal{C})]$ . We consider a concept-based explanation based on a representation  $h: \mathcal{X}(\Omega, \mathcal{C}) \to \mathcal{H}$  extracted from  $f$  (e.g. an intermediate layer of the neural network) and of the form
$$
e (x) = c [ h (x) ],
$$
where  $c: \mathcal{H} \to \{0,1\}^C$  maps any representation  $r \in \mathcal{H}$  to a binary vector in  $c[r] \in \{0,1\}^C$  indicating the presence/absence of  $C \in \mathbb{N}^+$  selected concepts. If the representation  $h$  is  $\mathcal{G}$ -invariant, then the explanation  $e$  is  $\mathcal{G}$ -invariant.
Remark D.15. We note that concepts activation vectors (CAVs) are obtained by fitting a linear classifier  $c$ . Concept activations regions (CARs), on the other hand, are obtained by fitting a kernel-based concept classifier.
$$
\begin{array}{l} e (\rho [ g ] x) = c [ h (\rho [ g ] x) ] \\ = c [ h (x) ] \quad (\text {I n v a r i a n c e} h) \\ = e (x), \\ \end{array}
$$
D.4 Enforcing Invariance
Finally, we prove Proposition 2.3 that allows us to turn any interpretability method into a  $\mathcal{G}$ -invariant method.