SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In conclusion, the results clearly demonstrate NMF as a method that strikes a balance between the two approaches as NMF demonstrates promising performance across all tested metrics. Henceforth, we will use the NMF to extract concepts without mentioning it.
The Last Layer as a Promising Direction The various methods examined, namely ACE, ICE, and CRAFT, generally rely on a deep layer to perform their decomposition without providing quantitative or theoretical justifications for their choice. To explore the validity of this choice, we apply the aforementioned metrics to each block's output in a ResNet50 model. Figure 3 illustrates the metric evolution across different blocks, revealing a trend that favors the last layer for the decomposition. This empirical finding aligns with the practical implementations discussed above.
Figure 3: Concept extraction metrics across layers. The concept extraction methods are applied on activations probed on different blocks of a ResNet50 (B2 to B5). Each point is averaged over 10 classes of ImageNet using 16k images for each class. We evaluate 3 concept extraction methods: PCA  $(\texttt{- - - })$ , NMF  $(\text{一} )$  , and KMeans  $(\cdot \cdot \cdot \cdot)$
3.2 Concept importance
In this section, we leverage our framework to unify concept importance scoring using the existing attribution methods. Furthermore, we demonstrate that specifically in the case of decomposition in the penultimate layer, it exists optimal methods for importance estimation, namely RISE [32],
Integrated Gradients [7], Gradient-Input [6], and Occlusion [13]. We provide theoretical evidence to support the optimality of these methods.
From concept importance to attribution methods The dictionary learning formulation allows us to define the concepts  $\mathbf{V}$  in such a way that they are optimal to reconstruct the activation, i.e.,  $\mathbf{A} \approx \mathbf{U}\mathbf{V}^{\top}$ . Nevertheless, this does not guarantee that those concepts are important for the model's prediction. For example, the "grass" concept might be important to characterize the activations of a neural network when presented with a St-Bernard image, but it might not be crucial for the network to classify the same image as a St Bernard [26, 16, 54]. The notion of concept importance is precisely introduced to avoid such a confirmation bias and to identify the concepts used to classify among all detected concepts.
We use the notion of Concept ATribution methods (which we denote as CATs) to assess the concept importance score. The CATs are a generalization of the attribution methods: while attribution methods assess the sensitivity of the model output to a change in the pixel space, the concept importance evaluates the sensitivity to a change in the concept space. To compute the CATs methods, it is necessary to link the activation  $\mathbf{a} \in \mathbb{R}^p$  to the concept base  $\mathbf{V}$  and the model prediction  $\mathbf{y}$ . To do so, we feed the second part of the network  $(\mathbf{g})$  with the activation reconstruction  $(\mathbf{u}\mathbf{V}^{\top} \approx \mathbf{a})$  so that  $\mathbf{y} = \mathbf{g}(\mathbf{u}\mathbf{V}^{\top})$ . Intuitively, a CAT method quantifies how a variation of  $\mathbf{u}$  will impact  $\mathbf{y}$ . We denote  $\varphi_{i}(\mathbf{u})$  the  $i$ -th coordinate of  $\varphi(\mathbf{u})$ , so that it represents the importance of the  $i$ -th concept in the representation  $\mathbf{u}$ . Equipped with these notations, we can leverage the sensitivity metrics introduced in standard attribution methods to re-define the current measures of concept importance, as well as introduce the new CATs borrowed from the attribution methods literature:
$$
\varphi_ {i} (\boldsymbol {u}) = \left\{ \begin{array}{l l} \nabla_ {\boldsymbol {u} _ {i}} \boldsymbol {g} (\boldsymbol {u} \mathbf {V} ^ {\top}) & \text {(u s e d i n T C A V : A C E , I C E)}, \\ \frac {\mathbb {E} _ {\mathbf {m} _ {\sim i}} (\mathbb {V} _ {\mathbf {m}} (\boldsymbol {g} ((\boldsymbol {u} \odot \mathbf {m}) \mathbf {V} ^ {\top}) | \mathbf {m} _ {\sim i}))}{\mathbb {V} (\boldsymbol {g} ((\boldsymbol {u} \odot \mathbf {m}) \mathbf {V} ^ {\top}))} & \text {(S o b o l : C R A F T)}, \\ (\boldsymbol {u} _ {i} - \boldsymbol {u} _ {i} ^ {\prime}) \times \int_ {0} ^ {1} \nabla_ {\boldsymbol {u} _ {i}} \boldsymbol {g} ((\boldsymbol {u} ^ {\prime} \alpha + (1 - \alpha) (\boldsymbol {u} - \boldsymbol {u} ^ {\prime})) \mathbf {V} ^ {\top}) d \alpha & \text {I n t . G r a d i e n t s}, \\ \underset {\delta \sim \mathcal {N} (0, \mathbf {I} \sigma)} {\mathbb {E}} (\nabla_ {\boldsymbol {u} _ {i}} \boldsymbol {g} ((\boldsymbol {u} + \delta) \mathbf {V} ^ {T})) & \text {S m o o t h g r a d .} \\ \dots & \end{array} \right.
$$
The complete derivation of the 7 new CATs is provided in the appendix. In the derivations,  $\nabla_{\boldsymbol{u}_i}$  denotes the gradient with respect to the  $i$ -th coordinate of  $\boldsymbol{u}$ , while  $\mathbb{E}$  and  $\mathbb{V}$  represent the expectation and variance, respectively. In Eq. 6,  $\mathbf{m}$  is a mask of real-valued random variable between 0 and 1 (i.e.  $\mathbf{m} \sim \mathcal{U}([0,1]^p)$ ). We note that, when we use the gradient (w.r.t to  $\boldsymbol{u}_i$ ) as an importance score, we end up with the directional derivative used in the TCAV metric [26], which is used by ACE and ICE to assess the importance of concepts. CRAFT leverages the Sobol-Hoeffding decomposition (used in sensitivity analysis), to estimate the concept importance. The Sobol indices measure the contribution of a concept as well as its interaction of any order with any other concepts to the output variance. Intuitively, the numerator of Eq. 6 is the expected variance that would be left if all variables but  $\boldsymbol{u}_i$  were to be fixed.
Evaluation of concept importance methods Our generalization of the concept importance score, using the Concept ATtributions (CATs), allows us to observe that current concept-based explainability methods are only leveraging a small subset of concept importance methods. In Appendix A, we provide the complete derivation of 7 new CATs based on the following existing attribution methods, notably: Gradient input [6], Smooth grad [5], Integrated Gradients [7], VarGrad [55], Occlusion [13], HSIC [10] and RISE [32].
With the concept importance scoring now formulated as a generalization of attribution methods, we can borrow the metrics from the attribution domain to evaluate the faithfulness [56, 32, 33] of concept importance methods. In particular, we adapt three distinct metrics to evaluate the significance of concept importance scores: the C-Deletion [32], C-Insertion [32], and  $\mathrm{C - }\mu$  Fidelity [33] metrics. In C-Deletion, we gradually remove the concepts (as shown in Figure 4), in decreasing order of importance, and we report the network's output each time a concept is removed. When a concept is removed in C-Deletion, the corresponding coordinate in the representation is set to 0. The final C-Deletion metrics are computed as the area under the curve in Figure 4. For C-Insertion, this is
(b)
Figure 4: (a) C-Deletion, C-Insertion curves. Fidelity curves for C-Deletion depict the model's score as the most important concepts are removed. The results are averaged across 10 classes of ImageNet using a ResNet50 model. (b) C-Deletion, C-Insertion and C- $\mu$ Fidelity across layer. We report the 3 metrics to evaluate CATs for each block (from B2 to B5) of a ResNet50. We evaluate 8 Concept Attribution methods, all represented with different colors (see legend in Figure 4(a). The average trend of these eight methods is represented by the black dashed line  $(- - - )$ . Lower C-Deletion is better, higher C-Insertion and C- $\mu$ Fidelity is better. Overall, it appears that the estimation of importance becomes more faithful towards the end of the model.
the opposite: we start from a representation vector filled with zero, and we progressively add more concepts, following an increasing order of importance.
For the C-  $\mu$  Fidelity, we calculate the correlation between the model's output when concepts are randomly removed and the importance assigned to those specific concepts. The results across layers for a ResNet50 model are depicted in Figure 4b. We observe that decomposition towards the end of the model is preferred across all the metrics. As a result, in the next section, we will specifically examine the case of the penultimate layer.
A note on the last layer Based on our empirical results, it appears that the last layer is preferable for both improved concept extraction and more accurate estimation of importance. Herein, we derive theoretical guarantees about the optimality of concept importance methods in the penultimate layer. Without loss of generality, we assume  $y \in \mathbb{R}$  the logits of the class of interest. In the penultimate layer, the score  $y$  is a linear combination of activations:  $y = a\mathbf{W} + \mathbf{b}$  for weight matrix  $\mathbf{W}$  and bias  $\mathbf{b}$ . In this particular case, all CATs have a closed-form (see appendix B), that allows us to derive 2 theorems. The first theorem tackles the CATs optimality for the C-Deletion and C-Insertion methods (demonstration in Appendix D). We observe that the C-Deletion and C-Insertion problems can be represented as weighted matroids. Therefore the greedy algorithms lead to optimal solutions for CATs and a similar theorem could be derived for  $\mathrm{C - }\mu$  Fidelity.
Theorem 3.1 (Optimal C-Deletion, C-Insertion in the penultimate layer). When decomposing in the penultimate layer, Gradient Input, Integrated Gradients, Occlusion, and Rise yield the optimal solution for the C-Deletion and C-Insertion metrics. More generally, any method  $\varphi(\mathbf{u})$  that satisfies the condition  $\forall (i,j) \in \{1,\dots,k\}^2, (\mathbf{u} \odot \mathbf{e}_i)\mathbf{V}^\top \mathbf{W} \geq (\mathbf{u} \odot \mathbf{e}_j)\mathbf{V}^\top \mathbf{W} \Rightarrow \varphi(\mathbf{u})_i \geq \varphi(\mathbf{u})_j$  yields the optimal solution.
Theorem 3.2 (Optimal C-  $\mu$  Fidelity in the penultimate layer). When decomposing in the penultimate layer, Gradient Input, Integrated Gradients, Occlusion, and Rise yield the optimal solution for the  $C - \mu$  Fidelity metric.
Therefore, for all 3 metrics, the concept importance methods based on Gradient Input, Integrated Gradient, Occlusion, and Rise are optimal, when used in the penultimate layer.
In summary, our investigation of concept extraction methods from the perspective of dictionary learning demonstrates that the NMF approach, specifically when extracting concepts from the penultimate layer, presents the most appealing trade-off compared to PCA and K-Means methods. In addition, our formalization of concept importance using attribution methods provided us with a theoretical guarantee for 4 different CATs. Henceforth, we will then consider the following setup: a
NMF on the penultimate layer to extract the concepts, combined with a concept importance method based on Integrated Gradient.
3.3 Unveiling main strategies
Figure 5: From global (class-based) to local (image-based) importance. Global importance can be decomposed into reliability and prevalence scores. Prevalence quantifies how frequently a concept is encountered, and reliability indicates how diagnostic a concept is for the class. The bar-charts are computed for the class "Espresso" on a ResNet50 (see Figure 1, left panel)
So far, the concept-based explainability methods have mainly focused on evaluating the global importance of concepts, i.e., the importance of concepts for an entire class [26, 29]. This point can be limiting when studying misclassified data points, as we can speculate that the most important concepts for a given class might not hold for an individual sample (local importance). Fortunately, our formulation of concept importance using attribution methods gives us access to importance scores at the level of individual samples (i.e.,  $\varphi(\pmb{u})$ ). Here, we show how to use these local importance scores to efficiently cluster data points based on the strategy used for their classification.
The local (or image-based) importance of concepts can be integrated into global measures of importance for the entire class with the notion of prevalence and reliability (see Figure 5). A concept is said to be prevalent at the class level when it appears very frequently. A prevalence score is computed based on the number of times a concept is identified as the most important one, i.e.,  $\arg \max \varphi (\boldsymbol {u})$ . At the same time, a concept is said to be reliable if it is very likely to trigger a correct prediction. The reliability is quantified using the mean classification accuracy on samples sharing the same most important concept.
Strategic cluster graph. In the strategic cluster graph (Figure 1 and Figure 6), we combine the notions of
concept prevalence and reliability to reveal the main strategies of a model for a given category, more precisely, we reveal their repartition across the different samples of the class. We use a dimensionality reduction technique (UMAP [57]) to arrange the data points based on the concept importance vector  $\varphi(\boldsymbol{u})$  of each sample. Data points are colored according to the associated concept with the highest importance - arg max  $\varphi(\boldsymbol{u})$ . Interestingly, one can see in Figure 1 and Figure 6 that spatially close points represent samples classified using similar strategies - as they exhibit similar concept importance - and not necessarily similar embeddings. For example, for the "lemon" object category (Figure 6), the texture of the lemon peel is the most prevalent concept, as it appears to be the dominant concept in  $90\%$  of the samples (see the green cluster in Figure 6). We also observe that the concept "pile of round, yellow objects" is not reliable for the network to properly classify a lemon as it results in a mean classification accuracy of  $40\%$  only (see top-left graph in Figure 6).
In Figure 6 (right panel), we have exploited the strategic cluster graph to understand the classification strategies leading to bad classifications. For example, an orange  $(1^{st}$  image,  $1^{st}$  row) was classified as a lemon because of the peel texture they both share. Similarly, a cathedral roof was classified as a lemon because of the wedge-shaped structure of the structure  $(4^{th}$  image,  $1^{st}$  row).
This article introduced a theoretical framework that unifies all modern concept-based explainability methods. Breaking down and formalizing the two essential steps in these methods, concept extraction and concept importance scoring, allowed us to better understand the underlying principles driving concept-based explainability. We leveraged this unified framework to propose new evaluation metrics for assessing the quality of extracted concepts. Through experimental and theoretical analyses, we justified the standard use of the last layer of an ANN for concept-based explanation. Finally, we harnessed the parallel between concept importance and attribution methods to gain insights into global concept importance (at the class level) by examining local concept importance (for individual samples). We proposed the strategic cluster graph, which provides insights into the strategy used by an ANN to classify images. We have provided an example use of this approach to better understand
Understand False Positive
Hallucinating lemon because of the peel texture
Hallucinating lemon because of lemon wedge shape
Hallucination of lemons from piles of round, yellow objects
Figure 6: Strategic cluster graph for the lemon category. Left: U-MAP of lemon samples, in the concept space. Each concept is represented with its own color and is exemplified with example belonging to the cluster. The concepts are  $\bullet$  the lemon wedge shape,  $\bullet$  a pile of round, yellow objects,  $\bullet$  green objects hanging on a tree, and finally  $\bullet$  the peel texture, which is the predominant strategy. The reliability of each concept is shown in the top-left bar-chart. Right: Example of images predicted as lemon along with their corresponding explanations. These misclassified images are recognized as lemons through the implementation of strategies that are captured by our proposed strategic cluster graph.
Hallucinating a lemon because of a green object hanging on a tree
the failure cases of a system. Overall, our work demonstrates the potential benefits of the dictionary learning framework for automatic concept extraction and we hope this work will pave the way for further advancements in the field of XAI.
This work was conducted as part of the DEEL project. Funding was provided by ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481), DARPA (D19AC00015), NIH/NINDS (R21 NS 112743), and the ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A0004). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant S10OD025181.
$\dagger$  https://www.deel.ai/
[59] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[60] Matthew Sotoudeh and Aditya V. Thakur. Computing linear restrictions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[61] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2014.
[62] Hassler Whitney. On the abstract properties of linear dependence. Hassler Whitney Collected Papers, pages 147-171, 1992.
[63] Sajad Fathi Hafshejani and Zahra Moaberfard. Initialization for non-negative matrix factorization: a comprehensive review. International Journal of Data Science and Analytics, 2023.
[64] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[65] Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications. Springer, 2018.