SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thomas Fel\*,1,2, Victor Boutin\*,1,2,  
Mazda Moayeri\*, Remy Cadene\*, Louis Bethune\* Léo Andeol\*, Mathieu Chalvidal\*,2, Thomas Serre\*,2
$^{1}$ Carney Institute for Brain Science, Brown University
$^{2}$ Artificial and Natural Intelligence Toulouse Institute
<sup>3</sup>Department of Computer Science, University of Maryland {thomas_fel,victor_boutin}@brown.edu
In recent years, concept-based approaches have emerged as some of the most promising explainability methods to help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual "concepts" buried within the complex patterns of ANN activations in two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce a unifying theoretical framework that recast the first step - concept extraction problem - as a special case of dictionary learning, and we formalize the second step - concept importance estimation - as a more general form of attribution method. This framework offers several advantages as it allows us:  $(i)$  to propose new evaluation metrics for comparing different concept extraction approaches;  $(ii)$  to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-art concept-based approaches and importance estimation techniques;  $(iii)$  to derive theoretical guarantees regarding the optimality of such methods.
We further leverage our framework to try to tackle a crucial question in explainability: how to efficiently identify clusters of data points that are classified based on a similar shared strategy. To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic cluster graph. Finally, we present Lens, a dedicated website that offers a complete compilation of these visualizations for all classes of the ImageNet dataset.
The black-box nature of Artificial Neural Networks (ANNs) poses a significant hurdle to their deployment in industries that must comply with stringent ethical and regulatory standards [1]. In response to this challenge, eXplainable Artificial Intelligence (XAI) focuses on developing new tools to help humans better understand how ANNs arrive at their decisions [2, 3]. Among the large array of methods available, attribution methods have become the go-to approach [4-14]. They yield heatmaps in order to highlight the importance of each input feature (or group of features [15]) for driving a model's decision. However, there is growing consensus that these attribution methods fall short of providing meaningful explanations [16-19] as revealed by multiple user studies [20-25, 20]. It has been suggested that for explainability methods to become usable by human users, they need to be able to highlight not just the location of important features within an image (i.e., the where information) but also their semantic content (i.e., the what information).
$\star$  The authors contributed equally.
One promising set of explainability methods to address this challenge includes concept-based explainability methods, which are methods that aim to identify high-level concepts within the activation space of ANNs [26]. These methods have recently gained renewed interest due to their success in providing human-interpretable explanations [27-30] (see section 2 for a detailed description of the related work). However, concept-based explainability methods are still in the early stages, and progress relies largely on researchers' intuitions rather than well-established theoretical foundations. A key challenge lies in formalizing the notion of concept itself [31]. Researchers have proposed desiderata such as meaningfulness, coherence, and importance [27] but the lack of formalism in concept definition has hindered the derivation of appropriate metrics for comparing different methods.
This article presents a theoretical framework to unify and characterize current concept-based explainability methods. Our approach builds on the fundamental observation that all concept-based explainability methods share two key steps: (1) concepts are extracted, and (2) importance scores are assigned to these concepts based on their contribution to the model's decision [27]. Here, we show how the first extraction step can be formulated as a dictionary learning problem while the second importance scoring step can be formulated as an attribution problem in the concept space. To summarize, our contributions are as follows:
- We describe a novel framework that unifies all modern concept-based explainability methods and we borrow metrics from different fields (such as sparsity, reconstruction, stability, FID, or OOD scores) to evaluate the effectiveness of those methods.
- We leverage modern attribution methods to derive seven novel concept importance estimation methods and provide theoretical guarantees regarding their optimality. Additionally, we show how standard faithfulness evaluation metrics used to evaluate attribution methods (i.e., Insertion, Deletion [32], and  $\mu$  Fidelity [33]) can be adapted to serve as benchmarks for concept importance scoring. In particular, we demonstrate that Integrated Gradients, Gradient Input, RISE, and Occlusion achieve the highest theoretical scores for 3 faithfulness metrics when the concept decomposition is on the penultimate layer.
- We introduce the notion of local concept importance to address a significant challenge in explainability: the identification of image clusters that reflect a shared strategy by the model (see Figure 1). We show how the corresponding cluster plots can be used as visualization tools to help with the identification of the main visual strategies used by a model to help explain false positive classifications.
Figure 1: Strategic cluster graphs for the espresso and zucchini classes. The framework presented in this study provides a comprehensive approach to uncover local importance using any attribution methods. Consequently, it allows us to estimate the critical concepts influencing the model's decision for each image. As a result, we introduced the Strategic cluster graph, which offers a visual representation of the main strategies employed by the model in recognizing an entire object class. For espresso (left), the main strategies for classification appear to be: bubbles and foam on the coffee, Latte art, transparent cups with foam and black liquid, the handle of the coffee cup, and finally the coffee in the cup, which appears to be the predominant strategy. As for zucchini, the strategies are: a zucchini in a vegetable garden, the corolla of the zucchini flower, sliced zucchini, the spotted pattern on the zucchini skin and stacked zucchini.
Kim et al. [26] were the first to propose a concept-based approach to interpret neural network internal states. They defined the notion of concepts using Concept Activation Vectors (CAVs). CAVs are derived by training a linear classifier between a concept's examples and random counterexamples and then taking the vector orthogonal to the decision boundary. In their work, the concepts are manually selected by humans. They further introduce the first concept importance scoring method, called Testing with CAVs (TCAV). TCAV uses directional derivatives to evaluate the contribution of each CAV to the model's prediction for each object category. Although this approach demonstrates meaningful explanations to human users, it requires a significant human effort to create a relevant image database of concepts. To address this limitation, Ghorbani et al. [27] developed an unsupervised method called Automatic Concept Extraction (ACE) that extracts CAVs without the need for human supervision. In their work, the CAVs are the centroid of the activations (in a given layer) when the network is fed with multi-scale image segments belonging to an image class of interest. However, the use of image segments could introduce biases in the explanations [34-37]. ACE also leverages TCAV to rank the concepts of a given object category based on their importance.
Zhang et al. [28] proposed a novel method for concept-based explainability called Invertible Concept-based Explanation (ICE). ICE leverages matrix factorization techniques, such as non-negative matrix factorization (NMF), to extract Concept Activation Vectors (CAVs). Here, the concepts are localized as the matrix factorization is applied on feature maps (before the global average pooling). In ICE, the concepts' importance is computed using the TCAV score [26]. Note that the Singular Value Decomposition (SVD) was also suggested as a concept discovery method [28, 30]. CRAFT (Concept Recursive Activation FacTorization for explainability) uses NMF to extract the concepts, but as it is applied after the global pooling average, the concepts are location invariant. Additionally, CRAFT employs Sobol indices to quantify the global importance of concepts associated with an object category.
3 A Unifying perspective
Notations. Throughout,  $||\cdot ||_2$  and  $||\cdot ||_F$  represent the  $\ell_{2}$  and Frobenius norm, respectively. We consider a general supervised learning setting, where a classifier  $f:\mathcal{X}\to \mathcal{Y}$  maps inputs from an input space  $\mathcal{X}\subseteq \mathbb{R}^d$  to an output space  $\mathcal{V}\subseteq \mathbb{R}^c$ . For any matrix  $\mathbf{X}\in \mathbb{R}^{n\times d}$ ,  $\pmb{x}_i$  denotes the  $i^{th}$  row of  $\mathbf{X}$ , where  $i\in \{1,\dots ,n\}$  and  $\pmb{x}_i\in \mathbb{R}^d$ . Without loss of generality, we assume that  $\pmb{f}$  admits an intermediate space  $\mathcal{H}\subseteq \mathbb{R}^p$ . In this setup,  $h:\mathcal{X}\rightarrow \mathcal{H}$  maps inputs to the intermediate space, and  $g:\mathcal{H}\rightarrow \mathcal{V}$  takes the intermediate space to the output. Consequently,  $\pmb {f}(\pmb {x}) = (g\circ h)(\pmb {x})$ . Additionally, let  $\pmb {a} = \pmb {h}(\pmb {x})\in \mathcal{H}$  represent the activations of  $\pmb{x}$  in this intermediate space. We also abuse notation slightly:  $\pmb {f}(\mathbf{X}) = (\pmb {g}\circ \pmb {h})(\mathbf{X})$  denotes the vectorized application of  $\pmb{f}$  on each element  $\pmb{x}$  of  $\mathbf{X}$ , resulting in  $(f(x_{1}),\ldots ,f(x_{n}))$ .
Prior methods for concept extraction, namely ACE [27], ICE [28] and CRAFT [29], can be distilled into two fundamental steps:
(i) Concept extraction: A set of images  $\mathbf{X} \in \mathbb{R}^{n \times d}$  belonging to the same class is sent to the intermediate space giving activations  $\mathbf{A} = \mathbf{h}(\mathbf{X}) \in \mathbb{R}^{n \times p}$ . These activations are used to extract a set of  $k$  CAVs using K-Means [27], PCA (or SVD) [28, 30] or NMF [28, 29]. Each CAV is denoted  $\pmb{v}_i$  and  $\mathbf{V} = (\pmb{v}_1, \dots, \pmb{v}_k) \in \mathbb{R}^{p \times k}$  forms the dictionary of concepts.
(ii) Concept importance scoring: It involves calculating a set of  $k$  global scores, which provides an importance measure of each concept  $v_{i}$  to the class as a whole. Specifically, it quantifies the influence of each concept  $v_{i}$  on the final classifier prediction for the given set of points  $\mathbf{X}$ . Prominent measures for concept importance include TCAV [26] and the Sobol indices [29].
The two-step process described above is repeated for all classes. In the following subsections, we theoretically demonstrate that the concept extraction step  $(i)$  could be recast as a dictionary learning problem (see 3.1). It allows us to reformulate and generalize the concept importance step  $(ii)$  using attribution methods (see 3.2).
3.1 Concept Extraction
A dictionary learning perspective. The purpose of this section is to redefine all current concept extraction methods as a problem within the framework of dictionary learning. Given the necessity for clearer formalization and metrics in the field of concept extraction, integrating concept extraction with dictionary learning enables us to employ a comprehensive set of metrics and obtain valuable theoretical insights from a well-established and extensively researched domain.
The goal of concept extraction is to find a small set of interpretable CAVs (i.e.,  $\mathbf{V}$ ) that allows us to faithfully interpret the activation  $\mathbf{A}$ . By preserving a linear relationship between  $\mathbf{V}$  and  $\mathbf{A}$ , we facilitate the understanding and interpretability of the learned concepts [26, 38]. Therefore, we look for a coefficient matrix  $\mathbf{U} \in \mathbb{R}^{n \times k}$  (also called loading matrix) and a set of CAVs  $\mathbf{V}$ , so that  $\mathbf{A} \approx \mathbf{U}\mathbf{V}^{\mathrm{T}}$ . In this approximation of  $\mathbf{A}$  using the two low-rank matrices  $(\mathbf{U}, \mathbf{V})$ ,  $\mathbf{V}$  represents the concept basis used to reinterpret our samples, and  $\mathbf{U}$  are the coordinates of the activation in this new basis. Interestingly, such a formulation allows a recast of the concept extraction problem as an instance of dictionary learning problem [39] in which all known concept-based explainability methods fall:
$$
\left(\mathbf {U} ^ {\star}, \mathbf {V} ^ {\star}\right) = \underset {\mathbf {U}, \mathbf {V}} {\arg \min } \| \mathbf {A} - \mathbf {U} \mathbf {V} ^ {\top} \| _ {F} ^ {2} s. t \left\{ \begin{array}{l} \forall i, \boldsymbol {u} _ {i} \in \left\{\mathbf {e} _ {1}, \dots , \mathbf {e} _ {k} \right\} (\text {K - M e a n s : A C E [ 2 7 ]}), \\ \mathbf {V} ^ {\top} \mathbf {V} = \mathbf {I} (\text {P C A : [ 2 8 , 3 0 ]}), \\ \mathbf {U} \geq 0, \mathbf {V} \geq 0 (\text {N M F : C R A F T [ 2 9 ] & I C E [ 2 8 ]}) \\ \mathbf {U} = \psi (\mathbf {A}), \| \mathbf {U} \| _ {0} \leq K (\text {S p a r s e A t o e n c o d e r [ 4 0 ]}) \end{array} \right. \tag {1}
$$
with  $e_i$  the  $i$ -th element of the canonical basis,  $\mathbf{I}$  the identity matrix and  $\psi$  any neural network. In this context,  $\mathbf{V}$  is the dictionary and  $\mathbf{U}$  the representation of  $\mathbf{A}$  with the atoms of  $\mathbf{V}$ .  $u_i$  denote the  $i$ -th row of  $\mathbf{U}$ . These methods extract the concept banks  $\mathbf{V}$  differently, thereby necessitating different interpretations*.
In ACE, the CAVs are defined as the centroids of the clusters found by the K-means algorithm. Specifically, a concept vector  $\boldsymbol{v}_i$  in the matrix  $\mathbf{V}$  indicates a dense concentration of points associated with the corresponding concept, implying a repeated activation pattern. The main benefit of ACE comes from its reconstruction process, involving projecting activations onto the nearest centroid, which ensures that the representation will lie within the observed distribution (no out-of-distribution instances). However, its limitation lies in its lack of expressivity, as each activation representation is restricted to a single concept  $(||\boldsymbol{u}||_0 = 1)$ . As a result, it cannot capture compositions of concepts, leading to sub-optimal representations that fail to fully grasp the richness of the underlying data distribution.
On the other hand, the PCA benefits from superior reconstruction performance due to its lower constraints, as stated by the Eckart-Young-Mirsky[43] theorem. The CAVs are the eigenvector of the covariance matrix: they indicate the direction in which the data variance is maximal. An inherent limitation is that the PCA will not be able to properly capture stable concepts that do not contribute to the sample variability (e.g. the dog-head concept might not be considered important by the PCA to explain the dog class if it is present across all examples). Neural networks are known to cluster together the points belonging to the same category in the last layer to achieve linear separability ([44, 29]). Thus, the orthogonality constraint in the PCA might not be suitable to correctly interpret the manifold of the deep layer induced by points from the same class (it is interesting to note that this limitation can be of interest when studying all classes at once). Also, unlike K-means, which produces strictly positive clusters if all points are positive (e.g., the output of ReLU), PCA has no sign constraint and can undesirably reconstruct out-of-distribution (OOD) activations, including negative values after ReLU.
In contrast to K-Means, which induces extremely sparse representations, and PCA, which generates dense representations, the NMF (used in CRAFT and ICE) strikes a harmonious balance as it provides moderately sparse representation. This is due to NMF relaxing the constraints imposed by the K-means algorithm (adding an orthogonality constraint on  $\mathbf{V}$  such that  $\mathbf{V}\mathbf{V}^{\mathrm{T}} = \mathbf{I}$  would yield an equivalent solution to K-means clustering [45]). This sparsity facilitates the encoding of
*Concept extractions are typically overcomplete dictionaries, meaning that if the dictionary for each class is combined,  $k > p$ , as noted in [29]. Recently, [29] and a more detailed work [41] suggest that overcomplete dictionaries are serious candidates to the superposition problem [42].
Figure 2: Most important concepts extracted for the studied methods. This qualitative example shows the three most important concepts extracted for the 'rabbit' class using a ResNet50 trained on ImageNet. The crops correspond to those maximizing each concepts  $i$  (i.e.,  $\pmb{x}$  where  $\mathbf{U}(\pmb{x})_i$  is maximal). As demonstrated in previous works [28, 29, 49], NMF (requiring positive activations) produces particularly interpretable concepts despite poorer reconstruction than PCA and being less sparse than K-Means. Details for the sparse Autoencoder architecture are provided in the appendix.
compositional representations that are particularly valuable when an image encompasses multiple concepts. Moreover, by allowing only additive linear combinations of components with non-negative coefficients, NMF inherently fosters a parts-based representation. This distinguishes NMF from PCA, which offers a holistic representation model. Interestingly, the NMF is known to yield representations that are interpretable by humans [28, 29]. Finally, the non-orthogonality of these concepts presents an advantage as it accommodates the phenomenon of superposition [38], wherein neurons within a layer may contribute to multiple distinct concepts simultaneously.
To summarize, we have explored three approaches to concept extraction, each necessitating a unique interpretation of the resulting Concept Activation Vectors (CAVs). Among these methods, NMF (used in CRAFT and ICE) emerges as a promising middle ground between PCA and K-means. Leveraging its capacity to capture intricate patterns, along with its ability to facilitate compositional representations and intuitive parts-based interpretations (as demonstrated in Figure 2), NMF stands out as a compelling choice for extracting meaningful concepts from high-dimensional data. These advantages have been underscored by previous human studies, as evidenced by works such as Zhang et al.[28] and Fel et al.[29].
Relative ℓ2(↓)	Sparsity (↑)	Stability (↓)	FID (↓)	OOD (↓)
Eff / R50 / Mob	Eff / R50 / Mob	Eff / R50 / Mob	Eff / R50 / Mob	Eff / R50 / Mob
PCA	0.60 / 0.54 / 0.73	0.00 / 0.00 / 0.0	0.41 / 0.38 / 0.43	0.47 / 0.17 / 0.24	2.44 / 0.36 / 0.16
KMeans	0.72 / 0.66 / 0.84	0.95 / 0.95 / 0.95	0.07 / 0.08 / 0.04	0.46 / 0.21 / 0.33	1.76 / 0.29 / 0.15
NMF	0.63 / 0.57 / 0.75	0.68 / 0.44 / 0.64	0.17 / 0.14 / 0.16	0.38 / 0.21 / 0.24	1.98 / 0.29 / 0.15
Table 1: Concept extraction comparison. Eff, R50 and Mob denote EfficientNetV2[46], ResNet50[47], MobileNetV2[48]. The concept extraction methods are applied on the last layer of the networks. Each results is averaged across 10 classes of ImageNet and obtained from a set of 16k images for each class.
Evaluation of concept extraction Following the theoretical discussion of the various concept extraction methods, we conduct an empirical investigation of the previously discussed properties to gain deeper insights into their distinctions and advantages. In our experiment, we apply the PCA, K-Means, and NMF concept extraction methods on the penultimate layer of three state-of-the-art models. We subsequently evaluate the concepts using five different metrics (see Table 1). All five metrics are connected with the desired characteristics of a dictionary learning method. They include achieving a high-quality reconstruction (Relative 12), sparse encoding of concepts (Sparsity), ensuring the stability of the concept base in relation to A (Stability), performing reconstructions within the intended domain (avoiding OOD), and maintaining the overall distribution during the reconstruction process (FID). All the results come from 10 classes of ImageNet (the one used in Imagenette [50]), and are obtained using  $n = 16k$  images for each class.
We begin our empirical investigation by using a set of standard metrics derived from the dictionary learning literature, namely Relative  $l_{2}$  and Sparsity. Concerning the Relative  $l_{2}$ , PCA achieves the highest score among the three considered methods, confirming the theoretical expectations based on the Eckart-Young-Mirsky theorem [43], followed by NMF. Concerning the sparsity of the underlying representation  $\mathbf{u}$ , we compute the proportion of non-zero elements  $||\mathbf{u}||_0 / k$ . Since
K-means inherently has a sparsity of  $1 / k$  (as induced by equation 1), it naturally performs better in terms of sparsity, followed by NMF.
We deepen our investigation by proposing three additional metrics that offer complementary insights into the extracted concepts. Those metrics are the Stability, the FID, and the OOD score. The Stability (as it can be seen as a loose approximation of algorithmic stability [51]) measures how consistent concepts remain when they are extracted from different subsets of the data. To evaluate Stability, we perform the concept extraction methods  $N$  times on  $K$ -fold subsets of the data. Then, we map the extracted concepts together using a Hungarian loss function and measure the cosine similarity of the CAVs. If a method is stable, it should yield the same concepts (up to permutation) across each  $K$ -fold, where each fold consists of 1000 images. K-Means and NMF demonstrate the highest stability, while PCA appears to be highly unstable, which can be problematic for interpreting the results and may undermine confidence in the extracted concepts.
The last two metrics, FID and OOD, are complementary in that they measure: (i) how faithful the representations extracted are w.r.t the original distribution, and (ii) the ability of the method to generate points lying in the data distribution (non-OOD). Formally, the FID quantifies the 1-Wasserstein distance [52]  $\mathcal{W}_1$  between the empirical distribution of activation  $\mathbf{A}$ , denoted  $\mu_{\mathbf{a}}$ , and the empirical distribution of the reconstructed activation  $\mathbf{U}\mathbf{V}^{\mathrm{T}}$  denoted  $\mu_{\mathbf{u}}$ . Thus, FID is calculated as  $\mathrm{FID} = \mathcal{W}_1(\mu_{\mathbf{a}},\mu_{\mathbf{u}})$ . On the other hand, the OOD score measures the plausibility of the reconstruction by leveraging Deep-KNN [53], a recent state-of-the-art OOD metric. More specifically, we use the Deep-KNN score to evaluate the deviation of a reconstructed point from the closest original point. In summary, a good reconstruction method is capable of accurately representing the original distribution (as indicated by FID) while ensuring that the generated points remain within the model's domain (non-OOD). K-means leads to the best OOD scores because each instance is reconstructed as a centroid, resulting in proximity to in-distribution (ID) instances. However, this approach collapses the distribution to a limited set of points, resulting in low FID. On the other hand, PCA may suffer from mapping to negative values, which can adversely affect the OOD score. Nevertheless, PCA is specifically optimized to achieve the best average reconstructions. NMF, with fewer stringent constraints, strikes a balance by providing in-distribution reconstructions at both the sample and population levels.