A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation
Authors: Thomas FEL
Victor Boutin
Louis Béthune
Remi Cadene
Mazda Moayeri
Léo Andéol
Mathieu Chalvidal
Thomas Serre
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Explainable AI
Concept-based explainability
Interpretability
Concept extraction
Concept importance
Attribution methods
TL;DR: We introduce a new framework that unifies all existing concept based explainability methods

Abstract: In recent years, concept-based approaches have emerged as some of the most promising explainability methods to help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual ``concepts'' buried within the complex patterns of ANN activations in two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce a unifying theoretical framework that recast the first step -- concept extraction problem -- as a special case of **dictionary learning**, and we formalize the second step -- concept importance estimation -- as a more general form of **attribution method**.
This framework offers several advantages as it allows us: (i) to propose new evaluation metrics for comparing different concept extraction approaches; (ii) to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-art concept-based approaches and importance estimation techniques; (iii)  to derive theoretical guarantees regarding the optimality of such methods. 

We further leverage our framework to try to tackle a crucial question in explainability: how to *efficiently* identify clusters of data points that are classified based on a similar shared strategy.
To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic cluster graph. Finally, we present Lens, a dedicated website that offers a complete compilation of these visualizations for all classes of the ImageNet dataset.

A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation
Thomas Fel*.1.2, Victor Boutin 11.2
Mazda Moayeri', Rémi Cadène Louis Bethune Léo Andéol", Mathieu Chalvidal Thomas Serre
1 Carney Institute for Brain Science, Brown University
Artificial and Natural Intelligence Toulouse Institute "Department of Computer Science, University of Maryland (thomas_fel) victorr bowuinnnbbbootooooooo edu
Abstract
In recent years, concept-based approaches have emerged as some of the most promising explainability methods help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual "concepts" buried within the complex patterns of ANN activations two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce unifying theoretical framework that recast the first step concept extraction problem as a special case of dictionary learning, and we formalize the second step concept importance estimation as more general form of attribution method. This framework offers several advantages as it allows us: (i) propose approaches; (ii) to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-ar concept-based approaches and importance estimation techniques; (iii) to derive theoretical guarantees regarding the optimality of such methods.
evaluation metrics for comparing different concept extraction
We further leverage our framework try tackle crucial question in explain- ability: how to efficiently identify clusters of data points that are classified based on a similar shared strategy. To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic clus- ter graph. Finally, we present Lens, dedicated website that offers complete compilation of these visualizations for all classes of the ImageNet dataset.
Introduction
The black-box nature of Artificial Neural Networks (ANNs) poses significant hurdle to their deployment in industries that must comply with stringent ethical and regulatory standards [1]. In response this challenge, eXplainable Artificial Intelligence (XAI) focuses on developing new tools to help humans better understand how ANNs arrive at their decisions [2, 3J. Among the large array of methods available, attribution methods have become the go-to approach [4-14]. They yield heatmaps in order to highlight the importance of each input feature (or group of features [15]) for driving model's 's decision. However, there is growing consensus that these attribution methods fall short of providing meaningful explanations [16-19] as revealed by multiple user studies [20-25, 20J. It has been suggested that for explainability methods to become usable by human users, they need be able to highlight not just the location of important features within an image (i.e., the where information) but also their semantic content (i.e., the what information).
The authors contributed cqually.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

One promising set of explainability methods to address this challenge includes concept-based ex- plainability methods, which are methods that aim identify high-level concepts within the activation space of ANNs [26]. These methods have recently gained renewed interest due to their success providing human-interpretalle
explanations [27-30] (see section for detailed description of the related work). However, concept-based explainability methods are still in the early stages, and progress relies largely on researchers' intuitions rather than well-established theoretical foundations. A key challenge lies in formalizing the notion of concept itself [31]. Researchers have proposed desiderata such as meaningfulness, coherence, and importance [27] but the lack of formalism in concept definition has hindered the derivation of appropriate metries for comparing different methods.
This article presents a theoretical framework to unify and characterize current concept-based ex- plainability methods. Our approach builds on the fundamental observation that all concept- based explainability methods share two key steps: (1) concepts are extracted, and (2) importance scores are assigned to these concepts based on their contribution to the model's decision [27]. Here, we show how the first extraction step can be formulated as a dictionary learning problem while the second importance scoring step be formulated an attribution problem in the concept space. To
summarize, our contributions are as follows:
We describe novel framework that unifies all modern concept-based explainability methods and we borrow metrics from different fields (such as sparsity, reconstruction, stability, FID, or OOD scores) to evaluate the effectiveness of those methods.
We leverage modern attribution methods to derive seven novel concept importance estimation methods and provide theoretical guarantees regarding their optimality. Additionally, we show how standard faithfulness evaluation metrics used to evaluate attribution methods (i.c., Insertion, Deletion [32], and uFidelity [33J) can be adapted to serve as benchmarks for concept importance scoring. In particular, we demonstrate that Integrated Gradients, Gradient Input, RISE, and Occlusion achieve the highest theoretical scores for 3 faithfulness metries when the concept decomposition is on the penultimate layer.
We introduce the notion of local concept importance to address a significant challenge ex- plainability: the identification of image clusters that reflect shared strategy by the model (see Figure 1). We show how the corresponding cluster plots can be used as visualization tools to help with the identification of the main visual strategies used by model to help explain false positive classifications.
Espresso strategic clustering graph
Zucchini strategic
clustering graph
Figure 1: Strategic cluster graphs for the espresso and zucchini classes. The framework presented in this study provides a comprehensive approach to uncover local importance using any attribution methods. Consequently, it allow us to estimate the critical concepts influencing the model's decision for each image. As results, we introduced the Strategic cluster graph, which offers visual representation the main strategies employed by the model in recognizing an entire object class. For espresso (left), the main strategies for classification appear to be: bubbles and foam the coffee, Latte art, transparent cups with foam and black liquid, the handle of the coffee cup, and finally the coffee in the cup, which appears to be the predominant strategy. As for zucchini, the strategies are: zucchini in a vegetable garden, the corolla of the zucchini flower, sliced zucchini, the spotted pattern the zucchini skin and stacked zucchini.

---

2 Related Work
Kim et al. [26] were the first to propose a concept-based approach interpret neural network internal states, They defined the notion of concepts using Concept Activation Vectors (CAVs). CAVs are derived by training a linear classifier between a concept's examples and random counterexamples and then taking the vector orthogonal to the decision boundary. In their work, the concepts are manually selected by humans. They further introduce the first concept importance scoring method, called Testing with CAVs (TCAV). TCAV uses directional derivatives to evaluate the contribution of each CAV to the model's s prediction for each object category. Although this approach demonstrates meaningful explanations to human users, it requires a significant human effort to create relevant image database of concepts. To address this limitation, Ghorbani et al. [27] developed an unsupervised method called Automatic Concept Extraction (ACE) that extracts CAVs without the need for human supervision. In their work, the CAVs are the centroid of the activations (in given layer) when the network is fed with multi-scale image segments belonging to an image class of interest. However, the use of image segments could introduce biases in the explanations [34-37]. ACE also leverages TCAV to rank the concepts of given object category based on their importance.
Zhang et al. [28] proposed novel method for concept-based explainability called Invertible Concept- based Explanation (ICE). ICE leverages matrix factorization techniques, such as non-negative matrix factorization (NMF), to extract Concept Activation Vectors (CAVs). Here, the concepts are localized as the matrix factorization is applied on feature maps (before the global average pooling). In ICE, the concepts' importance is computed using the TCAV score [26]. Note that the Singular Value Decomposition (SVD) was also suggested Recursive Activation Fac Toriiizzttion for explainability) uses NMF to extract the concepts, but as it is applied after the global pooling average, the concepts location invariant. Additionally, CRAFT employs Sobol indices to quantify the global importance of concepts associated with an object category.
concept discovery method [28, 30J. CRAFT (Concept
3 A Unifying perspective
Notations. Throughout, ||- 1122 and || || represent the & and Frobenius norm, respectively. We consider a general supervised learning setting, where a classifier f X - y maps inputs from an input space X' S Rd to an output space y C R For any matrix X E Raxd æ; denotes the 7th row of X, where E (1.....n) n and æ; E IRd. Without loss of generality, we assume that f admits an intermediate space H C Rp. In this setup, h X' - H maps inputs to the intermediate space, and g : H + y takes the intermediate space to the output. Consequently, f(x) = (g (xx). Additionally, let h(x) E H represent the activations of in this intermediate space. We also abuse notation slightly: f(X) (g o (X) denotes the vectorized application of f on each element of X, resulting in (f ...., f(,,)).
Prior methods for concept extraction, namely ACE [27], ICE [28] and CRAFT [29], can be distilled into two fundamental steps:
(i) Concept extraction: A set of images X E Rxdd belonging the same class is sent to the intermediate space giving activations A = h(X) E RRXp, These activations are used to extract set of & CAVs using K-Means [27], PCA (or SVD) [28, 30] NMF [28, 29]. Each CAV is denoted v; and V = .., vK) ) E R forms the dictionary of concepts.
(i) Concept importance scoring: It involves calculating set of ls global scores, which provides an importance measure of each concept v; to the class as a whole. Specifically, it quantifies the influence of each concept v; on the final classifier prediction for the given set of points X. Prominent measures for concept importance include TCAV [26] and the Sobol indices [29].
The two-step process described above is repeated for all classes. In the following subsections, we theoretically demonstrate that the concept extraction step (i) could be recast as dictionary learning problem (see 3.1). It allows us to reformulate and generalize the concept importance step (ii) using attribution methods (see 3.2).
3

---

3.1 Concept Extraction
dictionary learning perspective. The purpose of this section is to redefine all current concept extraction methods problem within the framework of dictionary learning. Given the necessity for clearer formalization and metrics in the field of concept extraction, integrating concept extraction with dictionary learning enables us employ a comprehensive set of metrics and obtain valuable theoretical insights from a well-established and extensively researched domain.
The goal of concept extraction is to find a small set of interpretable CAVs (i.e.. V) that allows us to faithfully interpret the activation A. By preserving linear relationship between V and A, we facilitate the understanding and interpretability of the learned concepts [26, 38]. Therefore, we look for a coefficient matrix E R (also called loading matrix) and a set of CAVs V, so that A ~ UV!. In this approximation of A using the two low-rank matrices (U, V), V represents the concept basis used to reinterpret our samples, and U are the coordinates of the activation in this new basis. Interestingly, such formulation allows recast of the concept extraction problem as an instance of dictionary learning problem [39] in which all known concept- based explainability methods fall:
A i, U. E (ei,...., ex} (K-Means: ACE [27]),
(1)
VTV= =I (PCA: 30D; 2 2 (NMF; [29] & ICE [28])
(2)
(U*, V*) = arg min min
m - UV s.t.
U.V
(3)
U = (b),, ||Ulo S K (Sparse Autoencoder [40]) (4)
with e; the i-th element of the canonical basis, I the identity matrix and ub any neural network. In this context, V is the dictionary and the representation of A with the atoms of V. u, denote the i-th row of U. These methods extract the concept banks V differently, thereby necessitating different interpretations'.
In ACE, the CAVs are defined as the centroids of the clusters found by the K-means algorithm. Specifically, concept vector v; in the matrix V indicates a dense concentration of points associated with the corresponding concept, implying a repeated activation pattern. The main benefit of ACE comes from its reconstruction process, involving projecting activations onto the nearest centroid, which ensures that the representation will lie within the observed distribution (no out-of -uito--distribution instances). However, its limitation lies in its lack of expressivity, as each activation representation is restricted to single concept (1/u|lo 1). As result, it cannot capture compositions of concepts, leading to sub-optimal representations that fail to fully grasp the richness of the underlying data distribution.
On the other hand, the PCA benefits from superior reconstruction performance due its lower constraints, as stated by the Eckart-Young-Aa covariance matrix: they indicate the direction in which the data variance maximal. An inherent limitation is that the PCA will not be able to properly capture stable concepts that do not contribute to the sample variability (e.g. the dog-head concept might not be considered important by the PCA to explain the dog class if it is present across all examples). Neural networks are known to cluster together the points belonging to the same category in the last layer to achieve linear separability ([44, 29... Thus, the orthogonality constraint in the PCA might not be suitable to correctly interpret the manifold of the deep layer induced by points from the same class (it is interesting to note that this limitation can be of interest when studying all classes at once). Also, unlike K-means, which produces strictly positive clusters if all points are positive (c.g., the output of ReLU), PCA has no sign constraint and can undesirably reconstruct nnttof-diisrrbution (OOD) activations, including negative values after ReLU.
theorem. The CAVs are the cigenvector of the
In contrast to K-Means, which induces extremely sparse representations, and PCA, which generates dense representations, the NMF (used in CRAFT and ICE) strikes a harmonious balance as provides moderately sparse representation. This is due to NMF relaxing the constraints imposed by the K-means algorithm (adding an orthogonality constraint on V such that VVT would yield an equivalent solution to K-means clustering [45]). This sparsity facilitates the encoding of
"Concept extractions are typically overcomplete dictionaries, meaning that if the dictionary for each class is combined, & >> p. as noted in [29]. Recently, [29] and more detailed work [41] suggest that overcomplete dictionaries serious candidates to the superposition problem [42].

---

K-Means
PCA
NMF
Sparse Autoencoder
C1
C2
Figure 2: Most important concepts extracted for the studied methods. This qualitative example shows the three most important concepts extracted for the 'rabbit' class using a ResNet50 trained on ImageNet. The crops correspond to those maximizing each concepts i (i.e., z: where (xx); is maximal). As demonstrated in previous works [28, 29, 491, NMF (requiring positive activations) produces particularly interpretable concepts despite poorer reconstruction than PCA and being less sparse than K-Means. Details for the sparse Autoencoder architecture are provided in the appendix.
compositional representations that particularly valuable when image encompasses multiple concepts. Moreover, by allowing only additive linear combinations components with non-negative coefficients, NMF inherently fosters parts-based representation. This distinguishes NMF from PCA, which offers a holistic representation model. Interestingly, the NMF is known to yield representations that are interpretable by humans [28, 29]. Finally, the non-orthogonality of these concepts presents an ad advantage as it accommodates the phenomenon of superposition [38]. wherein neurons within a layer may contribute to multiple distinct concepts simultaneously.
To summarize, we have explored three approaches to concept extraction, each necessitating a unique interpretation of the resulting Concept Activation Vectors (CAVs). Among these methods, NMF (used in CRAFT and ICE) emerges as a promising middle ground between PCA and K-means. Leveraging its capacity capture intricate patterns, along with its ability to facilitate compositional representations and intuitive parts-based interpretations (as demonstrated in Figure 2), NMF stands out compelling choice for extracting meaningful concepts from nigh-ddmensiioal data. These advantages have been underscored by previous human studies, as evidenced by works such as Zhang al.[28] and Fel [29].
Relative & ())
Sparsity (1)
Stability (L)
FID (L)
OOD (1)
Eff R50 Mob
/ R50 / Mob
Eff R50/ Mob
Eff
Mob
Eff R50/ / Mob
PCA
0.60/ 0.54 / 0.73
0.00 0.00/ / 0.0
0.41 /0.8 /044
0.47 / 0.17 / /024
2.44 / 0.36 / 0.16
KMeans
0 0.66/ 0.84
0,95 / 0.95 / 0.95
0.07 0.08 / 0.04
0.46 0.21 / /0.3
1.76 / 0.29 / 0.15
NMF
0.63/ 0.57 0.75
0.68 / 0.44 0.64
0.17 0.16
0.38 0.21 / 0.24
1.98 / 0.29 / 0.15
Table Concept extraction comparison. Eff, R50 and Mob denote Efficien(Ner ResNer50[47], MobileNe:V2[48], The concept extraction methods are applied on the last layer of the networks. Each results is averaged across 10 classes of ImageNet and obtained from a set of 16k images for each class.
Evaluation of concept extraction Following the theoretical discussion of the various concept extraction methods, we conduct an empirical investigation of the previously discussed properties to gain deeper insights into their distinctions and advantages. In our experiment, we apply the PCA, K-Means, and NMF concept extraction methods on the penultimate layer of three state-of-the-art models. We subsequently evaluate the concepts using five different metries (see Table 1). All five metrics are connected with the desired characteristics of dictionary learning method. They include achieving a high-quality reconstruction (Relative 12), sparse encoding of concepts (Sparsity), ensuring the stability of the concept base in relation to A (Stability), performing reconstructions within the intended domain (avoiding OOD), and maintaining the overall distribution during the reconstruction process (FID). All the results come from 10 classes of ImageNet (the one used in Imagenette [50]), and are obtained using n 16k images for each class.
We begin our empirical investigation by using set of standard metrics derived from the dictionary learning literature, namely Relative l2 and Sparsity. Concerning the Relative Ez, PCA achieves the highest score among the three considered methods, confirming the theoretical expectations based on the Eckart- -oung--iou theorem [43], followed by NMF. Concerning the sparsity of the underlying representation compute the proportion of nor-zero elements |u| lo/k. Since
5

---

K K-means inherently has sparsity of 1/k (as induced by equation 1), it naturally performs better in terms of sparsity, followed by NMF.
We deepen our investigation by proposing three additional metrics that offer complementary insights into the extracted concepts. Those metrics are the Stability, the FID, and the OOD score. The Stability (as it can be seen as a loose approximation of algorithmic stability [51]) measures how consistent concepts remain when they are extracted from different subsets of the data. To evaluate Stability, we perform the concept extraction methods N times on K-fold subsets of the data. Then, we map the extracted concepts together using a Hungarian loss function and measure the cosine similarity of the CAVs. If a method is stable, it should yield the same concepts (up to permutation) across each K-fold, where each fold consists of 1000 images. K-Means and NMF demonstrate the highest stability, while PCA appears to be highly unstable, which can be problematic for interpreting the results and may undermine confidence in the extracted concepts.
The last two metrics, FID and OOD, are complementary in that they measure: (i) how faithful the representations extracted are w.r.t the original distribution, and (ii) the ability of the method to generate points lying in the data distribution (non-OOD). Formally, the FID quantifies the 1- Wasserstein distance [52] Wi between the empirical distribution of activation A, denoted pa and the empirical distribution of the reconstructed activation UV denoted Yu- Thus, FID is calculated as FID (Ha M.) On the other hand, the OOD score measures the plausibility of the reconstruction by leveraging Deep-KNN KNN [53]. recent state-of of-the-art OOD metric. More specifically, we use the Deep-KNN score evaluate the deviation of reconstructed point from the closest original point. In summary. good reconstruction method is capable of accurately representing the original distribution (as indicated by FID) while ensuring that the generated points remain within the model's s domain (non-OOD). K-means leads to the best OOD scores because each instance is reconstructed as centroid, resulting in proximity to in-distribution (ID) instances. However, this approach collapses the distribution to a limited set of points, resulting in low FID. On the other hand, PCA may suffer from mapping to negative values, which can adversely affect the OOD score. Nevertheless, PCA is specifically optimized to achieve the best average reconstructions. NMF, with fewer stringent constraints, strikes a balance by providing in-distribution reconstructions at both the sample and population levels.
In conclusion, the results clearly demonstrate NMF as method that strikes balance between the two approaches as NMF demonstrates promising performance across all tested metrics. Henceforth, we will use the NMF to extract concepts without mentioning
The Last Layer as a Promising Direction The various methods examined, namely ACE, ICE, and CRAFT, generally rely deep layer to perform their decomposition without providing quantitative or theoretical justifications for their choice. To explore the validity of this choice, we apply the aforementioned metrics each block's output ResNet50 model. Figure illustrates the metric evolution across different blocks, revealing a trend that favors the last layer for the decomposition. This empirical finding aligns with the practical implementations discussed above.
Sparsity
Relative
Stability
0.6
0.4
1.6
0.3
0.8
0.25
0.0
82
83
84
85
82
83
84
85
82
83
84
85
62
83
84
85
82
83
84
85
Figure 3: Concept extraction metrics across layers. The concept extraction methods are applied on activations probed different blocks of a ResNet50 (B2 to B5). Each point averaged over 10 classes of ImageNet using 16k images for each class. We evaluate 3 concept extraction methods: PCA (---), NMF (-), and KMcans (......).
3.2 Concept importance
In this section, we leverage our framework to unify concept importance scoring using the existing attribution methods. Furthermore, we demonstrate that specifically the case of decomposition in the penultimate layer, it exists optimal methods for importance estimation, namely RISE [32],

---

Integrated Gradients [7], Gradient-Ine Input [6], and Occlusion [13]. We provide theoretical evidence to support the optimality of these methods.
From concept importance to attribution methods The dictionary learning formulation allows us to define the concepts V in such a way that they are optimal to reconstruct the activation, i.e., A UVT Nevertheless, this does not guarantee that those concepts are important for the model's prediction. For example, the "grass" concept might be important to characterize the activations of a neural network when presented with St-Bernard image, but might not be crucial for the network to classify the same image as St Bernard [26, 16, 54]. The notion of concept importance is precisely introduced avoid such confirmation bias and to identify the concepts used to classify among all detected concepts.
We the notion of Concept ATtribution methods (which denote CATs) to assess the concept importance score. The CAT's are a generalization of the attribution methods: while attribution methods assess the sensitivity of the model output change in the pixel space, the concept importance evaluates the sensitivity to a change in the concept space. To compute the CAT' methods, it is R the concept base V and the model prediction y. To do
necessary to link the activation so, we feed the second part of the network (g) with the activation reconstruction (u s a) so that y - -- VV Intuitively, CAT method quantifies how a variation of will impact y. We denote (u)) the i-th coordinate of (puu), ). so that it represents the importance of the i-th concept in the representation u. Equipped with these notations, we can leverage the sensitivity metrics introduced in standard attribution methods to re-define the current measures of concept importance, as well as introduce the new CAT's borrowed from the attribution methods literature:
Vn,g(uVT)
(used in TCAV: ACE, ICE),
(5)
Em. (Vm(g((u O V) m).))
(Sobol: CRAFT),
(6)
(u((um))VT)
(((u)=
(u- u() L (1-))(u=a'')VV
Int. Gradirrsssss,,
(7)
E (Vu.g(u++0)) 0)V"))
Smooth grad.
(8)
8~N(0.10)
The complete derivation of the 7 new CATs is provided in the appendix. In the derivations, V., denotes the gradient with respect to the i-th coordinate of u, while E and V represent the expectation and variance, respectively. In Eq. 6, m a mask of real-valued random variable between 0 and (i.e m ~ L((0 1l")). We note that, when we use the gradient (w.r.t to u.) as an importance score, we end up with the directional derivative used in the TCAV metric [26], which is used by ACE and ICE to assess the importance of concepts. CRAFT leverages the Sobol-Hoeffdie sensitivity analysis), to estimate the concept importance. The Sobol indices measure the contribution of a concept as well as its interaction of any order with any other concepts to the output variance. Intuitively, the numerator of Eq. 6 is the expected variance that would be left if all variables but u, were to be fixed.
decomposition (used in
Evaluation of concept importance methods Our generalization of the concept importance score, using the Concept ATtributions (CATs), allows to observe that current concept-based explainability methods are only leveraging a small subset of concept importance methods. In Appendix A, we provide the complete derivation of 7 new CATs based on the following existing attribution methods, notably: Gradient input [6], Smooth grad [5], Integrated Gradients [7], VarGrad [55], Occlusion [13], HSIC [10] and RISE [32].
With the concept importance scoring now formulated as a generalization of attribution methods, we can borrow the metrics from the attribution domain to evaluate the faithfulness [56, 32, 33] of concept importance methods. In particular, we adapt three distinct metries to evaluate the significance of concept importance scores: the C-Deletion [32], C-Insertion [32], and C- C-pFidelity [33] metrics. In C-Deletion, we gradually remove the concepts (as shown in Figure 4), decreasing order of importance, and we report the network's 's output each time a concept is removed. When concept removed in C-Deletion, the corresponding coordinate in the representation is set to O. The final
C- C-Deletion metrics computed as the area under the curve in Figure 4. For I-nserrton,, this is
7

---

C- D-Deletion
Cinsertion
1.00
1,00
C-Deletion
Cinsertion
G-MuFidelity
0.73
0.75
0.7
0.50
2
0.85
0.25
0.25
0.5
0.00
0.00
0.80
0.6
10
15
20
25
5
10
13
20
25
# of Concept
# of Concegt.
0.4
Saliency
Misiettrribuiioneehhdd Occlusion
SmoothGred
0.2
GradientInput
SobolAttributionnohhhdd
0.85
0.0
VarGrad
IntegratedCradlents
82 84
83
85
87
84
(a)
(b)
Figure 4: (a) C-Deletion, C-Insertion curves. Fidelity curves for C-Deletion depict the model's s score as the most important concepts are removed. The results are averaged across 10 classes of ImageNet using ResNet50 model. (b) C-Deletion, C-Insertion and C-uFidelity across layer. We report the metrics to evaluate CATs for each block (from B2 B5) ResNet50. We evaluate Concept Attribution methods, all represented with different colors (see legend in Figure 4(a). The average trend these cight methods is represented by the black dashed line (---). Lower C-Deletion is better, higher C-Insertion and C-uFidelity is better. Overall, it appears that the estimation of importance becomes more faithful towards the end of the model.
the opposite: start from a representation vector filled with zero, and progressively add more concepts, following an increasing order importance.
For the C-pFidelity, we calculate the correlation between the model's output when concepts are randomly removed and the importance assigned to those specific concepts. The results across layers for a ResNet50 model are depicted in Figure 4b. We observe that decomposition towards the end of the model is preferred across all the metrics. As result, in the next section, we will specifically examine the case of the penultimate layer.
note on the last layer Based on our empirical results, it appears that the last layer preferable for both improved concept extraction and more accurate estimation of importance. Herein, we derive theoretical guarantees about the optimality of concept importance methods in the penultimate layer. Without loss of generality, we assume y E R the logits of the class of interest. In the penultimate layer, the score y is a linear combination of activations: y = W for weight matrix W and bias b. In this particular case, all CAT' have a closed-fome (see appendix B), that allows to derive 2 theorems. The first theorem tackles the CATs optimality for the C-Deletion and Inserrtion methods (demonstration in Appendix D). We observe that the C-Deletion and C-Insertion problems can be represented as weighted matroids. Therefore the greedy algorithms lead to optimal solutions for CAT' and similar theorem could be derived for C-pFidelity.
Theorem 3.1 (Optimal C-Deletion, C-Insertion in the penultimate layer). When decomposing in the penultimate layer, Gradient Input, Integrated Gradients, Occlusion, and Rise yield the optimal solution for the C-Deletion and C-Insertion metrics. More generally, any method yp(u) that satisfies the condition V(i,j) E (l,..., 1:22 (u Ve, (uOe)) = sp(u); sp(u), yields the optimal solution.
Theorem 3.2 (Optimal C-uFidelity in the penultimate layer). When decomposing in the penultimate layer, Gradient Input, Integrated Gradients. Occlusion, and Rise yield the optimal solution for the C- C-uFidelity metric.
Therefore, for all 3 metrics, the concept importance methods based Gradient Input, Integrated Gradient, Occlusion, and Rise are optimal, when used in the penultimate layer.
In summary, our investigation of concept extraction methods from the perspective of dictionary learning demonstrates that the NMF approach, specifically when extracting concepts from the penultimate layer, presents the most appealing trade-off compared to PCA and K-Means methods. In addition, our formalization of concept importance using attribution methods provided us with theoretical guarantee for different CATs. Henceforth, we will then consider the following setup:
8

---

NMF on the penultimate layer to extract the concepts, combined with a concept importance method based on Integrated Gradient.
3.3
Unveiling main strategies
So far, the concept-based explainability methods have mainly focused on evaluating the global importance of concepts, i.e., the importance of concepts for entire class [26, 29]. This point can be limiting when studying misclassified data points, as we can speculate that the most important concepts for a given class might not hold for individual sample (local importance). Fortunately, our formulation of concept importance using attribution methods gives us access to importance scores at the level of individual samples (i.e., (p(u)). Here, we show how to use these local importance scores to efficiently cluster data points based on the strategy used for their classification.
Reliability
1.0
0.8
0.6
Global Importance
0.2
0.0
Prevalence
600
The local (or image- based) importance of concepts can be integrated into global measures of importance for the entire class with the notion of prevalence and reliability (see Figure 5). A concept is said to be prevalent the class level when it appears very frequently. A preva- lence score is computed based on the number of times a concept is identified as the most important one, i.e., arg max ((u)). At the same time, concept said be reliable if it is very likely to trigger a correct prediction. The reliability is quantified using the mean classification accuracy on samples sharing the same most important concept.
200
Figure 5: From global (class-based) to lo- cal (image-based) importance. Global importance can be decomposed into re- liability and prevalence scores. Preva- lence quantifies how frequently concept is encountered, and reliability indicates how diagnostic a concept is for the class. The bar-charts are computed for the class "Espresso" on a ResNet50 (see Figure 1, left panel)
Strategic cluster graph. In the strategic cluster graph (Figure 1 and Figure 6). we combine the notions of concept prevalence and reliability to reveal the main strategies of model for given category, more precisely, we reveal their repartition across the different samples of the class. We use a dimensionality reduction technique (UMAP [57]) to arrange the data points based the concept importance vector (p))) of each sample. Data points are colored according to the associated concept with the highest importance arg max (rt)). Interestingly, one see in Figure 1 and Figure 6 that spatially close points represent samples classified using similar strategies as they exhibit similar concept importance and not necessarily similar embeddings. For example, for the "lemon" object category (Figure 6). the texture of the lemon peel is the most prevalent concept, as it appears to be the dominant concept 90% of the samples (see the green cluster in Figure 6). We also observe that the concept "pile of round, yellow objects" is not reliable for the network to properly classify a lemon as it results mean classification accuracy of 40% only (see top-left graph in Figure 6).
In Figure 6 (right panel), we have exploited the strategic cluster graph to understand the classification strategies leading to bad classifications. For example, orange (1*t image, 1st row) was classified as a lemon because of the peel texture they both share. Similarly, a cathedral roof was classified as a lemon because of the wedge-shaped structure of the structure t4th image, 1e row).
Discussion
This article introduced a theoretical framework that unifies all modern concept-based explainability methods. Breaking down and formalizing the two essential steps in these methods, concept extraction and concept importance scoring, allowed us to better understand the underlying principles driving concept-based based explainability. We leveraged this unified framework to propose new evaluation metrics for assessing the quality of extracted concepts. Through experimental and theoretical analyses, we justified the standard use of the last layer of an ANN for concept-basee explanation. Finally, we harnessed the parallel between concept importance and attribution methods to gain insights into global concept importance (at the class level) by examining local concept importance (for individual samples). We proposed the strategic cluster graph, which provides insights into the strategy used by an ANN to classify images. We have provided an example of this approach to better understand

---

Reliability
Understand False Positive
0.0
Halluchating lemon because
Halusinaling lemon because wedge
Lemon strategic
clustering graph
Haluccattin of lemons from
Hallocinatngg lemon of girreen objeett henging
Figure 6: Strategic cluster graph for the lemon category. Left: U-MAP of lemon samples, in concept space. Each concept represented with its own color and is exemplified with example belonging to the cluster. The concepts are the lemon wedge shape, pile of round, yellow objects, green objects hanging on a tree, and finally the peel texture, which is the predominant strategy. The reliability of each concept is shown in the top-left bar-chart. Right: Example of images predicted as lemon along with their corresponding explanations. These misclassified images are recognized as lemons through the implementation of strategies that captured by proposed strategic cluster graph.
the failure cases of system. Overall, our work demonstrates the potential benefits of the dictionary learning framework for automatic concept extraction and we hope this work will pave the way for further advancements in the field of XAI.
Acknowledgements
This work was conducted as part of the DEEL project". Funding was provided by ONR (N00014-19 19- 1-2029), NSF (IIS-19) LIS-1912280 and 112743), and the ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-P134. 0004). Additional support provided by the Cameey Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant S10OD025181.
DARPA (D19AC00015), NIH/NINDS (R21 NS
'htps://www.dell.a/
10

---

References
[1] Paolo Tripicchio and Salvatore D' Avella. Is deep learning ready to satisfy industry needs? Procedia Manufacturing. 51:1192-1199, 2020.
[2] Finale Doshi- Velez and Been Kim. Towards a rigorous science of interpretable machine learning, ArXiv e-print, 2017.
[3] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 624 635, 2021.
[4] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop, Proceedings of the International Conference on Learning Representations (ICLR), 2013.
[5] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas. and Martin Wattenberg. Smoothgrad: removing noise by adding noise. In Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML), 2017.
[6] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje, Learning important features through propagat- ing activation differences. In Proceedings of the International Conference Machine Learning (ICML). 2017.
[7] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference Machine Learning (ICML.) 2017.
[8] Ramprasaath R. Sclvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2017.
[9] Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol- sensitivity analysis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[10] Paul Novello, Thomas Fel, and David Vigouroux. Making sense of dependence: Efficient black-box explanations using dependence measure. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[11] Thomas Fel, Melanie Ducoffe, David Vigouroux, Remi Cadene, Mikael Capelle, Claire Nicodeme, and Thomas Serre. Don't lie me! robust and efficient explainability with verified perturbation analysis. Workshop on Formal Verification of Machine Learning, Proceedings of the International Conference on Machine Learning (ICML), 2022.
[12] Mara Graziani, Iam Palatnik de Sousa, Marley MBR Vellasco, Eduardo Costa da Silva, Henning Müller, and Vincent Andrearczyk. Sharpening local interpretable model-agnostic explanations for histopathol- ogy: improved understandability and reliability. In Medical Image Computing and Computer Assisted Intervention (MICCAT). Springer, 2021.
[13] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2014.
[14] Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[15] Marouane 1I Idrissi, Nicolas Bousquet, Fabrice Gamboa, Bertrand Tooss, and Jean-Michel Loubes. On the coalitional decomposition of parameters interest, 2023.
[16] Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (NIPS), 2018
[17] Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[18] Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[19] Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards better understanding attribution methods. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

[20] Peter Hase and Mohit Bansal, Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
[21] Hua Shen and Ting-Hao Huang. How useful the maahine-generatie interpretations to general users? human evaluation guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 168-172, 2020.
[22] Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[23] Sunnie S. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2022.
[24] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic cvaluation scores. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[25] Leon Sixt, Martin Schuessler, Oana-Tuliana Popescu, Philipp WeiB, and Tim Landgraf. Do users benefit from interpretable vision? a user study, baseline, and dataset. Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[26] Been Kim, Martin Wattenberg. Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. In- terpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning. Proceedings the International Conference on Machine Learning (ICML). 2018
[27] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, pages 9273-9282, 2019.
[28] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Fhinger, and Benjamin IP Rubinstein. Invertible concept- based explanations for enn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35. pages 11682-11690, 2021.
[29] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[30] Mara Graziani, An-phi Nguyen, Laura O' Mahony, Henning Müller, and Vincent Andrearezyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML, 2023.
[31] James Genone and Tania Lombrozo. Concept possession, experimental semantics, and hybrid theories of reference. Philosophical Psychology, 25(5):717-742, 2012.
[32] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
[33] Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature- model explanations. In Proceedings of the International Joint Conference on Artificial Intelligence (ICAAI), 2020.
[34] Johannes Haug, Stefan Zürn, Peter El-Jiz, and Gjergji Kasneci. On baselines for local feature attributions. arXiv preprint 20Xiv:2101.0000 2021.
[35] Cheng Yu Hsich, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsich. Evaluations and methods for explanation through robustness analysis. In Proceddiggs of the International Conference Learning Representations (ICT.), 2021.
[36] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. 2019.
[37] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020.
[38] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dode Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint 220i 10652, 2022.
12

---

[39] Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Founda- tions and Trends in Computer Graphicx and Vision. 8(2-3)885-283, 2014.
[40] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. Proceedings of the International Conference on Learning Representations (ICTR), 2014.
[41] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby. Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden Mcl.ean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah, Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, http:/sssooeerrirrriuiiuuupaaa2e
[42] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hadfield-Dadds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuity Thread, 2022.
[43] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 13):211-218, 1936.
[44] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):2465-24466) 2020.
[45] Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM international conference on data mining, pages 606-610. SIAM, 2005.
[46] Huan Zhang, Tsui-Wei Weng, Pun-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. Advances Neural Information Processing Systems (NeurIPS), 2018.
[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[48] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching-Chich& Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IELE conference computer vision and pattern recognition, pages 4510-4520, 2018.
[49] Jayneel Parckh, Sanjeel Parckh, Pavlo Mozharovskyi, Florence d' Alché Buc, and Gaël Richard. Listen to interpret: Post-hoc interpretability for audio networks with nmf. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[50] Jeremy Howard. Imagenette dataset. URL http://ttt....ceeaaaaa/////att/e
[51] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2002.
[52] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer.
[53] Yiyou Sun, Yifei Ming. Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Leaming, pages 20827-20840. PMLR, 2022.
[54] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2017.
[55] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[56] Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? Proceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL), 2020.
[57] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint /802. 2018
[58] Thomas Fel, Lucas Hervier. David Vigouroux, Antonin Poche, Justin Plakoo. Remi Cadene, Mathieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop on Explainable Artificial Intelligence for Computer Vision (CVPR), 2022.
13

---

[59] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. In Proceedings of the International Conference Learning Reprexentations (ICLR), 2018.
[60] Matthew Sotoudeh and Aditya V. Thakur. Computing linear restrictions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[61] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of IEEE European Conference Computer Vision (ECCV), 2014.
[62] Hassler Whitney. On the abstract properties of linear dependence. Hassler Whitney Collected Papers. pages 147-171, 1992.
[63] Sajad Fathi Hafshejani and Zahra Moaberfard. Initialization for non-negative matrix factorization: comprehensive review. International Journal of Data Science and Analytics, 2023.
[64] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[65] Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithmms and applications. Springer, 2018.
14


Reviews:
[{"review_id": "JgL3nCto2q", "reviewer": "Reviewer_xA3z", "summary": "The paper provides a unifying perspective for the concept extraction / importance estimation method via the dictionary learning framework. Namely, depending on the constraints on $U$ and $V$, the framework includes ACE, PCA, CRAFT and ICE as special cases. Then, the paper compares three difference approaches, PCA / NMF / K-Means, for the dictionary learning framework using several metrics. Moreover, by generalizing the attribution methods in XAI, the concept importance values are also computed and compared across different attribution methods for the NMF concept extraction. The theory for the concepts extracted from the penultimate layer is also given. Finally, by defining reliability and prevalence of the concepts, the visual and qualitative results are shown, which are compelling. ", "questions": "- The strategic graph is obtained by the Integrated Gradient attribution method -- how would be different across different attribution method? \n- How would the strategic graph change for different test sets? \n", "limitations": "Please see the comment for weakness. \n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Unifying effort for the concept extraction / importance is sound and the theoretical results back the approach of the paper. \n- Combining the attribution method with concept extraction is interesting and the reliability / prevalence notions make sense. \n- The visualization results are compelling. \n", "weaknesses": "- I think it would be beneficial to show the failure case of the proposed method. Figure 1/2 show compelling results for three classes (Espresso, Zucchini, and Lemon) but would there be classes that the proposed method fails to give meaningful explanation? What would those classes be like? It would be informative to show such examples as well so that people who use the proposed technique could be cautious. \n- It seems like the local importance score depends on the entire test set -- namely depending on what other images exist in the test set, the reliability and prevalence should vary. What are the variance of those values depending on the test set? Maybe cross-validation like evaluation should be meaningful. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and the suggestions in the reviews. \n\nPlease find below a point-by-point answer:\n* **Reviewer : 'I think it would be beneficial to show the failure case of the proposed method. Figure 1/2 show compelling results for three classes (Espresso, Zucchini, and Lemon) but would there be classes that the proposed method fails to give meaningful explanation? What would those classes be like? It would be informative to show such examples as well so that people who use the proposed technique could be cautious.'**\nTo be entirely honest, we have not encountered any cases where the method \"does not work.\" As evidence, we have included  a website containing the 1000 classes of ImageNet (https://anon1285.github.io/neurips1285/)\nHowever, you are correct, and as requested by several reviewers, we will create a dedicated section on limitations of the framework.  \n\n\n* **Reviewer : 'It seems like the local importance score depends on the entire test set -- namely depending on what other images exist in the test set, the reliability and prevalence should vary. What are the variance of those values depending on the test set? Maybe cross-validation like evaluation should be meaningful. How would the strategic graph change for different test sets?'**  \nThe local importance score does not vary with the distribution as it is computed, for one single image, with the concept attribution score. However, you are correct for  the reliability and prevalence scores that depend on the aggregation of a set of points. Since the local importance for a point remains constant, the prevalence is exclusively influenced by the stability of the concept extraction method (i.e. its ability to give the same sets of concepts whatever the testing distribution), which we have measured in Figure 2, plot 5.  The prevalence we have computed is based on the NMF concept extraction method, which exhibits a good stability. Therefore, the prevalence is only slightly impacted by the distribution of the test images.  \n\n\n* **Reviewer : 'The strategic graph is obtained by the Integrated Gradient attribution method -- how would be different across different attribution method?'**\nThe strategic graph is computed using the last-layer representation, and we have demonstrated (in section “A note on the last layer”) that several attribution methods boil down to the same form during concept extraction on the last layer. Consequently, Integrated Gradient, Gradient Input, Rise, and Occlusion will yield the same strategic graph. Furthermore, we have demonstrated (theorem 3.1 and 3.2)  that these feature importance methods are optimal with respect to the three fidelity metrics. Hence, we exclude the other methods from consideration.  \n"}, {"review_id": "hWEWrHOseX", "reviewer": "Reviewer_Y6po", "summary": "In this paper, the authors propose a unified perspective on post-hoc concept-based explanation methods. The main intuition underlying the work revolves around the fact that typical concept-based explanations can be considered as a two-stage process whereby, initially, a concept vocabulary is learned and, lastly, each concept's importance is evaluated. In light of their theoretical analysis, the authors describe a variegated class of concept-based explanations, showing in particular that the penultimate layer is particularly suited for concept analysis. In addition, the authors derive that some concept-importance methods are theoretically sound in the penultimate layer.\nConcerning the experimental investigation, the authors propose a set of metrics for comparing different explainability methods in both the settings of concept dictionary learning and concept importance estimation. The experiments are specifically tailored to evidence differences between existing methods.\nThe authors also contribute to a dedicated interface for visualizing the clusters graph on the ImageNet classes. ", "questions": "Does the analysis offer guidance toward non-linear cases? Is it possible to extend the current approach to that scenario?\n\n", "limitations": "The interpretability of concepts is not discussed at length; this addition may strengthen the authors' findings and provide guidance to improve the research on concept-based explanations. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In this paper, the theoretical analysis is extensive and well curated and the presentation is clear and easy to follow. The main strength of the paper is introducing a unified framework for analyzing concept-based explanations, which reveals particularly suited to address the pros and cons limitations of existing methods. In particular, the linear decomposition of the activation matrix in the concept vectors covers several existing approaches. This is of particular help also for proceeding with the research in the field. \n\nThe integration of three new metrics for concept extraction helps to compare several strategies, among which NMF decomposition seems preferable. Similarly, the authors propose three metrics to evaluate concept importance methods, deriving theoretical guarantees for 4 methods in the penultimate layer. This strengthens the idea that concept analysis is best when conducted in the deepest layer of neural networks. The experiments are illustrative to provide a discrimination between different methods and complete the theoretical analysis conducted. \n\nI also found helpful the introduction of the strategic cluster graph, which provides an intuitive interface for visualizing concept-level explanations of the model. ", "weaknesses": "I have no major concerns about the presented material, which is very curated and high quality. \n\nNonetheless, I find two weaknesses on the general method proposed. First of all, learning a dictionary of concepts in unsupervised manner does not give any guarantee that the extracted concepts do actually possess a well-defined semantics. This requires a verification with an end user, as for example in [1], and, so, the considered methods for post-hoc concepts analysis are limited and do not ensure interpretability.\n\nAnother weakness amounts to extracting the concepts in a linear manner and it seems that classes are actually related to concepts in a kind of AND rule. This does not hold in general, where classes might depend on non-linear relations between concepts. As an example, take a class resulting from a XOR operation between two concepts, there is no way to extract this information with these methods.  \n\nI hope to see some discussion on these points.\n\n[1] Biswas, Shreyan, et al. \"CHIME: Causal Human-in-the-Loop Model Explanations.\" Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 10. No. 1. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very nice review. Once again, we extend our appreciation for the time and the very comprehensive review.\n\nPlease find below a point-by-point answer:\n\n* **Reviewer : 'Learning a dictionary of concepts in an unsupervised manner does not give any guarantee that the extracted concepts do actually possess a well-defined semantics. This requires a verification with an end user, as for example in [1], and, so, the considered methods for post-hoc concepts analysis are limited and do not ensure interpretability'.** We wish to highlight the fact that the dictionary learning technique is applied on images belonging to the same class (which might be interpreted as a ‘weak’ form or supervision). Therefore we have the guarantee that the learned concepts are well describing a given class. Nonetheless, we agree with the reviewer that there is no guarantee that the learned concepts are semantically meaningful. However, [20] has demonstrated that, in practice, the extracted concepts tend to align with human judgment (especially when extracted using an NMF).\nWe have expanded a section on limitations to address this point.\n\n* **Reviewer : 'Another weakness amounts to extracting the concepts in a linear manner and it seems that classes are actually related to concepts in a kind of AND rule. This does not hold in general, where classes might depend on non-linear relations between concepts. As an example, take a class resulting from a XOR operation between two concepts, there is no way to extract this information with these methods.'**\nThe main advantage of imposing a linear decomposition is that the resulting concepts are more easily interpretable. We refer the reviewer to [18] for an interesting discussion about linear interpretability. However we agree with the reviewer that such a constraint limits the expressivity of the learned concepts. We have expanded the limitation section to better highlight this point.\n\n* **Reviewer : 'Does the analysis offer guidance toward non-linear cases? Is it possible to extend the current approach to that scenario?**\nIndeed, the limitation you mentioned has been highlighted several times. As for the proof regarding C-Deletion and C-Insertion, we could potentially extend it using matroid theory to cover cases of submodular functions. However, it seems challenging to prove anything of interest in a non-linear case without additional assumptions.'\n"}, {"review_id": "BRNG6mfbM5", "reviewer": "Reviewer_d9UA", "summary": "This paper presents a novel framework for unifying existing post-hoc concept-based explainability methods in a cohesive, formalized form. Such a framework enables a principled study of key properties of state-of-the-art concept-based explainability methods, such as ACE, T-CAV, CRAFT, and ICE, and allows the authors to easily compare and contrast the quality of explanations generated by all of these approaches. To this end, this paper further introduces a large set of novel evaluation metrics for the quality of concept explanations together with a new simple framework to generate and visualize local concept-based explanations for a given prediction. These metrics are used to evaluate a variety of architectures in ImageNet and conclude that NMF-based concept-extraction approaches have significant potential compared to their clustering and PCA-based counterparts. Finally, the paper provides theoretical guarantees regarding the optimality of a set of concept-based explainability concept attribution methods, with respect to the fidelity and completeness of their explanations, when they are used to explain the latent space of the last layer of a DNN.", "questions": "Regarding the weaknesses discussed above, I will gladly consider increasing my score if some of these concerns are appropriately addressed by the authors (especially if they come from a misunderstanding on my end). To help with this discussion, I believe the following questions could be a good starting point:\n\n1. Could you please elaborate on how CCD fits within your framework and whether it would be feasible to include a topic-modelling-based concept extractor as part of the comparison in section 3.1? The lack of discussion of this method, a key algorithm in unsupervised concept learning, weakens the universal unification argument in this paper.\n2. One of the main claims of the paper is that the framework unifies “all modern concept-based explainability methods”. Yet, I fail to see how it can explicitly describe T-CAV in the form originally proposed by their authors. Could you please clarify how T-CAV , as defined in its original paper, can be explicitly framed in the proposed framework? I can see how it can be framed if one assumes that the latent space from which we extract the CAVs from can be explicitly reconstructed from all the concepts we learn (and is done so as part of inference). This, however, is in no way an assumption of the original T-CAV process and, in fact, it is a very strong constraint that in practice fails to apply. This is because we will most certainly not have all the necessary concepts for perfect reconstruction for most tasks.\n3. Regarding how ACE is framed in the proposed framework, I am still a bit doubtful that it is such a straight fit as claimed by this work. If I recall correctly, ACE uses clustering just to learn subsets of concept examples from which CAVs can later be learnt using linear separators. This paper, however, seems to suggest that ACE uses the cluster centroids themselves as the CAVs. If my understanding of that claim is correct, then I believe that’s wrong. Could you please clarify this point? If my understanding is correct, then this would require correctly updating a text where it claims that ACE falls within the clustering concept extractor family of methods.\n\nRegarding suggestions for the presentation, the paper is very well-written (thank you so much!). Nevertheless, the following possible errors and typos could be addressed before a camera-ready version:\n\n- **[important]** The appendices referenced in the main body of the paper seem not to be correctly aligned with their correct sections in the submitted supplementary material.\n- The use of bold symbols is a bit confusing. Sometimes it is used for tensors/vectors (e.g., $X$ and $x$ in line 93), sometimes it is used for functions even when their output is not a vector (e.g., $g$ in line 98), and sometimes it is used to refer to a scalar (e.g., line 509 in the appendix). Traditionally it is just used for vectors/tensors and not functions names or scalars. It is ok if this does not follow tradition, but it could benefit from consistency (as, say, some scalars are not bolded as the bias term $b$ sometimes).\n- Typo in line 112: it should probably be “This allows us…” rather than “It allows us…” as the “it” is unclear in that context.\n- In line 177 it should probably be “the best score” rather than “the highest score”  as PCA achieves the best score amongst all methods (which is the lowest, not the highest for relative $\\ell_2$.\n- The use of mix cases in “The Last Layer as a Promising Direction” in line 212 is inconsistent with the rest of the subtitles.\n- In line 214, a full stop “.” is missing before “To explore”.\n- The caption of Table 1 should be on top of the table to follow NeurIPS standards.\n- Line 220-221 read a bit off. Maybe a typo or a word is missing there.\n- Figure 3’s x-axis should be “# of Concepts” rather than “# of Concept”\n- In that same figure, it would be beneficial to use different colors or markers as it is otherwise hard to disentangle some of the methods (and then even harder to find the respective color in Figure 4 without the legend right there).\n- CAT is redefined in line 265.\n- In line 279 “zero” should be “zeros”.\n- Line 296 is missing “are” between $y \\in \\mathbb{R}$ and “the logits…”\n- In line 339, “this” should be “these”\n- nit: It is uncommon to have more than one paragraph in the abstract. Just pointing this out in case this comes from an accidental new line inserted in the abstract.", "limitations": "This work does not discuss any limitations of the proposed framework, metrics, or results. It could certainly benefit from a short section where it is discussed how this framework may fail to generalize to some existing concept-based methods and how it could possibly be expanded in the future. Similarly, the manuscript does not discuss possible societal impacts of the work. However, I believe no clear negative societal impacts could arise from this paper's contributions and methods.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thank you for this very interesting and amazingly-crafted work; I thoroughly enjoyed reading it. I believe this paper’s goal of introducing a unifying formal framework to study most concept-based XAI methods is very important in a field lacking principled and theoretical analysis. Overall, I believe that this paper’s main strengths are:\n\n1. This paper’s core motivation, the framework it introduces, and the metrics it proposes are all worthy of study and of interest to the wider XAI community. These can become particularly handy for other researchers to evaluate, compare, and study novel methods against existing ones.\n2. Similarly, the key insights developed by this paper, in particular the optimality theorems and their surprising results on NMF-based methods, are all novel and potential sources for interesting future research.\n3. The paper is extremely well-written, with the writing being very clear and the figures nicely complementing the rest of the text. In particular, I like the non-standard approach to writing this work (not strictly following the now-boring intro-methods-results-discussion pipeline) and think it is worth pointing out.\n4. There is a very good mix of theoretical results and interesting insights from very simple experiments. In particular, I found the use of matroids for the proofs of the proposed theorems to be very interesting.\n5. The introduction of a framework that enables both global and local concept-based explanations with one method is novel and particularly useful in practice.", "weaknesses": "Overall, I think this paper introduces a very useful framework and set of metrics that may be of real interest to the wider community. I am leaning towards acceptance for this reason. However, I have some concerns/questions regarding some specific bits of the paper as well as some possible gaps in this work.  Specifically, I believe the following are that paper's main limitations:\n\n1. Some key and more recent methods are not mentioned or cited in this work. These include Concept Gradients [1] (which I am not sure if it directly fits in the proposed framework, although this would be a good test for the proposed ideas) and, more importantly, as this is extremely related to how the reconstruction of the latent space is framed in this work, completeness-aware concept discovery (CCD) [2]. Similarly, traditional topic models, such as latent Dirichlet allocation (LDA), are highly related to some of the ideas discussed in the concept extraction part of this framework.\n2. I am still uncertain whether ACE and T-CAV, as originally proposed in their original papers, can be expressed using this paper’s framework. If so, this weakens this paper's unification claim, a key contribution according to the authors. This may be a misunderstanding on my end, but please take a look at my questions below to possibly clarify this.\n3. Against general convention, none of the figures and tables include error bars, making it harder to understand when differences are statistically significant.\n4. The paper is missing key details regarding their empirical evaluation. These include model sources, licenses, and references showing the training setups for the original pre-trained models if models were not trained from scratch.\n\n### References\n\n[1] Bai, Andrew, et al. \"Concept Gradient: Concept-based Interpretation Without Linear Assumption.\" *The Eleventh International Conference on Learning Representations*. 2022.\n\n[2] Yeh, Chih-Kuan, et al. \"On completeness-aware concept-based explanations in deep neural networks.\" *Advances in neural information processing systems* 33 (2020): 20554-20565.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to learn that the reviewer found the paper enjoyable and useful for the community. We extend our sincere gratitude to the reviewer for investing time and effort in providing a thorough review!\n\nPlease find below a point-by-point answer:\n\n* **Reviewer: Some key and more recent methods are not mentioned or cited in this work. These include Concept Gradients [1] (which I am not sure if it directly fits in the proposed framework, although this would be a good test for the proposed ideas) and, more importantly, as this is extremely related to how the reconstruction of the latent space is framed in this work, completeness-aware concept discovery (CCD) [2]. Similarly, traditional topic models, such as latent Dirichlet allocation (LDA), are highly related to some of the ideas discussed in the concept extraction part of this framework.** The authors were not aware of the CCD work, and we appreciate your observation. As you rightly pointed out, we have now added a discussion of CCD in the appropriate section. Regarding LDA, we acknowledge its potential as a good solution (also noted by reviewer  hQqh), and we believe it could serve as a promising starting point for future research on topic-modelling-based concept extraction.\n\n* **Reviewer: I am still uncertain whether ACE and T-CAV, as originally proposed in their original papers, can be expressed using this paper’s framework. If so, this weakens this paper's unification claim, a key contribution according to the authors. This may be a misunderstanding on my end, but please take a look at my questions below to possibly clarify this. One of the main claims of the paper is that the framework unifies “all modern concept-based explainability methods”. Yet, I fail to see how it can explicitly describe T-CAV in the form originally proposed by their authors. Could you please clarify how T-CAV , as defined in its original paper, can be explicitly framed in the proposed framework? I can see how it can be framed if one assumes that the latent space from which we extract the CAVs from can be explicitly reconstructed from all the concepts we learn (and is done so as part of inference). This, however, is in no way an assumption of the original T-CAV process and, in fact, it is a very strong constraint that in practice fails to apply. This is because we will most certainly not have all the necessary concepts for perfect reconstruction for most tasks.** T-CAV (from [18]),  is a sensitivity score that quantifies how important a concept is (a CAV) to a classification score. In the original paper, it is computed using a gradient projected on the CAV (i.e. the directional directive). We have shown in section 3.2., that the T-CAV sensitivity score is equivalent to the gradient of the activation reconstruction (i.e $g(uV^{T})$) w.r.t to the representation (i.e. $u$). Said differently, the T-CAV score is a particular CAT (concept attribution method) that leverages the gradient method. Note that in the original T-CAV article [18], the CAVs are extracted using human judgment (it is not an automatic concept extraction method), therefore we could not fit this non-automated concept extraction method in our framework. Concerning ACE, both the attribution and the concept extraction fits in our framework. The attribution method of ACE is a T-CAV (i.e. the gradient Concept ATtribution) and its concept extraction method is based on clustering. Note that the centroids themselves are not the CAVs in ACE, but the training of CAVs is based on the distances to the centroids. This strongly suggests that the vector orthogonal to the decision boundary points towards or is close to the centroid. However, we agree with you and have added this clarification to the paper. Thank you for bringing up this important detail, and we appreciate your input.\n\n* **Reviewer: The paper is missing key details regarding their empirical evaluation. These include model sources, licenses, and references showing the training setups for the original pre-trained models if models were not trained from scratch.** \nWe have included these details in a dedicated section in the appendix. The models used are pre-trained models on ImageNet from the Keras library, and we have provided the configurations of these models.\n\n* **Reviewer: This work does not discuss any limitations of the proposed framework, metrics, or results. It could certainly benefit from a short section where it is discussed how this framework may fail to generalize to some existing concept-based methods and how it could possibly be expanded in the future. Similarly, the manuscript does not discuss possible societal impacts of the work. However, I believe no clear negative societal impacts could arise from this paper's contributions and methods.** Another reviewer requested a limitation section, and as per their suggestion, we have included it in the appendix.\n\n* **Concerning the minor error and the typos** We appreciate the corrections regarding the typos, and we have taken them into account and made the necessary adjustments accordingly. As a result, the manuscript has been improved.\n"}, {"review_id": "W6UcvCzPxg", "reviewer": "Reviewer_hQqh", "summary": "This paper proposes a unifying framework to study common concept-based explainability methods. In particular, the authors observe that all the methods boil down to dictionary learning in the latent spaces of a neural network. Based on this observation, they (1) compare the different concept extraction/discovery methods and (2) extend established feature attribution methods to concept-based explanations. The authors investigation leads them to conclude that the penultimate layer is the best one where to extract concepts proving the optimality of concept-adapted metrics for attribution.", "questions": "1. As mentioned above, using the authors framework to suggest new ways to do concept extraction would particularly strengthen the paper. Could the authors comment, at least as future work, how they could leverage their framework to evaluate possible new concept extraction methods?\n2. (L. 128) Are the authors sure about the expressivity claim of K-means based methods? wouldn't expressity depend on the number of cluster K? \n3. (L. 135) This seems a drawback of doing class-based concept extraction, rather than a problem specific to PCA. For example, how would K-means be able to detect the dog-head concept, if all the samples in the dog class have the dog-head concept? \n4. The authors claim that NMF strikes an harmonious balance between K-Means and PCA. This somewhat should imply that NMF is a better choice. Could the authors elaborate on this? This would help strengthen the utility of their framework. The only argument is the claim that (L. 153) NMF is known to yield human-interpretable concepts. Is this not the case for PCA or K-Means? In the referenced papers, I cannot see a study on the utility of NMF concepts compared to PCA concepts.\n5. As an extension to the above, it would be interesting to leverage the authors framework to more systematically pinpoint the properties of a certain dictionary learning method that make the extracted concepts more or less interpretable from a user perspective. The authors can perhaps expand on this in a future work section.\n6. Evaluation of concept extraction: an additional evaluation could be performed by applying concept extraction methods to a model with ground truth concepts, e.g. a concept bottleneck model which was specifically trained with user-defined concepts as additional training signal.\n7. As mentioned above, the results (especially the theorems) are unsurprising/trivial. but I will concede that I might be wrong, if the authors could further elaborate on the differences between their results and Ancona's. As observed by the authors, the last classification layer is by construction linear wrt the penultimate layer. Since the dictionary learning view essentially reveals that the concepts extracted at the penultimate layer are essentially a linear projection on a different base of the penultimate layer latent space. Therefore, it is a trivial extension of Ancona et al.'s work, the fact that all attribution methods are essentially equivalent on linear classifiers, and furthermore, achieve optimal fidelity, i.e. correlation of 1, which again is unsurprising for linearly correlated variables. Consequently, it seems also trivial to me that C-deletion and C-insertion also achieve optimal values since insertion and deletion greedy strategies are trivially optimal for positively monotonically correlated variables. In essence, I would say that Theorem 3.2 is an extension of Ancona's work, while Theorem 3.1 is essentially a corollary  to 3.2.  \n8. As additional comment to the point above, it seems also unfair to evaluate different layers using these definitions of deletion and insertions since they would correctly evaluate only layers that are linearly related to the network outputs.", "limitations": "Authors do not discuss or comment about possible limitations of their framework and possible future work. In particular, can their framework be used to derive new concept attribution scores and relative evaluation metrics, in case somebody might be constrained to not use the penultimate layer?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- (Clarity) Paper is in general easy to follow and fairly clear at discussing the core concepts.\n- (Novelty) I am personally a big fan of unifying frameworks, so this paper seems like a reasonable steps towards understanding concept-based explainability methods\n- (Significance) The strategic cluster graph is quite compelling from an user-experience perspective \n- (Significance) While the results don't seem particularly surprising to me (more below), it is nice to see an analysis of concept-based methods laid out.\n- (Quality) Mathematical derivations seem sound.", "weaknesses": "1) *Novelty/Significance* \n  - *Theoretical results*: As mentioned above, I do not find the theoretical results particularly surprising. More details are reported below, but, in a nutshell, the results seem to me like a trivial extension of the work from Ancona et al. [1].\n  - *Strategic cluster graph*: I would have preferred to see a more in-depth/quantitative analysis of this \"tool\". The concept analysis of the graph is fairly hand-wavy and arbitrary, e.g. in the naming of the concepts (apart for one of the wrongly classified samples).\n\nAs a more actionable feedback, perhaps I would comment that personally I would have preferred to have a less extensive theoretical analysis (since I find it unsurprising), and a more detailed use case for the strategic cluster graph, maybe with an user study. Or alternatively the authors could have leveraged their framework to propose a new method for concept extraction inspired by other dictionary learning method, perhaps something probabilistic such as the latent dirichlet allocation (although on the top of my mind, it may not be very trivial).\n\n2) *Clarity*: some definitions are left unreferenced. For example (L. 273), there is no reference to the original feature-based metrics, so it might appear unclear how the C-metrics are an extension of those ones.\n\n3) More detailed feedback can be found below, in the \"Questions\" section.  \n\n[1] Ancona, Marco, et al. \"Towards better understanding of gradient-based attribution methods for deep neural networks.\" arXiv preprint arXiv:1711.06104 (2017).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading and providing a constructive review with relevant remarks and suggestions that will enhance the quality of the paper.\n\nPlease find below a point-by-point answer:\n* **About the lack of analysis of the Strategic Clustering graph:** The main goal of this article is to provide a theoretical framework to unify all concept-based methods. The strategic cluster graph was just proposed at the end as a use-case example and is secondary compared to the theoretical framework (which explains indeed the limited materials concerning those graphs). Further work, specifically focusing on strategic cluster graphs is in preparation.\n\n* **About the fact that the authors do not propose new concept extraction methods:** In this article, we have laid down the theoretical foundation, and proposed a series of benchmarks so that it becomes easier to propose and evaluate new concept extraction and attribution methods. We believe that this contribution is novel and important enough to deserve an entire article. Note that this opinion is also shared and supported by all other reviewers. Nonetheless, we agree with you that a natural follow-up of this article would be to investigate new methods (especially concept extraction methods). We actually plan to extend this article into a journal version where we aim to further investigate the topic by exploring probabilistic approaches such as Gaussian mixture and LDA. We appreciate this suggestion, as it aligns with our inclination to continue in this direction. In the meantime, we agree with you and have added a future work section that details these plans. Thank you for your valuable feedback and contribution to the improvement of the paper.\n\n* **About the unreferenced definition (L. 273)...** We have corrected these references. Thank you.\n\n* **Reviewer: About the expressivity claim of the K-Means (L. 128)**. The term expressivity was indeed poorly chosen, and we have now modified it. The intention was to highlight that for the reconstruction, a point could only belong to one cluster, which led to the inappropriate use of the term \"lack of expressiveness.\" We have replaced this sentence by referring to the absence of compositionality to better convey the concept. Thank you for pointing out this improvement, and we appreciate your valuable feedback.\n\n* **Reviewer: (L. 135) This seems a drawback of doing class-based concept extraction, rather than a problem specific to PCA. For example, how would K-means be able to detect the dog-head concept, if all the samples in the dog class have the dog-head concept?** We respectfully disagree. The goal of a PCA is to find the directions that maximize the variance, therefore it will tend to omit features that are common to all samples (as the dog-head features for samples representing dogs). On the other hand K-means (on the features) aims at finding centroids that minimize the distance to features’ sample. Therefore, if the dog-head is a prevalent feature, it is likely that the K-Means will represent this feature. Interestingly, in the specific case of K-means with only one concept, the average indeed becomes the centroid. However, the issue with PCA arises from the absorption of the mean, which is a well-known fact. Thank you for clarifying this point, and we appreciate your valuable input.\n\n* **About the lack of argument on the utility of the NMF methods.** With all due respect, we believe that there is a misleading interpretation of our work : our arguments in favor of the NMF are bolstered by both theoretical and empirical arguments.  While the reviewer suggests that the sole claim hinges solely on NMF yielding human-interpretable concepts, we want to emphasize that we provide multiple theoretical arguments in Section 3.1 (l135, l139, l143, l147, l150, l154, l155) and empirical evidence in the 'Evaluation of concept extraction' (l178, l181, l189, l205, l206) section. We believe that these supporting elements validate the significance and validity of our conclusion. In addition, we wish to highlight that the referenced paper that studies the NMF utility is actually comparing it to PCA and K-means (see Fig 3 and Table 2 in [20]).\n\n* **About systematically pinpointing the properties of different dictionary learning method.** Thank you for this excellent suggestion. We have indeed added a discussion on this topic in the future work section. \n\n* **Reviewer: The results (especially the theorems) are unsurprising/trivial given de work of Anconda ?** We respectfully disagree. We are well aware of the really interesting  works of Ancona. There is indeed some similarities with the Anconda work, but our propositions (on linear optimality) differ for 2 main reasons : i) Ancona does not analyze Gradient, Smoothgrad, VarGrad, SquareGrad, RISE, etc. and ii)  Ancona does not provide any proof concerning mu-fidelity, C-deletion, or C-insertion, which is why we have introduced our theoretical elements. Note that the Mu-Fidelity metric (our) is a relaxation of the sensitivity-n metric (Ancona), therefore our proposition encompasses the Ancona's one. In addition, our work is also extending the linear optimality with respect to 2 other new metrics (C-Deletion and C-Insertion) that are not related to the Ancona metric. We sincerely hope that our response serves to alleviate any misconceptions and foster a clearer understanding of our intentions. Our aspiration is for the reviewer to consider the points we've presented, potentially leading to a reassessment of the evaluation rating. Your feedback is valuable to us, and we genuinely appreciate your time and consideration."}]

Author rebuttal:

The reviewers found the paper "*extremely well-written*" "*thoroughly enjoyable to read*" and appreciating the "*non-standard approach*" taken. Regarding the theoretical work, you found it “*interesting and extensively curated*”, with reviewers finding the use of matroids in proofs "*very interesting*" and acknowledging the importance of introducing a formal framework for concept-based XAI methods in a field lacking theoretical analysis”. Furthermore, the paper receives positive feedback on its overall quality, with reviewers expressing gratitude for this "*very interesting and amazingly-crafted work*" and having "*no major concerns*" about its curated and high-quality content.

However, we acknowledge that you have raised certain points and identified weaknesses in the paper. During the rebuttal period, we meticulously examined the critiques provided by all four reviewers and have focused on addressing the following two main issues that have been highlighted: 

1. The need for more examples of the analysis of the Strategic Clustering Graph and its potential limitations. We have included an anonymised website that showcases the Strategic Cluster Graph for the 1000 classes of ImageNet (https://anon1285.github.io/neurips1285/). Upon analysis, we did not identify any specific cases of failure.

2. The need to include a future work section, explaining the potential next steps, particularly in terms of applying the framework to discover/test new extraction methods. We acknowledge that this framework can not only be used to study existing methods but also to propose novel ones. We are excited about the possibility of pushing the boundaries using this framework. In this article, we have chosen to focus on establishing the foundation of the framework and have started exploring new extraction methods with it. We have added a dedicated section to discuss future research directions, and we sincerely appreciate your valuable suggestions.

Furthermore, we have diligently incorporated the necessary citations and clarifications into our Related Work and Discussion sections. We will now proceed to address each reviewer's remaining comments directly.


Meta-review: This paper received four positive ratings: 7775. Authors support its contributions to unifying the concept extraction and importance and its value to the community, together with sound theoretical results and analyses. The concerns of reviewers are also well addressed. Overall, it is recommended to be accepted. And, based on the discussions with reviewers, more details and discussions need to be added in the final version.
Decision: Accept (spotlight)