Title: A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Explainable AI

Concept-based explainability

Interpretability

Concept extraction

Concept importance

Attribution methods
Abstract: In recent years, concept-based approaches have emerged as some of the most promising explainability methods to help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual ``concepts'' buried within the complex patterns of ANN activations in two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce a unifying theoretical framework that recast the first step -- concept extraction problem -- as a special case of **dictionary learning**, and we formalize the second step -- concept importance estimation -- as a more general form of **attribution method**.
This framework offers several advantages as it allows us: (i) to propose new evaluation metrics for comparing different concept extraction approaches; (ii) to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-art concept-based approaches and importance estimation techniques; (iii)  to derive theoretical guarantees regarding the optimality of such methods. 

We further leverage our framework to try to tackle a crucial question in explainability: how to *efficiently* identify clusters of data points that are classified based on a similar shared strategy.
To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic cluster graph. Finally, we present Lens, a dedicated website that offers a complete compilation of these visualizations for all classes of the ImageNet dataset.

A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation

Abstract

In recent years, concept-based approaches have emerged as some of the most promising explainability methods to help us interpret the decisions of Artificial Neural Networks (ANNs). These methods seek to discover intelligible visual 'concepts' buried within the complex patterns of ANN activations in two key steps: (1) concept extraction followed by (2) importance estimation. While these two steps are shared across methods, they all differ in their specific implementations. Here, we introduce a unifying theoretical framework that recast the first step concept extraction problem - as a special case of dictionary learning , and we formalize the second step - concept importance estimation - as a more general form of attribution method . This framework offers several advantages as it allows us: (i) to propose new evaluation metrics for comparing different concept extraction approaches; (ii) to leverage modern attribution methods and evaluation metrics to extend and systematically evaluate state-of-the-art concept-based approaches and importance estimation techniques; (iii) to derive theoretical guarantees regarding the optimality of such methods.

We further leverage our framework to try to tackle a crucial question in explainability: how to efficiently identify clusters of data points that are classified based on a similar shared strategy. To illustrate these findings and to highlight the main strategies of a model, we introduce a visual representation called the strategic cluster graph. Finally, we present Lens, a dedicated website that offers a complete compilation of these visualizations for all classes of the ImageNet dataset.

1 Introduction

The black-box nature of Artificial Neural Networks (ANNs) poses a significant hurdle to their deployment in industries that must comply with stringent ethical and regulatory standards [1]. In response to this challenge, eXplainable Artificial Intelligence (XAI) focuses on developing new tools to help humans better understand how ANNs arrive at their decisions [2, 3]. Among the large array of methods available, attribution methods have become the go-to approach [4-14]. They yield heatmaps in order to highlight the importance of each input feature (or group of features [15]) for driving a model's decision. However, there is growing consensus that these attribution methods fall short of providing meaningful explanations [16-19] as revealed by multiple user studies [20-25, 20]. It has been suggested that for explainability methods to become usable by human users, they need to be able to highlight not just the location of important features within an image (i.e., the where information) but also their semantic content (i.e., the what information).

One promising set of explainability methods to address this challenge includes concept-based explainability methods, which are methods that aim to identify high-level concepts within the activation space of ANNs [26]. These methods have recently gained renewed interest due to their success in providing human-interpretable explanations [27-30] (see section 2 for a detailed description of the related work). However, concept-based explainability methods are still in the early stages, and progress relies largely on researchers' intuitions rather than well-established theoretical foundations. A key challenge lies in formalizing the notion of concept itself [31]. Researchers have proposed desiderata such as meaningfulness, coherence, and importance [27] but the lack of formalism in concept definition has hindered the derivation of appropriate metrics for comparing different methods.

This article presents a theoretical framework to unify and characterize current concept-based explainability methods. Our approach builds on the fundamental observation that all concept-based explainability methods share two key steps: (1) concepts are extracted, and (2) importance scores are assigned to these concepts based on their contribution to the model's decision [27]. Here, we show how the first extraction step can be formulated as a dictionary learning problem while the second importance scoring step can be formulated as an attribution problem in the concept space. To summarize, our contributions are as follows:

We describe a novel framework that unifies all modern concept-based explainability methods and we borrow metrics from different fields (such as sparsity, reconstruction, stability, FID, or OOD scores) to evaluate the effectiveness of those methods.

We leverage modern attribution methods to derive seven novel concept importance estimation methods and provide theoretical guarantees regarding their optimality. Additionally, we show how standard faithfulness evaluation metrics used to evaluate attribution methods (i.e., Insertion, Deletion [32], and µ Fidelity [33]) can be adapted to serve as benchmarks for concept importance scoring. In particular, we demonstrate that Integrated Gradients, Gradient Input, RISE, and Occlusion achieve the highest theoretical scores for 3 faithfulness metrics when the concept decomposition is on the penultimate layer.

We introduce the notion of local concept importance to address a significant challenge in explainability: the identification of image clusters that reflect a shared strategy by the model (see Figure 1). We show how the corresponding cluster plots can be used as visualization tools to help with the identification of the main visual strategies used by a model to help explain false positive classifications.

Espresso strategic

clustering graph

clustering graph

2 Related Work

Kim et al. [26] were the first to propose a concept-based approach to interpret neural network internal states. They defined the notion of concepts using Concept Activation Vectors (CAVs). CAVs are derived by training a linear classifier between a concept's examples and random counterexamples and then taking the vector orthogonal to the decision boundary. In their work, the concepts are manually selected by humans. They further introduce the first concept importance scoring method, called Testing with CAVs (TCAV). TCAV uses directional derivatives to evaluate the contribution of each CAV to the model's prediction for each object category. Although this approach demonstrates meaningful explanations to human users, it requires a significant human effort to create a relevant image database of concepts. To address this limitation, Ghorbani et al. [27] developed an unsupervised method called Automatic Concept Extraction ( ACE ) that extracts CA Vs without the need for human supervision. In their work, the CAVs are the centroid of the activations (in a given layer) when the network is fed with multi-scale image segments belonging to an image class of interest. However, the use of image segments could introduce biases in the explanations [34-37]. ACE also leverages TCAV to rank the concepts of a given object category based on their importance.

Zhang et al. [28] proposed a novel method for concept-based explainability called Invertible Conceptbased Explanation ( ICE ). ICE leverages matrix factorization techniques, such as non-negative matrix factorization (NMF), to extract Concept Activation Vectors (CAVs). Here, the concepts are localized as the matrix factorization is applied on feature maps (before the global average pooling). In ICE , the concepts' importance is computed using the TCAV score [26]. Note that the Singular Value Decomposition (SVD) was also suggested as a concept discovery method [28, 30]. CRAFT (Concept Recursive Activation FacTorization for explainability) uses NMF to extract the concepts, but as it is applied after the global pooling average, the concepts are location invariant. Additionally, CRAFT employs Sobol indices to quantify the global importance of concepts associated with an object category.

3 A Unifying perspective

Notations. Throughout, || · || 2 and || · || F represent the /lscript 2 and Frobenius norm, respectively. We consider a general supervised learning setting, where a classifier f : X → Y maps inputs from an input space X ⊆ R d to an output space Y ⊆ R c . For any matrix X ∈ R n × d , x i denotes the i th row of X , where i ∈ { 1 , . . . , n } and x i ∈ R d . Without loss of generality, we assume that f admits an intermediate space H ⊆ R p . In this setup, h : X → H maps inputs to the intermediate space, and g : H → Y takes the intermediate space to the output. Consequently, f ( x ) = ( g ◦ h )( x ) . Additionally, let a = h ( x ) ∈ H represent the activations of x in this intermediate space. We also abuse notation slightly: f ( X ) = ( g ◦ h )( X ) denotes the vectorized application of f on each element x of X , resulting in ( f ( x 1 ) , . . . , f ( x n )) .

Prior methods for concept extraction, namely ACE [27], ICE [28] and CRAFT [29], can be distilled into two fundamental steps:

( i ) Concept extraction: A set of images X ∈ R n × d belonging to the same class is sent to the intermediate space giving activations A = h ( X ) ∈ R n × p . These activations are used to extract a set of k CAVs using K-Means [27], PCA (or SVD) [28, 30] or NMF [28, 29]. Each CAV is denoted v i and V = ( v 1 , . . . , v k ) ∈ R p × k forms the dictionary of concepts.

( ii ) Concept importance scoring: It involves calculating a set of k global scores, which provides an importance measure of each concept v i to the class as a whole. Specifically, it quantifies the influence of each concept v i on the final classifier prediction for the given set of points X . Prominent measures for concept importance include TCAV [26] and the Sobol indices [29].

The two-step process described above is repeated for all classes. In the following subsections, we theoretically demonstrate that the concept extraction step (i) could be recast as a dictionary learning problem (see 3.1). It allows us to reformulate and generalize the concept importance step (ii) using attribution methods (see 3.2).

3.1 Concept Extraction

A dictionary learning perspective. The purpose of this section is to redefine all current concept extraction methods as a problem within the framework of dictionary learning. Given the necessity for clearer formalization and metrics in the field of concept extraction, integrating concept extraction with dictionary learning enables us to employ a comprehensive set of metrics and obtain valuable theoretical insights from a well-established and extensively researched domain.

The goal of concept extraction is to find a small set of interpretable CA Vs (i.e., V ) that allows us to faithfully interpret the activation A . By preserving a linear relationship between V and A , we facilitate the understanding and interpretability of the learned concepts [26, 38]. Therefore, we look for a coefficient matrix U ∈ R n × k (also called loading matrix) and a set of CA Vs V , so that A ≈ UV T . In this approximation of A using the two low-rank matrices ( U , V ) , V represents the concept basis used to reinterpret our samples, and U are the coordinates of the activation in this new basis. Interestingly, such a formulation allows a recast of the concept extraction problem as an instance of dictionary learning problem [39] in which all known concept-based explainability methods fall:

with e i the i -th element of the canonical basis, I the identity matrix and ψ any neural network. In this context, V is the dictionary and U the representation of A with the atoms of V . u i denote the i -th row of U . These methods extract the concept banks V differently, thereby necessitating different interpretations * .

In ACE , the CAVs are defined as the centroids of the clusters found by the K-means algorithm. Specifically, a concept vector v i in the matrix V indicates a dense concentration of points associated with the corresponding concept, implying a repeated activation pattern. The main benefit of ACE comes from its reconstruction process, involving projecting activations onto the nearest centroid, which ensures that the representation will lie within the observed distribution (no out-of-distribution instances). However, its limitation lies in its lack of expressivity, as each activation representation is restricted to a single concept ( || u || 0 = 1 ). As a result, it cannot capture compositions of concepts, leading to sub-optimal representations that fail to fully grasp the richness of the underlying data distribution.

On the other hand, the PCA benefits from superior reconstruction performance due to its lower constraints, as stated by the Eckart-Young-Mirsky[43] theorem. The CAVs are the eigenvector of the covariance matrix: they indicate the direction in which the data variance is maximal. An inherent limitation is that the PCA will not be able to properly capture stable concepts that do not contribute to the sample variability (e.g. the dog-head concept might not be considered important by the PCA to explain the dog class if it is present across all examples). Neural networks are known to cluster together the points belonging to the same category in the last layer to achieve linear separability ([44, 29]). Thus, the orthogonality constraint in the PCA might not be suitable to correctly interpret the manifold of the deep layer induced by points from the same class (it is interesting to note that this limitation can be of interest when studying all classes at once). Also, unlike K-means, which produces strictly positive clusters if all points are positive (e.g., the output of ReLU), PCA has no sign constraint and can undesirably reconstruct out-of-distribution (OOD) activations, including negative values after ReLU.

In contrast to K-Means, which induces extremely sparse representations, and PCA, which generates dense representations, the NMF (used in CRAFT and ICE ) strikes a harmonious balance as it provides moderately sparse representation. This is due to NMF relaxing the constraints imposed by the K-means algorithm (adding an orthogonality constraint on V such that VV T = I would yield an equivalent solution to K-means clustering [45]). This sparsity facilitates the encoding of

K-Means

PCA

NMF

Sparse Autoencoder

C1

C2

compositional representations that are particularly valuable when an image encompasses multiple concepts. Moreover, by allowing only additive linear combinations of components with non-negative coefficients, NMF inherently fosters a parts-based representation. This distinguishes NMF from PCA, which offers a holistic representation model. Interestingly, the NMF is known to yield representations that are interpretable by humans [28, 29]. Finally, the non-orthogonality of these concepts presents an advantage as it accommodates the phenomenon of superposition [38], wherein neurons within a layer may contribute to multiple distinct concepts simultaneously.

To summarize, we have explored three approaches to concept extraction, each necessitating a unique interpretation of the resulting Concept Activation Vectors (CAVs). Among these methods, NMF (used in CRAFT and ICE ) emerges as a promising middle ground between PCA and K-means. Leveraging its capacity to capture intricate patterns, along with its ability to facilitate compositional representations and intuitive parts-based interpretations (as demonstrated in Figure 2), NMF stands out as a compelling choice for extracting meaningful concepts from high-dimensional data. These advantages have been underscored by previous human studies, as evidenced by works such as Zhang et al.[28] and Fel et al.[29].

Evaluation of concept extraction Following the theoretical discussion of the various concept extraction methods, we conduct an empirical investigation of the previously discussed properties to gain deeper insights into their distinctions and advantages. In our experiment, we apply the PCA, K-Means, and NMF concept extraction methods on the penultimate layer of three state-of-the-art models. We subsequently evaluate the concepts using five different metrics (see Table 1). All five metrics are connected with the desired characteristics of a dictionary learning method. They include achieving a high-quality reconstruction (Relative l2), sparse encoding of concepts (Sparsity), ensuring the stability of the concept base in relation to A (Stability), performing reconstructions within the intended domain (avoiding OOD), and maintaining the overall distribution during the reconstruction process (FID). All the results come from 10 classes of ImageNet (the one used in Imagenette [50]), and are obtained using n = 16 k images for each class.

We begin our empirical investigation by using a set of standard metrics derived from the dictionary learning literature, namely Relative l 2 and Sparsity. Concerning the Relative /lscript 2 , PCA achieves the highest score among the three considered methods, confirming the theoretical expectations based on the Eckart-Young-Mirsky theorem [43], followed by NMF. Concerning the sparsity of the underlying representation u , we compute the proportion of non-zero elements || u || 0 /k . Since

K-means inherently has a sparsity of 1 /k (as induced by equation 1), it naturally performs better in terms of sparsity, followed by NMF.

We deepen our investigation by proposing three additional metrics that offer complementary insights into the extracted concepts. Those metrics are the Stability, the FID, and the OOD score. The Stability (as it can be seen as a loose approximation of algorithmic stability [51]) measures how consistent concepts remain when they are extracted from different subsets of the data. To evaluate Stability, we perform the concept extraction methods N times on K -fold subsets of the data. Then, we map the extracted concepts together using a Hungarian loss function and measure the cosine similarity of the CAVs. If a method is stable, it should yield the same concepts (up to permutation) across each K -fold, where each fold consists of 1000 images. K-Means and NMF demonstrate the highest stability, while PCA appears to be highly unstable, which can be problematic for interpreting the results and may undermine confidence in the extracted concepts.

The last two metrics, FID and OOD, are complementary in that they measure: (i) how faithful the representations extracted are w.r.t the original distribution, and (ii) the ability of the method to generate points lying in the data distribution (non-OOD). Formally, the FID quantifies the 1Wasserstein distance [52] W 1 between the empirical distribution of activation A , denoted µ a , and the empirical distribution of the reconstructed activation UV T denoted µ u . Thus, FID is calculated as FID = W 1 ( µ a , µ u ) . On the other hand, the OOD score measures the plausibility of the reconstruction by leveraging Deep-KNN [53], a recent state-of-the-art OOD metric. More specifically, we use the Deep-KNN score to evaluate the deviation of a reconstructed point from the closest original point. In summary, a good reconstruction method is capable of accurately representing the original distribution (as indicated by FID) while ensuring that the generated points remain within the model's domain (non-OOD). K-means leads to the best OOD scores because each instance is reconstructed as a centroid, resulting in proximity to in-distribution (ID) instances. However, this approach collapses the distribution to a limited set of points, resulting in low FID. On the other hand, PCA may suffer from mapping to negative values, which can adversely affect the OOD score. Nevertheless, PCA is specifically optimized to achieve the best average reconstructions. NMF, with fewer stringent constraints, strikes a balance by providing in-distribution reconstructions at both the sample and population levels.

In conclusion, the results clearly demonstrate NMF as a method that strikes a balance between the two approaches as NMF demonstrates promising performance across all tested metrics. Henceforth, we will use the NMF to extract concepts without mentioning it.

The Last Layer as a Promising Direction The various methods examined, namely ACE , ICE , and CRAFT , generally rely on a deep layer to perform their decomposition without providing quantitative or theoretical justifications for their choice. To explore the validity of this choice, we apply the aforementioned metrics to each block's output in a ResNet50 model. Figure 3 illustrates the metric evolution across different blocks, revealing a trend that favors the last layer for the decomposition. This empirical finding aligns with the practical implementations discussed above.

3.2 Concept importance

In this section, we leverage our framework to unify concept importance scoring using the existing attribution methods. Furthermore, we demonstrate that specifically in the case of decomposition in the penultimate layer, it exists optimal methods for importance estimation, namely RISE [32],

Integrated Gradients [7], Gradient-Input [6], and Occlusion [13]. We provide theoretical evidence to support the optimality of these methods.

From concept importance to attribution methods The dictionary learning formulation allows us to define the concepts V in such a way that they are optimal to reconstruct the activation, i.e., A ≈ UV T . Nevertheless, this does not guarantee that those concepts are important for the model's prediction. For example, the 'grass' concept might be important to characterize the activations of a neural network when presented with a St-Bernard image, but it might not be crucial for the network to classify the same image as a St Bernard [26, 16, 54]. The notion of concept importance is precisely introduced to avoid such a confirmation bias and to identify the concepts used to classify among all detected concepts.

We use the notion of Concept ATtribution methods (which we denote as CAT s) to assess the concept importance score. The CATs are a generalization of the attribution methods: while attribution methods assess the sensitivity of the model output to a change in the pixel space, the concept importance evaluates the sensitivity to a change in the concept space. To compute the CATs methods, it is necessary to link the activation a ∈ R p to the concept base V and the model prediction y . To do so, we feed the second part of the network ( g ) with the activation reconstruction ( u V T ≈ a ) so that y = g ( u V T ) . Intuitively, a CAT method quantifies how a variation of u will impact y . We denote ϕ i ( u ) the i -th coordinate of ϕ ( u ) , so that it represents the importance of the i -th concept in the representation u . Equipped with these notations, we can leverage the sensitivity metrics introduced in standard attribution methods to re-define the current measures of concept importance, as well as introduce the new CATs borrowed from the attribution methods literature:

The complete derivation of the 7 new CATs is provided in the appendix. In the derivations, ∇ u i denotes the gradient with respect to the i -th coordinate of u , while E and V represent the expectation and variance, respectively. In Eq. 6, m is a mask of real-valued random variable between 0 and 1 (i.e m ∼ U ([0 , 1] p ) ). We note that, when we use the gradient (w.r.t to u i ) as an importance score, we end up with the directional derivative used in the TCA V metric [26], which is used by ACE and ICE to assess the importance of concepts. CRAFT leverages the Sobol-Hoeffding decomposition (used in sensitivity analysis), to estimate the concept importance. The Sobol indices measure the contribution of a concept as well as its interaction of any order with any other concepts to the output variance. Intuitively, the numerator of Eq. 6 is the expected variance that would be left if all variables but u i were to be fixed.

Evaluation of concept importance methods Our generalization of the concept importance score, using the Concept ATtributions (CATs), allows us to observe that current concept-based explainability methods are only leveraging a small subset of concept importance methods. In Appendix A, we provide the complete derivation of 7 new CATs based on the following existing attribution methods, notably: Gradient input [6], Smooth grad [5], Integrated Gradients [7], VarGrad [55], Occlusion [13], HSIC [10] and RISE [32].

With the concept importance scoring now formulated as a generalization of attribution methods, we can borrow the metrics from the attribution domain to evaluate the faithfulness [56, 32, 33] of concept importance methods. In particular, we adapt three distinct metrics to evaluate the significance of concept importance scores: the C-Deletion [32], C-Insertion [32], and Cµ Fidelity [33] metrics. In C-Deletion, we gradually remove the concepts (as shown in Figure 4), in decreasing order of importance, and we report the network's output each time a concept is removed. When a concept is removed in C-Deletion, the corresponding coordinate in the representation is set to 0 . The final C-Deletion metrics are computed as the area under the curve in Figure 4. For C-Insertion, this is

(a)

(b)

C-Deletion

C-Insertion

1.00

1.00

C-Deletion

C-Insertion

C-MuFidelity

1.0

0.75

0.75

0.7

0.90 -

0.50

0.50

0.6

-1

0.85

=

0.8 -

0.25

0.25

0.5

0.80 -

0.6

0.00

0.00

0

5

10

1520

25

0

5

101520

25

0.4

#of Concept

#ofConcept

0.75

0.4

Saliency

HsicAttributionMethod

0.3

/

SmoothGrad

Occlusion

0.2

0.70

0.2

GradientInput

SobolAttributionMethod

0.1

1

0.65

0.0-

VarGrad

IntegratedGradients

B2

B3

B4

B5

B2

B3

B4

B5

B2

B3

B4

B5

the opposite: we start from a representation vector filled with zero, and we progressively add more concepts, following an increasing order of importance.

For the Cµ Fidelity, we calculate the correlation between the model's output when concepts are randomly removed and the importance assigned to those specific concepts. The results across layers for a ResNet50 model are depicted in Figure 4b. We observe that decomposition towards the end of the model is preferred across all the metrics. As a result, in the next section, we will specifically examine the case of the penultimate layer.

A note on the last layer Based on our empirical results, it appears that the last layer is preferable for both improved concept extraction and more accurate estimation of importance. Herein, we derive theoretical guarantees about the optimality of concept importance methods in the penultimate layer. Without loss of generality, we assume y ∈ R the logits of the class of interest. In the penultimate layer, the score y is a linear combination of activations: y = a W + b for weight matrix W and bias b . In this particular case, all CATs have a closed-form (see appendix B), that allows us to derive 2 theorems. The first theorem tackles the CATs optimality for the C-Deletion and C-Insertion methods (demonstration in Appendix D). We observe that the C-Deletion and C-Insertion problems can be represented as weighted matroids. Therefore the greedy algorithms lead to optimal solutions for CATs and a similar theorem could be derived for Cµ Fidelity.

Theorem 3.1 (Optimal C-Deletion, C-Insertion in the penultimate layer) . When decomposing in the penultimate layer, Gradient Input , Integrated Gradients , Occlusion , and Rise yield the optimal solution for the C-Deletion and C-Insertion metrics. More generally, any method ϕ ( u ) that satisfies the condition ∀ ( i, j ) ∈ { 1 , . . . , k } 2 , ( u /circledot e i ) V T W ≥ ( u /circledot e j ) V T W = ⇒ ϕ ( u ) i ≥ ϕ ( u ) j yields the optimal solution.

Theorem 3.2 (Optimal Cµ Fidelity in the penultimate layer) . When decomposing in the penultimate layer, Gradient Input , Integrated Gradients , Occlusion , and Rise yield the optimal solution for the Cµ Fidelity metric.

Therefore, for all 3 metrics, the concept importance methods based on Gradient Input, Integrated Gradient, Occlusion, and Rise are optimal, when used in the penultimate layer.

In summary, our investigation of concept extraction methods from the perspective of dictionary learning demonstrates that the NMF approach, specifically when extracting concepts from the penultimate layer, presents the most appealing trade-off compared to PCA and K-Means methods. In addition, our formalization of concept importance using attribution methods provided us with a theoretical guarantee for 4 different CATs. Henceforth, we will then consider the following setup: a

NMF on the penultimate layer to extract the concepts, combined with a concept importance method based on Integrated Gradient.

3.3 Unveiling main strategies

Reliability

1.0

0.8

0.6

Global Importance

0.4

6

0.2

4

0.0

Prevalence

800

600

400

200

0

So far, the concept-based explainability methods have mainly focused on evaluating the global importance of concepts, i.e., the importance of concepts for an entire class [26, 29]. This point can be limiting when studying misclassified data points, as we can speculate that the most important concepts for a given class might not hold for an individual sample (local importance). Fortunately, our formulation of concept importance using attribution methods gives us access to importance scores at the level of individual samples ( i.e., ϕ ( u ) ). Here, we show how to use these local importance scores to efficiently cluster data points based on the strategy used for their classification.

The local (or image-based) importance of concepts can be integrated into global measures of importance for the entire class with the notion of prevalence and reliability (see Figure 5). A concept is said to be prevalent at the class level when it appears very frequently. A prevalence score is computed based on the number of times a concept is identified as the most important one, i.e., arg max ϕ ( u ) . At the same time, a concept is said to be reliable if it is very likely to trigger a correct prediction. The reliability is quantified using the mean classification accuracy on samples sharing the same most important concept.

Strategic cluster graph. In the strategic cluster graph (Figure 1 and Figure 6), we combine the notions of concept prevalence and reliability to reveal the main strategies of a model for a given category, more precisely, we reveal their repartition across the different samples of the class. We use a dimensionality reduction technique (UMAP [57]) to arrange the data points based on the concept importance vector ϕ ( u ) of each sample. Data points are colored according to the associated concept with the highest importance arg max ϕ ( u ) . Interestingly, one can see in Figure 1 and Figure 6 that spatially close points represent samples classified using similar strategies - as they exhibit similar concept importance - and not necessarily similar embeddings. For example, for the 'lemon' object category (Figure 6), the texture of the lemon peel is the most prevalent concept, as it appears to be the dominant concept in 90% of the samples (see the green cluster in Figure 6). We also observe that the concept 'pile of round, yellow objects' is not reliable for the network to properly classify a lemon as it results in a mean classification accuracy of 40% only (see top-left graph in Figure 6).

In Figure 6 (right panel), we have exploited the strategic cluster graph to understand the classification strategies leading to bad classifications. For example, an orange ( 1 st image, 1 st row) was classified as a lemon because of the peel texture they both share. Similarly, a cathedral roof was classified as a lemon because of the wedge-shaped structure of the structure ( 4 th image, 1 st row).

4 Discussion

This article introduced a theoretical framework that unifies all modern concept-based explainability methods. Breaking down and formalizing the two essential steps in these methods, concept extraction and concept importance scoring, allowed us to better understand the underlying principles driving concept-based explainability. We leveraged this unified framework to propose new evaluation metrics for assessing the quality of extracted concepts. Through experimental and theoretical analyses, we justified the standard use of the last layer of an ANN for concept-based explanation. Finally, we harnessed the parallel between concept importance and attribution methods to gain insights into global concept importance (at the class level) by examining local concept importance (for individual samples). We proposed the strategic cluster graph, which provides insights into the strategy used by an ANN to classify images. We have provided an example use of this approach to better understand

Reliability

1.0

0.5

UnderstandFalsePositive

0.0

Hallucinatinglemon

Hallucinatinglemonbecause

becauseofthepeeltexture

oflemonwedgeshape

Lemon strategic

clustering graph

Hallucinationoflemonsfrom

Hallucinatingalemonbecauseof

pilesofround,yellowobjects

agreenobjecthangingonatree

the failure cases of a system. Overall, our work demonstrates the potential benefits of the dictionary learning framework for automatic concept extraction and we hope this work will pave the way for further advancements in the field of XAI.

References

Paolo Tripicchio and Salvatore D'Avella. Is deep learning ready to satisfy industry needs? Procedia Manufacturing , 51:1192-1199, 2020.

Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. ArXiv e-print , 2017.

Alon Jacovi, Ana Marasovi´ c, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 624-635, 2021.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop, Proceedings of the International Conference on Learning Representations (ICLR) , 2013.

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. In Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML) , 2017.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the International Conference on Machine Learning (ICML) , 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML) , 2017.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017.

Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Paul Novello, Thomas Fel, and David Vigouroux. Making sense of dependence: Efficient black-box explanations using dependence measure. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Thomas Fel, Melanie Ducoffe, David Vigouroux, Remi Cadene, Mikael Capelle, Claire Nicodeme, and Thomas Serre. Don't lie to me! robust and efficient explainability with verified perturbation analysis. Workshop on Formal Verification of Machine Learning, Proceedings of the International Conference on Machine Learning (ICML) , 2022.

Mara Graziani, Iam Palatnik de Sousa, Marley MBR Vellasco, Eduardo Costa da Silva, Henning Müller, and Vincent Andrearczyk. Sharpening local interpretable model-agnostic explanations for histopathology: improved understandability and reliability. In Medical Image Computing and Computer Assisted Intervention (MICCAI) . Springer, 2021.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV) , 2014.

Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017.

Marouane Il Idrissi, Nicolas Bousquet, Fabrice Gamboa, Bertrand Iooss, and Jean-Michel Loubes. On the coalitional decomposition of parameters of interest, 2023.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (NIPS) , 2018.

Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In Proceedings of the International Conference on Machine Learning (ICML) , 2020.

Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems (NeurIPS) , 34, 2021.

Sukrut Rao, Moritz Böhle, and Bernt Schiele. Towards better understanding attribution methods. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2020.

Hua Shen and Ting-Hao Huang. How useful are the machine-generated interpretations to general users? a human evaluation on guessing the incorrectly predicted labels. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 8, pages 168-172, 2020.

Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In Proceedings of the IEEE European Conference on Computer Vision (ECCV) , 2022.

Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Leon Sixt, Martin Schuessler, Oana-Iuliana Popescu, Philipp Weiß, and Tim Landgraf. Do users benefit from interpretable vision? a user study, baseline, and dataset. Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning . Proceedings of the International Conference on Machine Learning (ICML), 2018.

Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems , pages 9273-9282, 2019.

Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11682-11690, 2021.

Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Mara Graziani, An-phi Nguyen, Laura O'Mahony, Henning Müller, and Vincent Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML , 2023.

James Genone and Tania Lombrozo. Concept possession, experimental semantics, and hybrid theories of reference. Philosophical Psychology , 25(5):717-742, 2012.

Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC) , 2018.

Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , 2020.

Johannes Haug, Stefan Zürn, Peter El-Jiz, and Gjergji Kasneci. On baselines for local feature attributions. arXiv preprint arXiv:2101.00905 , 2021.

Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsieh. Evaluations and methods for explanation through robustness analysis. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. 2019.

Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill , 2020.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652 , 2022.

Julien Mairal, Francis Bach, Jean Ponce, et al. Sparse modeling for image and vision processing. Foundations and Trends® in Computer Graphics and Vision , 8(2-3):85-283, 2014.

Alireza Makhzani and Brendan Frey. K-sparse autoencoders. Proceedings of the International Conference on Learning Representations (ICLR) , 2014.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread , 2022.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1 (3):211-218, 1936.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Chris Ding, Xiaofeng He, and Horst D Simon. On the equivalence of nonnegative matrix factorization and spectral clustering. In Proceedings of the 2005 SIAM international conference on data mining , pages 606-610. SIAM, 2005.

Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Jayneel Parekh, Sanjeel Parekh, Pavlo Mozharovskyi, Florence d'Alché Buc, and Gaël Richard. Listen to interpret: Post-hoc interpretability for audio networks with nmf. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Jeremy Howard. Imagenette dataset. URL https://github.com/fastai/imagenette/ .

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2002.

Cédric Villani et al. Optimal transport: old and new , volume 338. Springer.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning , pages 20827-20840. PMLR, 2022.

Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2017.

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) , 2020.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Thomas Fel, Lucas Hervier, David Vigouroux, Antonin Poche, Justin Plakoo, Remi Cadene, Mathieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Bethune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop on Explainable Artificial Intelligence for Computer Vision (CVPR) , 2022.

Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradientbased attribution methods for deep neural networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.

Matthew Sotoudeh and Aditya V. Thakur. Computing linear restrictions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV) , 2014.

Hassler Whitney. On the abstract properties of linear dependence. Hassler Whitney Collected Papers , pages 147-171, 1992.

Sajad Fathi Hafshejani and Zahra Moaberfard. Initialization for non-negative matrix factorization: a comprehensive review. International Journal of Data Science and Analytics , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations (ICLR) , 2015.

Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications . Springer, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unifying framework to study common concept-based explainability methods. In particular, the authors observe that all the methods boil down to dictionary learning in the latent spaces of a neural network. Based on this observation, they (1) compare the different concept extraction/discovery methods and (2) extend established feature attribution methods to concept-based explanations. The authors investigation leads them to conclude that the penultimate layer is the best one where to extract concepts proving the optimality of concept-adapted metrics for attribution.\n\nSTRENGTHS:\n- (Clarity) Paper is in general easy to follow and fairly clear at discussing the core concepts.\n- (Novelty) I am personally a big fan of unifying frameworks, so this paper seems like a reasonable steps towards understanding concept-based explainability methods\n- (Significance) The strategic cluster graph is quite compelling from an user-experience perspective \n- (Significance) While the results don't seem particularly surprising to me (more below), it is nice to see an analysis of concept-based methods laid out.\n- (Quality) Mathematical derivations seem sound.\n\nWEAKNESSES:\n1) *Novelty/Significance* \n  - *Theoretical results*: As mentioned above, I do not find the theoretical results particularly surprising. More details are reported below, but, in a nutshell, the results seem to me like a trivial extension of the work from Ancona et al. [1].\n  - *Strategic cluster graph*: I would have preferred to see a more in-depth/quantitative analysis of this \"tool\". The concept analysis of the graph is fairly hand-wavy and arbitrary, e.g. in the naming of the concepts (apart for one of the wrongly classified samples).\n\nAs a more actionable feedback, perhaps I would comment that personally I would have preferred to have a less extensive theoretical analysis (since I find it unsurprising), and a more detailed use case for the strategic cluster graph, maybe with an user study. Or alternatively the authors could have leveraged their framework to propose a new method for concept extraction inspired by other dictionary learning method, perhaps something probabilistic such as the latent dirichlet allocation (although on the top of my mind, it may not be very trivial).\n\n2) *Clarity*: some definitions are left unreferenced. For example (L. 273), there is no reference to the original feature-based metrics, so it might appear unclear how the C-metrics are an extension of those ones.\n\n3) More detailed feedback can be found below, in the \"Questions\" section.  \n\n[1] Ancona, Marco, et al. \"Towards better understanding of gradient-based attribution methods for deep neural networks.\" arXiv preprint arXiv:1711.06104 (2017).\n\nQUESTIONS:\n1. As mentioned above, using the authors framework to suggest new ways to do concept extraction would particularly strengthen the paper. Could the authors comment, at least as future work, how they could leverage their framework to evaluate possible new concept extraction methods?\n2. (L. 128) Are the authors sure about the expressivity claim of K-means based methods? wouldn't expressity depend on the number of cluster K? \n3. (L. 135) This seems a drawback of doing class-based concept extraction, rather than a problem specific to PCA. For example, how would K-means be able to detect the dog-head concept, if all the samples in the dog class have the dog-head concept? \n4. The authors claim that NMF strikes an harmonious balance between K-Means and PCA. This somewhat should imply that NMF is a better choice. Could the authors elaborate on this? This would help strengthen the utility of their framework. The only argument is the claim that (L. 153) NMF is known to yield human-interpretable concepts. Is this not the case for PCA or K-Means? In the referenced papers, I cannot see a study on the utility of NMF concepts compared to PCA concepts.\n5. As an extension to the above, it would be interesting to leverage the authors framework to more systematically pinpoint the properties of a certain dictionary learning method that make the extracted concepts more or less interpretable from a user perspective. The authors can perhaps expand on this in a future work section.\n6. Evaluation of concept extraction: an additional evaluation could be performed by applying concept extraction methods to a model with ground truth concepts, e.g. a concept bottleneck model which was specifically trained with user-defined concepts as additional training signal.\n7. As mentioned above, the results (especially the theorems) are unsurprising/trivial. but I will concede that I might be wrong, if the authors could further elaborate on the differences between their results and Ancona's. As observed by the authors, the last classification layer is by construction linear wrt the penultimate layer. Since the dictionary learning view essentially reveals that the concepts extracted at the penultimate layer are essentially a linear projection on a different base of the penultimate layer latent space. Therefore, it is a trivial extension of Ancona et al.'s work, the fact that all attribution methods are essentially equivalent on linear classifiers, and furthermore, achieve optimal fidelity, i.e. correlation of 1, which again is unsurprising for linearly correlated variables. Consequently, it seems also trivial to me that C-deletion and C-insertion also achieve optimal values since insertion and deletion greedy strategies are trivially optimal for positively monotonically correlated variables. In essence, I would say that Theorem 3.2 is an extension of Ancona's work, while Theorem 3.1 is essentially a corollary  to 3.2.  \n8. As additional comment to the point above, it seems also unfair to evaluate different layers using these definitions of deletion and insertions since they would correctly evaluate only layers that are linearly related to the network outputs.\n\nLIMITATIONS:\nAuthors do not discuss or comment about possible limitations of their framework and possible future work. In particular, can their framework be used to derive new concept attribution scores and relative evaluation metrics, in case somebody might be constrained to not use the penultimate layer?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework for unifying existing post-hoc concept-based explainability methods in a cohesive, formalized form. Such a framework enables a principled study of key properties of state-of-the-art concept-based explainability methods, such as ACE, T-CAV, CRAFT, and ICE, and allows the authors to easily compare and contrast the quality of explanations generated by all of these approaches. To this end, this paper further introduces a large set of novel evaluation metrics for the quality of concept explanations together with a new simple framework to generate and visualize local concept-based explanations for a given prediction. These metrics are used to evaluate a variety of architectures in ImageNet and conclude that NMF-based concept-extraction approaches have significant potential compared to their clustering and PCA-based counterparts. Finally, the paper provides theoretical guarantees regarding the optimality of a set of concept-based explainability concept attribution methods, with respect to the fidelity and completeness of their explanations, when they are used to explain the latent space of the last layer of a DNN.\n\nSTRENGTHS:\nThank you for this very interesting and amazingly-crafted work; I thoroughly enjoyed reading it. I believe this paper’s goal of introducing a unifying formal framework to study most concept-based XAI methods is very important in a field lacking principled and theoretical analysis. Overall, I believe that this paper’s main strengths are:\n\n1. This paper’s core motivation, the framework it introduces, and the metrics it proposes are all worthy of study and of interest to the wider XAI community. These can become particularly handy for other researchers to evaluate, compare, and study novel methods against existing ones.\n2. Similarly, the key insights developed by this paper, in particular the optimality theorems and their surprising results on NMF-based methods, are all novel and potential sources for interesting future research.\n3. The paper is extremely well-written, with the writing being very clear and the figures nicely complementing the rest of the text. In particular, I like the non-standard approach to writing this work (not strictly following the now-boring intro-methods-results-discussion pipeline) and think it is worth pointing out.\n4. There is a very good mix of theoretical results and interesting insights from very simple experiments. In particular, I found the use of matroids for the proofs of the proposed theorems to be very interesting.\n5. The introduction of a framework that enables both global and local concept-based explanations with one method is novel and particularly useful in practice.\n\nWEAKNESSES:\nOverall, I think this paper introduces a very useful framework and set of metrics that may be of real interest to the wider community. I am leaning towards acceptance for this reason. However, I have some concerns/questions regarding some specific bits of the paper as well as some possible gaps in this work.  Specifically, I believe the following are that paper's main limitations:\n\n1. Some key and more recent methods are not mentioned or cited in this work. These include Concept Gradients [1] (which I am not sure if it directly fits in the proposed framework, although this would be a good test for the proposed ideas) and, more importantly, as this is extremely related to how the reconstruction of the latent space is framed in this work, completeness-aware concept discovery (CCD) [2]. Similarly, traditional topic models, such as latent Dirichlet allocation (LDA), are highly related to some of the ideas discussed in the concept extraction part of this framework.\n2. I am still uncertain whether ACE and T-CAV, as originally proposed in their original papers, can be expressed using this paper’s framework. If so, this weakens this paper's unification claim, a key contribution according to the authors. This may be a misunderstanding on my end, but please take a look at my questions below to possibly clarify this.\n3. Against general convention, none of the figures and tables include error bars, making it harder to understand when differences are statistically significant.\n4. The paper is missing key details regarding their empirical evaluation. These include model sources, licenses, and references showing the training setups for the original pre-trained models if models were not trained from scratch.\n\n### References\n\n[1] Bai, Andrew, et al. \"Concept Gradient: Concept-based Interpretation Without Linear Assumption.\" *The Eleventh International Conference on Learning Representations*. 2022.\n\n[2] Yeh, Chih-Kuan, et al. \"On completeness-aware concept-based explanations in deep neural networks.\" *Advances in neural information processing systems* 33 (2020): 20554-20565.\n\nQUESTIONS:\nRegarding the weaknesses discussed above, I will gladly consider increasing my score if some of these concerns are appropriately addressed by the authors (especially if they come from a misunderstanding on my end). To help with this discussion, I believe the following questions could be a good starting point:\n\n1. Could you please elaborate on how CCD fits within your framework and whether it would be feasible to include a topic-modelling-based concept extractor as part of the comparison in section 3.1? The lack of discussion of this method, a key algorithm in unsupervised concept learning, weakens the universal unification argument in this paper.\n2. One of the main claims of the paper is that the framework unifies “all modern concept-based explainability methods”. Yet, I fail to see how it can explicitly describe T-CAV in the form originally proposed by their authors. Could you please clarify how T-CAV , as defined in its original paper, can be explicitly framed in the proposed framework? I can see how it can be framed if one assumes that the latent space from which we extract the CAVs from can be explicitly reconstructed from all the concepts we learn (and is done so as part of inference). This, however, is in no way an assumption of the original T-CAV process and, in fact, it is a very strong constraint that in practice fails to apply. This is because we will most certainly not have all the necessary concepts for perfect reconstruction for most tasks.\n3. Regarding how ACE is framed in the proposed framework, I am still a bit doubtful that it is such a straight fit as claimed by this work. If I recall correctly, ACE uses clustering just to learn subsets of concept examples from which CAVs can later be learnt using linear separators. This paper, however, seems to suggest that ACE uses the cluster centroids themselves as the CAVs. If my understanding of that claim is correct, then I believe that’s wrong. Could you please clarify this point? If my understanding is correct, then this would require correctly updating a text where it claims that ACE falls within the clustering concept extractor family of methods.\n\nRegarding suggestions for the presentation, the paper is very well-written (thank you so much!). Nevertheless, the following possible errors and typos could be addressed before a camera-ready version:\n\n- **[important]** The appendices referenced in the main body of the paper seem not to be correctly aligned with their correct sections in the submitted supplementary material.\n- The use of bold symbols is a bit confusing. Sometimes it is used for tensors/vectors (e.g., $X$ and $x$ in line 93), sometimes it is used for functions even when their output is not a vector (e.g., $g$ in line 98), and sometimes it is used to refer to a scalar (e.g., line 509 in the appendix). Traditionally it is just used for vectors/tensors and not functions names or scalars. It is ok if this does not follow tradition, but it could benefit from consistency (as, say, some scalars are not bolded as the bias term $b$ sometimes).\n- Typo in line 112: it should probably be “This allows us…” rather than “It allows us…” as the “it” is unclear in that context.\n- In line 177 it should probably be “the best score” rather than “the highest score”  as PCA achieves the best score amongst all methods (which is the lowest, not the highest for relative $\\ell_2$.\n- The use of mix cases in “The Last Layer as a Promising Direction” in line 212 is inconsistent with the rest of the subtitles.\n- In line 214, a full stop “.” is missing before “To explore”.\n- The caption of Table 1 should be on top of the table to follow NeurIPS standards.\n- Line 220-221 read a bit off. Maybe a typo or a word is missing there.\n- Figure 3’s x-axis should be “# of Concepts” rather than “# of Concept”\n- In that same figure, it would be beneficial to use different colors or markers as it is otherwise hard to disentangle some of the methods (and then even harder to find the respective color in Figure 4 without the legend right there).\n- CAT is redefined in line 265.\n- In line 279 “zero” should be “zeros”.\n- Line 296 is missing “are” between $y \\in \\mathbb{R}$ and “the logits…”\n- In line 339, “this” should be “these”\n- nit: It is uncommon to have more than one paragraph in the abstract. Just pointing this out in case this comes from an accidental new line inserted in the abstract.\n\nLIMITATIONS:\nThis work does not discuss any limitations of the proposed framework, metrics, or results. It could certainly benefit from a short section where it is discussed how this framework may fail to generalize to some existing concept-based methods and how it could possibly be expanded in the future. Similarly, the manuscript does not discuss possible societal impacts of the work. However, I believe no clear negative societal impacts could arise from this paper's contributions and methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a unified perspective on post-hoc concept-based explanation methods. The main intuition underlying the work revolves around the fact that typical concept-based explanations can be considered as a two-stage process whereby, initially, a concept vocabulary is learned and, lastly, each concept's importance is evaluated. In light of their theoretical analysis, the authors describe a variegated class of concept-based explanations, showing in particular that the penultimate layer is particularly suited for concept analysis. In addition, the authors derive that some concept-importance methods are theoretically sound in the penultimate layer.\nConcerning the experimental investigation, the authors propose a set of metrics for comparing different explainability methods in both the settings of concept dictionary learning and concept importance estimation. The experiments are specifically tailored to evidence differences between existing methods.\nThe authors also contribute to a dedicated interface for visualizing the clusters graph on the ImageNet classes.\n\nSTRENGTHS:\nIn this paper, the theoretical analysis is extensive and well curated and the presentation is clear and easy to follow. The main strength of the paper is introducing a unified framework for analyzing concept-based explanations, which reveals particularly suited to address the pros and cons limitations of existing methods. In particular, the linear decomposition of the activation matrix in the concept vectors covers several existing approaches. This is of particular help also for proceeding with the research in the field. \n\nThe integration of three new metrics for concept extraction helps to compare several strategies, among which NMF decomposition seems preferable. Similarly, the authors propose three metrics to evaluate concept importance methods, deriving theoretical guarantees for 4 methods in the penultimate layer. This strengthens the idea that concept analysis is best when conducted in the deepest layer of neural networks. The experiments are illustrative to provide a discrimination between different methods and complete the theoretical analysis conducted. \n\nI also found helpful the introduction of the strategic cluster graph, which provides an intuitive interface for visualizing concept-level explanations of the model.\n\nWEAKNESSES:\nI have no major concerns about the presented material, which is very curated and high quality. \n\nNonetheless, I find two weaknesses on the general method proposed. First of all, learning a dictionary of concepts in unsupervised manner does not give any guarantee that the extracted concepts do actually possess a well-defined semantics. This requires a verification with an end user, as for example in [1], and, so, the considered methods for post-hoc concepts analysis are limited and do not ensure interpretability.\n\nAnother weakness amounts to extracting the concepts in a linear manner and it seems that classes are actually related to concepts in a kind of AND rule. This does not hold in general, where classes might depend on non-linear relations between concepts. As an example, take a class resulting from a XOR operation between two concepts, there is no way to extract this information with these methods.  \n\nI hope to see some discussion on these points.\n\n[1] Biswas, Shreyan, et al. \"CHIME: Causal Human-in-the-Loop Model Explanations.\" Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 10. No. 1. 2022.\n\nQUESTIONS:\nDoes the analysis offer guidance toward non-linear cases? Is it possible to extend the current approach to that scenario?\n\nLIMITATIONS:\nThe interpretability of concepts is not discussed at length; this addition may strengthen the authors' findings and provide guidance to improve the research on concept-based explanations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a unifying perspective for the concept extraction / importance estimation method via the dictionary learning framework. Namely, depending on the constraints on $U$ and $V$, the framework includes ACE, PCA, CRAFT and ICE as special cases. Then, the paper compares three difference approaches, PCA / NMF / K-Means, for the dictionary learning framework using several metrics. Moreover, by generalizing the attribution methods in XAI, the concept importance values are also computed and compared across different attribution methods for the NMF concept extraction. The theory for the concepts extracted from the penultimate layer is also given. Finally, by defining reliability and prevalence of the concepts, the visual and qualitative results are shown, which are compelling.\n\nSTRENGTHS:\n- Unifying effort for the concept extraction / importance is sound and the theoretical results back the approach of the paper. \n- Combining the attribution method with concept extraction is interesting and the reliability / prevalence notions make sense. \n- The visualization results are compelling.\n\nWEAKNESSES:\n- I think it would be beneficial to show the failure case of the proposed method. Figure 1/2 show compelling results for three classes (Espresso, Zucchini, and Lemon) but would there be classes that the proposed method fails to give meaningful explanation? What would those classes be like? It would be informative to show such examples as well so that people who use the proposed technique could be cautious. \n- It seems like the local importance score depends on the entire test set -- namely depending on what other images exist in the test set, the reliability and prevalence should vary. What are the variance of those values depending on the test set? Maybe cross-validation like evaluation should be meaningful.\n\nQUESTIONS:\n- The strategic graph is obtained by the Integrated Gradient attribution method -- how would be different across different attribution method? \n- How would the strategic graph change for different test sets?\n\nLIMITATIONS:\nPlease see the comment for weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}