CRAFT: explaining using Concepts from Recursive Activation FacTorization
Authors: Thomas FEL
Agustin Martin Picard
Louis Béthune
Thibaut Boissin
Julien Colin
David Vigouroux
Remi Cadene
Thomas Serre
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Explainability
Concept
Matrix Factorization
Implicit Differentiation
Attribution Methods
Sensitivity Analysis

Abstract: Despite their considerable potential, concept-based explainability methods have received relatively little attention, and explaining what’s driving models’ decisions and where it’s located in the input is still an open problem. To tackle this, we revisit unsupervised concept extraction techniques for explaining the decisions of deep neural networks and present CRAFT – a framework to generate concept-based explanations for understanding individual predictions and the model’s high-level logic for whole classes. CRAFT takes advantage of a novel method for recursively decomposing higher-level concepts into more elementary ones, combined with a novel approach for better estimating the importance of identified concepts with Sobol indices. Furthermore, we show how implicit differentiation can be used to generate concept-wise attribution explanations for individual images. We further demonstrate through fidelity metrics that our proposed concept importance estimation technique is more faithful to the model than previous methods, and, through human psychophysic experiments, we confirm that our recursive decomposition can generate meaningful and accurate concepts. Finally, we illustrate CRAFT’s potential to enable the understanding of predictions of trained models on multiple use-cases by producing meaningful concept-based explanations.

Interpreting the decisions of modern machine learning models such as neural networks remains a major challenge. The need for robust and reliable explainability methods has never been more urgent 21 as machine learning is being applied to an ever increasing range of domains, including safety critical CRAFT: Concept Recursive Activation FacTorization for Explainability
Anonymous Author(s)
Affiliation Address email
Abstract
Despite their considerable potential, concept-based explainability methods have received relatively little attention, and explaining what's driving models' decisions and where it's located in the input is still an open problem. To tackle this, revisit unsupervised concept extraction techniques for explaining the decisions deep neural networks and present CRAFT framework to generate concept-based explanations for understanding individual predictions and the model's high-level logic for whole classes. CRAFT takes advantage of novel method for recursively decomposing higher-level concepts into more elementary ones, combined with a novel approach for better estimating the importance of identified concepts with Sobol indices. Furthermore, we show how implicit differentiation can be used to generate concept-wise attribution explanations for individual images. We further demonstrate through fidelity metrics that our proposed concept importance estima- tion technique more faithful the model than previous methods, and, through human psychophysic experiments, we confirm that our recursive decomposition can generate meaningful and accurate concepts. Finally, we illustrate CRAFT's potential to enable the understanding of predictions of trained models on multiple use-cases by producing meaningful concept-basee based explanations." .
2
6
10
11
12
13
14
15
18
17
Introduction
19
22 ones. The application of the General Data Protection Regulation law (GDPR) [1] in the European
23 Union has drawn the attention of the general public to the rights they should have on their data.
24 This kickstarted race for other needs, with more and more regulation agencies asking for the right
25 for AI decisions to be explainable to users c.g. European AI act [2], EASA concepts for design
26 assurance [3].
27
In order to try to meet this need, an array of explainability methods have already been proposed. Most of these methods aim explaining what inputs (or pixels image) are driving the model's decision. These so-called attribution methods yield heatmaps that indicate the importance of individual pixels. Among the most notable ones is LIME [4], which was initially developed to try to locally that is.
28
29
30
31 at an instance level understand models' predictions to identify possible biases in vision models.
32 Multiple improvements have since been introduced either by better harnessing the information
33 provided by gradients to estimate the importance of individual pixels [5, 6, 7, 8, 9, 10, 11, 12].
34 leveraging image perturbations to evaluate the sensitivity of model's output [13, 14] or, more
35 recently, via the use of formal methods to generate explanations [15].
"Our code is available anonymous. 4open. att.ee/ea//aaaattoooooeeeaaaaaaa
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

36
more refined and However, all the aforementioned methods focus on one side of explainability - answering the question 37 of where- ii.. where in an image are the pixels that are critical to the decision located. They leave
38 the question of what - i.e., what visual features are actually driving decisions entirely open. We argue that this limitation is one of the main reasons why these methods fail some cases to help users, for instance, identify the source of a system's 's bias or its failure cases shown in [16]. Feature visualization methods [17. 18] characterize the selectivity of individual neurons (or neural channels or arbitrary directions in the neural activation space) via the synthesis of input stimuli which maximize their responses and can partially answer this question. Still in this vein, [19, 20, 21] proposed to use the training dataset to identify the samples that contribute the most to the model's 's decision. Finally, closer to our work, a new line of research has recently been initiated [22] based on high-level concepts. The goal of this branch is to find humanly interpretable concepts in the activation space of a layer in neural network. This approach can give positive results, but in its original formulation, it requires prior knowledge on the relevant concepts, and more importantly, the labeling of dataset for each of the concepts we want to extract. Hence, several works have proposed to automate the concept search based only on the training dataset and without explicit human supervision. The most prominent of these techniques, ACE [23], uses a combination of segmentation and clustering techniques, but requires heuristies to remove outliers. This method unlocks the possibility of large scale concept extraction without additional labeling or human supervision. Nevertheless, it suffers from several problems: each segment can only belong to one cluster, the choice of the layer from which to retrieve the concepts is not clear, and the amount of information lost during the outlier rejection phase can
39
40
41
42
43
45
46
47
48
49
50
51
52
53
54
55
56 be a cause of concern. More recently, [24] proposes to leverage matrix decompositions on internal feature maps to discover concepts.
57
58 It is important to note that current work does not offer link between their global and local ex- planations, nor do they offer an answer to the question of which layer to choose to perform the decomposition. Building up on these conclusions, we revisit these concept extraction techniques by using Non- -egativee Matrix Factorization (NMF) and propose 3 different ingredients to answer these questions simultancously, thereby introducing CRAFT, a new automatic concept extraction method.
59
60
61
62
63
We can summarize our main contributions as follows:
64
A novel approach for the automated extraction of high-level concepts learned by deep neural networks.
66
A recursive procedure to automatically decompose concepts into sub-concepts, starting with the last layer of the model and working our way inwards. We validate the benefit of this recursivity - i.c. decomposing concepts into sub- sub-concepts with human psychophysic experiments which show that (i) that the decomposition of a concept yields more coherent sub-concepts (ii) the groups of points formed by these sub-concepts appear meaningful to humans (expert or non-expert).
67
68
69
70
71
72
A novel technique to quantify the importance of individual concepts on a model's predictions using Sobol indices coming from the field of Sensitivity Analysis.
73
74
A novel Concept Attribution Map (CAM) method to backpropagate each of the concept values independently into the pixel space by leveraging the implicit function theorem. allowing us to locate the concept in given input image. This effectively unlocks the ability to apply all the white-box [5, 6, 7, 8, 9, 12, 25] and box [4, 26, 13, 14] explainability techniques in the literature to obtain concept-wise attribution maps.
75
76
77
78
79
A demonstration of the approach combining local and global explanations to accurately explain predictions and understand complex failure cases.
80
2
Related Work
81
82 Explaining where The widespread use of black-box machine learning methods including deep 83 convolutional neural networks in myriads of computer vision tasks prompted a need to understand 84 where in the input image the model looked to make predictions. These explanatory heatmaps can 85 be generated through completely different approaches depending on whether access to gradients is
86 provided. If is indeed the case, there's 's plethora of different methods that harnesses intermediary information inside the neural network to create these explanations [5, 8, 7, 28, 6, 29, 9, 12). However, 88 they have been found to induce confirmation bias [30] and to be vulnerable to adversarial attacks [31]. Somewhat differently, there are other methods [10. 11] that harness gradients to optimize masks
87
90 to maximize the impact on the predictions, and thus determine the most important parts of the
91 input for the model. However, if only the input and its corresponding output are available, other
2

---

Concept global Importance
0.30
0.25
0.20
0.15
0.10
0.05
4
0
2
18 21 22
Concept
Figure 1: CRAFT Results for the prediction 'Chain Saw', First, our method uses NMF to extract from the train set (ILSVRC2012 [27]) the most relevant concepts used by the network (ResNet50V2). Then, the global influence of these concepts on predictions measured using Sobol indices (right panel). Finally, the method provides local explanations through Concept Atriibution Maps (heatmap associated to concept, and computed using grad-CAM by backpropagating through the NMF concept values with implicit differentiationn. Besides, concepts can be interpreted by looking at crops that maximize the NMF coefficients. For the class 'Chain Saw', the detected concepts seem to be: Co for the chainsaw engine, C2 for the saw blade, C₁ for the human head, C1s for the vegetation, C21 for the jeans and C22 for the tree trunk.
82 techniques exist that enable the generation of attribution maps by locally estimating the importance of each input pixel: LIME [4], RISE [13], and more recently, attribution method based Sobol indices [14, 32]. Crucially, they propose to input perturbed versions of the example one wishes to explain and either construct linear model to determine the importance of each region of the input, leverage Monte-Carlo methods to this end, or compute the Sobol indices [32] associated to them as a measure of their influence on the model. Concretely, we will be exploiting all this literature to locate the important parts of images with respect to what we will call "high-level" concepts by generating concept-wise attribution maps.
94
96
98
99
Explaining <what> There have been studies [33, 34] that indicate that CNNs trained the ImageNet dataset [27] rely heavily on textures to classify, and largely disregard the shapes. For this reason, some researchers suggest that attribution maps might not be enough to explain models' predictions [17], and that explainability methods revealing the role of the textures are a must. Namely, generated the inputs that would maximize the neural activation of a given layer with respect to a given class. However, these explanations may not be easily interpretable by humans. Finally, other approaches suggest modify the structure of the neural network, either by constraining the convolutional layers to naturally provide visual explanations [35], or by forcing it to generate prototypes for the classes [36], but our main foeus post-hoc methods that can be applied
101
103
in [18] and [17], explanations
105
106
107
108
109
to pre- pre-tranned neural networks and don't 't need further training.
110
Concept discovery. In [22], Kim et al. proposed an alternative to explaining the what: they built a database with different concepts (such as "stripes") to extract concept vector in the latent space of given layer. Then they proposed to estimate the importance of this concept vector using the directional derivative of the model's predictions with respect to this concept vector. However, it is a supervised approach, and thus, only applicable when we have prior knowledge of the concepts in play. The natural extension of this idea is automatic discovery of concepts in unsupervised fashion, without the need for prior knowledge or labelled concept datasets. As such, in [23], a technique is proposed to discover these "high-level" concepts: they perform segmentation at different resolutions on patches of images, cluster them and select the most significant based on perception and Testing with Concept Activation Vectors (TCAV) [22] scores. However, the quality of the result is highly dependent on the segmentation scheme and on the layer used for perception scores. Building up on this technique, [24] propose to generate bank of concepts for each class by performing dimensionality reductions on the activation maps flattened over the channel dimension. Once the factorization done, the reconstruction
112
113
114
115
116
117
118
119
120
121
122
123
of the activation of the image can then be interpreted as a combination of a set of concepts and a coefficient associated to these concepts. Not all factorization-base methods are equal though. Their large-scale scale human experiments show an interesting trend: Non-negative Matrix Factorization (NMF) widely preferred over Principal Component Analysis (PCA) or ACE for generating meaningful
125
3

---

(1) Layer
Layer t+ (2)
Layer I+a
Layer I
(3)
Layer tm
Layer
Layer (-X
C2
C,1
22
C33
Figure 2: (1) Neural Collapse (Amalgamation) classifiers need to be able to linearly separate each class at the last layer, and to do this, the activations of the same class must merge during the forward pass until they all converge to the one-hot vector of the class in the logits layer. This may result in activations that are too concentrated to be broken down into meaningful concepts. (2) Recursive process When a concept is not understood (e.g.. C), we propose decompose it into multiple sub-concepts (e.g., C1, C2, C3) using the activations from an earlier layer to overcome the aforementioned neural collapse issue. (3) Example of concept recursive decomposition using CRAFT on the class 'Parachute' of ILSVRC2012 [27].
127
concepts for humans. Finally, [37] defines the notion of completeness a concept bank and proposes method to learn complete set of concepts using Shapley values [26].
Overview of the method
129
130
In this section, we first describe our Concept Activations Factorization method by pointing out the differences that set technique apart from previous work. We then proceed introduce the three new ingredients that make up CRAFT: (1) a method to recursively decompose concepts into sub-concepts. (2) and (3) how we unlock any attribution method to create Concept Attribution Maps, using implicit differentiation [38, 39, 40].
132
approach to better estimate the importance of extracted concepts
134
136
Notation this work, we consider a general supervised learning setting, where (æ1, .n) E d are points and (yI,..., Un)) E ) their associated labels. Unless specified, all points are assumed to have the same labels. We are given smachine-learn) black-box predictor f X + y, which at assume that f is neural network composed of & layers, and we denote f(x) = h2 o h o... o h2 (x) with (2(2) C R'' being the intermediate activations for the layer / and h;(x), an activation for the same layer. Further, we require non-negative activations: hx(3);; 2 0 Vi E (1....,)) P)). which amounts to choosing a layer whose activation function (x) 2 0. In particular, this assumption is verified by any architecture that utilizes ReLU, but any non-negative activation function works. Finally, we denote h4, the function going from the layer 7 to the output of the model f.
138
some test input æ predicts the output f(x). Without loss of generality,
140
142
144
146
3.1
Concept Activations Factorization
As illustrated in Fig.3, propose to Non- Von-negative matrix factorization activations find basis of concepts. Inspired by ACE [23], we will use sub-regions of images to attempt to identify coherent concepts. Instead of using segmentation which naturally introduces artifacts due the inpainting required by a baseline value we start by taking random crops of each image in our dataset (e.g, set of points that the model predicts belonging the same class) form an auxilary dataset X E Raad such that X, = T(x:) with T crop function. Given layer I, we obtain the activations for the random crops A hu (X) E Roxp, In the case where is a convolutional neural network, a global average pooling is applied on the activations. We recall that all the elements of A are non nen-negative real numbers.
148
150
152
153
154
155
196
We are now ready to apply Non Negative Matrix Factorization (NMF) to decompose the positive activations A, into product of non-negative, low rank matrices U (A) Raxr and W E Rexr, with:
158
min | -1--UW7]
(1)
W20
Where || denotes the Frobenius norm One of the appealing properties of NMF the low rank constraint & min(n. p). Simply put, NMF can be understood as the joint learning of W. a dictionary of CAVs "concept bank" Figure that maps a RRP basis onto R and U the coefficients of vectors A expressed in this new basis. The minimization of the reconstruction error
159
161

---

X
(x) U())"
(.)
Y
Concepts coofficients
Activations
Concept bank
Implicit differentiation
Sobol global importance
C C3
Figure 3: Overview of CRAFT. Starting from a set of crops X containing a concept C (e.g., crops images of the class Parachute), we send random crops to layer 1 to get activations it (X). We then factorize the activation into two lower rank matrices, U and W. W is what we call a concept bank (a base of concepts), while U corresponds to the coefficients in this new basis. We then extend the method with new ingredients: (1) the recursivity by proposing to re-decompose a concept (e.g.. take new set of point containing Ci) at an earlier layer I' < I, (2) a better importance estimation using Sobol indices and (3) leveraging implicit differentiation to generate Concept Attribution Maps allowing to localize concepts in an image.
2 - U ensures that the new basis contains (mostly) relevant concepts. Intuitively, the non negativitty constraints U 2 0, W 2 O encourage (i) the sparsity of W (useful for creating disentangled concepts), (ii) the sparsity of U (convenient for selecting minimal set of useful concepts) and (ii) the imputation of missing data [41]. which corresponds to the sparsity pattern of Relt- activations A. We shall also note that each original activation A, coming from the W. This
184
166
167
input x; can be approximated by its reconstruction (x:)) = U.W = approach is attractive as each activation can be understood as a composition of concepts.
168
169
170
While other methods in the literature solve similar problem (such as low rank factorization using SVD or ICA). the NMF has stepped up as both fast, effective and is known yield meaningful concepts to humans [42, 43, 24]. Finally, once the concept bank W is precomputed, we can associate
171
172
173
the concept coefficients (ax) to any new input x (e.g a full image) by solving the underlying Non-Negatin Least Squares (NNLS) problem minu20 / (l]((z)) U(x) WT WT P. and therefore
174
175
have its decomposition in the concept base.
178
In essence, the core of our method can be summarized as follows: using a set of images, we re-interpret their embedding at a given layer I as a composition of concepts that can be easily understood by humans. In the next section we show how we can recursively apply concept activation factorizations on a layer? 1' < for an image containing a previously computed concept.
177
178
179
180
3.2
Ingredient 1: A Recursive Flavor
181
One of the most apparent issues in previous works [23, 24] is the choice of the layer at which the activation maps are extracted. Depending on this, certain concepts start getting amalgamated [44]
182
183
into one, resulting in incoherent and indecipherable clusters, as illustrated in Fig 2. We posit that this can be solved by iteratively applying our decomposition at different layer-depths, and for the concepts that remain difficult to understand, look for their sub-concepts at carlier layers by isolating the images that contain them. This allows us to build hierarchies of concepts for each class.
194
185
186
187
We offer a simple solution consisting of reapplying our method to a concept by performing a second step of Concept Activation Factorization on set of points that contain the concept C in order to refine it and create sub-concepts (e.g., decompose C into (C1, C2, C2}) see see Fig.2 for an illustrative example. Note that we generalize current methods in the sense that taking points (æt 1. ..., xx,) that are clustered in the logits layer (belonging to the same class) and decomposing them in previous valid recursive step. For a more general case, let assume that a set of points that contain a common concept is obtained using a first step of Concept Activation Factorization. We then look for a set of points with high coefficient for the concept of our choice to perform the next factorization. Formally, with factorization for a layer I UW* and concept practice, we assume to be equal to the 90th percentile of the values of U. Given this new set of points, we can then re-apply the Concept Matrix Factorization method to a earlier layer with l' < L- to obtain the sub-concept's decomposition from the initial concept.
188
189
190
191
192
layer- as done in [23, 24]
193
194
195
index i, this set of points is defined as C = (T : UA,); ) Z A]
196
197
198
5

---

200
3.3
Ingredient 2: Sobol indices for enhanced concept importance estimation
201
A common concern with concept extraction methods is that what makes sense to humans is not necessarily what is being used by the model to predict. To avoid this kind of confirmation bias during our concept analysis phase, we can estimate the global importance of the extracted concepts. To do So, [22] proposed an estimator based directional derivatives: the partial derivative of the model output with respect to the concept vector. While this measure is theoretically well founded, it relies
203
205
206
on the same principle as gradient-based methods, and thus, suffers from the same pitfalls: neural network models have noisy gradients [5, 7]. Hence, the farther the chosen layer is from the output,
207
208
the noisier the directional derivative score will be.
209
Since we essentially want to know which concept has the greatest effect on the output of the model, it is natural to consider the field of Sensitivity Analysis [45, 46, 32, 47]. In this section, we briefly recall the classical total Sobol indices and how to apply it to our problem. The complete derivation of the Sobol-Hoeffding decomposition is presented in the appendix D.
211
212
213
Formally, we place ourselves layer/ and perform our Concept Activation Factorization, providing us with U. W. A natural way to estimate the importance of concept U, is to measure the fluctuation of the model's output h (UUT)) in response to meaningful perturbations of the concept coefficient U,. Concretely, with M- = (M .... M,) E [0, 1]", here an i.idd sequence of real-valued random variables, we introduce concept fluctuation to reconstruct perturbated activation Á = (U @ M) (e.g.. the masks can be used to put concept value to zero). We can then propagate this perturbated activation to the model output Y = h (A). Thus, an important concept will have a large variance on the model output while unused concept will barely change it.
215
218
219
220
221
Finally, we can capture the importance that a concept might have as a main effect- - along with its interactions with other concepts
the model's output by calculating the expected variance that would remain if all the indices of the masks except the M, were to be fixed. This yields the general definition of the Total Sobol indices.
223
The total Sobol index Sr. which measures the contribution of a concept U; as well as its interactions of any order with any other concepts to the model output variance, is given by:
Definition 3.1 (Total Sobol indices).
226
ST. EM (VM, (Y MM))) V(Y)
EMM.
(b @ ())T))M..) V(U S M)WT)
228 In a practical way, this index can be calculated efficiently [48, 49, 50, 51, 52], more details on the sampling (Quasi-Monte Carlo) and the estimator used are left in appendix D.
229
230
3.4 Ingredient 3: Unlocking Concept Attribution Map
Attribution methods useful for determining the regions deemed important by the model for the decision, but they lack the information about what exactly triggered it. We have seen that we can already extract this information from the matrices U and W, but as is, we cannot know to which part of the image the model associates each concept, and thus, better comprehend the model's decisions. In this section, we will show how we can unlock the set of attribution methods (forward and backward mode) to find where a concept is located in the input image (see Fig. Forward attribution methods don't 't rely on gradients and only use inference information, whereas backward methods require to back-propagate through the network's 's layers. By application of the chain rule, 2U
232
233
234
235
236
237
238
239
computing a0 dz
requires access to JA'
To do so, it could be tempting to solve the linear system UWT- = A. However, this problem is ill-posed since W is low rank. A standard approach is calculate the Moore-Penrose pseudo- inverse ( (WT)+, + which solves rank deficient systems by looking at the minimum norm solution [53]. In practice (W is computed with the Singular Value Decomposition (SVD) of WT. Unfor- tunately, SVD is also the solution to the unstructured minimization of 1| - UWT I by the
240
241
242
243
244
245
Eckart-Young. frounn-Mrrsyy theorem [54]. Hence, the non negativity constraints i. U O, W 2 0-of of
W = AT from succeeding. the NMF are ignored, which prevents approaches based on solving UT
246
247
Other issues stem from the fact that the U, W decomposition is generally not unique.
248
Our third contribution consists on tackling this problem to allow the use of attribution methods - i.e. Concept Atribution Maps - by proposing strategy to differentiate through the NMF layer.
249
250
Impicit differentiation of NMF layers The NMF problem NP-hard [55], and it is not convex with respect to the input pair (U, W). However, fixing the value of one of the two factors and

---

Church
Garbage truck
English springer
CRAFT
ACE
Figure 4: Qualitative comparison. We compare concepts found by our method (top) to those extracted with ACE [23] (bottom) for the classes Church, Garbage truck and English springer from ILSVRC2012 [27].
252 optimizing the other turns the NMF formulation into a pair of Non Negative Least Squares (NNLS) problems (see Equation 3), which are convex. This ensures that alternating minimization (a standard approach for NMF) of (U, W) factors will eventually reach a local (and global) minimum:
254
1 _ A U
U4+1 arg
arg min U20
W++ arg min 1 IA U,
(3)
W20
255
Each of the NNLS problem fulfills the KKT conditions(56, 57]. which can be encoded in the so-called
256
optimality function F, see Equation 10 Appendix C.2. The implicit function theorem [39] allows us
aU to use implicit differentiation [38, 39, 58] to efficiently compute the Jacobians DA and aW without
257
0A
258
requiring back- pook-prop through each of the iterations of the NMF solver:
a(U, W. W.U.W S.U.W) W) = (0, -0,
(4)
0A
However, this requires the dual variables U and W. which not computed by Scikit-learn's 's [59] popular implementation' Consequently, we leverage the work of [62] and we re-implement our own solver with Jaxopt [40] based on ADMM [63], a GPU friendly algorithm (see Appendix C.2).
280
262
We start by performing Concept Activations Factorization i.e we precompute the concept bank W by solving the NMF Concept Attribution Maps of new input æ are calculated by solving the NNLS problem minu [x) - I'? The implicit differentiation of NMF layer 2A is integrated into classical pack-propagation to obtain SU
2U
284
Most interestingly, this technical advance unlocks all white- box explainability methods [5, 6, 7, 9, 12] to generate concept-wise attribution maps and trace the part of the image that triggered the detection of the concept. Additionally, it is even possible to employ black-box methods [4, 13, 26, 14] since it only amounts to solving an NNLS problem.
Bay
286
268
Experimental evaluation
289
270
We used CRAFT to explain a ResNer50V2 trained on the ILSRVC2012 [27] data set (ImageNet). We selected subset of 10 classes, each containing 1000 images (those recommended by ImageNette") In all of our experiments, r = 25, like in [23] and the cropping function т consists on randomly choosing 10 square 64 64 patches for each image. We start by qualitatively validating CRAFT by showing that: (1) the method yields concepts that are easy to interpret (see Fig. 4), (2) the combination
271
272
273
274
275
of local and global explanations allows to explain complex failure cases otherwise unexplainable with only the attribution methods (see Fig. 5). Then, we validate independently the new ingredients brought by the method by showing quantitatively that (3) recursivity allows us to refine concepts, making them more meaningful to humans with the help of two psychophysies experiments, and (4) Sobol indices allow for better estimation of concept importance. Additional experiments, including
278
277
279
Scikit-leamn uses a Block coordinate descent algorithm [60, 61], with randomized SVD initialization. 'https:// httpg/ggttttttt/////egeeegtebbtt http//fasaaiimmmgeeee
7

---

Figure 5: This is a Shovel. We compare a heatmap generated by RISE [13] (left) with the Concept Attribution Maps generated with our implicit differentiation pipeline and Grad-CAM (right) on the explanations of the two most influential concepts that drove the ResNet50's decision. We found first concept that seems to be associated with textures of dirt commonly found in the images of the class Shovel. The second concept clucidated by CRAFT is located on the astronaut's pants, which he confuses with the ski suits of people clearing snow from their driveway with a shovel.
280 a sanity check and an example of activation maximization (Deep dream) on the concept bank, as well many other examples of local explanations for randomly picked images from ILSVRC2012, included appendix B.
282
We leave discussion on the limitations of this method and on the broader impact in appendix A.
4.1
Example of CRAFT concepts
285
Figure 4 compares the examples of concepts found by CRAFT against those found by ACE [23] for classes of Imagenet. For each class the concepts ordered by importance (the highest being the most important). ACE uses a clustering technique and TCAV to estimate importance, while CRAFT the method introduced in and Sobol to estimate importance. These examples illustrate one of the weaknesses of ACE: the segmentation used can introduce biases through the baseline value used [64, 10]. The concepts found by CRAFT seem distinct: (vault, cross, stained glass) for the Church class, (dumpster, truck door, two-wheeler) for the garbage truck, and (eyes, nose, fluffy ears)
287
289
290
291
292
for the English Springer. More examples can be found in the appendix.
293
4.2
Explaining complex failure cases
294
One of the goals of explainability is to explain the failure cases of the models studied. Figure shows example of an incorrect prediction: the model in question here still ResNet50 predicts 'shovel'. Moreover, the at- tribution method the left here RISE [13] does not tell us much Odds Ratios
Experts (n = 36)
Laymen (n 37)
295
Intruder
297
Acc. Concept
70.19%
61.08%
Acc. Sub-Concept
74.81% (p = 0.18)
67.03% (p = 0.043)
299
Hinary choice
Sub-Concept
76.1% (p 0.001)
74.95% (p < 0.001)
301
3.53
2.99
except that the evidence for shovel seems to be located at the level of the ground and the lower torso and legs
303
Table 1: Results from the psychopshysics experiments.
305
of the astronaut. With CRAFT, we can however study the concepts found by the model at these locations. There are two of them: the first concept in green, aims the lunar ground and refers the
306
307
rocks often seen next to shovels in the dataset. The second concept in purple is aimed at the legs of the astronaut and refers the legs of person, often in ski suit, which he takes for the astronaut's
308
309
4.3
Validation of Recursivity
To evaluate the meaningfulness of the extracted high-level concepts, we performed psychopshysic experiments with human subjects, to whom we requested to answer survey two phases. Further- more, distinguished two different audiences: the one hand, experts in machine learning, and on
311

---

313
the other hand, people with no particular knowledge in computer vision. Both groups of participants were volunteers and didn't receive any monetary compensation. Some examples of the developed interface are available the appendix E.
314
315
316
Intruder detection experiment, we make users identify the intruder out of a series of five segments belonging to a certain class, with the odd one being taken from different concept but from the same class. Now, we compare the results of this intruder detection with a concept (e.g., C1) coming from a layer and one of its sub-concepts (e.g., C12 in Fig.2) extracted using our recursive method. If the concept (or cub-concept) is meaningful, then it should be easy for the users to find the intruder. Table I summarizes our results, showing that indeed both concepts and sub-concepts are meaningful, and that recursivity can lead to a slightly higher understanding of the generated concepts (significant for non-experts, not significant for experts) and might suggest a way to make concepts more interpretable.
318
320
322
323
324
Binary choice experiment, In order to test the improvement of the meaningfulness of the sub- concept generated with recursivity with respect to the larger parent concept, we showed participants a segment belonging to subcluster and to the parent cluster (c.g., ((x) C C11 C C1) without specify- ing why those images are grouped together. We then we asked which of the two clusters (i.c., C₁ or C1) seemed to accommodate the image the best. If our hypothesis is correct, then the concept refinement brought by recursivity should help form more coherent clusters. The results in Table are satisfying. since in both the expert and non-expert groups, the participants chose the sub-cluster by more than 74% of the times. We measure the significance of our results by fitting a binomial logistic regression to our data, and we find that both groups more likely to choose the sub-concept cluster (at p < 0.001).
325
327
329
331
332
333
334
4.4
Fidelity analysis
335
We propose to simultaneously verify that the concepts are faithful to the model and
loghts score
0.8
1.0
Sobol KNF
Ttav XMF
12
1.0
336
337
that the concept importance estimator per- forms better than TCAV [22] by using the fidelity metrics introduced in [23, 24]. These metrics are similar to the one used for attribution methods, which consist on studying the change of the logit score when removing/adding pixels considered impor- tant. Nevertheless, we do not make these modifications in the pixel space but in the concept space: once U, W are computed, we reconstruct the matrix A R UWT us-
0.6
0.8
0.4
338
0.2
0.4
Tat KMF
340
0.0
0.2
Sobol NVF
10
20
15 # of Concepts
20
25
3 removed Concepts
342
Figure 6: (Left) Deletion curve (lower is better). (Right) Insertion curves (higher is better). Whether in deletion or insertion, the score calculated on more than 100,000 images - shows that using Sobol indices yield to better estimates of important concepts.
344
345
346
348
ing only the most important concept (or removing the most important concept for deletion), and study the score in output of the model. As
349
350
can be seen from Fig. 6, ranking the extracted concepts using Sobol's importance score results in much steeper curves than when they are sorted by their TCAV scores. We confirm these results with other matrix factorization techniques (PCA, ICA, RCA) in the Appendix F.
351
352
Conclusion
353
354
In this paper, we introduced a method for automatically extracting numan-scrutabll concepts from Deep Neural Network: CRAFT. Our method allows to explain pre-trained model both in a per-class and per-instance basis by highlighting both what the model saw when predicting the class label and where it is located, which, as we have shown, exhibits complementary benefits. The approach relies on three novel ingredients: 1) exploiting the recursive nature of the feature extraction chains in CNNs to find decompositions where each concept is clearly understandable; 2) measuring concept importance through Sobol indices to more accurately identify which concepts influence a model's decision for given class; and 3) harnessing implicit differentiation to backpropagate through NMF blocks, thus enabling the use of any attribution method to create concept-wise local explanations that
356
357
358
360
362
we call Concept Attribution Maps. Human experiments confirmed the validity of the approach and that concepts identified by CRAFT are meaningful. We hope that this work will guide further efforts
364
365
in the search for concept-based explainability methods and that further connections between local
366
and global explanations will be made.
9

---

References
367
388
[1] Margot E Kaminski and Jennifer M Urban. The right to contest ai. Columbia Law Review, (1(7))1957-2045 2021. 1
370
[2] Mauritz Kop. Eu artificial intelligence act: The european approach ai. Stanford-Vienna Transatlantic Technology Law Forum, Transatlantic Antitrust... 2021. 1
371
372
[3] Christoph Torens, Umut Durak, and Johann C Dauer. Guidelines and regulatory framework for machine learning in aviation. AIAA Scitech 2022 Forum, page 1132, 2022.
373
374
[4] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135- 1144, 2016. 1, 3, 7, 24
378
377
[5] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint (xXiv:170.00382, 2017. 1, 2. 6, 28
378
379
380
[6] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In In Workshop at International Conference Learning Representations. Citeseer, 2014. I, 2,
381
[7] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319-3328. PMLR, 2017. 1, 6,
384
385
[8] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017. 2,
386
387
388
389
99] Jost Tobias Springenherg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 2014. 1,
[10] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3433 2017. 1, 2,8
392
383
394
[11] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2950-2958, 2019. 1
395
396
397
[12] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2921-2929, 2016. 2,
399
400
[13] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07422, 2018. 3, 3, 7, 8, 24, 26
401
402
[14] Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with basel sensitivity analysis. Advances in Neural Information Processing Systems, 34, 2021. 1, 3,7,2
403
404
405
[15] Thomas Fel, Mclanie Ducoffe, David Vigouroux, Remi Cadene, Mikael Capelle, Claire Nicodeme, and Thomas Serre. Don't lie to me! robust and efficient explainability with verified perturbation analysis. arXiv preprint arXiv:2202.07728 2022.
407
408
[16] Thomas Fel, Julien Colin, Rémi Cadène, and Thomas Serre. What cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. arXiv preprint arXiv:2112.04417, 2021. 2
409
410
[17] Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. b/bui///////iiiiiigg-bbbbbaaba 2,3
412
10

---

414
[18]
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. 2, 3, 16, 21
415
https://distill.pub/200
416
[19]
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
417
418
[20]
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020. 2
420
421
[21] Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018. 2
422
423
[22]
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 1677. PMLR, 2018. 2, 3, 6, 9, 21, 26
425
427
[23] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Neural Information Processing Systems, 32, 2019. 2, 4,
429
8, 99, 26
[24] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for enn models with non-negative concept activation vectors. arXiv preprint arXiv:2006. 1030006..544 15417, 2020. 2, 5,5,9,2
432
433
434
[25] Thomas Fel, Mélanie Ducoffe, David Vigouroux, Rémi Cadène, Mikael Capelle, Claire Nicodème, and Thomas Serre. Don't lie to me! robust and efficient explainability with verified perturbation analysis. arXiv preprint arXiv:2202.07728 2022. 2
436
437
[26] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. 2, 4,7
438
439
[27] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009, 3, 4, 7, 7,26,,7 27
440
441
[28] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Müller, and Wojciech Samek. On pixel-wise explanations for non-lineat classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1 07 2015.
443
445
[29] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-888888 18-333. Springer, 2014. 2
446
[30] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018. 28
448
419
450
[31] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180-186, 2020. 2
452
453
[32] Ilya M Sobol. Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and computers in simulation, 55(1-3):271-280 2001. 3, 6
454
455
[33] Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 13:00000-19015, 2020. 3
457
[34] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 2018. 3
459
11

---

461
[35] Quanshi Zhang, Ying Nian Wu, and Song-Chun Chun Zhu. Interpretable convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition.
462
pages 8827-8836, 201 8.
464
[36] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019. 3, 16
465
466
467
[37]
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awee aware concept-ban based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554 2020. 4
468
469
470
[38] Steven George Krantz and Harold R Parks. The implicut function theorem: history, theory, and applications. Springer Science & Business Media, 2002. 4, 7, 24
472
[39] Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008. 4, 7, 24
474
[40] Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-
475
López, Fabian Pedregosa, and Phinlppil Vert. Efficient and modular implicit differentiation.
476
arXiv preprint arXiv:2105.15183, 2021. 4, 24
477
[41] Bin Ren, Laurent Pueyo, Christine Chen, Élodie Choquet, John H Debes, Gaspard Duchêne, François Ménard, and Marshall D Perrin. Using data imputation for signal separation in high-contrast imaging. The Astrophysical Journal, 892(2):74, 2020. 5
478
479
480
[42] Yu-Xiong Wang and Yu-Jin Zhang. Nonnegative matrix factorization: A comprehensive review. IEEE Transactions on Knowledge and Data Engineering, 25(6): 2013. 5
481
482
[43] Xiao Fu, Kejun Huang, Nicholas D Sidiropoulos, and Wing-Kin Ma. Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications. IEEE
483
484
Signal Process. Mag., 36(2):59-B))) 80, 2019. 5
485
[44] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40)224622 24662,,3,, 2020. 5
486
487
488
[45] Bertrand Iooss and Paul Lemaître. A review on global sensitivity analysis methods. In Uncer- tainty management comulaation-opimmzaatii of complex systems, pages 101-122. Springer, 2015. 6
489
491
[46] Ilya M Sobol. Sensitivity analysis for non- mathematical models. Mathematical modelling and computational experiment, 1:407-414, 1993. 6, 25
492
493
[47] RI Cukier, CM Fortuin, Kurt Shuler, AG Petschek, and Ho Schaibly. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. theorry The Journal of chemical physics, 5987:387333387, 1973.
494
495
496
[48] Andrea Saltelli, Paola Annoni, Ivano Azzini, Francesca Campolongo, Marco Ratto, and Stefano Tarantola. Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index. Computer physics communications, 18 181(2)2559-770 2010. 6
437
499
[49] Amandine Marrel, Bertrand Iooss, Beatrice Laurent, and Olivier Roustant. Calculations of sobol indices for the gaussian process metamodel. Reliability Engineering & System Safety, 94(3)774--51, 2009.
501
[50] Alexandre Janon, Thierry Klein, Agnes Lagnoux, Maëlle Nodet, and Clémentine Prieur. Asymp- totic normality and efficiency of two sobol index estimators. ESAIM: Probubility and Statistics, 18:342-364, 2014. 6, 25
503
[51] Art B Owen. Better estimation of small sobol' sensitivity indices. ACM Transactions on 505
Modeling and Computer Simulation (TOMACS), 23(2):1-17, 2013. 6
12

---

507
[52] Stefano Tarantola, Debora Gatelli, and Thierry Alex Mara. Random balance designs for the estimation of first order global sensitivity indices. Reliability Engineering & System Safery. 91(6):717-727, 2006. 6
508
509
510
[53] João Carlos Alves Barata and Mahir Saleh Hussein. The moore- penrose pseudoinverse: A tutorial review of the theory. Brazilian Journal of Physics, 42(1)):14-165 2012.
511
512
[54] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936. 6
513
514
[55] Stephen A Vavasis. On the complexity of nonnegative matrix factorization. SIAM Journal on Optimization, 2037 6
515
[56]
William Karush. Minima of functions of several variables with inequalities as side constraints.
517
M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939. 7, 23
518
[57] Harold W Kuhn and Albert W Tucker. Nonlinear programming proceedings of the second berkeley symposium on mathematical statistics and probability, Neyman, pages 481-492, 1951. 7. 23
520
521
[58] Bradley M Bell and James Burke. Algorithmic differentiation of implicit functions and optimal values. In Advances in Automatic Differentiation, pages 67-77. Springer, 2008. 7, 24
523
[59] Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of muchine learning research, 12(0ct):2825-2830 2011. 7
524
526
527
[60]
Andrzej Cichocki and Anh-Huy Phan. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE transactions on fundamentals of electronics, communications
528
529
and computer sciences, 92(3):70-- 2009. 7, 24
[61] Cédric Févotte and Jérôme Idier. Algorithms for nonnegative matrix factorization with the 530
8-divergence. Neural computation, 23(9):2421-2455) 2011. 7, 24
532
[62] Kejun Huang, Nicholas D Sidiropoulos, and Athanasios P Liavas. A flexible and efficient algorithmic framework for constrained matrix and tensor factorization. IEEE Transactions on Signal Processing, 64(19):5052-5000 -00555555555555..-- 2016. 7.23
533
534
535
[63] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed opti- mization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3112: 2011. 7, 16, 23, 24
536
537
538
664] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020. 8
539
665]
Kaiyu Yang, Jacqueline Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfuscation in imagenet. arXiv preprint arXiv:2103.06191, 2021. 14
541
542
666]
Yunhao Ge, Yao Xiao, Zhi Xu, Meng Zheng, Srikrishna Karanam, Terrence Chen, Laurent Itti, and Ziyan Wu. A peek into the reasoning of neural networks: Interpreting with structural visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2195-2204, 2021. 16
544
[67] Thomas Fel, Lucas Hervier, David Vigouroux, Antonin Poche, Justin Plakoo, Remi Cadene, Mathieu Chalvidal, Julien Colin, Thibaut Boissin, Louis Béthune, Agustin Picard, Claire Nicodeme, Laurent Gardes, Gregory Flandin, and Thomas Serre. Xplique: A deep learning explainability toolbox. Workshop, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 21
547
549
[68] Magnus R Hestenes and Eduard Stiefel. Methods of conjugate gradients for solving. Journal of research of the National Bureau of Standards, 49(6)4409, 1952. 23, 24
$52
553
[69] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 630-645. Springer, 2016. 26
13

---

Checklist
556
S56
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
558
contributions and scope? [Yes]
559
(b) Did you descrihe the limitations of your work? [Yes]
560
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
582
563
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
565
(b) Did you include complete proofs of all theoretical results? [Yes] In the appendix.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] As URL
568
569
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [N/A]
570
571
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [Yes] For the experiments comparing TCAV scores to our concept importance score based on Sobol indices
572
573
574
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
577
(a) If your work uses existing assets, did you cite the creators? [Yes]
578
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
580
581
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
582
583
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] The ILSVRC2012 dataset contain personally identifiable information [65]
585
586
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] Screenshot of the experiments are in the appendix
588
509
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
590
591
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
14


Reviews:
[{"review_id": "tgD-EtHK4dS", "reviewer": "Reviewer_B8eQ", "summary": "This work presents a novel method to provide local and global explanations for the decision process of neural convolutional models. Their approach tries to divide agglomerated concepts from later layers into disentangled sub-concepts in earlier layers to provide more human-comprehensible explanations/ concepts. For this purpose, they make use of three “ingredients”, namely 1) a reiteration to find early-stage concepts that are human-comprehensible, 2) an approach from the variance-based sensitivity analysis, the Sobol indices, to estimate concept importance in the input for the prediction and 3) a method to provide an attribution map through implicit differentiation for the found concepts such that we know where the concept is contained in the input. Experiments, including some ablations studies and a user study, are provided to prove the function of the proposed method.", "questions": "Suppose the decision is made based on these concepts. How exactly does your method differ and compare to prototype networks? To some extent, a concept seems similar to a prototype, especially as you show multiple prototypical (parts) examples when you want to answer the “what” question. \n\n\nFurther remarks not affecting the evaluation:\n\n- link to anonymous GitHub does not work\n- Line 64 meanignful -> meaningful\n- Line 105 vectorin -> vector in\n- In general, quite some typos, capitalization, and punctuation errors\n- Please incorporate fig 7 in your main text if you refer to it multiple times\n- The appendix is a bit messy; many typos and references to figures are missing, etc.\n- Line 245 What are KKT conditions?", "limitations": "No real discussion on limitations. Neither from a technical nor from a societal view. Also no discussion on future work. That’s a big downside of this work, especially if you motivate your work with societal reasons, i.e. “the right to explanations” for end-users. What impact could your method have?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- S1 Using the NMF to find promising concepts is a good idea, especially to only find sparse and meaningful concepts. Using Sobol indices for feature importance seems sensible. Also, combining the three ingredients to solve this task is exciting and overall yields a novel approach to explainability. This work presents a multitude of solid mathematical solutions for practical problems in this task.\n- S2 The figures used to conceptually show the function of their method is very good and intuitive.\n- S3 The story is in general clear.\n\n**Weaknesses**:\n- W1 Overall, the experimental evaluation is weak. Many claims are only shown conceptually, especially comparing ACE and the failure cases. That is a big drawback. Also, the user study is very sparse. It shows that sub-concepts help to a small degree in finding an intruder. Attributing this finding to more meaningful concepts is a vast step. Also, no comparison of what other methods achieve is not given.\n- W2 The work claims that the method rather answers what than where, but this question remains unanswered despite intuitively making sense. The idea is to not only highlight where a concept is attributed in the input but also what the concept looks like. However, still, it is hard to understand what the concept is. The shovel examples left me somewhat confused. I agree that this approach targets the “what” but only to a limited extent. Here, I am missing a discussion on this claim.\n- W3 In general, a fair discussion on limitations like computational expense, advantages, and disadvantages, in general, are missing. How efficient is this method? How computationally expensive is this method? If it only works with ReLUs, that is a significant drawback; what could be the following steps to work with other activation functions?\n- W4 The intro motivation is a bit weak. Explainability is essential, yes. However, as far as I understood, the GDPR’s “right to explanations” does not ask for an explanation this work is proposing. The GDPR is mainly about privacy rights, which means the “right to explanations” refers to the data a model is trained with, which your work does not really target. Instead, the proposed European AI act targets the “right to explanations” in your way.\n\nBased on the exciting idea I tend to accept this paper. However, there are concerns about the evaluation. If the authors address these concerns, e.g. on the missing limitation discussion, I am happy to change my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "anXUsoZmW48", "reviewer": "Reviewer_tkuX", "summary": "In this paper the authors propose the “Concept Recursive Activation FacTorization” (CRAFT), a framework to generate concept-based explanations from trained networks. CRAFT aims at addressing current limitations in state-of-the-art concept-based techniques (e.g. ACE [1]) including: (i) the association of *one concept only* for each image segment/patch, (ii) the instability/noise of gradients w.r.t. concept vectors, (iii) the localization of concepts in the input space to provide local explanations. To address these limitations, the authors equip CRAFT with three methods: (i) a recursive procedure to automatically decompose concepts into sub-concepts, (ii) an alternative scoring function for concept importance based on Sobol indeces, and (iii) a method to localize concepts in the input (pixel) space. Following Zhang et al. [2], CRAFT identifies coherent bases of concepts using the Non-negative Matrix Factorization (NMF) rather than k-Means as in ACE [1]. In the experiments the authors show that CRAFT can: (i) identify meaningful and coherent concepts, (ii) localize concepts in the input space allowing users to explain misclassified images, (iii) decompose high-level concepts into coherent subconcepts, and (iv) provide a more robust ranking concept w.r.t. TCAV in terms of  fidelity scores.\n\n[1] Ghorbani, Amirata, et al. \"Towards automatic concept-based explanations.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[2] Zhang, Ruihan, et al. \"Invertible concept-based explanations for cnn models with non-negative concept activation vectors.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 13. 2021.", "questions": "Addressing the main limitations mentioned above would significantly improve the paper and, as a consequence, reviewers’ feedback and scores. However, I understand that the authors may not have enough time to address them all in the rebuttal. For this reason I list here a few minor items which can give the authors a better chance to improve their work in the short term:\n1. A few sentences may need further clarification as they may contain typos or have grammar issues (e.g. ““The second concept elucidated by craft is located on the astronaut’s pants that **he confuse** with the ski suits” (Figure 5)). The authors may also improve the flow using the active voice more frequently (e.g. “Namely, in [38] and [39], explanations are generated as the inputs” (line 97)). Other sentences which may benefit from rephrasing include: “Once the factorization done, and the reconstruction of the activation of the image can then be interpreted as a combination of a set of concepts and a coefficient associated to these concepts.” (line 116), “A common concern with concept extraction methods is that make sense to humans does not mean it” (line 189), “Finally, we can capture the importance that a concept might have as a main effect – by itself – as well 210 as by interacting with other concepts on the model’s output by calculating the expected variance that 211 would remain if all the indices of the masks except the Mi were to be fixed.” (line 209). \n2. A few acronyms need to be spelled out (e.g. TCAV, line 112).\n3. The authors reference Figure 3 before Figure 2. I would suggest to swap the figures.\n4. Some claims may require to be rephrased as in their current form may not align with actual results. For instance, I find the possibility of using CRAFT for explaining complex failure cases quite exciting, but the only experimental result is on one cherry-picked example which may not be enough to prove the efficacy of the method (Section 4.2).  Another example is when authors state “We use our approach and combine local and global explanation, to *accurately* explain predictions” (line 72). Here I was expecting a *quantitative* result in the experimental section showing the actual *accuracy* scores of CRAFT’s explanations. A third example is when authors state that “CRAFT yields more interpretable concepts than ACE” (line 74). The main issue here is that ACE and CRAFT are different under many aspects. These differences make a fair comparison a bit tricky as each difference may introduce a confounding factor in the comparison. However, the results here are only qualitative (Figure 4 and appendix) which makes the claim a bit weaker than it could have been. Moreover, the experiment design does not allow to identify which ingredient (i.e. Sobol indeces? NMF? or $\\tau$?) from ACE or CRAFT is the one that is actually making the difference. A fourth example is when authors state “These examples illustrate one of the weaknesses of ACE: the segmentation used can introduce biases through the baseline value used. Moreover, this segmentation does not facilitate the interpretation of the concepts” (line 276). However the evidence here is only based on a qualitative visual comparison which may not be enough to support such strong claims.", "limitations": "As mentioned before, the authors may consider to include a description for the limitations of their work. Including relevant limitations would significantly improve the paper as it would allow other scientists to understand how to use and improve CRAFT advancing the research field. A possible limitation worth mentioning is that the experimental evidence is mostly qualitative (Section 4.1, 4.2, and 4.3) and/or possibly subject to confirmation or selection biases (Section 4.3).", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#  Strengths\nThe automatic decomposition of high-level concepts into sub-concepts is the most interesting contribution of this paper w.r.t. existing concept-based approaches. This is a significant step forward w.r.t. state-of-the-art approaches such as ACE as it enables the identification of coherent hierarchies of concepts. In particular concept hierarchies may facilitate human experts in navigating the concept space. Even though the methods are not new (e.g. NMF, or Sobol indeces), they are effectively used in new contexts. In addition, the experiments show how CRAFT can be used in practice as a powerful tool to “debug” misclassified samples by localize concepts in the input space.\n\n# Limitations\nWhile this paper introduces a few interesting steps forward, the authors may consider a few suggestions for improving their contributions. In particular the authors may consider to:\n1. Broaden the experimental section as it is currently based on one dataset only, thus questioning the generality of the results. I would strongly encourage the authors to broaden their experiments to include additional benchmarks commonly used in concept-based XAI papers (e.g. CUB [1], TabulaMuris [2], or CelebA [3]).\n2. Include a description for the limitations of their work. Including relevant limitations would significantly improve the paper as it would allow other scientists to understand how to use and improve CRAFT advancing the research field. A possible limitation worth mentioning is that the experimental evidence is mostly qualitative (Section 4.1, 4.2, and 4.3) and/or possibly subject to confirmation or selection biases (Section 4.3).\n3. Include critical information which is currently missing for understanding and reproducing the experiments. For instance, the authors do not mention what error bars represent (e.g. Figure 6) nor how the images where selected for the human experiments i.e., they just mention that “the choice was randomized” (line 701), but they do not specify how (e.g. random selection from cherry-picked examples or from all images? From groups of image crops or from the original images?). Another issue is that it is not clear how the authors pick CRAFT hyperparameters (e.g. $\\tau$, line 263) and how sensitive the results are to hyperparameters’ changes.\n4. Reorganize some sections, include missing information, or rephrase a few sentences to improve flow and clarity. For instance, in the abstract the authors do not mention any knowledge gap which makes a bit unclear why CRAFT might be needed. On the contrary, in the introduction the authors mention four knowledge gaps (lines 31, 41, 47, and 51) which makes a bit unclear what is the *main* purpose of CRAFT. Making the introduction and contributions more focused would significantly improve flow and clarity and it would save the authors some space for describing methods, results, and limitations.\n\n[1] Wah, Catherine, et al. \"The caltech-ucsd birds-200-2011 dataset.\" (2011).\n\n[2] Cao, Kaidi, Maria Brbic, and Jure Leskovec. \"Concept learners for few-shot learning.\" arXiv preprint arXiv:2007.07375 (2020).\n\n[3] Liu, Ziwei, et al. \"Large-scale celebfaces attributes (celeba) dataset.\" Retrieved August 15.2018 (2018): 11.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Yz0llkIbY1R", "reviewer": "Reviewer_KNMa", "summary": "The authors have done a nice job by improving the concept based explanation generation techniques from different perspectives. An automatic concept extraction followed by a recursive procedure towards achieving a set of concepts and sub-concepts are proposed in this work. A concept-based heatmap generation method is also proposed here that backpropagate concept values to pixel space that is very similar to widely used class-based saliency methods.", "questions": "L31: What do you mean by information methods?\n\nL73-74: If some figure in the supplementary section is referred to in the main paper, it is a good practice to clearly mention that.\n\nL137: I request the authors to explain the notation h_{l,k}, as l and k represent some layer and the total number of layers respectively, but it was not clear to me how they're used in a single notation.\n\nL163: Please write the full form when the short form is used for the first time in the paper. The full form of NNLS (i.e. Non Negative Least Square) should have been mentioned here.\n\nOther questions: \na) Please explain the notations of Table 1 in detail. It is not very clear what 'n' represents here.\n\nb) Here the concepts found by NMF are claimed to be human understandable. If any or some of the concepts are exceptions, I am interested to know whether these concepts are specific to a particular category?\n\nc) As the gradient based explanation generation techniques generally produce noisy saliency maps, did you see any such phenomenon for concepts? Do the heatmaps for all the images are as nice as shown in fig.1? If yes, which part of the method (related to concept attribution map generation) helped you to achieve that specifically?", "limitations": "The authors did not add any section to address the limitations and potential negative societal impact of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \na) Several important directions are worked on here, that start with concept generation in a recursive manner from multiple layers of the network. This technique helps the cases where concepts generated from a very deep layer don't make much sense and decomposing them with sub-concepts can develop an improved understanding of the model. \n\nb) This is the first method, I believe, to work on saliency based explanations for each concept. The authors have shown the effectiveness of this approach by explaining different concepts in a single image. \n\n\nWeaknesses: \na) Although the quality of a technical literature shouldn't be judged by the language quality, I still believe that there are many scopes to polish the language and take care of some typos to improve the readability of the paper.\n\nb) I couldn't access the code related to the paper from the provided link. If by any chance the code was not ready for sharing by the submission time, it would have been better to skip that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "RWgq8DnvFU7", "reviewer": "Reviewer_p9rf", "summary": "This paper proposed a concept-based explainability method, CRAFT, to generate both global and local concept-based explanations.\nThis paper introduces some key contributions: (1) recursively decomposing higher-level concepts into more elementary ones.\n(2) concept importance estimation based on Sobol's indices. (3) Concept Attribution Maps, which connect global concepts to local image level explanation.", "questions": "(1) How could we use the explanation to explain the error cases made by the model?\n\n(2) It would be good to compare with a related paper: \"A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts\" which also provides both global and local concept-level explanations.", "limitations": "More quantitative experiments to show the performance and compare it with other baselines would be better.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n(1) Concept-level explanation is a challenge and an important direction. The proposed method is useful and novel, especially for the recursive decomposition to get different level concepts and the linking between global high-level concepts and local attribution map.\n\n(2) The three ingredients based on the concept activation factorization are promising and inspiring.\n\n(3) The paper is well organized and easy to follow.\n\n**Weaknesses**\n\n(1) How to verify the faithfulness of the discovered concepts? What if the human-understandable concepts are not the correct reasoning logic for the original neural networks.\n\n(2) How to evaluate the scalability to larger classes? \n\n(3) Are there quantitative results to show the performance and compare it with other baselines?\n\n==========================Pose Rebuttal=============================\n\nThanks for your feedback. I will keep my original score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject