Uncovering Unique Concept Vectors through Latent Space Decomposition
Authors: Mara Graziani
Laura O'Mahony
An-phi Nguyen
Henning Müller
Vincent Andrearczyk
Venue: Accepted by TMLR ()

Abstract: Interpreting the inner workings of deep learning models is crucial for establishing trust and ensuring model safety. Concept-based explanations have emerged as a superior approach that is more interpretable than feature attribution estimates such as pixel saliency. However, defining the concepts for the interpretability analysis biases the explanations by the user’s expectations on the concepts. To address this, we propose a novel post-hoc unsupervised method that automatically uncovers the concepts learned by deep models during training. By decomposing the latent space of a layer in singular vectors and refining them by unsupervised clustering, we uncover concept vectors aligned with directions of high variance that are relevant to the model prediction, and that point to semantically distinct concepts. Our extensive experiments reveal that the majority of our concepts are readily understandable to humans, exhibit coherency, and bear relevance to the task at hand. Moreover, we showcase the practical utility of our method in dataset exploration, where our concept vectors successfully identify outlier training samples affected by various confounding factors. This novel exploration technique has remarkable versatility to data types and model architectures and it will facilitate the identification of biases and the discovery of sources of error within training data.

Published in Transactions on Machine Learning Research (November/2023)
Uncovering Unique Concept Vectors Space Decomposition
through Latent
Mara Graziani
mara.graziani@ hevs.ck
Institute of Informatics
University of Applied Sciences of Western Switzerland (Hes-so Valais)
Sierre, CH 3960, Switzerland
Laura O' Mahony
LauraA. OMahony @ ul.ie
Department of Computer Science and Information Systems University of Limerick
Limerick, V94 T9PX, Ireland
An-Phi Nguyen
an-phi.nguyen@@ biognosys.com
Biognosys
Zurich, CH 8952, Switzerland
Henning Müller
meminng.mulller@hert
Institute of Informatics
University of Applied Sciences of Western Switzerland (Hes-so Valais) Sierre, CH 3960, Switzerland
Department of Radiology and Medical Informatics
University of Geneva
Geneva, CH 1211, Switzerland The Sense Innovation and Research Center Sion, CH 1950, Switzerland
Vincent Andrearczyk
Institute of Informatics
University of Applied Sciences of Western Switzerland (Hes-so Valais) Sierre, CH 3960, Switzerland
Reviewed OpenReview: http://///geeeeeeee.ee//////rror
Abstract
Interpreting the inner workings of deep learning models is crucial for establishing trust and ensuring model safety. Concept-based explanations have emerged as a superior approach that is more inter- pretable than feature attribution estimates such as pixel saliency. However, defining the concepts for the interpretability analysis biases the explanations by the user's expectations on the concepts. To address this, we propose a novel post-hoc unsupervised method that automatically uncovers the concepts learned by deep models during training. By decomposing the latent space of layer in sin- gular vectors and refining them by unsupervised clustering, uncover concept vectors aligned with directions of high variance that are relevant to the model prediction, and that point to semantically distinct concepts. Our extensive experiments reveal that the majority of our concepts are readily understandable to humans, exhibit coherency, and bear relevance to the task at hand. Moreover, we showcase the practical utility our method in dataset exploration, where our concept vectors successfully identify outlier training samples affected by various confounding factors. This novel exploration technique has remarkable versatility to data types and model architectures and will facilitate the identification of biases and the discovery of sources of emrr within training data.

---

Published in Transactions on Machine Learning Research (November/2023)
1 Introduction
Understanding and explaining the inner workings of deep learning models is challenging, yet of outmost importance in high-risk applications where reliable and intuitive explanations are crucial for decision making. Model validation plays a vital role in ensuring trustworthy predictions and to avoid actions that may negatively impact society.
Because of their ease of use and understandability, the use of user-defined queries to inquire about the relevance of high-level concepts such as objects, shapes, and textures, has shown great promise in model validation. These concept- based queries are addressed through explainability methods that leverage concept vectors, representing vectors in the latent space of a layer that are representative of concept (Kim et al. (018)) Methods further developing on concept vectors were proposed for multiple applications and tasks (Goyal et al. 2019 Graziani et al. 2020 Koh et al., 2020) Chen et al. 2020). However, concept vectors are generally obtained as a response to user-defined queries and are, as such, biased towards the user's knowledge and expectations, a phenomenon known as the experimenter bias (Rosenthal & Fode 1963). Users may bias the analysis by only querying the model for some concepts while neglecting others. The exhaustive coverage of all possible concepts, besides, is unfeasible and in some domains such as biology chemistry, the concepts may be unclear difficult to define. Because of this, model interpretability with concept vectors is, as of now, limited to concepts that are well-identifiable and easy to label.
Achieving broader interpretability with concept vectors requires a reverse engineering approach that focuses on au- tomating concept identification. In this case, the only assumption made on the concepts is that they are relevant for the models' downstream task. Few methods have ventured in the unsupervised exploration of imaging representations through activation maximization Erhan al. al.] (2009); Olah al. (2017: 2020), clustering of the latents (Ghorbani et al. 2019. Klindt al., 2023) matrix factorization McGrrah al. (2022); ; Fell et al. (2023), but only limited analyses exist that match the discovered concepts with the downstream task activity (Fel et al. 2023 Considering the model's sensitivity to a concept, however, is essential to ensure that the concept discovery method focuses only on directions in the latent space that impact the model's output. In this work, we propose a we propose novel framework that merges factorization and clustering of the latents together with output-sensitivity for concept discovery. As a result, we are able to isolate directions in the latent space that respond to vell-distnnuuishable unique concepts. Our novel approach can be applied with high versatility across multiple architectures, tasks, and data types.
To evaluate the effectiveness of our concept discovery approach, we apply it to the task of natural image classification. The discovered concepts align with human-understandabl interpretations and demonstrate consistency with previous research findings. For instance, texture and object parts emerge as prominent in vision models, reflecting the work of Kim et al. (2018) on recognizing zebra stripes and Ghorbani et al. (2019) on detecting vehicle parts to classify police vans. Furthermore, user evaluation studies confirm that the discovered concepts are easy to understand, and show that they provide valuable insights about the model's decision making process.
2 Related Work
This work aims at explaining already trained models, focusing on post-hoe interpretability. Despite having acknowl- edged limitations (Rudin) 2019 Adebayo et al. 2018)), post-hoc approaches allow gather important insights de-facto standard models for vision tasks such as Inception (Szegedy et al. 2016) and ResNet (He et al. 2016), which are largely reused in real-world applications.
Post-hoc methods for vision models are numerous, and were categorized according to different taxonomies and crite- ria GGazziani et al. 2023). Four main groups can be identified covering (i) feature attribution, (ii) feature visualization. (iii) concept attribution and (iv) surrogate explanations. Feature and concept attribution methods, i.e. (i) and (iii) respectively, aim identifying what characteristics in the data are the most related to the prediction. In the case of (i), this is done in the input space, with methods that evaluate the impact of perturbations by accessing the model gradients or estimating occlusion sensitivity. De-facto standard approaches are Class activation maps (CAM) (Zhou et al. 2016) and the gradient-weighted variant Brad-CAM by Selvaraju et al. (2017). Grad-CAM was actually demonstrated to be generalization of CAM up to a normalization constant. Feature visualization (ii), initially proposed as activation maximization by Erhan et al. (2009) and further developed in the works Olah et al. (2017) and Olah et al. (2020). aims at uncovering the patterns that are learned at intermediate layers through visualizing what maximally activates each model unit. Finally, surrogate methods (iv) refer to techniques that require surrogate model to generate expla- nations Ribeiro al. (2016) The idea of concept attribution (iii) is particularly related to this work. Concept activation
2

---

Published in Transactions on Machine Learning Research (November/2023)
vectors (CAVs) Kim et al. 2018) were the first post-hoc method to introduce the hypothesis that linear combinations of units carry information on user-defined high-level concepts. Ghorbani et al. further extended the method to remove the need for user-defined queries (Ghorbani et al. 2019). The algorithm identifies image segments that cluster in the latent space, and uses CAVs to determine if the direction of a cluster is a relevant concept for the model. The identified vectors, however, depend on the quality of the image segments, which is not guaranteed for medical images (Graziani et al. 202T ). Moreover, CAVs are unlikely to be orthogonal in the latent space. Counterintuitively, two CAVs for unrelated concepts may be closely aligned in the latent space. To alleviate this, Chen al. (2020) proposed to train a concept whitening layer that ensures the orthogonality of the concept vectors in its latent space.
One could argue that research on concept discovery has started with mechanistic interpretability, and precisely with the interpretation of individual neuron activations through methods such as network dissection (Bau et al. 2017). activation maximization (Erhan et al., 2009) and feature visualization (Olah et al., 2017). These methods isolate one neuron in the model and consider it as a direction that spans the latent space. Different approaches were then proposed to obtain a visualization of what maximises the neuron's 's response. Research in this direction, however, has pointed out that neuron activations seem to respond multiple unrelated features at the same time, phenomenon known polysemanticity (Nguyen et al. 2016 Olah et al. 2017 2020) and that they do not seem to align single interpretable concept (Klindt et al.) 2023), but rather to multiple ones at the same time (O DDMMhony et al. 2023). Based on these observations, much of this research has shifted from considering individual neuron directions, towards considering combinations of multiple neurons at the same time (Bricken et al., 2023). If we consider the latent space as spanned by the neuron basis, such combinations of neurons can actually be interpreted as vectors pointing to specific areas of the latent space. Clustering methods were used in Ghorbani et al. (2019);
: Klindt et al.) (2023); O M'Mhooyy et al.) (2023) identify clusters of features and characterize them concepts. Similarly, matrix factorization has been used by McGrath et al. 2022) Klindt et al. (023); Fel al. (023) obtain a decomposition of the latent space into
basis that can be used for discovery purposes. The work by Fel et al. (2023). in particular, emphasises the importance of considering the network sensitivity alongside these directions, being in line with the observations in this paper. However, in contrast to this work, our method considers the additional use of clustering as a means of refining the concept directions after the sensitivity analysis, with the intent of isolating unique concepts, namely concepts with a clearly distinct meaning and a unique interpretation. K-means clustering was shown particularly beneficial to this end in recent works (Klindt et al. 2023 O' Maaony et al. 2023).
Not related to this work, but still relevant in the concept discovery realm are transversally different approaches that aim at using multimodality as means of identifying concepts. For example, language models are used in Yan et al. al.](202) (2023);
; Kalibhat et al. (2023) to identify coincise and descriptive attributes that are used in visual recognition, and that could therefore be considered as concepts. None of these methods, however, is highly versatile to different architectures and data types, and they mainly rely on additional multi-modal data such as images associated with textual captions and the use of multimodal foundation models such as CLIP (Rafford et al., 2021). On a parallel line, an analysis of deep networks based on SVD was proposed by Raghu et al. (017)), where canonical correlation is used to test the similarity of two compressed representations of different model layers. Here the activations of single neuron represent vector in the canonical basis of the latent space. Following this line, non- negative matrix factorization was used to identify meaningful vectors to interpret AlphaZero (McGrath et al. 2022).
Our work takes inspiration from all of these works to introduce a novel method that does not rely on user-defined concepts or pixel segmentation, that does not require the training of additional concept whitening layers, and that leads independent vectors pointing to unique concepts with a distinct semantic meaning.
3 Methods
Based on the observations of Kim et al., we assume that directions in the latent space of layer can identify the concepts used by a model for inference (Kim et al. 2018). However, while they rely user-defined concept examples to find the concept vectors, here we aim to reverse-engineer the problem. Our objective is to identify concept vectors that directly emerge as relevant directions impacting the model decision function. We further assume, as suggested by [hen et al. (2020), that concept vectors should be orthogonal to each other to maximize the separability of the concepts. the objective of our method is to identify the set of orthonormal vectors in the latent space that carry most of the information, i.e. that most impact the model's prediction.
The method consists of three phases:
3

---

Published in Transactions on Machine Learning Research (November/2023)
1. variance alignment via the singular value decomposition (SVD) of the activations of a layer;
ranking of the singular vectors based on directional output sensitivity;
selection of the top ranked vectors and refinement of the direction to isolate unique concepts.
3.1 Notation
We consider the prediction function of a neural network f I + R" from an m-dimensional input vector to an n--dimensional output vector. We assume the model was already trained using a dataset consisting of N labeled pairs of input data points and labels (e),, C R" x R". Given an arbitrary layer I, the neural network can be seen as composition of a feature extractor Ø' IIm + I2 and downstream predictor vif : Rdd + I i.e. (x) - (x).. In the following, to simplify notation, and since we consider a single layer at a time, we drop the superscript identifying the layer Is. Furthermore, for given input I;, use the shorthand o, Φ(z;). We are interested in finding M orthonormal vectors U1, ... UM E R
In the following, we present the method for n = 1 and for densely connected layers. The method can be extended to n > 1, for example multi-class classification by applying the same construction to each of the outputs. Similarly, for convolutional layers, a pooling operation is introduced to obtain a d-dimensional representation. Details about both extensions are given in Section 3.4
3.2 Step 1: SVD
Step one aims at identifying the vectors that best summarize the encoding of the dataset in the latent space of a layer (with dimension d). To this end, we apply SVD to the entire layer's response to the input dataset. Such matrix & E RaxN is obtained by column-stacking the latent representations di. with i 1,..., N. By applying SVD, we obtain:
$ UEVT
Note, U E Raddd VT E NNN, and is a diagonal matrix of singular values d....., od The left singular vectors are the columns of the matrix U, and they align with the variance in 4 The singular values rank the directions from the largest to the lowest observed variance.
3.3 Step 2: Sensitivity-based ranking
At this point, we are only making use of the feature extractor o. and we are not considering whether the singular vectors are actually used by the downstream predictor. This means that the vectors found in the previous step might not be particularly relevant to the downstream predictive task. This second step evaluates the sensitivity of the output function along the directions of the singular vectors. The sensitivity represents the impact of perturbations of the feature representation of each input along the direction of the singular vector. As in Kim et al. (2018), this is computed as the derivative of the model output along the direction of the singular vectors. This operation is computed for all the singular vectors in U. Consider the gradient of the downstream predictor &b (Section 3.1 with respect to the latent space, which we denote V the for input o;. To compute the directional derivatives, we rotate the gradients to align with the singular vectors in U: Voti = U'V Vath. At this point, it is important to note that an input data point may have large gradients for low-activation features, and vice-versa, high activation values may be annihilated by close-to-zent gradients. Therefore, consider the joint impact of gradients and activations together. This operation is fundamental, as it reduces the gradient noise that can be derived by small gradient values at the end of model training. Let
a. U"T Φ:
denote the coefficients of d; rotated to align with the singular vectors. We consider the (element-wise) product between the coefficients of the rotated activations 6; and of the rotated gradients V The
0
(3)
Finally, to compute the ranking, we consider the overall importance of singular vector to the prediction. This is simply given by the sample mean of the ga over all inputs i in the dataset: ğ N Σ, Øt- Note, ğ IRdd For simplicity
4

---

Published in Transactions on Machine Learning Research (November/2023)
of the notation, drop the tilde in the rest of the paper. Simply put, this step replaces the conventional ranking of the singular vectors given by the values in E with a new ranking based on the value of the coefficients of the rotated gradients.
3.4 Extension to Vision Classification Tasks
Let demonstrate how to extend Steps and 2 to vision classification tasks. We consider convolutional neural network (CNN) / Rd xw xe' + R& classifying input image in n K classes. Particularly, Ji,: is the predicted probability of imput 2; belonging to class k namely p(y k/æ rz). NN is the number of samples in class k, with N -=Ek= N₁. For convolutional layers, the feature extraction is of IR xg''x'xx image z; to d feature maps of width and height h.
Rhxwxd and it maps input
To compute the SVD of Step 1, we aggregate the spatial information by a global average pooling operation, hence & E R where is the number of channels. More precisely, by pooling, we reduce each o(x,) E R xxuxd to a d-dimensional vector.
Step is also modified. For each class in K- K-classification task, we compute separate Is Rd To obtain the ranking, further evaluate how each gk compares the values obtained for the other classes. For instance, we compare 9h to the distribution of the 9g obtained for all the input data points that are not of class k hence for all the K classes except k.
Formally, for each class k we compute the average of the projection of gk on the singular vectors:
Σ U
9h
9t,kr
(4)
NR
where A Idd is the global average pooling of o(x;) O Votie deviation of the values obtained for all the rest of the data, namely for all classes except k. The mean is obtained, for instance, by averaging over all the samples I; such that ys + k:
We then compute the sample mean and standard
9k
Σ UT gi.-
(5)
N - Nk
Similarly, the variance is computed as:
2
Σ (UTTI.
gt, - gwd
(6)
N N
Finally, the importance scores for each singular vector in U are given by:
Sk 9k
(7)
where 2k.j is the importance score of the j-th column in U. This measure is then used to obtain a class-specific ranking, from which we retain the first M positions to identify M concept vectors.
3.5 Step 3: Identification of unique concept vectors
This step focuses isolating directions that act pointers unique concept representations. Note, by unique concept refer to distinct and individual idea notion that is well-distinguishable from others. Therefore, aim at isolating a specific and well-defined characteristic or pattern that is relevant to the task at hand and semantically distinguishable from other identified concepts.
At this stage, the singular vectors found by SVD provide no guarantee on whether they are pointing to unique nor human-unte underttadeett concepts. To account for this, we propose an additional step, namely the disentanglement of
Note, the pooling is here applied after element-wise product of the features maps and the gradients
5

---

Published in Transactions on Machine Learning Research (November/2023)
singular value directions into unique concept vectors that possess distinct semantic interpretation. This refinement process employs an unsupervised clustering methodology that was previously investigated by ('MMhoony et al. 2023 on individual neuron directions. The clustering requires six passages. We start by (i) aligning each latent representation oi in accordance with the directions indicated by the singular vectors. Then, (ii) we isolate the representations di exhibiting the highest coefficients along each direction. This is done by looking at the magnitude of the components in each direction and retaining the inputs with the highest values. In (iii) we use hierarchical clustering to identify clusters of inputs with minimal intra-cluster distances. This step is used to identify the optimal number of clusters, which is determined based on distance threshold parameter that controls the granularity of the concepts. At point (iv), if more than one cluster is identified as the optimal number of clusters, we perform an additional k-means clustering that further enhances the quality of the clusters. (v) Any outliers and insignificant clusters that contain an insufficient number of samples (i.e. fewer than five images) are removed. Finally, in (vi) we find the candidate concept vectors as the vectors pointing to the centroids of the newly identified clusters, namely by taking the mean activations of the respective clusters.
Human-interfacing is the last action of this step, necessary to interpret the discovered knowledge. The simplest way to obtain insights about a candidate vector is by data analysis and visualization. Depending the data type and the model architecture, multiple approaches can be used to analyze the data. The simplest approach is to visualize the imput data corresponding to the representations laying in the region pointed by the candidate concept vector. In dense models, the candidate vectors can also be used as feature importance estimates. The elements of a candidate vector u are used as weights that are multiplied element-wise to the feature importance values obtained by back- back-propagating the model's s gradients all the way to the input. Section Section3.66 discussss more in detail how to obtain concept visualizations and interpretations in detail.
3.6 Concept Maps and Segmentation Masks
The main challenge of concept discovery is the visualization and interpretation of the discovered information itself. As McGrath et al. (2022) discuss, the discovered information may not always be interpretable nor associable with a human- aaaann concept, and research is needed in this direction. concept vector discovered by our method can be interpreted the concept vectors, and feature perturbations along these directions have the largest impact on the output function. Visualizing how a concept activates given a certain input can be particularly beneficial to the interpretation of the discovered information in human-understandable terms. We define concept activation maps as the visualization of the model's 's response, for a given concept, to a certain input. The computation of such maps is straightforward for convolutional networks. For a discovered concept direction u E R and an input image 24, we can visualize the layer's activation responding to the discovered concept as the sum of the feature maps for each channel Φi,1, 1.l...... Di.d weighed by the concept vector coefficients. In other words, one can take the feature maps of each channel and weight them by the coefficient values of the concept vecto Note, if we were to consider concept that is fully aligned with the nith feature map (with < d, e.g.. (0, ..., 0, 1, O, ..., 0) (non-zero only for U,,), the visualization for x; would correspond to the n-th feature map itself. Concept segmentation masks can be directly derived by the concept maps as in Bau et af. (2017). namely by retaining the input pixel values with concept activation higher than the 80-th percentile in the concept maps.
latent discriminant direction for the downstream task. The variance of the activations aligns with
3.7 Dataset Exploration and Outlier Detection
We showcase the utility of concept discovery for dataset exploration, demonstrating how the discovered concept vec- tors can be used to find input samples that are mislabeled or that contain confounding factors. The data is rotated to align in the space spanned by the singular vectors and anomalous samples are identified based on the statistical dispersion of the data For instance, we isolate 10% of the representations that fall outside of the interquartile range of the representation coefficients for the training data. The inputs corresponding to these representations are flagged for further inspection, they are outliers that may contain artefacts or potentially misleading confounding factors.
Being the concept vector singular vector, has norm one.

---

Published in Transactions on Machine Learning Research (November/2023)
A. 2D UMAP of images activating concepts for bubble, police van and zebra
B.
Unique Concept
C
Unique concect
E
Concapt
F.
Unique Concept
Bubble
KMTT
Polico
Max. activated by:
Max, activated by:
D. Bisementic Concept
Clustering
2D UMAP
Figure 2D UMAP of images activating the discovered concept vectors for bubble, police van and zebra. The images form three seperate clusters. We visualize concept segmentation masks for the images in each of the clusters. B. Concept segmentation masks and maximally activating input image for the class bubble. C. Concept segmentation masks for zebra. The concepts discovered in B. and C. are unique concepts. D. A bisemantic concept is identified for the class police van, activating for both vehicle parts and dog paws. After the refinement through clustering, the concept is split in two separate vectors representing either the graphics of the police logo or the dog fur. E. Unique concept representing the police log on the van structure. F. Unique concept deriving from the disentanglement of the polysemantic concept in
4 Results
This section presents the comprehensive evaluation of our concept discovery approach across various tasks and mod- els. Our experiments focus on widely adopted models with publicly available pre-trained weights, which have been extensively studied in the field of interpretability. Specifically, we present the results obtained using Inception V3 (IV3) (Szegedy al., 2016) with pretrained weights on the ImageNet ILSVRC2012 dataset (Russakovsky al. 2015). In addition, demonstrate the versatility of concept discovery beyond image-based applications by showcas- ing its effectiveness in classification tasks involving tabular data (details provided in the Supplementary Materials).
4.1 Controlled validation with ground truth
To validate our method's ability accurately discover concepts, we perform controlled validation experiments. Fol- lowing the approach in Kim et al. (2018), define binary classification tasks where the crucial concept influencing the outcome was introduced by design. We selected the church and tench classes from ImageNet and build two binary classifiers with Inception V3 backbones. For the first classifier, denoted as VV3_noised the church training images are augmented by red squared stamp, and the fish ones by green stamp. The stamps were randomly placed within the images, with a noise level of zero, making them perfeetly correlated with the class label. Our expectation was that the model would rely on the presence and color these stamps for classification. When evaluating this model's performance on test images with and without the stamps, we could test this hypothesis. This analysis gives us, in fact, the ground-truth causal concept effect (CaCH) of the stamps, revealing the actual impact of the stamps the
7

---

Published in Transactions on Machine Learning Research (November/2023)
A.
C.
B.
0.4
0.2
Figure 2: Visualization of IV3_ "3_noise0 heatmaps obtained by A. GradCAM and B. our concept activation maps method for the most important concept discovered by our method. The first two images tench inputs misclassified churches. Both methods point to the red jumper instead than to the green square. Similarly both methods point to the red squares in the correct images. C. Comparison of how frequently the two methods successfully hit the stamps.
model prediction (Goyal et al. 2019). After training IV3_ "3_noiseO from scratch for 100 epochs using stochastic gradient descent and standard parameters, the model achieved a remarkable test accuracy of 99.79% on images augmented with the squares. However, when the squares were removed from the test data, the accuracy plummeted to the level of random guessing (i.e., 51.73%), demonstrating a high CaCE. In contrast, the second model, named _ is trained on a dataset where the color assignment noise is set to one. Red and green squares were added randomly to the training images. In this case, the model reaches 95.43% test accuracy and the CaCE is low since the accuracy remains high 93.34% even when the squares are removed from the test images.
We perform concept discovery in both models. The color of the red square is the most important concept discovered for ""__oiseeO. As expected, this concept cannot be discovered in IV3 v3__noisel. This shows that our discovery method only points to the stamps when these are actually used by the classifier. We quantify these observations in terms of the number of stamp hits over the entire dataset. Precisely, we count in how many images the heatmaps overlap with the stamp region and we then normalize this quantity by dividing it for the number of images total. For IV3_ _3__noiseO, our method correctly pinpoints the ground-truth concepts in the images with frequency 83% at the Mixed 6e layer, and of 60 % at Mixed 7b. These values drop drastically for the concepts that we identify in IV3_noisel, for instance to 40% and 10%, respectively.
We compare both qualitatively and quantitatively our results to the object localizations obtained by GradCAM. A qualitative comparison is presented in Figures[A and B. From the quantitative comparison of the hit rate in Figure GradCAM has higher true positive rates of the detection of the red squares, but shows higher rate of false positives than our method. This suggests that our approach has reduced risk of misleading explanations for local explanations, e.g. for single datapoint. Moreover, our method provides us with the additional information of the concept vector, thus allowing to act on the concept itself by modifying, removing it isolating it in the latent space.
4.2 Confirmation of existing findings in discovered concepts
Our method automatically identifies concepts associated with image categories. Our results are line with specific findings that were already discussed by previous research (Kim et al., 20188) Ghorbani et al.) 2019), specifically for the object categories of zebra, police van and bubble. Additional results for other classes than the selected ones and for ResNet50 are provided in the Supplementary Materials A and an integral analysis of the concepts discovered for ImageNet classes will be made accessible online.
We identify three orthogonal vectors as candidate concept vectors (i.e. one for each of the analyzed classes, with M = 1). The discovered concepts encompass a diverse range of visual elements, including patterns observed in the coat of zebra, graphical features and tires associated with a police van, and glossy reflections as seen in a bubble, as demonstrated in Figure [ The maximally activating images are either visually similar or exaggerate the motifs shown by the activation maps. The maximally activating input for bubble, for example, exaggerates the glossy- reflections and the round shapes generally observed in soap bubbles. The text of the menu restaurant contains black and white
8

---

Published in Transactions on Machine Learning Research (November/2023)
stripes, which also in zebra images. Importantly, these illustrations were obtained without any prior annotation of the concepts as detailed in Section 3.6
Upon closer examination of the police van class [Fi our analysis reveals an interesting observation. The candi- date concept vector identified for this class exhibits responses to multiple unrelated features. In addition to activating for van tires and police graphies, it also responds to dog chests, which may seem unexpected at first. However, this phenomenon not entirely unexpected and it can be attributed the concept of superposition, which has been ob- served in CNNs (Olah et al., 2020). Superposition refers to the ability of neural networks to represent more features than the number of neurons they possess. This results in the presence of polysemantic neurons, which exhibit activa- tion patterns that are semantically unrelated. Previous research highlighted the existence of such polysemantic neurons that activate simultaneously for car hoods and animal paws (Olah et al. 2020 O' O'Mahony et al. (003)). aligning with our findings. Our method demonstrates its capability to successfully disentangle the two activations associated with the police van class by identifying two distinct clusters that refine the concept direction into separate vectors, as illus- trated in Figures [T and F. In this process, the orthogonality of the direction is compromised to achieve unique concept directions for each activation. The disentanglement of the two patters allows us capture the distinct semantic mean- ings associated with each activation, providing more refined and comprehensive understanding of the underlying concepts.
4.3 Coherence and human interpretability of discovered concepts
To assess the coherence and human interpretability of the discovered concepts, we conducted a human evaluation test following the methodology outlined in previous works (Ghorbani et al.) 2219). The test involved 30 par- ticipants, and the details of the evaluation test can be accessed at the following link: https: /frrmss.///omomoom gle/ MJ63G984ERvoZuFF
The evaluation comprised three main experiments, namely (i) intruder detection (ii) con- cept meaningfulness and (iii) inter-user agreement. Participants were familiarized with the test format by a prior introduction with an exemplar question and the corresponding correct answer.
In the intruder detection experiment (i) illustrated in Figure BJA participants were presented with ten questions, each featuring visualization of four concept segmentation masks associated with a specific concept vector, and one random mask from different concept vector. The task was to identify the outlier image that conceptually differed from the others. Participants successfully recognized the intruders with average accuracy of 0.88. Concept meaningfulness (ii) was measured by asking participants to label the concepts based on the visualization of the concept segmentation masks for three images. Despite the small image segments depicted the concept segmentation masks, all participants accurately labeled all the images, indicating that the concept segmentation masks were easily interpretable and could be associated with specific concepts. Inter-user agreement (iii) was evaluated by requesting participants to agree or disagree with concept labels provided by other participants. Out of the 30 users, inter-user agreement exceeded 91% for all questions. Notably, concepts identified for dog breeds showed relatively lower agreement scores, as users disagreed on the specific type of animal depicted in the concept segmentation masks.
4.4 Quantitative evaluation of concept relevance
We further measure the significance of the discovered concept vectors by quantifying the impact of concept removal using two different approaches, namely occlusion in the input space and weight annihilation at the intermediate layer.
Following the methodology Ghorbani et al. (2019), occlusion is applied to the input pixels with high values in the concept map of each concept, specifically targeting the pixels with values above the 80th percentile. We quantify the smallest destroying concepts (SDC) as the smallest number of concepts to remove in order to see a performance degradation on at least 80% of the dataset. Starting with the most important concept, we gradually remove concepts while monitoring the drop in performance. As depicted in Figure BC, we observe a degradation of least 80% in the prediction accuracy when the first concept is removed for nearly 400 classes. The performance degradation extends to almost all ImageNet classes (962 out of 1000) when we remove the first five concepts. Supplementary Figure B.9 in the Supplementary Materials provides a comparison the model predictions before and after the removal of the SDC for each class.
One limitation of SDC is is the introduction of distribution shift in the modified images due to the replacement of pixels in the input space. It becomes challenging disentangle whether the prediction change is solely due to
9

---

Published in Transactions on Machine Learning Research (November/2023)
A.
Inasddeddetection
Intruder detection
B.
Concept labeling
C
Concegt occlusion
Concopt removal weight annihllation
Ounost
incomet
bike
frame i
30
bicycle?
0.00
airplane
7.5
randon, peroval
6.00
rendar.eeooul
plane
@
removed concagus 10000 graakcannppp
removed concapts (-200 neightt neigtascoroed
E. Fine-grained concepts for nugby ball
Fine-grained concepts
Fina-grained concapts
Tst Concept
Concept
Boogle
Sheopdoog
2nd Concect
3rd Concept
2nd Concapt
3rd Concept
Figure 3: A. Detailed results of the intruder detection experiment with human participants. Out of ten classes, par- ticipants were able detect with very high accuracy the intruding concept. B. Wordelouds of the concept labelling results for two concepts. The font size corresponds to the word frequency. C. Concept importance measured in terms of the impact of removing concepts in the input space by occlusion, namely by masking out areas with high values in the concept activation masks. Performance is evaluated on all ImageNet classes while gradually removing concepts starting from the most important. D. Concept importance measured as the impact of removing concepts in the latent space by annihilating the model weights that are linked to the concept. Performance is evaluated on all ImageNet classes while gradually removing concepts starting from the most important. Fine-grainee concept activation maps for the first, second and third concept vectors differentiating rugby from soccer balls. The map adopts symmetric colorbar with red for positive-, blue for negative- and green for zero-valued activations. Images are sorted from the largest absolute projection values on the concept vector. F. Fine-graines concept activation maps with symmetric col- orbar for the concept activation maps on ImageWoof categories.
removing the concept or the distribution shift (Hooker et al.) 2019). To address this concern, we conduct an additional analysis where concepts are dropped directly inside the layer, comparing the change in prediction accuracy against dropping random directions. To drop a concept, we retain the layer weights with low concept vector components and we set the remaining weights to zero. For example, to remove the first concept of the class lionfish, we set to zero 204 parameters out of the original 2, 048 (keeping the remaining 1, 844 weight values untouched), based on the 80-th percentile of the components of the concept vector for that class. We compare the impact of this ablation against dropping random weights to disentangle the effect of removing specific concept direction from randomly setting weights to zero.
Figure 3D shows the accuracy drop observed when removing the concept directions from the first most important to the fifth, computed on the training images and 1000 validation images. Removing the first five concept directions causes an accuracy drop from 0.920 to 0.785 on train data, and from 0. 0.2 to 0.676 on test data. The same drop cannot be observed instead when setting random weights to zero (i.e. accuracy 0.89 on train and 0.734 on test data).
4.5 Fine-grained analyzes
Concept discovery can be conducted in fine-grained manner, focusing on specific subsets of classes. This targeted approach enables the identification of concepts that are tailored and relevant smaller group of similar classes. Such granular analyses are particularly valuable for classes that pose classification challenges or hold specific importance for interpretability analysis.
We consider a subset of ten dog breeds classes in ImageNet, referred as ImageWoof (Howard 2019). This subset presents a significant challenge as the differences between the dog breeds are subtle (more details in the Supplementary Materials B). Our method identifies one concept for each class, totalling ten concepts. For Dingo Beagle and Old English sheepdog visualize the images that maximally activate the concept for each class, hence with the largest
10

---

Published in Transactions on Machine Learning Research (November/2023)
projection the concept vector (Figure 3G). High activation values are around the head for all dogs, and confusing factors are ignored (e.g. the woman in sheepdog). Each dog has very distinctive head shape (the dingo has straight ears, the beagle long bent ears and the sheepdog has long black and white fur) and for this reason, different concept vectors are found for each class. The beagle also shows a diffused attention that includes the tail and paws of the dogs. This probably to distinguish this class from the English foxhound.
By running concept discovery on pairs of classes, we identify concepts that provide informative insights into what distinguishes one class from another. For instance, Figures EE and F showcase the concepts that differentiate a Soccer ball from Rugby ball. In this analysis, discovered concepts for each class in order of importance. The most important concept for Rugby ball emerges to be the shape of the ball, as shown by the activations around the ball in Figure On the other hand, the concept activation maps for Soccer ball identify the knitted pattern of the ball as the most important concept, and the soccer ball shape emerges only as the second most important concept. We can also see the attention to the context, particularly in the third concept. The maps activate on the players on the field, showing sensitivity to the player's i jerseys and socks. Some of these findings align with the hypotheses found in (Ghorbani et al. 2019) about the relevance of jerseys to classify images in the basketball category. Thus, jerseys and socks may introduce confounding factors in the classification of these categories. Finally, in the inputs for the rugby class, the maps also activate on the players' hands holding the ball, or on the act of tackling, whereas in the soccer images the maps activate on the acts of kicking the ball.
set the number of concepts to find for each class M = 3 and present three
4.6 Concept Uniqueness
We evaluate the uniqueness of the concepts identified by our method, showing that it is possible to disentangle super- position and identify vectors that point to unique concepts. Firstly, it is important to note that the discovered concepts do not align with individual model units. To quantify the alignment, we compute the cosine similarity between the discovered concept vectors for each category in ImageNet (considering M = and a total of 100 concept vectors) and the basis vector of the latent space. The cosine similarity is consistently lower than 0.5 for various layers such as Mixed_7b 7b, Mixed_5b 5b, Mixed_5d, Mixed_ _óce and Mixed_ A cosine similarity close to zero indicates the inde- pendence of the two directions, suggesting that the concept vectors and the latent space vectors not aligned. In contrast, if the vectors were aligned, the absolute value of the cosine similarity would be close to one.
Next, we examine the outcomes of applying the disentanglement method proposed in O' MaMMoo et al. 2023). Specifi- cally, we compare how many distinct clusters are suggested by the disentanglement method when applied to individual ncurons as opposed to the singular vectors identified by singular value decomposition in our method. This analysis is informative on whether the basis of the singular vectors can facilitate capturing inherently distinct concept represen- tations, i.e. unique concepts. Figure A for example, illustrates how three separate clusters are identified as unique concepts starting from one of the singular vectors. The disentanglement outcome provides us with 3 distinct concepts for babies, teddy bears, and necklaces. By applying this approach to all of the singular vector directions within layer Mixed_7b we quantify how many singular vector directions were already pointing to unique clusters and how many required disentanglement. As shown by Figures and C, most of the singular vectors in our method already point to unique concept directions, and only a few require disentanglement. As opposed to individual neuron directions, fewer directions show polysemanticity. This means that the singular value decomposition step in our method is a necessary approach that facilitates the identification of unique concepts. The remaining singular vector directions that are not unique are then disentangled by our clustering method in distinct concepts.
4.7 Effective Outlier Detection
Our method successfully identifies 23% (equivalent 0.2% of the ImageNet training set) of the training images as outliers based on their representation in the latent space. Among these images, 184 (1.8% of the training set) are misclassified by the model. Note, the flagged images were seen during training. hence they are challenging training examples that confuse the model rather than improving its robustness. This observation is supported by the lower top-1 accuracy on the flagged images at 0.90 compared to 0.93 on the rest of the training images.
Furthermore, when compared to random neurons or random directions, our method demonstrates higher accuracy in identifying these challenging training examples. The accuracy on the flagged images obtained through our method is 0.90, whereas random neurons or vectors achieve an accuracy of 0.92. This reaffirms the notion that the concept vectors discovered by our method offer more informative directions for dataset exploration than random alternatives.
11

---

Published in Transactions on Machine Learning Research Novemberr20223)
A
B.
10000
medtom with LnIIga coroasts
800
000
000
200
unque
1200
600
Kumber of unique contapts
Figure 4: A. UMAP the images maximally activating polysemantic singular vector. Hierarchical clustering is used to identify the optimal number of clusters, i.c. three. Three concept vectors pointing to the centroids each cluster are thus derived obtain unique pointers concepts. Number of clusters identified by hierarchical clustering for the images maximally activating individual neurons. The high number of neurons with least two clusters shows that the directions are highly polysemantic, confirming existing research. C. Number of clusters identified by hierarchical clustering for the images that maximally activate our singular vectors. The high number of directions with single cluster shows that the singular vectors are more likely to identify unisemantic concepts.
Prodiction: Comio book
Prediction: Can opener Label: Safety pin
Prediction: Lawn Label: Shavel
Prediction:
Label: Weasel
Labal:
Figure 5: A. Results of dataset exploration. Our method identifies training images with distinet issues, such style shift to drawings (Ist image), poor resolution (2nd), multiple correct labels (3rd) and optical illusions (4th).
Figure shows some examples of the outliers identified by our method. These images present notable variations in style and resolution (e.g., the first two images from the left) or contain confounding factors. For example, in the third image from the left, multiple labels (shovel and lawn mower) are equally correct. Additional examples are shown in the Supplementary Materials.
12

---

Published in Transactions on Machine Learning Research (November/2023)
5 Discussion
The automatic discovery the latent concepts used by complex models remains an under-explored research direction. Our approach offers a novel yet effective means automatically discover concept vectors that point image clusters with a unique semantic meaning. Our experiments demonstrated its potential and accuracy through diverse evalua- tions. We successfully identified the influential concepts in controlled experiments and we illustrated how removing the most important concepts impacts model performance.
A key strength of our concept discovery method is its independence from manual annotation, supervision and image preprocessing. This significantly reduces the time and effort required in traditional manual labeling methods, and it makes our method highly flexible and generalizable. Scientific domains such as biochemistry could strongly benefit from this novel approach, using it to uncover biases in heterogeneous datasets and provide insights into the molecules driving reaction prediction. In medical imaging, could support knowledge discovery. such identifying visual cues predictive of genetic alterations from biological tissue slides. The uniqueness of the discovered concepts is another noteworthy aspect of our approach. We demonstrate that our concept vectors can more appropriately capture independent and distinct visual attributes than individual neuron directions.
However, it is essential to acknowledge some limitations of this work. We did not compare our method to concept- based approaches like TCAV (Kim et al. (2018) and similar derivations (Graziani et al. 2018). This omission, however, is only due to the fact that these methods rely on sensitivity scores that our approach is designed to maximize in step 2. Thus, direct comparisons using these metrics would be biased in favor of method. Additionally, concerns regarding the scalability of SVD on large datasets can be mitigated by reliable results with only 1% of the dataset. Challenges may arise when applying our method to mulli--odal models, where concepts may combine multiple modalities. Addressing this challenge would require back-propagating the information to individual modalities by saliency methods. Moreover, our methods's effectiveness relies on the quality of the input data used to compute SVD. If the network fails to capture relevant concepts during training, our method may not be able to discover them effectively. Finally, the evaluation of concept meaningfulness relies on human participants, which finally introduces some degree of subjectivity.
Nevertheless, our method proved effective in detecting input images with confounding issues, offering valuable in- sights on the models. The flagged outliers highlight images splotlight images with variations in style. resolution. or contain confounding factors. This information is valuable for improving model robustness and understanding the limitations and challenges associated with specific classes or image characteristics.
6 Acknowledgements
MG and VA devised the project, the main conceptual ideas and proof outline. MG worked out the technical details, car- ried out the implementation and experiments and wrote the manuscript with input from all authors. HM contributed to the revision of the paper and funding. LOM worked out the technical details and implementation of the clustering step, carried out experiments and contributed to the editing of the manuscript. AN contributed to the technical details and proof outline of the project. VA supervised the project, contributed to the technical details, planned the experiments, aided in extending the validation and editing the manuscript. All authors contributed to writing and editing.
MG acknowledges the support of the EU Horizon2020 project AI4Media (951911). LOM acknowledges the support of Science Foundation Ireland (18/CRT/6049). VA acknowledges the project TARGET (KF9S-5544..02-2022-))
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances neural information processing systems, 31, 2018.
Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelezyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. OpenXAI: Towards a transparent evaluation of model explanations. Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
URL https:
7/openreview. net mett:/oorumm.
13

---

Published in Transactions on Machine Learning Research (November/2023)
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6541-6549, 2017.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):722-72, 2020.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009,
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2711-2721, 2023.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:190..07165. 2019.
Mara Graziani, Vincent Andrearczyk, and Henning Müller. Regression concept vectors for bidirectional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications. pp. 124-132. Springer, 2018.
Mara Graziani, Vincent Andrearezyk, Stephane Marlhand-Maillet, and Henning Müller Concept attribution: Ex- plaining enn decisions physicians. Computers in Biology and Medicine, 123:103865, 2020. ISSN 0010-4825. doi: https::/doi.org// htt.:///g/....11/...oooooopooooo 10116f.compbiomed.2222 103865..
Mara Graziani, lam Palatnik de Sousa, Marley MBR Vellasco, Eduardo Costa da Silva, Henning Müller, and Vin- cent Andrearczyk. Sharpening local interpretable model-agnostic explanations for histopathology: Improved un- derstandability and reliability. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 540-549. Springer, 2021.
Mara Graziani, Lidia Dutkiewicz, Davide Calvaresi, José Pereira Amorim, Katerina Yordanova, Mor Vered, Rahul Nair, Pedro Henriques Abreu, Tobias Blanke, Valeria Pulignano, et al. A global taxonomy of interpretable ai: unifying the terminology for the technical and social sciences. Artificial intelligence review, 56(4):3473-354, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods deep neural networks. Advances in neural processng systems, 32, 2019.
processing
Jeremy Howard. imagenette, 2019. URL
Kareem L Jordan and Tina L. Freiburger. The effect of race/ethnicity on sentencing: Examining sentence type, jail length, and prison length. Journal of Ethnicity Criminal Justice, 13(3):17--166, 2015.
Neha Kalibhat, Shweta Bhardwaj, C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi Identifying interpretable subspaces in image representations. In International Conference on Machine Learning. PMLR, 2023,
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. (668-2677. PMLR, 2018.
14

---

Published in Transactions on Machine Learning Research (November/2023)
David Klindt, Sophia Sanborn, Francisco Acosta, Frédéric Poitevin, and Nina Miolane. Identifying interpretable visual features in artificial and biological neural systems. arXiv preprint arXiv:2310.1143/) 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang, Concept bottleneck models. In International Conference on Machine Learning, pp. 5338 5348, PMLR, 2020.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47)):2206625111 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv: 1602.0360 2016.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(1 2017.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An intro- duction to circuits. Distill, 5, 03 2020. doi: //.11///iiilll.00024.000
Laura O' Mahony, Vincent Andrearczyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3769-3774, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748 8763. PMLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '1., pp. 1335-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 2939778
97834503423 doi: 10. 114444996772299577 29397777 httttttts:///:...comm htttpsss//oo...ror/0..111/333333
Robert Rosenthal and Kermit L. Fode. The effect of experimenter bias the performance of the albino rat. Behavioral Science, 13)):183-189 1963. doi: https:///oi wwwp:////......//...ooooo
htw.////.....000000022222 URL httthhttt///oolll httptpsss///ommmeelbbaaaa
coley.com/dae dww/abs///b///// 10. www0/bs........... 10..0.///......0000...00 3830080302
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
Cynthia Rudin, Caroline Wang. and Beau Coker. The age of secrecy and unfaimess in recidivism prediction, 11 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-B Brrddeent-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), volume 128, pp. 618-626, 2017. doi: 10. 10.100//CC222017..44
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1831--8266, 2016.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong. Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3000-3100, 2023.
15

---

Published in Transactions on Machine Learning Research (November/2023)
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for dis- criminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
A Additional Results on IV3 and ResNet50
Note: In our study, we apply the methodology to the 1000 classes in ImageNet. To ensure computational feasibility and accommodate our infrastructure capabilities, we employ an undersampling technique. Specifically, we retain only one out of every ten images from the ImageNet training dataset. This strategic choice enables us to manage memory requirements effectively and streamline the computational process. Nevertheless, we demonstrate how concepts can also be obtained by utilizing the complete set of available images for select classes, providing more granular analysis in Section 4.5 Throughout the experiments, unless explicitly stated otherwise, we primarily focus on the concatenation layer named Mixed 7b. This layer is situated deep within the model, close to the end, and is expected to capture complex and high-level concepts. However, we emphasize that similar analyses can be conducted at different depths, layers, and even on entirely different architectures, as showcased in Appendix [
We provide additional results for 16 classes, namely airliner, clock, corkscrew, albatross, border collie, road sign, flamingo, mushroom, mushroom,artichokke hammerhead shark, screwdriver. iPod, tench fish, suspension bridge, umbrella and cucumber. The concept segmentation masks for the first most important concept are shown in Figure E.13 The concepts were resized to fit in the square, but they originally appear at multiple scales in the input images.
Results obtained on ResNet50 (at layer (.0rr_4.0.ddd) are shown in Figures E.14 and A.6 The visualizations point to concepts that are similar to those highlighted in IV3 visualizations, such as the fish fins, the zebra stripes, the glass- like reflections and van tires and logos. value on the concept vector for the four analysed classes. We can see the striped pattern emerging again for the class zebra, although in this case the color of the stripes seems be given less importance. For the class bubble, the shape of the bird belly and their repeated presence may share similarity with the images in the bubble class.
A.6sshow the input images in the dataset that have the largest projection
Lionfish
Police Van
Bubble
Zebra
Maximally activating input
Maximally activating input
Maximally activating input
Maximally activating input
Figure A.6: Results on ResNet50. Visualization of the inputs with the largest projection value on the concept vectors for the classes lionfish, police van, bubble and zebra. Next to the input images we visualize the concept segmentation masks.
B Additional Results on ImageWoof
The ImageWoof dataset is a subset of the following ImageNet classes: Australian terrier, Border terrier, Samoyed, Beagle, Shih-Tzu, English foxhound, Rhodesian ridgeback, Dingo, Golden retriever, Old English sheepdog. An input image for each class is shown in Figure 12 This dataset is challenging to classify because the classes represent multiple dog breeds with subtle differences. As Figure B.7 shows, the confusion matrix obtained on training images also presents misclassified samples.
Figure B.8 shows additional concept maps for the dog breeds (M = 1).
We show the prediction change before and after the removal of the smallest number of destroying concepts on Image- Woof in Figure B.9, No prediction change would appear as a diagonal line, whereas in our case we can clearly see that the prediction becomes random after removing the concept. The SDC is for Shih-Tzu, Rhodesian ridgeback, Beagle, Australian terrier and Golden retriever, for the Old English sheepdog, 3 for Samoyed and Dingo and for the Border Terrier.
16

---

Published in Transactions on Machine Learning Research (November/2023)
O. E. Sheepdog
Shih- Tzu
Rhodesian
Beagle
Foxhound
Border T.
Australian T.
Golden R.
Samoyed
Dingo
Shih-Tzu
430.0
0.0
0.0
10
0.0
Rhodesian
0.0
434.0
0.0
0.0
10
0.0
0.0
0.0
0.0
0.0
Beagle-
0.0
0.0
449.0 0
11:0
0.0
0.0
0.0
0.0
0.0
Foxhound
0.0
100
23.0
360.0
0.0
0.0
0.0
0.0
0.0
0.0
Border T.
0.0
Australian T.
0.0
0.0
0.0
0.0
100
419.0
0.0
10
0.0
0.0
Golden R.
0.0
10
10
0.0
440.0
0.0
0.0
O. E. Sheepdog
0.0
0.0
0.0
0.0
0.0
0.0
0.0
489.0
0.0
0.0
Samoyed
0.0
Dingo
0.0
2.0
0.0
0.0
D0
0.0
0.0
0.0
10
413.0
Figure B.7: Detail of the IV3 confusion matrix on Image Woof classes.
C Additional Results from the User Evaluation
Figure C.10 reports in detail the performance of the end users on each question in the evaluation study part (i).
D Benchmark results on COMPAS
Here we demonstrate the applicability of concept discovery non-imaging applications such as COMPAS. The model is, in this case, a densely connected graphic features of nearly 19 thousand defendants to predict the likelihood of a new offense. Previous studies largely discussed whether race, gender and age of the defendants are sensible variables affecting the system's 's fairness (Jordan] & Freiburger 2015) Rudin et al. 2018). While we agree that a multilayer perceptron is unnecessarily complex for this task (Rudin) 2019), we recognize its importance as a contradictory task in the model interpretability literature.
network. The COMPAS system uses criminal records and demo-
Figure D.II shows three discovered concepts for this task (M 3). Note, gradient backpropagation is used to visualize the importance of each input feature for the concept vector. The values in the plot represent the importance given to the input features to project any input on the concept vector. The three concepts are linear combination of multiple input features, and high relevance is given to the age of the defendant (age, in the first plot), the number of prior arrests (no_priors, in all plots), and the length of previous detentions (length_stay, two plots on the right).
The concept scores in the plot give a global explanation of the model. Local explanations about single inputs can be obtained by multiplying the scores with the feature values of each input. We benchmark the faithfulness of our explanations against other explainability methods by the Prediction Gap Important feature perturbation (PGI) and
17

---

Published in Transactions on Machine Learning Research (November/2022) (02)))
Rhodesian
Shih-Tzu Pefta com
English Foxhound
Australian Terrier
Samoyed
Golden Retriever
Border Terrier
Figure B.8: Concept maps for the classes in ImageWoof. The images are ranked in order of their projection value on the concept vector (largest to lowest).
18

---

Published in Transactions on Machine Learning Research (lovember20023)
1000
prediction after concept removal (input space)
800
600
400
200
0
200
300
400
500
600
700
800
original prediction
Figure B.9: IV3 predictions on ImageWoof before (original) and after concept removal in the input space.
19

---

Published in Transactions on Machine Learning Research (lovember2022))
Intruder correctly detected Intruder not detected
30
22.5
15
7.5
0
1
2
3
4
5
6
7
8
9
10
Figure C.10: Detailed performance on part i. the human evaluation study.
20

---

Published in Transactions on Machine Learning Research November/2023)
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
age
2y recid
no priors
length stay degree gender
race
age
2y recid no priors
length stay c. gender
race
age
2y rrrecdd no .o.priors
length stay c. degree gender race
Figure D.11: Top concepts discovered on COMPAS. The bar values illustrate the feature importance values to project each input on the discovered concept vector.
Unimportant feature perturbation (PGU), in Angarwall al. (2022). High PGI and low PGU values desirable. The former indicates that the explanation identifies important features, while the latter that the method ignores unimportant obtaining 0.27 PGI and 0.0026 PGU. These values are comparable with other standard explainability methods, outperforming the benchmark results obtained by gradient-based approaches and LIME, as shown in the Appendix Table
ones. We compute the PGI and PGU following the benchmark code |
Table | benchmarks our method in the OpenXAI benchmark (Agarwal et al. 2022). The results in table are taken, except for concept discovery. from the online leaderboard of the benchmark at open-xai open-xai.github.b3 github. The hyperparameters of the explainer were illustrated in the benchmark instructions, i.e. (protected_cm cl ass positive_outconme 1; perturbation_stere 0.3).
Table 1: Explanation faithfulness for COMPAS.
Method
PGI
PGU
LIME
0.232
0.247
Vanilla Gradient
0.240
0.240
Integrated Gradient
0.240
0.238
Gradient Input
0.254
0.216
SHAP
0.274
0.194
SmoothGrad
0.324
0.106
concept discovery
0,268
0.0026
E Additional Visualizations
21

---

Published in Transactions on Machine Learning Research (November/2023)
3-ngglish, fochound
6-golden retriever
0 Tzu
L-hoddesian, rideebak
beogle
4-Horder terrier
OOld English, sheepdog 0-Samoyed
d-dingo
Figure E.12: Dog breeds in ImageWoof.
Woof.
Airiner
Border Collie
Clock
Road Sign
LDD
BEW
L
4
ood
A
HBB
5
Corkscrew
Flamingo
MAAA
Albatross
Mushroam
Artichoke
Tench fish
Hammerhead
Suspension bridge
Screwdriver
Umbrella
iPod
Cucumber
Figure E.13: IV3 concepts for additional classes. We show the resized segmentation masks.
22

---

Published in Transactions on Machine Learning Research (lovember20023)
Lionfish
Police Van
Bubble
Zebra
Input
Concept
Input
Concept
Input
Concept
Input
Concept
More concept examples:
More concept examples:
More concept examples:
More concept examples:
Figure 14: Results RestNet50. Segmentation masks of the concept vectors in layer4.add of ResNet50 for the same classes and input images in Figure
23


Reviews:
[{"review_id": "uRFS0jIi4O", "reviewer": "Reviewer_SoMH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nAutomatic Concept Discovery: The paper's method for automatically discovering concepts reduces the need for user-defined concepts, enhancing the method's versatility.\n\nWide Applicability: The approach shows promise across various architectures, tasks, and data types, increasing its practicality.\n\nBroad Utility: The paper highlights the versatility of the method in dataset exploration and the identification of outlier samples, which can be valuable for quality control in machine learning applications.\n\n**Weaknesses:**\n\nInterpretability Challenge: While the method identifies concept vectors, the paper acknowledges the challenge of interpreting these vectors, especially when they may not align with easily understandable human concepts. Further research in this area is needed.\n\nComplexity: The method involves several steps, including singular value decomposition and clustering, which may make it computationally expensive and less straightforward to implement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method for automatically discovering concepts learned by deep learning models during training. Through singular value decomposition and unsupervised clustering of the latent space of the model, the authors identify concept vectors aligned with directions of high variance that are semantically distinct and relevant to model predictions. The paper demonstrates the application of this method in natural image classification and its utility in dataset exploration and outlier detection. This work has the potential to enhance the interpretability of deep learning models and improve dataset quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify Algorithm Steps: Provide a more detailed breakdown of each step of the proposed algorithm. Elaborate on the input, processes, and expected outcomes at each stage. This will assist readers in grasping the methodology with precision.\n- Expand on Evaluation Metrics: If possible, extend the discussion of evaluation metrics used in the user studies. Explain the rationale behind the selection of specific metrics and how they relate to the method's performance. Include statistical analyses where applicable.\n- Address Computational Efficiency: Given the potential computational complexity mentioned in the paper, discuss strategies or optimizations to address this issue. Explore how the method's efficiency can be improved for practical deployment.\n- Generalization and Transferability: Emphasize the generalizability and transferability of the proposed method to different domains and applications. Discuss potential challenges or limitations when applying the approach to various contexts.", "broader_impact_concerns": "- Interpretability Challenge: The paper acknowledges that interpreting the discovered concept vectors, especially when they do not align with easily understandable human concepts, remains a challenge. This issue raises broader concerns about the practical applicability of the method in real-world scenarios where interpretability is crucial for decision-making.\n\n- Computational Complexity: The method involves several steps, such as singular value decomposition and clustering, which may make it computationally expensive. This could limit its usability in resource-constrained environments, and addressing computational efficiency should be considered.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NfKtQ9k8in", "reviewer": "Reviewer_HnoJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-\tThe paper exhibited a well-organized overall structure, and including visual information and illustrative examples was beneficial for enhancing comprehension.\n-\tThe method proposed in the paper for quantitatively evaluating the effectiveness of concept vectors is novel.\nWeaknesses\n-\tThe absence of a comparison with existing methods for deriving concept vectors makes it challenging to ascertain the advancements made by the proposed method.\n-\tThe evaluation method for the learned concept vector, presented in section 4.2, is based on a small sample size of 30 participants, and the number of labels mentioned in each section is around 10, which is considerably limited. These few samples constrain an objective assessment on the efficacy of the method.\n-\tWhile the method describes the process of deriving the gradient through the element-wise product of the feature gradient and the feature, it lacks an explanation for adopting this approach over utilizing the feature gradient only.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Understanding the inner workings of a deep learning models is crucial. One of the methods for understanding the results of deep learning models is concept vectors. Existing methodologies identified concept vectors through manual annotation or supervision. However, the authors propose a method that automatically identifies concept vectors using an unsupervised approach. A significant advantage of this method is its independence from the model architecture, task, and data type due to its automatic determination of concept vectors.\nMoreover, the authors have empirically and quantitatively evaluated how well the deep learning model has learned the label information. Particularly, in the quantitative evaluation, they introduced a novel approach which involves reevaluating the performance of the model after retaining the weights of low-concept vector components in the intermediate layers while setting the weights of other significant components to zero. The effectiveness of the learned concept vectors becomes evident when observing maximal performance degradation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "One area for improvement in this paper could be enhancing the objectivity of its results. The findings lack comparison against other pre-existing methods for deriving concept vectors. While I agree with the authors on the necessity of automatically deriving concept vectors using the proposed unsupervised method, this paper may benefit from further evidence to ascertain whether the proposed method outperforms other approaches. Therefore, it would be advantageous to incorporate comparative performance results alongside other methods.", "broader_impact_concerns": "I do not see any concerns regarding ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Pe0Prc0mF", "reviewer": "Reviewer_Zkv5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The concept discovery approach proposed here is simple to understand and quite straightforward. I particularly liked step 2 of this work, which measures the sensitivity of the model's output to the SVD directions.\n- The paper demonstrates the effectiveness of the proposed approach across several kinds of tasks, e.g., outlier detection, showing that the output of the model is sensitive to changes in the concept, and showing that the concepts obtained are coherent, and unique.\n- The paper also considers some challenging tasks such as fine-grained classification as was done for the dog breed setting.\n\n\n### Weaknesses\n- **Writing and Clarity**: the paper's writing could be much more improved. I'll raise some minor points later, but there are issues that detract from understanding the paper more carefully: 1) there are certain statements that are made without justification, e.g., \"concept-based explanations have emerged as a superior approach that is more interpretable than feature attribution estimates such as pixel saliency\". This statement is unjustified and likely doesn't belong in the abstract. 2) Typos(see last portion of the reviews). 3) Certain portions of the paper is quite dense to follow. For example, Section 3.5, which talks about the identification of unique concept vectors is quite unclear. This section should be rewritten to make the exact steps much more clear. For example, what is being clustered? Also, is section 3.6 part of the entire scheme? It seems like Section 3.6 is more a visualization step.\n\n- **Evaluation & Synthetic Evaluation**: It is difficult to understand how reliable the approach presented here is. This is because the method is applied across a variety of settings and the results read like case studies. For example, the authors say that they confirm the previous research of Kim et. al. 2018 and Ghorbani et. al. 2019. However, one issue that plagues interpretability research and a lot of concept studies is the inherent confirmation bias of these procedures. Here is my suggestion for the authors. Can you design a toy setting where you train a model to rely on a specific concept apriori. You can then check to see whether you concept discovery approach is able to reliably recover that concept. One example of a testbed that might be useful here is the Spotcheck benchmark of Plumb et. al. from the paper \"Towards a More Rigorous Science of Blindspot Discovery\". In that test bed, they are able to manipulate the features that a model relies on. You can use a setting like that to demonstrate that this approach is effective. With a toy setting, you will be able to compute metrics that indicate how effective this approach should be compared to those of others. I believe this toy setting will be useful across the entire paper, for example, you can compute the effectiveness of the method for outlier detection, the uniqueness of the concept discovered compared to a ground truth. It is not enough to apply this method to realistic models, since we have no way to judge effectiveness in those settings; a synthetic controlled setting is required.\n\n- **Related Work**: There is a large literature on using SVD/PCA on latent representations to discover important features that a model is relying on. This work should engage with that literature. For example, a closely related paper is ModelDiff: A Framework for Comparing Learning Algorithms by Shah et. al. 2023. It'll be important for this paper to clearly discuss these papers as well.\n\n- **Details of the method**: There are a few degrees of freedom/hyper-parameters that need to be selected to use this method. For example, do you keep all the SVD directions, details of the clustering step etc. These details need to be added to the appendix.\n\n\n**Some Typos, Writing Issues**\n- Use citet and citep properly, e.g., Rosenthal & Fode (1963) in the introduction.\n- Repeated 'we propose' in third to last line of the second paragraph of introduction.\n- Discuss details of the evaluation test instead of linking to the form\n- Section 4.3: 'is' repeated\n- Section 4.5: what is M?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach to identify the key concepts, i.e., directions in latent space, that a model is relying on for its output. The approach consists of three steps: 1) perform SVD of the latent/representation space to identify the key directions of variance, 2) compute the sensitivity of the model's output along each important directions, and 3) decomposing the important SVD directions into concept directions via clustering. This approach is then used to analyze an Inception-v3 model (Resnet in appendix) trained on Imagenet. The paper also demonstrates several benefits of the proposal: e.g., outlier detection, and coherence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'll describe the key broad changes here.\n\n**Writing**:\nSee the weaknesses section. Here I believe the Section 3.5 needs to be rewritten to make all the key steps clear. \n\n**Synthetic Example with Ground Truth**:\nSee the discussion from the weaknesses section. The key addition that I am seeking is a way to judge the effectiveness of the proposed method. I would like a setting where the key ground truth concept that a model relies on is known. The method here can then be tested against that setting to test whether this approach returns the key features that the model is relying on.", "broader_impact_concerns": "I don't see any ethical concerns with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method for automatically discovering concepts learned by deep learning models. The approach identifies concept vectors in an unsupervised manner by performing SVD of the latent representation, ranking the singular vectors through sensitivity-based analysis, and decomposing the important SVD directions into concepts via clustering. Initially, the reviewers provided positive feedback on the submission, highlighting its generality and capability to handle any model architecture or task. They also noted its usefulness for dataset exploration and outlier detection. However, they raised concerns about the paper's clarity, the complexity of the approach, the lack of objectivity in evaluation, and the absence of comparison to explainability methods from the literature. During the discussion period, the authors addressed these concerns by modifying section 3.5 for improved clarity, conducting additional experiments for quantitative assessment, and providing comparisons to GradCam. They also explained why a direct comparison with concept methods might not be fair. Following these revisions and discussions, there was a consensus among reviewers to accept the paper.

 
The AE has carefully reviewed the submission and the discussions. The AE appreciates the rationale of the approach, which is sound and can bring important benefits in the field of concept explainability. The proposed unsupervised concept learning is convincingly validated by evidence. Particularly, the additional experiments and discussions provide much stronger validation of the proposed method. The applicability of the approach for dataset exploration and outlier detection is also highly valuable. While the discovered concept vectors may not be human understandable, the AE acknowledges this limitation, which can be addressed by future works in the field. The AE appreciates the improvements made in the paper's presentation and the inclusion of changes that were requested by the reviewers during the discussion period. Therefore, the AE recommends acceptance. As a minor change, the AE suggests that the authors expand the related work section to better position the paper's focus and put it into perspective with respect to state-of-the-art explainability approaches and the discussion in section 5.
Decision: Accept (poster)