Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees
Authors: Johanna Vielhaben
Stefan Bluecher
Nils Strodthoff
Venue: Accepted by TMLR ()

Abstract: The completeness axiom renders the explanation of a post-hoc eXplainable AI (XAI) method only locally faithful to the model, i.e. for a single decision. For the trustworthy application of XAI, in particular for high-stake decisions, a more global model understanding is required. To this end, concept-based methods have been proposed, which are however not guaranteed to be bound to the actual model reasoning. To circumvent this problem, we propose Multi-dimensional Concept Discovery (MCD) as an extension of previous approaches that fulfills a completeness relation on the level of concepts. Our method starts from general linear subspaces as concepts and does neither require reinforcing concept interpretability nor re-training of model parts. We propose sparse subspace clustering to discover improved concepts and fully leverage the potential of multi-dimensional subspaces. MCD offers two complementary analysis tools for concepts in input space: (1) concept activation maps, that show where a concept is expressed within a sample, allowing for concept characterization through prototypical samples, and (2) concept relevance heatmaps, that decompose the model decision into concept contributions. Both tools together enable a detailed global understanding of the model reasoning, which is guaranteed to relate to the model via a completeness relation. Thus, MCD paves the way towards more trustworthy concept-based XAI. We empirically demonstrate the superiority of MCD against more constrained concept definitions.

Published in Transactions on Machine Learning Research (05/2023)
Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees
Johanna Vielhaben
johanna.vielhatenn www.haaee@hhhhhh.haaaaa.a...
Paplainable Artificial Intelligence Group
Praunhofer Heinrich-Hers Hert----ttt---ttttuuee Institute
Stefan Blüccher
bluecher@luu-erliinee
Machine Learning Group
TU Berlin
Nils Strodthoff
wils.strodthof@@uol.dd de
Division Al4Health
Oldenburg University
Reviewed on OpenReview: //tps://ort openreview. net/ forum? id- KиBQP=7HKh
https: //
Abstract
The completeness axiom renders the explanation of post-hoc eXplainable AT (XAT) method only locally faithful to the model, i.e. for a single decision. For the trustworthy application of XAI, in particular for high-stake decisions, a more global model understanding is required. To this end, concept-based based methods have been proposed, which are however not guaran- teed to be bound the actual model reasoning. To circumvent this problem, we propose Multi-dimensional Concept Discovery (MCD) an extension of previous approaches that fulfills completeness relation the level of concepts. Our method starts from general linear subspaces as concepts and does neither require reinforcing concept interpretability nor re-training of model parts. We propose sparse subspace clustering to discover improved concepts and fully leverage the potential of mulli-liiensionall subspaces. MCD offers two complementary analysis tools for concepts in input space: (1) concept activation maps, that show where a concept expressod within sample, allowing for concept characteri- zation through prototypical samples, and (2) concept relevance heatmaps, that decompose the model decision into concept contributions. Both tools together enable a detailed global understanding of the model reasoning, which is guaranteed to relate to the model via completeness relation. Thus, MCD paves the way towards more trustworthy concept-based XAL. We empirically demonstrate the superiority of MCD against more constr concept. definitions.
1 Introduction
Explainable AT (XAT) allows to peek insight the black box of inherently complex deep learning models. Local interpretability methods are particular valuable, as they measure attributions for an individual instance, which are casily comprehensible for any kind of end-users,
(Covert et al., 2021 Lundberg & Lee 2017
Montavon et al. 2018 Samek et al. 2021] for reviews. For example, local methods make prediction interpretable on the level of single images or individual bank customers for an image credit risk classifier, respectively. Importantly, the commonly employed completeness ariom (altributions sum up to the model prediction) ensures a meaningful interpretation attributions (Lundberg Lee 2017 Sundararajan et al, 2017). However, to actually comprehend the model reasoning we require global model understanding, which reliably explains the model behavior across multiple instances (e.g. a group of female vs. male bank customers). We stress that it is not viable to require end-user aggregate local attributions into common

---

Published in Transactions on Machine Learning Research (05/2023)
DI: canonical directions
D2: orthggona directions
D3: a atbiuary directions
D4: linear subspaces of arbitrary dimensionality Cg
C
C
C
C
C
allow rotation
allow mo-oothagrrality
allow multi- dimensionallity
Figure 1: We strive for the most general decomposition of the hidden feature space, spanned by the neurons C2, into linear structures that form the concepts C³. The most constrained approach is to identify concepts with single neurons (D1), i.e. directions in feature space aligned with canonical basis vectors. If one allows for arbitrary rotations the concept directions, one arrives at D2. Leaving aside the orthogonality constraint, D3 allows concepts to form arbitrary directions in feature space. Finally, allowing concepts to form multi-dimensional subspaces, we arrive at the most general approach D4. Previous concept- based methods are based on D1, D2 and D3. We choose the most general approach D4, to discover concepts that are faithful.
model features (concepts). Such a procedure is prone to human confirmation bias and it is not clear how the imagined concepts align with the actual model reasoning. This urges for novel local and concept- based interpretability methods, which allow to understand shared model structures (used across multiple samples) for an individual instance. This idea was first formalized by Kim et al. (2018) and further developed by ACE (Ghorbani et al. 2019: and its successors (Ych et al.) 2020; Zhhag et al., 2021 Crucially, our work re-introduces completeness within the context of concept-based explanations. Thereby, concepts obtained within our dillti-dimensional concept discovery (MCD) scheme are locally and globally interpretable in terms of a well-defined completeness decomposition. We outline the benefits of MCD in the following paragraphs.
Concepts as multi-dimennional subspaces Indisputably, coucept discovery in neural networks inher- ently linked structures in the activations of intermediate feature layers. In Figure [ we illustrate different approaches to decompose the hidden feature space (union of all possible activations) into meaningful con- cepts, which are mathematically formalized as linear structures. As an illustrative example, consider the activations of the last convolutional layer just before average pooling and a linear classification head. Tn this case, the model part remaining after the intermediate feature layer can only exploit linearly separable concepts, hence justifying the linearity assumption here. The most constrained definition (left most panel, D1) is to directly identify concepts with canonical basis vectors of the feature space (Bau et al, 2017). our example, D1 identifies each convolutional channel with a concept. A slightly more general definition is allow concepts lie on orthogonal directions other than the unit axes in feature space (D2). In our example, this means, concepts are formed by orthogonal linear combinations of convolutional channels. Such concept decomposition can be obtained via a principal component analysis (PCA) of the feature space (hhang ct al. 2021). Going one step further, we disregard the orthogonality constraint and allow arbitrary directions in feature space (D3) (Ghorbani et al.) 2019 Kim el al., 2018 Yeh et al.) 2020 Zhang el al. 2021). Thereby, we can characterize related concepts which are linearly independent but not orthogonal. This is sensible bocause in general, the model has no mechanism that enforces orthogonality of concepts (for example different parts of an animal). Allowing for arbitrary general definition of linear decomposition (D4). Coming back to the CNN example, D4 allows concept.
subspaces unfolds the most
2

---

Published in Transactions on Machine Learning Research (05/2023)
to lie on a hyperplane spanned by multiple directions of the convolutional channels. We argue, that this general approach enables the most failhful concepts among D1-D4, as it allows to capture any meaningful linear structure within the hidden feature layer (benefit 1).
Multi--iimensionality ensures concise explanations Concepts strive to organize the information about the global model reasoning in a concise manner. To this end, we want to cover the relevant feature space with only few concepts and avoid fragmentation into large number of low domdddiiiie subspaces. As first step, we propose a concept completeness score, which mcasures the fraction of model prediction jointly covered by all concepts. We find that it requires significantly fewer multi-dimensional MCD concepts to reach a specified level of completeness as compared to more constrained concept definitions (D1-D3), i.c., MCD provides more concise explanations (benefil 2).
Re-establishing completeness for concepts To define concept relevances, in Section 2.3 we uniquely decompose the hidden activations in conjunction with the model prediction into concept parts. To this end, we restrict to a high-level feature layer which is only succeeded by linear operations (e.g. a linear classification head with global pooling). Then, the concept relevances follow a completeness relation (benefit 3), i.e., summing all concept relevances equals the final prediction. Thus, we restore the often-desired completeness property mentioned above for concept explanations. We stress, that our concept relevances follow directly from the decomposition into concept parts and do not invoke any additional XAI method nor retraining model parts. Phrased differently, MCD can completely capture the model reasoning solely in terms of linear operations on concept subspaces.
In summary, MCD is a consistent framework to discover faithful concepts, which are guaranteed to rely the actual model reasoning via the completeness relation. Our framework provides several possibilities to investigate the discovered reasoning structure in input space. Thus, we see the main utility of MCD in the domain of model understanding and certification. Concepts provide insights into model behavior that generalize across samples and are therefore a valuable tool for systematic investigations of spurious correlations (model biases) (Lapuschkin et al.) 2019 Palatnik de Sousa et al.) 2021 Weber et al. (021), as
well as for scientific cliscovery (Blücher et al. 2020 Hägele et al. 2020 McGrath et al. 2022 Sarčevíć et al. 2022), where the model serves as a proxy for the unknown relationships in the data.
2 Multi-dimennional Concept Discovery (MCD)
We organize this methodological section into three parts: First, we introduce our novel concept definition Second, we describe practical concept discovery procodures that align with this definition in Section 2.2 Third, we introduce concept decomposition and discuss how to construct local and global concept importance that fulfill concept completeness relation in Section 2.3
in Section 2.1
Fig. g presents a schematic summary of our MCD framework, which we shortly summarize here. During the training phase, the approach discovers multi-dimensional subspaces (concepts) in the hidden feature space of model, which we mathematically capture as concept bases. These concept bases are obtained from (i) clustering feature vectors (using some particular algorithm) and (ii) characterizing clusters by their dominant principal directions (lower left pancl). During the testing phase, we can compare now feature vectors (e.g. from a new test. sample) with these concept bases and thereby maallye/characteriie all concepts (right panel). Here, we provide two complementary tools: concept activation maps, which highlight strongly expressed regions of the concept in input space and concept relevance maps, which show how indicative a particular concept. is for the class predictions.
2.1 Concept definition
Concepts are inherently tied to the hidden representations of intermediate feature layers. For our concept definition, we split the model ff into two parts, f = g o h, where the mapping to a hidden fcature layer, which is mapped to the prediction by g. Our definition then relies on hidden representations h(a) E #xxXFF of input samples (height H, width W and number of features F, see upper left panel in Figure 2
3

---

Published in Transactions on Machine Learning Research (05/2023)
Testing
Model
concept layer
Feature
Decompuse Jeature vector into concept
concept activation map
input sample
R
representation mapping h((a)
prediction mapping
police car
pejaas into and rescale to ipput
wklh W
features
concept completeness
Multi-dimensionl concept discovery (MCD)
Training
pooling
prediction
sumples
ii Chustering
(ii) Construct
concept buses
feature vectors
PCA
day
{+}
mostil day input sue
Basis C+
d
concept relevance heatmap
Figure 2: Schematic illustration of the MCD framework for concept discovery. The upper left panel illustrates how the model is split into representation and prediction mapping. Feature vectors extracted from the representation mapping of sample. The lower left panel illustrates the concept discovery methodology of MCD (Section 2.2 First, randomly choose and cluster set of feature vectors {Ø} from a selection ion of samples (using any clustering algorithm). Second, construct subspace bases for all clusters C via PCA (intrinsic dimension d'). The upper right panel corresponds to the construction of concept activation maps and the lower right panel shows the construction of concept relevance healmaps, both laid out in Section 2.3
We spatially deconstruct the feature maps h(a)) and obtain feature vecto 6 E RF for cach location (z, y) E {1,..., H) x f1...., W}. We now strive to identify concepts as (linear) structures in this F- dimensional feature space and pose no additional restrictions (one-dimensionality and/or orthogonality) on the structure the subspaces.
Definition 1.
We define concept C d' dimensional lincar subspace in the F-dimensional feature space, spanned by the basis vectors ej,
C² = span (fc/li= 1,, ,d'))
-.....dd)
(1)
In particular, the dimensionality d' vary among the concepts 1,..., ne- We denote the number of concepts n. and present constructive way to determine it in Section 2.3 assune that the concept subspaces are pairwise disjoint.
In Figure I we illustrate the linear structures that concepts could possibly form in hidden feature space: from single directions (D1-D3) to linear subspaces (D4). The MCD concept definition above corresponds D4, which is the most general linear structure, i.e., arbitrarily orientated multi-dimennsonnll linear subspaces. Note, that exploring even more general, non-lincar concept structures, such sub-manifolds in feature space, is an interesting idea for layers where non-linear operations follow. For these, linear multi-diimensionll subspaces represent an improvement over the previous, more constrained linear concept definitions in terms of faithfulness. However, for a last hidden layer that is only followed by linear classification head, which we specialize to in Section [.33 linear subspaces are the most general structure that can be separated, and thus form a faithful model concept definition.
1 Vectors denoted lower-case bold (ø E RF).

---

Published in Transactions on Machine Learning Research (05/2023)
2.2 Concept Discovery
Typically, concept discovery, ie., obtaining concepts as defined by Equation | can be subdivided into two steps: (i) cluster a user-defined sct of feature vectors {d into clusters C ......"* C and (ii) identify a representative basis C² = {cylú = ..) for each concept cluster C (lower left panel in Figure
(1) Clustering feature vectors In principle, any clustering method can be considered to discover concept clusters in feature space. This includes well-established baselines such as k-means clustering or PCA. Both have previously been proposed in (Zhang et al., 2021) to identify one-dimensional subspaces. However, k- means does not incorporate any information about the final objective to identify linear subspaces as opposed to general clusters and PCA is restricted to orthogonal, one-dimensional subspaces. We, therefore, propose dedicated approach for this particular purpose to discover miltii-iimesioonll linear subspaces and draw the rich body of literature on sparse subspace clustering (SSC) (You et al., 2016a Soltanolkotabi & Candes, 2012 You el. al. 2016b Elhamilar & Vidal 2013). SSC clusters datapoints that lie on a union of separate low-dimensional subspaces embedded in a high-dimensional space. It is based on the idea, that a feature vector & can be expressed as a linear combination of other feature vectors from the same
C², As nicely laid out in (Elhamifar & Vidal 2013), SSC is idcally suited to identify clusters of linear subspaces and provides number of advantages over standard clustering algorithms, which directly applied to the data: SSC does not rely the spatial proximity of the data, it can be implemented robustly against noise and outliers and does not require specifying the cluster dimensionalities in advance.
We start out with a user-specified set of samples S for which we aim to discover concepts. The sample selection S is unrestricted: the user can decide on class-specific samples/ /concepts or use all training samples to obtain completely unspecii concepts. Then, the first step of SSC is to find a sparse self-representation matrix that expresses each Φ in terms of minimal number of other feature vectors (d% Second, spectral clustering is applied to the self-representation to obtain clusters C². We provide technical details on the particular subspace algorithm in Appendix A
(ii) Constructing concept bases We have now identified clusters Cl...., Ch., which contain all feature vectors th. from the training set. Next, we want to obtain general concepts and become independent from the original specific cluster members. To this end, we aim to identify a basis C¹ that robustly covers the cluster C¹. By construction, all Φ. E C² lie within a linear subspace, hence linear tool like principal component analysis (PCA) constructs an accurate basis C' for the cluster C². We determine the intrinsic dimension d' of the subspace using heuristic proposed by Fuuunnaaa & Olsen (1971) and implemented by Bac et al. (2021). The PCA components up to the intrinsic dimension d' then serve as basis vectors c' for the subspace C². Given two subspaces, we can quantify their relation in terms of their Grassmann distance,
see Appendix B
2.3 Concept decomposition
Previously, have laid out how to discover expressive set of concepts C.. Next, we discuss how new features vectors { (obtained from test set sample B) and the weights of the final linear classifier layer can be analyzed via a decomposition into concept contributions. To this end, we propose concept activation maps, concept relevance heatmaps and a global concept relevance score. These are complementary tools that form the final concept explanation, informing about the overall meaning of a concept (concept activation maps) and its impact on the classification of a specific sample (concepl relevance healmaps) or on a global level (global concept relevance score).
To ensure that the union of all concepts spans the entire feature space, we define C to be the orthogonal complement of the subspace spanned by all concepts, i.e., C+ span(C",... ...., Cme)t. To simplify the notation, we identify CB = =C+, In the following, we assume that the concept subspaces are pairwise disjoint.
2This assumption was never violated in our experiments. necessary, this could he enforced by remaving the intersection between the subspaces from both and considering it as a separate concept.
5

---

Published in Transactions on Machine Learning Research (05/2023)
Concept activation maps quantify the activation of chosen concept at certain spatial location in the input space of a sample B. For this purpose, we decompose the feature vectors ( into its unique concept contributions. Since the union of all concepts (including the orthogonal complement) forms a basis for the entire feature space, we can uniquely decompose any feature vector Φ as
d'
+1++
(2)
1=1
1=1
where op are the components of Φ in the given basis. Now, one can interpret 10'12 as a mcasure for the extent to which a certain concept is expressed in the given feature vector. Performing this step for every feature leads a concept activation map | whose spatial
vector within a sample 3, i.e., &².y = " h $2 dimensions match those of the feature layer. For fixed sample B, we normalize & such that the maximum length across all elements of the feature layer is 1, ie., we divide the vectors elementwise by max..y For CNNs, we follow the example of afSSevlaajjj et al. (2020) and compute the corresponding concept activation map in input space by bilinear upsampling in the spatial dimensions. Our concept activation maps extend the concept visualization of Zhang et al. (2021) to dinetiiaimenional concepts (upper right panel in Figure 2 For the final explanation, we also use them to characterize a concept in terms of prototypical examples. To this end, for each concopt 1, we sort test set samples by the maximum activation maxzy (f and choose the top-k samples as concept prololypes.
We stress, that our methodology is applicable beyond CNNs. In particular, one can decompose feature reprosentations of any model based on MCD. However, the prerequisite for showing concept (activation) maps in input space is the locality of the trained model, i.e, the ability to associate locations in feature and input space. Whereas this locality is built in as an inductive bias into convolutional architectures, it also emerges for vision transformer models during training, as manifested for example in localized attention maps (Caron et al.] 2021p. To substantiate these claims, we show the first concept- explanations for a vision transformer model in Section [ As a final remark, we emphasize that concept activation maps can be evaluated for any model, including self-supervised pretrained models before finetuning with classifiaction head, and any layer in the model. They reveal the learned structures in feature space and are only constrained by the restriction to linear subspaces instead of more general non-linear structures.
Concept relevance heatmaps and completencss relation As a general roquirement, any concept-basod XAI method should quantify the relevance of a concept in terms of its impact on the classification decision. To this end, we specialize to the last hidden layer, which is only followed by linear operations (e.g., mean pooling and linear classification head). We discuss the broad class of models to which this applies in the last paragraph of this section and empirically in Section
For a given class, the weight vector w E R linearly connects the final F-dimensional feature space with the scalar class prediction. First, we consider the feature vector after pooling & = t" = WH En,y &h, ( E FRF) in this very layer (see Figure 2 lower right panel). Now, we are interested in a local (per-sample) concept relevance. For this, we can decompose the class logit under consideration, & w + b, up to the bias term b, as
Me+1
Rc+1
Φ =
Σ
Φ
w
(3)
The decomposition above defines a local concept relevance p² = &' w. Aggregating relevances of from all concepts recovers the class logit prediction (up to the bias term), and thus, Equation 3 defines a completeness relation.
Second, we apply Equation (3) to the feature vectors & before pooling. This leads to a relevance heatmap ph 2% w that has the same spatial dimension as the feature layer. Importantly, rly. reduces to pd after
IIn the special case of one-dimensional concepts, o' reduces to the local concept relevance in /Zang et al. 2021
"We briefly comment on the remaining commonly desired Shapley axioms Lundberg & Lee (017)) The local concept relevance trivially fulfills them since it is built linear additive model. Formally, the hidden activation dg of given sample are segmented into concept contributions/ /unique features of. Thus, the value function corresponding to the underlying Shapley values is given by vp(5)= Eies 8/- w (linear in Φf) for 5 E (l,..., + 1}.
6

---

Published in Transactions on Machine Learning Research (05/2023)
spatial pooling. As for the concept activation maps, we spatial upsampling to map rhy back to the imput space and obtain concept relevance healmaps. Since upsampling preserves the completeness relation, these decompose the local relevance maps, commonly referred to as class activation maps (CAMs) (Zhou et al. 2016), Tx,y = WH Bay w into concept contributions.
Global relevance and completeness score Next, we establish a global (model-wide) concept relevance score, which measures the extend to which the concepts explain/cover the overall prediction strategy of the model. Recall, that all cf defined above represent basis for the feature space R... Hence, can directly decompose the weight vector into (analogously to Equation
(2)
Rc+1 1 d'
no+1
wfc
= Σ
w'
(4)
1=1
where w' Ei= wte! and by construction, w' --- for 1 L,... Te. In this case, we have
no+1
The
/ = 12 +
w
12
100122 +
1w' 110 w²))
(5)
1=1 1=1
1,1=11
The first equality allows to define
(((C)) = 101--w0110 1/ 1///
(6)
as a completeness score (fraction of which is explained by all concepts (C',,..,C':)) with respect to a given class. To the best of our knowledge, we are the first to introduce concept completeness score directly based on model parameters. Previous work (Yeh et al. 2020 defined related measure based model accuracy. Later, we will fix to define the number of concepts Th.- Note, that for an orthonormal basis (e.g., MCD-SSC-ortho, see below) the second term in Equation 5 (cosine) disappears. Then l/w| can be directly interpreted as (global) concept relevances, which sum up to the previous completeness score over all concepts. Further, the angles Equation (5) are lower- and upper-bounded by the corresponding minimal or maximal principal a between the two corresponding subspaces, i..., ghl min I L(w" R w') maxm Ok1 ght max. This means we can lower- and upper- bound |w by
Rh
no+1
Σ
Σ LK=11/4
|w'|| cos(
|wo
12012+
Σ 1,k=1,
|w'llw cos(
Obviously, the lower and upper bound coincide in the case of orthogonal subspaces. This implics, that the [w'l are also informative in the non-orthogonal case, provided the principal angles between the different subspaces are given. This highlights the intricate connection between (global) relevances and the geometry in feature space, iee,, the relative orientation of the concept spaces (specified via principal angles between pairs).
Finally, briefty comment the applicability of our approach for local and global concept relevances via Equation 3 and Equation (4. In the form described above it can be used for any model with linear layer as final layer, potentially preceded by a global pooling layer, if one aims to spatially resolve the relevances instead of considering only pooled feature vectors. This latter category covers broad range of modern CNN architectures such as ResNets, Inception-based based models but also vision transformers, that do not base their prediction on a CLS token, such as Swin transformers (Liu et al.] 2021). We envision, that our approach is even applicable, in approximate form, other feature layers apart from the final hidden layer if one locally approximates the remainder of the model by linear model, similarly as it is done by Ribeiro et al. (2016)
or by Selvaraju et al. (2020) to generalize (Zhou et al. 2016).
2.4 Alternative MCD variants
In Section 2.2
describe our algorithmic choices for the two steps of concept discovery, namely SSC for clustering of feature vectors and PCA for basis construction. As consequence of the modularity of the
A formal definition of principal angles is given in Appendix R

---

Published in Transactions on Machine Learning Research (05/2023)
MCD framework, we can easily define alternative variants of MCD, that also serve for ablation study later. To differentiate the original MCD) flavor described in Section 2.2 we name it MCD-SSC. Replacing SSC with other clustering methods gives rise to the following two MCD variants:
MCDD-means We consider k-means clustering directly applied to the features. Like SSC, it leads to multi-dimensional and in general non-orthogonal subspaces. However, the clustering algorithm does not include any information about the linear subspaces desired clustering target.
ICE/MCD-PCA We consider PCA applied the feature vectors directly. This corresponds the concept discovery algorithm considered by TCE (Zhang al., 2021). Note, that this approach already encompasses the basis identification step and directly leads to one-dimensional, orthogonal subspaces by construction.
MCD-SSC does not assume that two different subspaces C¹ and C'" are orthogonal, as there is no mechanism that promotes this during model training. Still, concept orthogonality could be enforced through the use of dedicated orthogonal subspace clustering methods (Rahmani & Atia 2017a), however, at the potential cost of slightly sub-optimal subspace clusters (Rahmani & Atia 2017b). Alternatively, this could be implemented by sequentially rotating each identified subspace into the orthogonal complement of its predecessors. The latter leads to the last MCD flavor:
MCD-SSC-orth We construct orthogonal subspaces from those discovered by MCD-SSC in an it- eralive fashion. Starting with an emply set, we explore the effect of adding one of the subspaces on the completeness defined in Equation ( and choose the one that leads to the largest increase. Iteratively, we consider adding another subspace rotated into the orthogonal complement of the span of the subspaces in the set so far, again selecting the candidate that leads to the largest completeness increase.
Later, find evidence that orthogonal are less faithful to the model than those that allow for arbitrary rotation.
3 Related Work
ACE (Ghorbani et al., 2019) uses a superpixel segmentation algorithm and k-means clustering to identify class-specific concept candidates for TCAV (Kim et al.) 2018). The concept discovery scheme of ACE has several shortcomings: The segmentation into candidate concept patches is model-indepenndnt and thus, segments are not necessarily meaningful as perceived by the model. To enable clustering of intermediate CNN activations, segments are resized and mean padded to the original input shape. This leads to artificial, off- manifold samples with potentially distorted aspect ratios and discards the overall scale information. Finally, ACE relies multiple heuristics to discard segments/clustenn both before and after k-means clustering. In contrast, MCD is coherently based on hidden model representations without relying on additional pre- post-processing. Similar limitations apply to methods that rely on ACF-discovered labeled concepts, like (Li et al.) 2021), which uses Shapley values for concept importance, and (Wu et al., (020)), which occludes particular neurons for neuron-wise relevances and transforms them into concept importances via concept classification. Recently, Crabbé & van der Schaar (2022) proposed a generalization of TCAV by invoking the kernel trick, which generalizes the concept dcfinition towards non-linear structures. However, unlike MCD, it does not allow quantifying the relevance of concept towards the model prediction and can only verily predefined concepts instead of discovering them.
ICE (Zhang et al., 2021 defines concepts as directions in feature space. Technically, this is achieved via dimensionality reduction techniques applied to concatenated flattened feature maps. ICE measures the im- portance of its blass-wise concepts using TCAV. Interestingly, ICE introduces the notion of a concept weight, which is analogous to our concopt relevances on the logit layer. However, they do not consider spatially re- solved concept relevance heatmaps and only address the special case of single-dimensional subspaces. Given these restrictions, ICE can be seen as special realization of the MCD framework, which uses dinensionality
8

---

Published in Transactions on Machine Learning Research (05/2023)
reduction methods like PCA clustering algorithms. Other methods learn concept vectors and a mapping to feature space either for all classes simultaneously (ConceptSHAP (Yeh et al. 20201) or for each class separately (MACE (Kumar et al.) 2021). PACE (Kamakshi et al. 2021)). ConceptShap, MACE and PACE all use additional regularizers to enforce concept dissimilarity. In contrast, MCD restricts the concept dis- covery process as little as possible. Importantly, each method above defines a custom measure for concept importance, which is based on approximations of the original model. In contrast, the local and global con- cept relevance within MCD is solely based on the original model parameters. Other approaches (Chormai et al. 2022) use concept definition similar to ours but use information from external attribution methods as well as orthogonality constraints to restrict the discovered concepts, whereas MCD works without such restrictions.
There is a complementary line of work of frameworks that try to identify concepts associated with particular neurons in hidden CNN representations, in conjunction with (Bau et al. 2017) or without (Achtibat et al. 2022) special concept-annotated datasets. Network Disscction (Bau et al. 2017] investigates the alignment of human-unddrstanddabl concepts and particular single hidden features (neurons). Net2vec (Fong & Vedaldi 2018 extended this by allowing concepts to be represented by combinations of neurons.
Lastly, there is a line of rescarch that constructs inherently interpretable concept models by design with (Koh
et al. 2020 Radenovic et al. 2022 Chen et al.
2020
Marconato et al. 2022 Zarlenga et al. 2022] or without
relying on concept annotations (Chen et al. 2019 The objective of the former ante-hoc concept models
is to discover concepts (Koh et al. 2020 Chen et al., 2020 Marconato et al. 2022: Zarlenga et al. 2022)
in feature space in order to identify them with known factors from the underlying data- gatt-ggnnrating process. In contrast, the objective of MCD is to recover the true concepts learned by an arbitrarily trained model, which do not have to align with concepts underlying the data-generating process. Even though technically feasible (Chen & Feng, 2012), we do not equip MCD with a mechanism that enforces identifiability with known concepts, e.g., through weak concept supervision. There is crucial difference in enforcing concept interpretability in the sense of identifiability with known concepts between ante-hoc and post-hoc approachos. Regularizing concept interpretability of post-hoc explanations might obfuscate the explanation and make the model appear more interpretable than it actually is. Concerning the latter category of concept models, our approach is best comparable with (Chen et al. 2019 as both can be reduced to lincar model operating on concepts that can be characterized via prototypes. We stress the essential difference, that our approach does not require retraining (with special training objectives) but is an interpretable reformulation of the original
model.
4 Results
We carry out experiments on ImageNet (Deng et al.) 2009). As model architectures, consider ResNet models (He et al. 2016) using original weights as provided by torchuision and updated weights provided by timm (Wightman 2019) with an improved training procedure (Wightman et al., 2021). We also present re- sults for a swin vision transformer (Liu et al. 2021), again using weights provided by bimm (SwinS3base221). In the following, we will refer to these models as ResNet50, ResNet50v2 and, Swin-T, respectively. We base all our experiments on images from a diverse selection of ten ImageNet classes, which roughly align with CIFAR10 c
4.1 Completeness arithmetic
First, we provide concrete example for an MCD explanation and showcase its completeness relation intro- duced in Section 2.3 (benefit 3). To this end, Figure 3 shows an MCD-SSC explanation of a ResNet50v2 prediction for sample of the police van class in ImageNet. The number of concepts was chosen such that the completeness measure in Equation ( reaches 77 0.5. The three information components of the explanation all provide complementary information:
namelyy (airliner, beach wagon, hummingbird, siamese cat, ox, golden retriever, tailed frog, zebra, container ship, police van)
9

---

Published in Transactions on Machine Learning Research (05/2023)
(1) Concept relevance heatmaps show the alignment of feature vector component o' associated with concept O' and the weight vector of specific class. Roughly speaking, this alignment indicates how typical the network perceives the particular instantiation of the concept for the class under consideration. Applying mcan pooling leads to a corresponding decomposition of the class logit under consideration (up to the bias term) into contributions corresponding to different concepts. This demonstrates the completeness relation on the level of concept relevance heatmaps as well on the level of logits, which represents unique feature of the MCD framework. Interestingly, for the explanation in Figure only the orthogonal complement concept contributes negatively to the class logit. The contributions of the first two concepts clearly dominate the class logit.
(2) Concept activation maps have positive scores showing how much a particular feature vector aligns with a specific concept subspace. These maps identify imput regions where the concept is highly expressed. We color-code concept activation maps as transparent overlay over the image where transparent regions indicate high activation. To guide the eyc, we also include a yellow contour line at a threshold value of 0.5 and a white one al value of 0.4.
(3) Concept prototypes allow characterizing concept subspace through examples. Here, we display the concept activation maps of three test set samples that show the highest activation with the given concept. In many cases, an untuitive meaning of a concept can be inferred most easily from these samples and numerous previous approaches present concepts in this way (Zhang et al. 2021 Achtibat et al. 2022 Yeh et al. 2020) In case the explanation in Figure 3 this could be windows/livery livery, blue lights, building, tires (and the orthogonal complement covering mainly the background). In addition, we also indicate the global concept relevances for the different concepts according to Equation
At this point, we deem it worthwhile discussing the complementary nature of concept relevance maps and concept activation maps:
Concept activation maps facilitate concept identification as they do not entangle concept activation and model prediction. More preciscly, concept activation maps provide insight into the structures present in the feature space, including those that may not directly contribute to the prediction. This is particularly evident in the case of the orthogonal complement, which is slightly activated in Figure 3 but has no positive relevance.
Concept the positive parts of the concept relevance maps correlate with the activation maps: Among all test set samples, we find mean Pearson correlation of 0.45 for the concepts of the CIFAR10 classes between the positive part of each concept. relevance map and the corresponding concept activation map (for MCD-SSC and ResNet500v2. This result serves sanity check and confirms that concept relevance is high in sample arcas where the respective concept is strongly activated. Importantly, the negative parts in the concept. relevance maps represent additional information.
In summary, the sample in Figure 3 is classified as a police van mainly due to its windows perceived as typical for the class by the network and are also the most relevant concept for the class globally, Further, all other concepts are expressed in the sample and contribute positively, except for the orthogonal complement. Thus, we can confirm that MCD-SSC concepts indeed capture and focus on the relevant model reasoning structure.
which are
4.2 Empirical evaluation
We compare MCD with sparse subspace clustering (MCD-SSC), MCD with alternative clustering, and pre- vious methods listed in Table 1 in terms of (1) faithfulness (benefit 1) and (2) conciseness (benefit 2) of the explanations.
4.2.1 Comparing faithfulness via concept flipping
In order compare the methods in Table in terms of faithfulness, we invoke the Smallest Destroying Concepts (SDC) benchmark proposed in GGhorann et al. 2019) and (Wu et al. 2020 For concepts that
10

---

Published in Transactions on Machine Learning Research (05/2023)
concept C
C
C
C
C
class logit bias = 7.77
+ 3.14
0.52
+ 0.94
+ (-1.63)
10.74
mean
mean
mean
mean
mean
mean
LOCAL
local relevance
activation
global relevance
0.52
0.42
0.35
0.22
0.86
GLOBAL
prototypes
concept space
complement
Figure 3: Completeness relation for the police van class in ImageNet. Concepts are discovered via MCD-SSC for ResNet50v2. The number of concepts is chosen such that the completeness score reaches y 0.5. We distinguish between local (sample-specífic) and global properties (characterizing a set of samples). Locally, we consider concept relevance maps, which quantify the spatially resolved contribution a concept to the prodiction. These satisfy completeness relation, as explicitly shown the first line. Concept activation maps provide complementary information and indicate how much a concept is activated depending on the spatial location in the sample. Globally, the overall relevance of particular concept is quantified by the global relevance scores. Finally, we also present concept prototypes (concept activation maps of the most strongly activated samples) to characterize a particular concept.
Table 1: Summary of concept discovery methods considered in this work.
Arbitrary
Method
Multi-dim. orientation
MCD-SSC
MCD-SSC-ortho
X
MCD-kmeans
/
/
ICE/MCD-PCA (Zhang et al. 2021 )
X
X
ACE
(Ghorbani al. 2019
X /
reflect the model's actual reasoning structure in feature space and faithful concept relevance scores, SDC should show a sharp decline of the model accuracy with the number of flipped concepts.
To evaluate SDC, we subsequently remove concepts, as represented by concept masks in input space, in order of their sample-wise (local) relevance starting from high to low. To inpaint the removed segments, we use classical imputation algorithm (Bertalmio et al., (200]) which leads to comparably realistic imputed images. Thus, the model is evaluated on-manifold in contrast imputing with gray patches as often done in the literature (Smek et al. 2017). For similar reasons, avoid the Smallest Sufficient Concepts (SSC)
11

---

Published in Transactions on Machine Learning Research (05/2023)
0.8
MCD-SSC
sample
MCD-sSC
MCD-SSC-erthe
MCD-SSC-ontt
0.7
MCD -m-aam
ICE
prediction accuracy
ACE
0.5
0.4
MCD-kmneas
ICE
ACE
0.2
0.0
0.2
0.4
0.6
percenlage of deleted pixels
Figure 4: Left: Concepts are flipped one at a time in descending order of local concept Importance/TCAl score, respectively. We measure the decline in model accuracy and show the mean accuracy across CIFAR10 classes against the fraction of deleted pixels. Meaningful concept discovery and quantification methods are supposed to show a sharp decline in this figure, but the decline should not happen after flipping only single concept (i.e. the whole object). Right: Qualitative comparison between hard concept assignments.
benchmark, which would require high-quality imputation algorithms to avoid evaluating the model far from the data manifold. We obtain concept masks, i.e., hard concept assignments, in input space by taking the argmax of the corresponding concept activation maps over all concepts including the orthogonal complement. After the argmax operation, we disregard (do not remove) the orthogonal complement during the SDC experiments. For each concept mask we obtain local relevance scores by pooling the corresponding concept relevance heatmaps over the respective regions. This provides concept masks in input space which are ordered according to their importance. ACE does not provide a measure of per-sample concept relevance. Therefore, we revert to the order of their (global) TCAV scores after discarding concepts where statistical testing in comparison to random input samples fails to stay below p 0.05. In contrast to previous studies [Ghorbani ct al. 2019 Wu et al. 2020), we report the model performance depending the fraction of occluded pixcls, which is essential for comparability since the segment size varies between different approaches. In order to show meaningful average of the samples across all classes we flip only as many concepts as are present for the class with minimum number of concepts ne for each method. We base our evaluation on the CIFAR10 classes described above and work with the ResNet50v2 model, for which we extract concepts from the last hidden layer. For all methods within the MCD framework, we fix the number of concepts a class-dependent way such that we reach a completeness score of 17 0.5.
In the left panel of Figure A we show the results of the SDC experiment. As mentioned above, meaningful concept discovery and quantification method should show a sharp decline in Figure However, in principle, the sharpest decline is achievable via assigning the complete object to a single concept. Such a concept would simply highlight the entire relevant region, i.c., this would not provide any insights beyond those that can be inferred by standard (non-concept-basse)) attribution methods such as LRP (Bach et al., 2015), PiedDiff (Blücher et al. 2 or Shapley values (Lundberg & Loe, 2017). The SCD results show, that ICE/MCD- PCA and MCD-SSC-ortho consistently detect only single relevant concept, as the accuracy curve stagnates after flipping the first concept. Thus, ICE/MCD-PCA and MCD-SCC-ortho, counteract the benefits of concept-based explanations. Since both approaches rely on orthogonal concepts, this constraint seems unfitted for a fine-grained analysis of related but distinct characteristics in feature space. Among the remaining algorithms, MCD-SSC shows the strongest decline as compared to MCD-kmeans and ACE, hence its discovered concepts are the most faithful.
To provide a qualitative impression of the concept relevance heatmaps across methods, we show them together with concept activation maps for selected sample of the golden retriever class (one of the CIFAR10 classes)
12

---

Published in Transactions on Machine Learning Research (05/2023)
concept space
complement
4.63
2.32
+
1.45
1.01
0.03
+ -1.54
MCD-SS SSC
MCD MCDD-mmeans MCD -SC--o
8.87
0.0
+ 0.46
+
0.09
0.02
+ -1.34
6.93
0.86
1.72
-1.6
6.28
7.04
0.1
+
-0.27
-0.4
-0.06
0.11
+
-0.57
+
+
ICE
ACE
+
Figure 5: Concept heatmaps and activation maps for ResNet50v2 and randomly chosen sample from the golden retriever class in ImageNet. The number of concepts is chosen such that the completeness score reaches 7) 0.5. Concepts are ordered from left to right according to global concept relevance. Concept heatmaps are titled by the pooled local concept relevance that sums to the prediction logit minus the bias. For ICE, we only show the first six out of 142 and for ACE the first six out of 25 concepts. For ACE, complement exists. For MCD-kmeans, relevance is distributed over all three concepts and also four of the five MCD-SSC concepts have notable relevance. In contrast, among the MCD flavors with orthogonal concepts (MCD-SSC-ortho and ICE), only one concept notably contribules to the prediction.
in Figure 5 and more equivalent results for other classes in Figures and S In Figure (right panel),
also show hard concept assignments for example image of the golden retriever class, which form the basis of the concept flipping experiment described above. These visually support the findings of the concept flipping experiment. Most approaches only discover single concept for the dog (apart from potential genuine background concept). In particular, consider the concept assignments in Figure 4 on the right: Here, both orthogonal approaches do not distinguish between the dog head and fur parts on the image. contrast, the unconstrained MCD-SSC approach can successfully distinguish these two correlated regions features space (incorporate two similar concepts) and shows the most fine-grained decompasition.
"Correspondong prototypes be found in Figures
13

---

Published in Transactions on Machine Learning Research (05/2023)
To summarize the results of the concept flipping experiment, our general MCD definition leads to the most faithful concepts, as the two unconstrained MCT) flavors (MCD-SSC and MCD-kmeans), show the steepest descent among all methods without reverting to the non-informative solution of a single relevant concept.
4.2.2 Conciseness of explanations
For an accessible explanation, it is desirable, to explain the model reasoning with as few meaningful concepts as completely as possible, i.c. to deliver concise concept explanations. To compare the conciseness of concept explanations measure the number of concepts required to reach a certain completeness score n) i.c. how many concepts are necessary to cover the whole relevant feature space. Again, there is a trivial solution, namely leveraging more and more dimensions to cover the majority of feature space via a single concept. Therefore, we additionally evaluate the average subspace dimension di and the mcan (scaled) Grassmann distance AK,, as defined in Equation (10), between all concept pairs (A: I) within one class e lo quantily how dissimilar two concepts are. In summary, we argue that concepts should be concise (small ne), but dissect the feature space into meaningful building blocks of model reasoning. While the latter is difficult to quantify, we argue that there is a trade-off between (1) covering feature space with very few concepts of high dimensionality and potentially small distance vs. (2) dissecting it into high number of concepts with small dimensionality (extreme case: one-dimensional). To support this reasoning, we also inspect the visual impression of concepts for a selection of classes. Again, we base our evaluation on the CIFARI0 classes and concepts for the last hidden layer of the ResNet5002 model. For all methods within the MCD framework, we fix the number of concepts in a class-dependent way such that we reach a completeness score of n) = 0.5.
We list the number of concepts ne that, is required to reach a completeness score of 1/ 0.5 and d, in Section [.2.]] To provide visual comparison of the concepts discovered by these methods, we show concept activation maps of prototypes for basketball, golden retriever and airliner class in ImageNet in Figures g to I and judge how broad they appear in input space. MCD-kmeans discovers the smallest number of concepts with the highest. mean concept dimensionality of 74.7 and the smallest inter-concept distance (mean(A = 0.83) among all methods. This is reflected in the visual appearance of the concept prototypes, which are visually broad and difficult to distinguish. MCD-SSC discovers on average 4.8 concepts with a 41% smaller mean concept dimensionality of 44.2. Visually, concepts are medium broad and are easier to distinguish in input space than for MCD-kmeans, which is reflected in higher inter-concept distance of 1.19. When requiring orthogonality the Grassman angle is fixed to mean(A and ICE). For orthogonal concept one concept is medium broad in input space while all others are almost not activated. Most likely, the orthogonality constraint hinders the concepts to reflect a natural similarity between certain concepts. This aligus with the conclusions drawn from the SDC benchmark. Most notably, to achieve a comparable model faithfulness (completeness score of 17 = 0.5) 30 times more one-dimensional ICE concepts than multi-dimensional MCD concepts are required, meaning this method delivers concept explanations that are not concise. Intuitively, a single concept is split up into several concepts, which is also reflected in their weak activation on test set samples. Lastly, the visual impression of ACE concepts is fixed by the choice of the superpixel algorithm. While ACE concepts are all one-dimensional, they do not provide a mechanism to quantify how complete they are, thus we cannot quantify ne required to reach a completeness of 50%. As an overall summary, MCD-SSC is superior in dissecting the feature space into enclosed and meaningful concepts.
T/2 = 1.57 (MCD-SSC-ortho
4.3
Use case: MCD concepts reveal differences in classification strategies between model architectures and training procedures
Finally, we showcase how MCD can unravel different classification strategies depending on the model ar- chitecture (ResNet50 vs. Swin-T) and the training strategies (ResNet50 vs. ResNet50v2), most notably the fact that the ResNet50v2 was trained using a multilabel loss. The test accuracies for the subset of CIFAR1O-classes are 0.80 (ResNet50), 0.84 (ResNet500v) and 0.86 (Swin-T). Here, we focus on MCD-SCC and, as before, restrict ourselves to concepts in the last hidden feature layer. First, we compare the discov-
We scaled version of the original Grassmann distance that aggregates the principle angles (in radian) between two subspaces, for which 0 5 A < w. Two special cases are A 0, meaning that subspace bases vectors are perfectly aligned, and A&l n/2, meaning that they are orthogonal.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: Summary of concept discovery methods considered in this work in comparison to prior work from Zhang et al. (2021) (ICE/MCD-PCA) and Ghorbani et al. (2019) (ACE). We measure average subspace dimension d' and the number of concepts
that is required to reach a completeness score of = 0.5
for ResNet50v2 on the CIFARIO classes. A small number of relevant concepts ne is desirable since this summarizes the complete model into an accessible and meaningful format. Here, multi-dimensional concepts have an advantage. Additionally, we evaluate the mean (scaled) Grassmann distance Akl, see Equation (10). between all concept pairs (k;,) within one class e to quantify the distinct between concepts. The visual inspection is based on prototypes of the basketball, golden retriever and airliner class concepts in Figures Medium broad and distinct concepts are the most informative.
to 11
Method
mean(d")
mean(n.)
mean(4.)
Visual inspection
MCD-SSC
44.2
4.8
1.19
medium broad
MCD-SSC-ortho
44.2
4.8
1.57
only one broad (rest narrow)
MCD-kmeans
74.7
2.7
0.83
very broad
ICE/MCD-PCA
146.7
1.57
only one broad (rest narrow)
ACE
n.a.
n.a.
medium broad
ered concepts between the models by the activation maps of concept prototypes for the beach wagon class of ImageNet in Figure We fix the number of concepts to n. 5. For Swin-T, we only apply a spatial upsampling of the concept activation maps from the feature the input space 14 14 in order to account for the 16 16 patch tokenization. We find that ResNet50 concepts, which could roughly be identified as (car body, windows, car roof, wheels, street), are more narrow than the expression of Swin-T and ResNet50v2 concepts. The latter are related to broader views of the car, such as concepts (1, 2, 4) for ResNet50v2 and concepts (1, 3) for Swin-T. Interestingly ResNet50v2 concepts reach a much lower completeness score of = 0.49 than ResNet50 (17 = 0.89) and Swin-T (n = 0.84) for fixed n. = 5. In Figure G we show the relation between the total concept space dimensionality, the number of concepts n. and the completeness score 17 across the CIFAR1O-classes. Even for n. 30, the ResNet50v2 concepts have a lower 7) than those of the ResNet50 for n. = 3, although the former covers already much larger part of the concept space. These observations support the statement that feature space of the ResNet50v2 exhibits a comparably richer structure than ResNet50. Thus, MCD-SSC concepts can reveal interesting differences in the character of the feature space as a consequence of two different training procedures for the same architecture. Interestingly, the dependence of on ne for the concepts between two models with different architectures, ResNet50 and Swin-T, T, is quite similar. This also aligns with the visual appearance of the concepts.
To summarize, Swin-T and ResNet50 build on broader and more versatile concepts. In comparison, ResNet50v2 builds more narrow and thus specific concepts for its classification strategy. These broad concepts are not unexpected for a transformer architecture like Swin-T with coarse self-altention windows, but rather surprising finding for ResNet50 in comparison to ResNet50v2.
5 Summary and Discussion
In this work, we put forward MCD, general framework for concept discovery based on the hidden repre- sentation of trained deep neural network. Unlike prior work in the field, propose general concept definition (incorporating previous approaches) as multi-dimensional linear subspaces without restricting to single directions or enforcing orthogonality between subspaces. We use concept activation maps to visualize concepts in input space. Considering the final hidden layer representation, we can reformulate the original model as a linear classifier acting on linear concept subspaces without the need to retrain with a special objective. This leads to completeness relation, i.c., a natural decomposition of class logits into contribu- tions corresponding to specific concepts and allows to resolve their spatial importance in terms of concept relevance heatmaps. As particularly suited realization of our framework, we put forward MCD-SCC, which
15

---

Published in Transactions on Machine Learning Research (05/2023)
0.55
ILS4
1.0
D:4
0:41
concegge Sgaacc
0.45
2.11
completenexsy
0.7
0.6
ResNet5022
ResNec50
Swin-T
200
400
600
800
1000
1200
conceptspace dimensionality Y
Figure 6: Left: Mean concept space completeness score v for the CIFAR10 classes across architectures against the dimensionality of the union all concept subspaces E, d'. The number concepts can be inferred from the points on the line where the first point on each line corresponds to n. = and the last one to The 30. ResNet50v2 shows a much lower completeness score at roughly the same n. and Σ d' ResNet50. The feature space dimensionality is F 2048 for (esNet50(v2) and F 768 for Swin-T. Right: We show MCD-SSC concept activation maps for concept prototypes for ResNet50, ResNet5002 and Swin-T and the beach wagon class in ImageNet. We fixed the number of concepts to n. 5. In this way, ResNet50v2 reaches = 0.49, ResNet50 = 0.89 and Swin-T f = 0.84. Each row shows a single concept and is titled by its global concept importance. The last row shows the orthogonal complement of the concept space.
relies sparse subspace clustering for concept discovery. Based on qualitative and quantitative insights, we show the superiority of MCD-SCC over other MCD flavors that build on traditional clustering algorithms.
We showcase the ability of MCD via discriminating between hidden representations obtained from different model architectures and training strategies. This paves the way towards further novel use-cases for MCD concepts such as gaining insights in the natural sciences, e-g-, identifying sub-classes of cancerous cells in histopathology or summarizing model behavior beyond single examples and thereby systematically discover- ing model biases. MCD prioritizes faithfulness of concepts over identifiability with known human concepts by regularizers that could obfuscate the original structures learned
not including any by the model. In this way, we can guarantec to cover the original model reasoning structure, which is crucial for auditing models or scientific discovery use cases. However, this might obfuscate non-expert human users first sight, since deep learning models will most likely not rely entirely on human-like features.
Code to reproduce experiments is publicly available at http://gghbbbb.cco/////laaaaaaaa XAI
Acknowledgments
This work was supported by the German Ministry for Education and Research (BMBF) through BIFOLD (refs. 01IS18025A and 01TS18037A).
References
Reduan Achtibat, Maximilian Dreyer, Πloaa Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From "where" to "what': Towards human-understandable explana- tions through concept relevance propagation. arXio preprint 2206. 03208, 2022.
Bac, Evgeny M. Mirkes, Alexander N. Gorban, Ivan Tyukin, and Andrei Zinovyev. Scikit-dimension: python package for intrinsic dimension estimation. ar preprint 2109.02 009.02596, 2021.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klbes-Robert Müllrr, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7))e0130140, 2015.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network disscction: Quantifying interpretability of deep visual representations. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2017.
Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. Navier-stokes, fluid dynamics, and image and video inpainting. JEEE Conference on Computer Vision and Pattern Recognation. TEEE, 2001.
Stefan Blücher, Lukas Kades, Jan M Pawlowski, Nils Strodthoff, and Julian M Urban. Towards novel insights in lattice field theory with explainable machine learning. Physical Review D, 101(9):094507, 2020.
Stefan Blücher, Johanna Vielhaben, and Nils Strodthoff. Preddiff: Explanations and interactions from conditional expectations. Artificial Intelligence, 312:103774, 2022. ISSN 0004-3702.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Com- puter Vision, pp. 9630-9640, 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in Newrol Information Processing Systems, 32, 2019.
Weifu Chen and Guocan Feng. Spectral clustering: A semi-supervised approach. Neurocompuling, 77(1): 229-242, 2012.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, Dec 2020.
Pattarawat Chormai, Jan Herrmann, Klaus-Robert M'uller, and Gr'egoire Montavon. Disentangled expla-
nations of neural network predictions by finding relevant subspaces. arXiv preprinl 2212. 14855, 2022.
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209): 90, 2021.
Jonathan Crabbé and Mihacla van der Schaar. Concept activation regions: A generalized framework for concept-based explanations. Advances in Neural Information Processing Systems, 35, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-244-220 288-255,-4 2009.
Ehsan Elhamilar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. In IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2765 2781. IEEE, 2013.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8730-8738. 2018.
Keinosuke Fukunaga and David Olsen. An algorithm for finding intrinsic dimensionality of data. IEEE Transactions on Computers, 100(2):176 183, 1971.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based expla- nations. Advances in Neural Information Processing Systems, 32, 2019.
Miriam Hägele, Philipp Soogerer, Scbastian Lapuschkin, Michael Bockmayr, Wojeiech Samek, Frederick Klauschen, Elaus-Robbrt Müller, and Alexander Binder. Resolving challenges in deep learning based analyses of histopathological images using explanation methods. Scientific Reports, 10:6423, 2020.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Jihun Hamm. Subspace-based based learning with Grassmann kernels. PhD thesis, University of Pennsylvania, 2008
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recogration, pp- 770- 170-778 2016.
Camille Jordan. Essai sur la géométrie à n dimensions. Bulletin de la Société Mathématique de France, 3: 103 174, 1875.
Vidhya Kamakshi, Uday Gupta, and Narayanan C Krishnan. Pace: Posthoc architecture-agnostic concept extractor for explaining cmns. In International Joint Conference on Newral Networks, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International Conference on Machine Learning, pp.
PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daun'e III and Aarti Singh (eds.), International Conference on Machine Learning, volume 119, pp. 5338-5348, 2020.
Ashish Kumar, Karan Schgal, Prerna Garg, Vidhya Kamakshi, and Narayanan Chatapuramkershnan. Mace: Model agnostic concept extractor for explaining image classification networks. IEEE Transactions on Artificial Intelligence, 2(6): 583, 2021.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10:1096, 2019.
Jiahui Li, Kun Kuang, Lin Li, Long Chen, Songyang Zhang, Jian Shao, and Jun Xiao. Instance-wist wise or class-wisc?? a tale of neighbor shapley for concept-based explanation. In ACM International Conference on Mullimedia, pp. 3664-3672, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference Computer Vision, 2021.
Scott M Lundberg and Su-ln Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
Emanuele Marconato, Andrea Passcrini, and Stefano TSso. Glancenets: Interpretabile, leak--rooff concept- based models. UAI 2022 Workshop on Causal Representation Learning, 2022.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 222047) 19, 2022.
Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Tam Palatnik de Sousa, Marley MBR Vellasco, and Eduardo Costa da Silva. Explainable artificial intelligence for bias detection in covid et-scan classifiers. Sensors, 21(16):5657, 2021.
Filip Radenovic, Abhimanyu Dubey, and Dhruv Mahajan. Neural basis models for interpretability. Advances in Neural Information Processing Systems, 35, 2022.
Mostafa Rahmani and George Atia. Innovation pursuit: A new approach to the subspace clustering problem. In International Conference on Machine Learning, volume 70, pp. 2874-2882, 2017a.
Mostafa Rahmani and George K. Atia. Subspace clustering via optimal direction search. IEEE Signal Processing Letlers, 24(12):1793 1797, 2017b.
18

---

Published in Transactions on Machine Learning Research (05/2023)
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should trust you?": Explaining the predic- tions of any classifier. In International Conference Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Miller. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Newral Networks and Learning Systems, 28(11):2600-2777 2017. doi: 10.1109 10099/NNNS.220006 259982.
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3))247-275) 2021.
Ana Sarěević, Damir Pintar, Mihaela Vranié, and Agneza Krajna. Cybersecurity knowledge extraction using xai. Applied Sciences, 12(17):8669, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Inter- national Journal of Computer Vision, 12:32):336 359, 2020.
Mahdi Soltanolkotabi and Emmanucl J Candes. A geometric analysis of subspace clustcring with outliers. The Annuls of Slalistics, 40(4):2195 2238, 2012.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, volume 70, pp. 3319-3322 JMLR, 2017.
Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395-416, 2007.
Manuel Weber, David Kersting, Lale Umutlu, Michael Schäfers, Christoph Rischpler, Wolfgang Fendler, Irêne Buvat, Ken Herrmann, and Robert Seifert. Just another "clever hans"? neural networks and fdg pet-ct to predict the outcome of patients with breast cancer. European journal of nuclear medicine and molecular imaging, 48(10))3141 3150, 2021.
Ross Wightman. Pytorch image models.
2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arxiv preprint 2110.00 110.00447 76, 2021.
Weibin Wu, Yuxin Su, Xixian Chen, Shenglin Zhao, Irwin King, Michael R. Lyu, and Yu- Wing Tai. Towards global explanations of convolutional neural networks with concept attribution. In IEEE Conference on Computer Vision and Pallern Recognition, pp. 8649 8658, 2020.
Chih-Kuan Ych, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradcep Ravikumar. On completenes--wware concept-based explanations in deep neural networks. Aduances in Neural Information Processing Systems, 33, 2020.
Chong You, Chun-Guang Tá, Daniel P Robinson, and René Vidal. Oracle based active set algorithm for scalable elastic net subspace clustering. In IEEE Conference Computer Vision and Pattern Recognition, pp. 0928-3937, 2016a.
Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In IEEE Conference Computer Vision and Pattern Recognition, pp. 3918 118-3997,44 2016b.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Concept embedding models. In Alice Н. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invert- ible saccept-based explanations for enn models with non-negative concept activation vectors. In AAAI Conference Artificial Intelligence, pp. 11682-11690, 2021.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Bolci Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In IEEE Conference on Compuler Cision and Pallern Recognilion, pp. 2921 2929, 2016.
A SSC Algorithmic Details
Concept-deternining self-reprreentatiion We compute sparse sell-representations R for random sub- collection of -N- H-W feature vectors {d sampled from S. Here, the term reff-reerrsnntationn refers to coefficient matrix that expresses cach sample as lincar combination of all other samples. More specifically, using the notation from (Elhamifar & Vidal 2013), given the feature vectors = (oi...., .n.] E RPxn, we identify a sparse coefficient matrix = Ti...., Tn. E R=*x such that
Φj fry where Ts 0.
The particular kind of sparsity constraints that are imposed on Equation and how it is optimized depends on the chosen SSC algorithm. Here, we use clastic net subspace clustering You et al.) 2016a), which is robust against noise and scales well for large sample sizes. In all our experiments, we fix the hyperparameter 7, which balances sparsity vs. robustness, variation of this parameter over range of values from 5 to 50. As computation time for SSC is dependent this parameter, chose 7 such that this is minimized.
7 10. We confirmed that the results are not sensitive
We remove outliers based on the fr-norm as in (Sotanolkootbii & Candes 2012), where we empirically fix the percentile threshold to 0.75 and re-fit the sparse self-representation for the remaining elements.
Another scalable alternative to the elastic net clustering orthogonal matching pursuit (OMP) (You et al. 2016b), which is, however, not robust against noise and does allow for outlier removal via thresholding. Finally, the original sparse subspace clustering method from (Elhamifar & Vidal 2013) is robust against noise and outliers but does not scale to large datasets. The particular robustness and scalability properties make elastic net subspace clustering (with thresholding) an ideal choice for the first step of our concept discovery method.
(OMP)
Spectral clustering In a second step, we perform spectral clustering with the affinity matrix W = == |
which encodes the similarity of two feature vectors according to their relf-representations. We determine the number of clusters ne either via the largest gap in the spectrum of the Laplacian (Von Luxburg 2007) predetermined value. This step assigns every input. feature &; to a particular cluster Cr,...y , Cr2 to the set of outliers.
B Characterizing relations between subspaces by principal angles
In this section, we briefly review the definition of principal angles, which can be used to characterize the rela- tion between two linear subspaces. The principal angles 8,1% (Jordan 1875 (i = =.....,,in min (ini A, dim B)) between two linear subspaces A, B, are defined recursively via
cos 0,B
max T aEA.EE
b
(9)
where the maximum is taken subject to the orthogonality constraints a"aa = 0 and b²¹,, = 0 for j 1.-.. To quantify the similarity between two subspaces A and B, we use scaled version of their Grassmann
distance Hamm 2008), which is defined as,
A AB = 1 1 min(dim A, dim B) (0/BB) +... +...+ + (BAB mint dim A,dim )²
(10)
This allows comparing the similarity of concepts within a given class or across classes regardless of the concept subspaces' dimensionality.
20

---

Published in Transactions on Machine Learning Research (05/2023)
C
Qualitative results
For qualitative comparison between of the concept activation maps and relevance heatmaps between the methods in Section 4.2 we provide results for selected samples in Figures 5
7
and
In Figures
to
we
show the respective concept prototypes for all concept discovery approaches.
concept space
complement
-2.33 + 11.55 + 0.53 + -1.59
+ -3.39
MCD- -SC
+
MCD-SSC- SSC-ootto
3.51
+
4.0
+ 0.89
-0.24
+
-3.39
+
13.38
-4.29
-1.15
-3.16
4.78
MCD MCD-kmes kmeans
4.68
+
1.86
+
-1.84
-0.64
-0.27
-0.65
+ -1.23
+
ICE
0
ACE
Figure 7: Concept heatmaps and activation maps for ResNet50v2 and a randomly chosen sample from the basketball class in ImageNet. The number of concepts is chosen such that the completeness score reaches global concept relevance. Concept heatmaps
77 0.5. Concepts are ordered from left to right according are titled by the pooled local concept relevance that sums to the prediction logit minus the bias. For ICE, only show the first six out of 105 and for ACE the first six out of 25 concepts.
21

---

Published in Transactions on Machine Learning Research (05/2023)
concept space
complement
1.64
-1.92
1.64 + 3.96
0.77
+ 2.78
+ -0.3
0.09
MCD-SSC
9.09
+ 0.43
+ -0.13
+ -0.13 + -0.04 + -0.12
+ -0.52
+ 0.09
MCD-SSC-ortin
1.87
1.18
6.3
-0.69
MCD-kmeans
8.66
7.74
-0.5
0.33
0.2
-0.09
0.41
-0.14
-0.44
ICE
ACE
Figure 8: Concept heatmaps and activation maps for ResNet50v2 and a randomly chosen sample from the airliner class ImageNet. The number of concepts chosen such that the completeness score reaches 7 = 0.5. Concepts are ordered from left to right according global concept relevance. Concept heatmaps are titled by the pooled local concept relevance that sums to the prediction logit minus the bias. For ICE, only show the first seven out of 141 and for ACE the first seven out of 25 concepts.
22

---

Published in Transactions on Machine Learning Research (05/2023)
MCD-SSC
MCD-SSC-ortho
MCD-kmeans
ICE
ACE
Figure 9: Concept activation maps for concept prototypes for basketball class of ImageNet. The last row shows prototype for the complement, except for ACE, where no complement exists. For ICE, we only show the first six out of 105 and for ACE the first six out 25 concepts.
23

---

Published in Transactions on Machine Learning Research (05/2023)
MCD-SSC
MCD--SCCorrtho
MCDD-mmeans
UPAA SYD
ICE
ACE
Figure 10: Concept activation maps for concept prototypes for golden retriever class of ImageNet. The last row shows prototype for the complement, except for ACE, where complement exists. For ICE, we only show the first six out of 142 and for ACE the first six out of 25 concepts.
24

---

Published in Transactions on Machine Learning Research (05/2023)
MCD-SSC
MCD-SSC-ortho
MCD-kmeans
ICE
ACE
Figure 11: Concept activation maps for concept prototypes for airliner class of ImageNet. The last row shows prototype for the complement, except for ACE, where no complement exists. For ICE, we only show the first seven out of 141 and for ACE the first seven out of 25 concepts.
25


Reviews:
[{"review_id": "CPDjSBKZyC", "reviewer": "Reviewer_2vJn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Empirical results seem better\n\nWeakness\n\n- Presentation is not very clear. Many related concepts are not fully explained, and motivations are not clearly written. Hence, one cannot fully judge the motivation or understand the advantages of the proposed approach. \n- Overall, I find the paper lacks sound justification, and mostly are hand wavy explanations. (although I recognize that maybe this is standard in XAI field due to lack of theoretical understanding and very subjective.)\n\n\nComments:\n- XAI is not explained in abstract.\n- \"directly identify concepts with a neural directions\": is it one direction or 3?\n- losing orthogonality seems to lead to overlap of concepts. I'm not sure arbitrary orientation is a desired property for concise explanation, since they likely contain redundant information. More discussion (esp. theoretical justification) here would be helpful. \n\nSection 4.2.1 studies this aspect in more details. However, it is not fully clear to me why MCD-SSC is better than MCD-SSC-ortho. \"the greedy way these orthogonal subspaces are constructed\" - also very informal. If one purposely control more concepts to be desired (maybe via some hyperparameters tuning), would it result in more fine-grained concepts for MCD-SSC-ortho?\n\n- \"Multi-dimensionality ensures concise explanations\": although this is discussed in introduction, I don't find this argument clear. Later section also lacks a clear mathematical justifications on why higher dimension is better than one-dimension. In addition, what would a good way to determine the dimension size?\n- Some hyperparameter choices are not well motivated, via ablation study for example. For example, number of concepts.\n- How does the propose method differ from many disentangled representation learning or prototype learning?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework  for concept discovery based on hidden representation of neural networks. Different from existing approaches, authors proposed a multi-dimensional linear subspaces without restricting to single directions or enforcing orthogonality between subspaces with complete relations. Empirical results show better performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "clear and sound justification on why multi-dimension and non-orthogonality are desired. ", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "64oNJPqyRR", "reviewer": "Reviewer_MjCK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strong aspects\n\n#### Strength 1: Generalization of \"concepts\"\nThe major contribution of this study lies in the definition of \"concepts\" as subspaces in latent space.\nIn previous research, \"concepts\" are typically defined as vectors in latent space.\nHowever, it would not be natural to consider that complex \"concepts\" can be represented as a single vector in high-dimensional latent space.\nOn the other hand, subspaces can more naturally represent richer and more complex \"concepts\" by adjusting their dimensions appropriately.\nThis perspective would be an important step forward in the study of \"concepts\".\n\n#### Strength 2: Benefits of subspace\nThe strengths of this study is not only the generalization of the definition of \"concepts\", but also the extraction of prototypical data and the visualization through heatmaps, which will further help users to understand deep image classification models.\nFurthermore, the extraction of these prototypical data and the visualization through heatmaps are naturally realized through the basis expansion of the latent vectors as well as the weight coefficients of using the basis vectors of the subspaces.\nThis suggests that defining \"concepts\" as subspaces is also useful as a model interpretation tool.\n\n\n### Weak aspects\nI did not find any crucial weakness.\nAs minor suggestions, I recommend the authors to enlarge images in Figures 3 and 5 as thy are too small to inspect.\nI also recommend the authors to move some (not necessarily all) of Figures 6, 7, and 8 to the main body of the paper.\nThey are essential ingredients and should be presented in the main body.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method to extract \"concepts\" in deep image classification models as subspaces in latent space.\nIn previous research, \"concepts\" are typically defined as vectors in latent space.\nThe idea in this paper can be thus interpreted as a generalization of existing \"concepts\".\nThe authors pointed out that existing sparse subspace clustering techniques can be used to estimate these \"concept\" subspaces.\nThe authors also proposed a method to quantify the contribution of each \"concept\" in image classification using the basis vectors of each subspace, and to extract prototypical data for each \"concept\".\nFurthermore, the authors enabled to visualize the contribution of each \"concept\" in the image through heatmaps.\nThe generalization of the notion of \"concepts\", the extraction of prototypical data, and the visualization through heatmaps are useful tools for users to understand deep image classification models.\nThe authors also conducted experiments using ResNet and Swtin Transformer, demonstrating that the proposed method is more effective in extracting \"concepts\", and that there are differences in the found \"concepts\" between models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "#### Request 1.\nI recommend the authors to enlarge images in Figures 3 and 5 as thy are too small to inspect.\n\n#### Request 2.\nI recommend the authors to move some (not necessarily all) of Figures 6, 7, and 8 to the main body of the paper.\nThey are essential ingredients and should be presented in the main body.", "broader_impact_concerns": "There is no ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QlqzbYGWI3", "reviewer": "Reviewer_FPuj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ English is good.\n+ Contribution generalizes existing approaches in a sensible manner.\n+ Contribution is of interest to the XAI/CBM community.\n+ Related work is well done.\n\n- The structure of the text should be improved.\n- Some comments are not entirely motivated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a novel post-hoc, concept based explainer for neural networks.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Writing*  English is good, and I could not spot any major linguistic mistakes.  Structure and clarity, however, should be improved.\n\n- The introduction is far too detailed, the description of the method is not detailed enough.  I think the structure should be changed to improve clarity and flow.  Specifically,\n\n 1) The technical paragraphs at the end of the introduction are very hard to follow at this stage (as the reader is not yet familiar with the intent of the various steps and with the techniques involves), and should be moved later, preferably between Sec 2.1 and 2.2.\n\n 2) The schematic overview of the algorithm should be moved between to the beginning of Sec 2, and it should briefly outline the clustering, basis construction, and relevance computation steps, clearly stating what are the expected inputs and outcomes of the various steps *formally*.\n\n 3) All algorithmic details should be kept in Sections 2.2+.  Intuition about important steps of the procedure are currently missing.  The authors should briefly describe:\n - What is subspace clustering?\n - How does SSC work?\n - Why is PCA enough to recover usable bases from the SSC clusters?\nShort, high-level explanations of all these build blocks would help tremendously.  The intended inputs and outputs to all steps should be specified.\n\n 4) The MCD-kmeans and MCD-PCA approaches mentioned in p 5 should be moved to the experiments.  This would keep the focus on the proposed approach.\n\n 5) The complement subspace $C^{n_c+1}$ defined in p 4 should be introduced later, when the need for it becomes apparent.  Also, $C^{n_c+1}$ does not roll easily on the tongue.  I suggest to introduce a more obvious mnemonic, say $C^\\bot$, $D$, or similar.\n\n 6) In the experiments, clearly state all reasearch questions and provide clear descriptions of all data sets and evaluation metrics.\n\n- Eq. 2: I suggest to use use a different symbol for $\\phi^l_i$, to avoid confusion with the feature vector $\\phi$.\n\n- p 6: Footnote 4 should be moved next to footnote 3.\n\n- p 6: In the first sentence on p 6, change \"i.e., $max_{x,y} ...$\" -> \"i.e., we divide the vectors elementwise by $max_{x,y} ...$\".\n\n- p 6: In the second sentence, what is $|\\phi^l|$?  Is it the Euclidean norm?  If so, please use $\\|\\phi\\|_2$ instead.  Otherwise, please explain what $|\\cdot|$ means.\n\n- p 8: Sec 3. \"The segmentation into candidate concept patches is model-independent and thus, segments are not necessarily meaningful as perceived by the model\".  But by locality - which holds in CNNs and (as the authors mention) empirically also for ViTs - superpixels generally map to subsets of latent features.\n\n- p 9: What does \"yellow(white)\" and \"0.5(0.4)\" mean?\n\n- p 9: \"the the\"\n\n- Everywhere: \"true-to-the-model\" - I suppose the authors refer to \"faithfulness\".  If so, the latter term should be preferred as it is more common in the XAI literature.\n\n- Why are some of the equations boxed?  The boxes are unnecessary and should be removed.\n\n- Fig 2: It would be useful to draw $\\phi^\\alpha_{xy}$ as a tube.  Also, please rotate the $x$ and $y$ coordinates, for readability.\n\n\n*Quality* I have some issues with the message conveyed by the paper.  Specifically:\n\n- p 2: \"Allowing for arbitrary multi-dimensional subspaces unfolds the most general deﬁnition of a linear decomposition. Thus, this general approach enables true-to-the-model concepts as it allows to capture any meaningful linear structure within the hidden feature layer (benefit 1).\"  I agree with the first sentence - linear subspaces are the most complex linear substructure possible for a linear space - but not with the second.  Why are linear subspaces \"true-to-the-model\"?  Please, either remove the second sentence (as it doesn't add much), make it relative (e.g., \"truer-to-the-model\" or \"more faithful\"), or provide arguments in support for it.  The reason why I am insisting on this is that the \"true-to-the-model\" slogan is repeated again in the text - but to the best of my knowledge it is not justified: the model may be carrying out non-linear operations during prediction, so faithfulness is not guaranteed.  I think it's important to discuss this point thoroughly, as faithfulness is one of the main selling points of the technique.\n\n- p 4: Correlated to the above:  the authors write \"Note, that exploring even more general structures, such as concepts as sub-manifolds in feature space, is an interesting idea. However, these do not allow for a decomposition of the feature vector and hence do not lead to a completeness property, which is central to the deﬁnition of concept relevance maps\".  The meaning is unclear to me.  I'd appreciate if the authors could elaborate on what they mean.  A second observation is that this seems to be in conflict with the \"true-to-the-model\" motto: if the model does reason in terms of manifolds rather than linear subspaces, explanations given in terms of subspaces (that as such satisfy the completeness property) cannot possibly be faithful to the model.  So I'm not sure what benefits completeness yields - especially from the perspective of faithfulness.  My understanding is that they are both useful properties, but quite orthogonal to each other.  They authors should discuss this briefly.\n\n- p 4: \"assume without loss of generality that their subspaces are pairwise disjoint.\"  Why is this w.l.o.g.?\n\n- p 4: \"does not take advantage of the spatial proximity of the data\" -> \"does not rely on ...\" (the sentence, as it is, implies a negative connotation)\n\n\n*Experiments*: \n\n1) The SCD dataset is not described, so I don't know whether this evaluation is in any way useful.\n\n3) Ideally, MCD should be compared to ground-truth explanations.  To this end, the authors could define data sets where the ground-truth explanations are known, for instance, using data sets with known concept annotations and defining synthetic labels based on simple logical combinations of concepts, then testing whether MCD can recover the concepts responsible for the label.  Right now, it is very hard to know whether the MCD explanations are faithful to the concept's reasoning.  This is optional, but it would help immensely.\n\n\n\n*Related Work*  The related work is well done.  Two aspects that may be worth discussing are:\n\n1) Automatically extracted concepts - even those featuring nice theoretical properties like completeness - are not necessarily human understandable.  This is a very important aspect.  In fact, assuming concept interpretability is viewed as concept identification - as done in:\n\n Marconato et al., \"GlanceNets: Interpretabile, Leak-proof Concept-based Models\", NeurIPS'22.\n\nand implicitly in other works on concept-based explanations - it is very hard to ensure concepts extracted automatically from networks are interpretable.  This is exactly the reason why TCAV, Concept-Bottleneck Models, Concept Whitening, and GlanceNets require a modicum of concept supervision.  The authors should explain what makes concepts extracted by MCD intepretable (regardless of the how the network under examination is structured and learned), and discuss possible limitations of their (unsupervised) approach.\n\n2) In the same vein, it would be worth briefly discussing whether concept supervision could be injected into the explanation procedure to guide the subspace decomposition.\n\n3) The idea of implementing concepts as subspaces has been considered in:\n\n  Zarlenga et al., \"Concept embedding models\", NeurIPS'22.\n\nIt would make sense to discuss the relationship between their (ante-hoc) solution to the proposed (post-hoc) approach.", "broader_impact_concerns": "As I mentioned, explanations extracted by *unsupervised* concept extraction methods always risk being potentially misinterpreted by stakeholders.  The authors should discuss this in the conclusion.  See:\n\n Chen et al., \"Does the explanation satisfy your needs?: A unified view of properties of explanations\"\n\nfor a reference.  This is a very recent overview of properties of explanations and mentions pitfalls in interpreting them.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new post-hoc explainable AI method, which leverages multi-dimensional linear subspaces without orthogonality restrictions to discover hidden concepts. Empirical results demonstrate the potential of the proposed method.

The reviewers agree about the contribution of the work to the problem of concept-based explanation. The presentation issues of the first draft were fixed during the revision. While some reviewers have concerns about the theoretical justifications of the proposed method, the empirical results are convincing enough to validate the proposed method and support the claim on the possible advantages of multi-dimensional concepts without orthogonality restrictions. All reviewers lean acceptance after the revision.


Decision: Accept (poster)