Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors
Authors: Angus Nicolson
Lisa Schut
Alison Noble
Yarin Gal
Venue: Accepted by TMLR ()

Abstract: Concept-based explanations translate the internal representations of deep learning models into a language that humans are familiar with: concepts. One popular method for finding concepts is Concept Activation Vectors (CAVs), which are learnt using a probe dataset of concept exemplars. In this work, we investigate three properties of CAVs: (1) inconsistency across layers, (2) entanglement with other concepts, and (3) spatial dependency. Each property provides both challenges and opportunities in interpreting models. We introduce tools designed to detect the presence of these properties, provide insight into how each property can lead to misleading explanations, and provide recommendations to mitigate their impact. To demonstrate practical applications, we apply our recommendations to a melanoma classification task, showing how entanglement can lead to uninterpretable results and that the choice of negative probe set can have a substantial impact on the meaning of a CAV. Further, we show that understanding these properties can be used to our advantage. For example, we introduce spatially dependent CAVs to test if a model is translation invariant with respect to a specific concept and class. Our experiments are performed on natural images (ImageNet), skin lesions (ISIC 2019), and a new synthetic dataset, Elements. Elements is designed to capture a known ground truth relationship between concepts and classes. We release this dataset to facilitate further research in understanding and evaluating interpretability methods.

Published in Transactions on Machine Learning Research (02/2025)
Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors
Angus Nicolson
ongus.nicokkom@een..occc
Institute of Biomedionl Engineering University of Owford
Lisa Schut
schut@@roboss.oz.ccc
OATML, Department Computer Science University of Oxford
Alison J. Noble
alison.nobleeeng.rr..
Institute of Biomedical Engineering University of Oxford
Yarin Gal
yarin.gal@cs.co oz
OATML, Deportment of Computer Science Universily of Oxford
Reviewed on OpenReview: https: // openreview. net/ forum? id= 7CU ulaI
Abstract
Concept-based explanations translate the internal roprosentations of deep learning models into a language that humans are familiar with: concepts. One popular method for find- ing concepts is Concept Activation Vectors (CAVs), which are learnt using a probe dataset of concept exemplars. In this work, we investigate three properties of CAVs: (1) incon- sistency across layers, (2) entanglement with other concepts, and (3) spatial dependency. Each property provides both challenges and opportumities in interpreting models. We intro- duce tools designed to detect the presence of these properties, provide insight into how cach property can lead to misleading explanations, and provide recommendations to mitigate their impact. To demonstrate practical applications, we apply our recommendations to a melanoma classification task, showing how entanglement can lead to uninterpretable results and that the choice of negative probe set can have substantial impact the meaning of CAV. Further, we show that understanding these properties can be used to advan- tage. For example, we introduce spatially dependent CAVs to test if model is translation invariant with respect to specific concept and class. Our experiments are performed on natural images (ImageNet), skin lesions (ISIC 2019), and a new synthetic dataset, Elements. Elements designed to capture a known ground truth relationship between concepts and classes. We release this dataset to facilitate further research in understanding and evaluating interpretability methods.
1
Introduction
Deep learning models have become ubiquitous, achieving performance reaching or surpassing human experts across a variety of tasks. However, currently, the inherent complexity of these models obfuscates our ability explain their docision-making process. they are applied in growing number of real-world domains, there an increasing need to understand how they work (Doshi- Velez & Kim 2017 Reyes et al. 2020 This transparency allows for easier debugging and better understanding of model limitations.

---

Published in Transactions on Machine Learning Research (02/2025)
Model explanations take many forms, such imput features, prototypes concepts. Recent work has shown that explainability methods that focus on low-level features can incur problems. For example, saliency methods, which determine the sensitivity of a deep learning model to individual pixels, can suffer from confirmation bias and lack model faithfulness (Adebayo et al. 2018). Even when faithful, saliency maps only show 'where' the model focused the image, and not. 'what" it focused on (Achtibal et al. 2022: Colin ct al. 2022).
To address these problems, concept- based methods provide explanations using high-level terms that humans are familiar with. A popular method is concept activation vectors (CAVs): linear representation of a concept found in the activation space of specific layer using probe dataset of concept examples (Kim et al. 2018). However, concept-based methods also face challenges, such as their sensitivity to the specific probe dataset (Ramaswamy et al. 2022a Soni et al. 2020).
In this paper, we focus on undorstanding three propertics of concept vectors:
1. They cannot be consistent across layers,
2. They can be entangled with other concepts,
3. They can be spatially dependent.
We provide tools to analyse each property and show that they can affect testing with CAVs (TCAV) ( 6.1 6.2 and 6.3p and lead to misleading explanations. To minimise the impact these effects can have, we recommend: creating CAVs for multiple layers, verifying expected dependencies between related concepts, and visualising spatial dependence ($7). These properties do not imply that CAVs should not be used. On the contrary, may be able to use these properties to better understand model behaviour. For example, we introduce a modified version of CAVs that are spatially dependent and can be used to identify translation invariance in convolutional neural networks (CNNs). Additionally, we do not claim that these propertics are necessarily unexpected. Instead, the aim of this paper to formally define each one, confirm when it can occur, and then clearly examine the consequences of these properties on the interpretation of TCAV scores. In each case, even when expected, these properties can cause misleading explanations.
To provide concrete example of how the properties ct al. (2023) which uses CAVs in the context of skin cancer diagnosis ($ 7). We demonstrate how to use our recommendations to sanity-check TCAV results and, in this specific case, we show that entangled concepts lead to uninterpretable explanations. Our results also indicate that the choice of negative probe dataset can have a substantial impact to the meaning of a CAV. This use-case, combined with our recommendations, can be used by practitioners as guide for how to effectively use CAVs in practice.
affect experiments, we examine the use-case of Yan
To help explore these properties, we created configurable synthetic dataset:: Elements ($ This dataset provides control over the ground-truth relationships between concepts and classes in order to understand model behaviour. Using the Elements datasct, rescarchers can study (1) the faithfulness of concept-basod explanation method and (2) the concept entanglement in a network.
2 Background: Concept Activation Vectors
A CAV (Kim el al. 2018) is a vector representation of a concept found in the activation space of a layer of neural network (NN). Consider a NN which can be decomposed into two functions: (x(x) a; E Rm which maps the input R" to a vector aj in the activation space of layer I, and h((a;) which maps a, to the output. To create CAV for concept (concept examples), and negative samples X_ (randomly sampled in-distribution images). For the sets X and x we create corresponding set of activations in layer t:
need a probe dataset D. consisting of positive samples X
A+ (x,)) Va, E } and Ar, = (a(x.) Vz, E }
(1)
We find the CAV Ved by training a binary linear classifier to distinguish between the sets At and AT;
Dott + bas Vay AtD and a Val b S 0 Van A
(2)
2

---

Published in Transactions on Machine Learning Research (02/2025)
Consistency
Entanglement
Spatial Dependence
ai
a122
sky
9c
blue
91
f(ar,
Moch
Vacla
ge
Gea
01
(1l
91
Use Multiple Layers
Visualise Concept Similarities
Visualise Spatial Norms
trange
E31
0.3
A30
Wlefi
422
822
500
10
303
E37
EC2
321
riihht
0.0
Figure Concept Activation Vectors can be: inconsistent across layers, i..,, we cannot find two concept vectors in different layers that have the same additive effect (left), entangled (middle) and spatially de- pendent (right). The top panel illustrates each of these different properties. The bottom panels show our recommendations on how to mitigate the impact these effects can have: creating CAVs for multiple layers (left), verifying expected dependencies ween related concepts (middle), and visualising spatial dependence (right).
where Vaa the normal vector of the hyperplane separating the activations At and A and bag is the intercept
To analyse model's sensitivity to Vad, Kim et al. 2018 introduce testing with CAVs (TCAV), which determines the model's conceptual sensitivity across an entire class. Let Xx be a set of inputs belonging to class The TCAV score defined as
1x E Xx S.,,,(()) > 0H
TCAV
(3)
[xxl
where the directional derivative of the concept, Se,k,, is defined as
hE (g1(x) + eV.,) h1 (g((x)) Vhia (n(())) Wa.t
(an,(w)) lim c-
(4)
where Vhy is the partial derivative of the NN output for class k with respect to the activation. The TCAV
score measures the fraction of class & inputs whose activation at layer / is positively influenced by concept A statistical lest comparing the scores of CAVs to random vectors is used to determine the concept's significance (see Appendix 10.1
3 CAV Hypotheses
To use CAV-based explanation methods in practice, it is important to understand how they work. Therefore, we study thrce propertics of CAVs and their effects on TCAV scores. We focus on these hypotheses as they provide insight into network representations and into the meaning encoded by concept vectors.
We formalise each property through null hypothesis, which provide evidence to reject later in the paper. In the following text, use the typesetting concept to denote concept.
3.1 Layer Consistency
In general, we want to understand model behaviour. However, CAVs explain whether a model is sensitive to concept in a specific layer. In practice, analysing all layers may be computationally infeasible, and it
Fng. assumes that the linear classifier has hard boundaries. In practice, classifiers typically achieve 80-95% accuracy.
3

---

Published in Transactions on Machine Learning Research (02/2025)
is unclear which layers to choose. Therefore, our first hypothesis explores the relationship between CAVs found in different layers. Recall that the TCAV scores depend the directional derivative: how the model output changes for en infinitesimal change of the activations in the direction of CAV. By perturbing the activations in the direction of CAV, we explore whether two concept vectors found in different layers can have the same aflect on the model output. We refer to this property as layer consistency (see Figure for a schematic overview).
Definition 1 (layer consistency) Assume we have function f(-) that maps the activations from layer l into activations in layer 12, where 4 < 122 Concept vectors, Vell and Velt, are consistent across layers iff for every input a and corresponding activations as, and 012. f(an, + ved1) = at + Ve.l.
If two CAVs are consistent across layers then they have the same downstrean affect on the model when activations are perturbed in their direction, i.e., even though they are different layers, they have an equivalent effect on the model output and therefore the model assigns them the same meaning, Our first hypothesis is:
Null Hypothesis 1 (NH1): Concept vector representations are consistent across layers
In $6.1 we formally explore this hypothesis, and perform empirical evaluations on the Elements and ImageNet DDeng el al. (209) datasets. We show theoretically the conditions must meet for layer consistent vectors Ua, and Ve.2 to exist.
3.2 Entangled concept vectors
Consider the meaning encoded by concept vector. We label CAV using the corresponding label of the probe dataset. For example, CAV may be labelled striped or red. This implicitly assumes that the label is a complete and accurate description of the information encoded by the vector. In practice, the CAV may represent several concepts e.g,, continuing the example above, the vector may encode both striped and red simultaneously. We refer to this phenomenon as concept entanglement. Mathematically, we formulate this follows. A concept vector Ve, is more similar the activations corresponding to images containing the concept. than activations for images not containing the concept, i.e. it satisfies
a Vost a Ve.1 Va AT, a A..
Assune have concepts ei and 02, with probe datasets De, and Day, respectively. For each probe datasel, find the activation sets: A.,, {At,, , and Aea. [At,1 U A
Definition 2 (entangled concepts) CAV Vess for concept e₁ is entangled with concept 02 if
at T Degd azga Vay
Varg At a Arigt
(6)
Our second hypothesis explores concept entanglement:
Null Hypothesis 2 (NH2): A CAV represents only the concept corresponding to the concept label of its probe dataset
If concepts entangled, it is not possible separate the model's scnsitivity to concept from its sensitivily to related concepts therefore, if we measure the TCAV score for Cı, we will unknowingly incorporate the effect of c2.
In 6.2 we demonstrate that visualising similarity matrices can be used to explore CAV entanglement and discuss how entanglement can alfect TCAV.
For simplicity we write the concept vectors without a scaling term, but all experiments scale by some small y and the mean norm of the activations that layer Appendix 12.1 for details.
4

---

Published in Transactions on Machine Learning Research (02/2025)
3.3 Spatial Dependence
Here, explore the influence of spatial dependence concepts. Tet De,py and Daps denote two datasets containing the same concept but in different locations μ4 # ug. By location mean the location of the concept relative to the frame of the image. The exact form of this will depend on the specific dataset and concept in question. For example, in the Elements dataset we use binary labels such as left/right or top/bottom but more complex representations could be used (e.g. a segmentation map of which pixels contain a concept). For example, De,u may contain exemplars of striped on the left of the image, and De,uz exemplars of the striped on the right of the image an example is shown in fig. - As before, we construct latent. representations Ac,,,1 and Acd,,p for datasets De,p and Bagg respectively. Let Det be the concept vector found using probe dataset De
Definition 3 (activation spatial dependence) Let ai be the activations corresponding to input æ; in layer I, and let Pe, be the location of concept e in æ; A layer has a spatially dependent representation of concept iff there erists some function & which maps ar to Pes for all inputs E;:
3 Va, E X Mans) Me,,
Activation spatial dependence in a NN may be due to architecture design, training procedure and/or the training dataset. In CNNs, it is the natural consequence of the receptive field of convolutional filters con- taining different regions of the input. If the NN has spatially dependent activations and the probe datasct has spatial dependence, it may be possible to create a concept vector with spatial dependence.
Definition 4 (concept vector spatial dependence) concept vector Ve,t is spatially dependent with respect to the locations μi and U2 iff'
a aalmm Degt a acl.p Dell Vatlan E A Atian atrps A edl.p
(8)
If a CAV spatially dependent then, by the definition above, is more similar to the activations from images containing the concept in specific location. This means the CAV represents not only the concept label, but the concept label at a specific location, e.g. striped objects on the right of the image, rather than striped objects in general. As done for the other two properties, we propose a hypothesis and aim to reject it later in the paper:
Null Hypothesis 9 (NH3): Concept activation vectors cannot be spatially dependent
We reject this hypothesis in 6.3 by analysing how the concept location in the probe dataset influences the spatial dependence of concept vectors. Rejecting NH3 motivates the introduction of spatially dependent CAV's (8 6.3), which can be used to test a model is translation invariant with respect to a specific concept and class.
4 Elements: A configurable synthetic dataset
To explore these hypotheses, we introduce a new synthetic datasct: Elements. In this datasct we can control: (1) the training dataset and class definitions, allowing to influence model properties, such concept correlation in the embedding space, and (2) the probe dataset, allowing us to test concept vector propertios, such as concept vector spatial dependence. We further elaborate on these advantages in Appendix
Figure 2 shows examples of images in the Elements datasets. Each image contains n elements, where an clement defined by seven propertics: colour, brightness, size, shape, texture, texture shift, and coordinates within the image. The dataset can be configured by varying the allowed combination of properties for each element. The ranges and configurations used for each property is given in Appendix 11
5

---

Published in Transactions on Machine Learning Research (02/2025)
(b) Armpee Striped Probe 5et
(c) Ammppeeee Striped Left Probe
(d) Agmppeeriht
Negative Probe Set
Striped Right Probe
Example images from Flements probe datasets. (a) Negative probe set. A random selection of images equivalent to images found in the model training set. (b) Positive probe set for stripes. (c) Positive probe set for stripes on the left. (d) Posilive probe set for stripes on the right.
Figure 2:
5
Related Work
Concept Correlation and Entanglement Chen et al. (020 discuss how concept vectors can be corre- lated, making it challenging to create vector that solely represents one concept. While their work focuses on de-correlating concepts during Iraining, we focus on analysing the impact of correlated concepts afler training and show how they can lead to misleading explanations ($6.2). Fong & Vedaldi (2018) use cosine similarity demonstrate that the similarity between concepts varies based on the vector creation method. In our work, we also use cosine similarity to compare concept vectors. The distinction lies in our focus on CAVs and the insights they provide into the dataset and model. There have been several works analysing correlated concepts in interpretable-by-ee besign networks (Heidemann et al. 2023 Zarlenga et al. 2023). Our work complements these works by studying standard neural network architectures and the post-hoc expla- nations of TCAV. Raman et al. (2024) examine the effect of eeker-concept relationships in CAVs and provide several useful metrics for measuring how well these relationships are represented, but they do not explore its effect on TCAV scores.
Spatial Dependence Biscione & Bowers 2021] describe how CNNs are not inherently translation in- variant but can learn to be (under certain conditions on the dataset). This finding challenges the com- mon assumption that CNNs possess inherent translation invariance. Through spatially dependent CAVs, demonstrate translation invariance with respect to specific concept and class, rather than in general, providing more detailed information about model. Raman et al. (2023) examine the locality of concept bottleneck models (CBMs) and find that, in some cases, CBMs make concept predictions using information far from the object of interest. In this work we examine post-hoc concept-based explanations, rather than predictions of an interpretablo-by by-ll-gg model. Additionally, our definition of spatial dependence is more related to whether we can learn CAVs which mean stripes on the right of the image, rather than whether a CAV is using information near/far from where the concept is located in the image.
What concept representations does our analysis apply to? Most. concept-based interpretability methods represent concepts as vectors in the activation space of trained neural network {Kim et al.) 2018
Fong & Vedaldi 2018 Zhou et al. 2018 Ghorbani et al.] 2019 Zhang el al.) 2020
Ramaswamy al.) 2022b
Fel et al.
2023
However, some concept- based methods use different representations: individual neurons
[Bau et al.
201
regions of activation space (Crabbé & van der Schaar 2022 or non-lincar concepts (Bai
et al. 2022
Li et al. 2023 Our work focuses on the properties of concept vectors.
How is our work relevant in practice? To give insight into when the various properties may be relevant,
we performed a review of computer vision papers which use CAVs in (1) the high-stakes applicalions of
medical imaging (including skin cancer, skin lesions, breast cancer, and histology (Yan et al.) 2023 Fürböckk
ct al. 2022 Pfau al. 2020)) ), and (2) computer vision rescarch on models trained with well-known datasets
[Krizhevsky 2209 Lin al. 2014 Wah al., 2011 Zhou al. 2017 Sagawa et al.) 2020 Deng et al. 2009)
A summary table can be found in Appendix 15 We found that the following papers could have benefited
6

---

Published in Transactions on Machine Learning Research (02/2025)
from evaluating: consistency (aan et al.
2023
Ramaswamy et al.
2022a;
Fürböck et al
2022
Yuksekgonul
et al.
2023
Ghosh et. al.
2023
Lucieri et al. 2020).
entanglement Yan et al. 2023
Ramaswamy et al.
2022a;
Fürböck et al.] 2022;
Yuksekgonul et al. 2023
Ghosh et al.
2023
Graziani et al. 2020 McGrath
ict al.
2022
Lucieri et al.
2020
Pfau et al. 2020 and spatial dependence Yann et al.) 2023
Ramaswamy
let al.
2022a
Fürböck et al.) 2022:
Yuksekgonul et al. 2023:
Ghosh et al.) 2023
McGrath et al.
2022 Lucieri
ct al.
2020
Pfau ct al.) 2020). We provide a detailed example, using the application of skin cancer diagnosis
Yan et al.
2023), in $
Datasets
While several datasets have been introduced for evaluating interpretability methods, they differ from ours in a few key ways. There are three questions we need our dataset to help answer:
Is the concept represented in the network?
2. Is the concept used for the network's prediction?
3. How does the network represent correlated concepts?
Existing datasets either do not allow insight into all three, or they have other practical reasons for being unsuitable. The Benchmarking Interpretability Method (BIM) (Yang & Kim 2019) inserts objects into scene images. While it benefits from utilizing real images and complex concepts (dog bedroom), it also presents challenges. One drawback is that relying on real images makes it challenging to establish the ground truth relationship between concepts and class predictions or to know the similarities between concepts. As such, it does not give us insight into (2) or (3). The CLEVR dataset (Johnson et al. 2016) could give insight into all 3, but because it renders 3D shapes is too slow for our purposes. Elements generates images in 0.004s compared to CLEVR's 4s. This translates to significant time saving when more data is required 4s for 1000 images with Elements versus Ih for CLEVR. Analyzing CAV properties is our core focus, Elements, with its speed and flexibility, allows us to create the many different dataset versions required for experiments. The Navon and Trifeature datasets, used by Hermann & Lampinen (2020) to study feature reprosentations, could also give insight into the three questions, with associated concepts of shape, color and texture relating to each image. However, there is only one large object in each image so our experiments on spatial dependence would not have been possible. The syuthetic dataset in Yeh et al.) (2020) is similar to our datasot but was designed for concept discovery, featuring images where each object corresponds to a single concept (shape). In our dataset, each object contains multiple concepts, allowing us to create associations between them. We focus on explanation faithfulness by ensuring that the concepts must be used correctly by the model to achieve high accuracy. So, for accurate model, we have ground truth understanding of how each concept is used. dSprites Matthey et al. (2017) and 3D Shapes Burgess & Kim (2018) are probably the most similar datasets to ours but Elements is a far larger dataset. For example, even for the simple version of the Elements dataset, there are approximately 10 different images after just a single object has been placed in the image. This is considerably larger than the 737, 280 and 480, 000 total images in the dSprites and 3D Shapes datasets, respectively. The additional complexity of Elements means NN needs to approximate more complex function and having multiple objects per image ensures the model has to learn pooject/locations based representations of the concepts, rather than image-based, e.g., to answer whether striped triangle is present it is not sufficient to simply determine if stripes occur anywhere in the image. An extended literature review can be found in Appendix 15
6 Results: Exploring Concept Vector Properties
We explore the hypotheses on consistency (NH1), entanglement (NH2) and spatial dependency (NH3) in respectively. We perform experiments using CAVs on the Elements and ImageNet datasets. Implementation details can be found in Appendix 10.
6.1
$ 6.2
and $ 0.3
6.1 Consistent CAVs
Theory We begin investigating NHI, which states that CAVs are consistent across layers, i.c. f(a, + Veci,) 011 + Vel22- Let ai, and di, be linear perturbations to the activations in layers in and 12, respectively:
7

---

Published in Transactions on Machine Learning Research (02/2025)
01; (11 Vell
(9)
di a12 Ve,l2 f(a,, + Ve.22
(10)
We want to investigate if va,,, and va,lg have the same effect on the activations (and hence the model), i.c. if:
[(ar.) 012 f(a,, Vcl1,) = f((a.) Ve,12-
(11)
(12)
Assuming is continuous and differentiable, in Appendix g we prove the result that Eq. T2 can hold if and only if f is equal to
=(a)))=(0) = -((,,, + Mas, + b.
(13)
Where g(-) is a periodic function with period Nad and M E Xmu; a non-zero linear term with constant R2 More intuitively, we can obtain layer consistent vectors Va,, and Vo.t and only f is composed of periodic function with period Ve,, and a non-zero lincar term M. In principal, f could approximate a universal approximators (Sonoda & Murata 2017 However,
function of this form since neural networks it seems reasonably unlikely that even if the model was modeling a periodic function it would have period of exactly the same direction a CAV. Throughout the rest of this section provide empirical evidence that, in practice, layer consistent CAVs are not found.
Experiments
Our goal is to investigate the question are the concept vectors found using TCAV consistent across layers? We measure the consistency of two perturbations using the consistency error:
Econsistency = [lf(ā:)) 0z, = llf(a, + vo,,,) (ar, + ved22)l
(14)
In our experiments, activation remains in distribution see Appendix 12.1 for details. If two perturbations have a consistency error of 0, then they have the same effect on the model. We include the following benchmarks:
scaling term to reduce the size of Ve.l, and Vc.1g to ensure the perturbed
Optimised CAV (lower bound): TCAV may not find a Vol2 that has a consistency error of with Val1 Therefore, we use gradient decent on Val1, to minimise the consistency error, which acts as a lower bound.
Projected CAV: the error between f(Ve))) and Valyg which measures how consistent the vectors when projected into the next layer. If f(-) conserves vector addition, the projected CAVs would have 0 error.
Random (upper bound): We include two benchmarks. Random CAVs found using probe datasets containing random images, and a Random Dircction vector: Va,l, Uniform(-1,1) If the consistency error is similar to random, suggests that the CAVs between layers are as similar to each other as random directions.
Figure shows the Cconeistency for different Ve,t, across different training runs (see Appendix I2 for details). The concept CAV obtains nonzero consistency error, suggesting that CAVs across different layers are not consistent. When compare it with the benchmarks, we find:
The consistency error for the optimised CAVs lower, implying that the standard approach to find CAVs does not find optimally layer consistent CAVs. However, the nonzero error for optimised CAVs suggests it is not possible to find consistent vectors across these layers.
As expected, the projected CAVs have nonzero error, indicating that vector addition is not pre- served.
The random CAVs have a higher error, suggesting the concept CAVs are more similar than random vectors.
8

---

Published in Transactions on Machine Learning Research (02/2025)
22
Econsistency
2.0
1.8
1.6
Optimised CAV Concept
Projected Random Random Direction
Figure 3: Empirical evidence for inconsistent CAVs across layers. The consistency error for different Ve,12 for striped in the penultimate convolutional layer of a ResNet-50 trained on ImageNet. The optimised CAV acts as lower bound, whereas the random CAV and Direction act as baselines that provide an intuitive upper bounds. Concept CAV: striped CAVs, trained as normal. Projected CAV: striped CAVs from layer 4 projected into layer l2, f(Vo,t))
The inability to find consistent concept vectors across layers suggests that the directions encoded by CAVs different layers are not equivalent; instead we speculate that they represent different components of the same concept. This result in unsurprising given that previous works have demonstrated that model representations are more complex later in the NN (Mordvintsev et al.; 2015: Olah et al. 2017 Bau et al. 2017), therefore it is unlikely that the same aspects of & concept are represented in different layers (discussed further in Appendix 12.3). Consequentially, TCAV scores across layers can vary as they perform different Lests - they measure the class sensitivity to a different version of the concept.
Figure scoss that concept vectors found in different layers of a model can give contradictory TCAV scores (further examples available in Appendix 12.4h. In the Flements dataset, shape concepts are encoded in each layer as the test accuracy for each layer is above 93%. Therefore, of these layers. However, the TCAV scores for cross in the Elements dataset contradict each other across 'layers.3' and 'layers.4', suggesting a positive and negative influence, respectively. This contradiction makes it difficult to draw a conclusion about the model's class sensitivity to cross.
expect to be able to use TCAV on each
On the right of fig. 5c we show the TCAV scores for striped for various classes in ResNet-50 model trained on ImageNet. The accuracy for the striped vectors in ImageNet is above 96% for all layers tested, suggesting that the concept is encoded by the model in each of the layers. As in Elements, we do not observe consistent TCAV scores across layers. Instead, we observe large change in the TCAV scores for striped in the penultimate layer, compared to earlier layers. 'layer4.1' suggests striped positively influences the likelihood of the classes tiger and leopard. However, earlier layers suggest that the class is not is not sensitive to the concept. This shows how, depending on the layers that are tested, different conclusions can be drawn.
In order to determine how often TCAV scores are inconsistent. across layers, we introduce a new metric, the TCAV layer consistency score. It measure of how well the significant TCAV scores for some concept, c, and class, k agree with each other across different layers 4, 1...L L. It is based on how many layers are on the same side of the null TCAV score. The null TCAV score is the mean TCAV score for a set of random CAVs (see Appendix 10.1 for more detail). Mathematically the metric, Sconsistent> is defined as
1 Σ TCAV c.k.l. > TCAV, -0.5l Sconsistent = =12(()
(15)
where indicates the TCAV score is calculated using random CAVs. It is difficult to make confident statements about the sensitivity of the model when the layer consistency score is close to 0 as it indicates the TCAV scores from different layers tend to disagree on the direction of the sensitivity.
We obtain a mean TCAV consistency score of 0.841 across all mean TCAV consistency score of 0.868 for a selection of classes and 16 concepts for ResNet-50 trained on ImageNet (See Appendix 12.5 for further details). The scores indicate that although complete disagree- ment across layers is not common (5.5% and 2.1% of scores were equal to 0 for Elements and ImageNet.
for Elements and a
9

---

Published in Transactions on Machine Learning Research (02/2025)
1.00
square
0.75
-115
4.22
432
806
4.30
0.55
049
<.33
A.3
(30
4:22
045
308
0.50
plus
+44
6:00
4:24
30M
8:01
6:30
6.84
4.00
4664
9:40
6.00
6.00
US3
042
216
3.10
4:30
080
310
618
6:26
5.
MEZ
0.25
CS50h Senlarly
C22
6:44
622
428
1222
0:01
0.03
0.40
568
-022
8:30
0.31
6311
912
1310
6221
031
631
032
0.00
green
0:33
<.00
4:43 060 300 300
6.00
4.00
300
800
833
C31
6.5
003
0.5
4.35
0.19
0.44 965
4.4
spots
630 6.00
365 031 642 004
100
8.8
044
6.30
367
307
6:6
331
030
045
4.30
151
031
EM
035
030 M67 312
EM
6:4
314
055
627
-0.75
stripes
630
808
3x
4:39
622 MM
212
E2
4.
Square
crdee
blangeo
plus
red
green
blse
solld
stripess
square
cirle
triangite
plus
red
green
blse
sotid
strppet
square
drcle
triangle
Blus
red
gress?
blud
soldd
stripes
-1.00
Figure Cosine similarities demonstrating entangled concepts. Mean pairwise cosine similarities for all concepts from different versions of the simple Elements dataset, with increasing association between red and triangle from left to right: E1, Eg and Ez.
respectively), it can occur, and a not insubstantial number of concepts/clasee have a reasonable amount of disagrecment across layers (23% and 14% of scores were I 0.5 for Elements and ImageNet, respectively).
6.2 Entanglement
Different concepts may be associated with each other. For example, consider blue and the sky a funda- mental aspect of the sky is that it is oflen blue. These concepts are inherently linked and should not be treated as independent. This section will discuss how to discover these associations using CAVs and the implications for TCAV.
To explore entanglement, we quantify and visualize concept associations by computing average pairwise cosine similarities between CAVs (we compute multiple CAVs for each concept). We investigate three models trained on different versions of the Elements dataset. Each dataset identical aside from the association between red and triangle:
F1: each combination of colour, shape and texture is equally likely,
Ez: the only shape that is red is triangles,
Eg: the concepts of red and triangle only ever co-occur.
In fig. 4 we show plot for each dataset. For E,, observe positive association between the concepts. In Ez, we observe small positive association between the triangle and red concepts. Lastly, in Eg, the cosine similarity between the red and triangle CAVs approaches the similarity of the concept with itself.
The trend between E,, Ea and Eg is likely due to the underlying association between the red and triangle increasing. We perform similar analyses on ImageNet in Appendix 13
Interestingly, we often obscrve a negative cosine similarity between mutually exclusive concepts. The model has encoded concepts that camnot co-occur (e-g., each element can only have single colour) in directions negatively correlated with each other. The presence of the red diminishes the likelihood of the blue green being present, and by having these concepts negatively associated with each other the model builds in this reasoning. This means that the red CAV does not solely signify red, it also encapsulates not blue and not green.
Next, we investigate the effect of entangled concept vectors on TCAV scores. We analyse the TCAV scores for the 'striped triangles' class in E, and Ez. The class label depends solely on the presence stripes and triangle. Therefore, we expect all other concepts to obtain low TCAV scores (indicating negative
10

---

Published in Transactions on Machine Learning Research (02/2025)
1.00
1.00
kyers.2
TOAV Score
0.75
kyyrs.3
TCAV Score
0.75
0.50
0.50
0.25
0.25
0.00
1.00
0.00
1.00
TCAV Score
0.75
Score
0.75
0.50
0.25
0.25
square
circte
triangle
plus
red
green
blue
saild
spots
stripes
0.00
triangle triangle
triangle right
stripes
strices left
stripes right
left
(a) Entangled CAVs increase red TCAV scores for Eg. (b) TCAV scores spatially dependent for the 'striped TCAV scores for all concepts in Br (top) and Ez (bot) triangles on the left' (top) and 'striped triangles on the for the class of striped triangles.
right' (bot) classes in Elements.
1.0
1.0
layers.2
ToAy scare
0.8
layers.3
layers.4
0.8
layer3.3
layer
0.6
1.yer3.3
0.4
1eyer3.4
layer4,0
0.2
bayer4.1
square
arde
triangle
plus
cross
0.0
7ebra
Eggar
Iopand
libn
(c)
Inconsistent TCAV scores across layers. Left: TCAV scores for shape concepts for the 'solid red
squares' class in Elements. Right: TCAV scores for striped for subset of ImageNet classes.
Figure 5: Consistency, entanglement, spatial dependence can affect TCAV scores. The standard deviation is black or red for significant and insignificant results, respectively. The null for each layer is shown horizontal black line.
sensitivity), as their presence makes the class less likely, or insignificant TCAV scores, the concept is uninformative 3
The results for E₁ and E2 are shown on the top and bottom of fig. 5a respectively. For Ej (the unalterod dataset), we find that only the stripes and triangle vectors have high TCAV score across multiple layers. For E2 (the altered dataset), however, the model appears to be sensitive to red, triangle and stripes, with high TCAV scores for each. This is due to the association between the red and triangle CAVs. 2, 37445/ 000 images in the test dataset contain striped triangles. None of these are incorrectly classified, so it is unlikely that the model uses the red concept for its prediction. Instead, the association between CAVs causes a misleadingly high TCAV score for the red concept. In conclusion, associated CAVs can lead to mislcading explanations.
6.3 Spatial Dependence
Finally, we investigate NH3: are CAVs spatiolly dependent? In the case of convolutional based neural network, where the activations are of shape H x W x D, can reshape the CAVs back into the original shape of the activations, and compute the channel-wise norm follows:
Set ||reshape( (H, W, D))
(16)
where Sa, E H and || ||2 the La norm across the channel dimension. We refer to this array as the spatial norms of the CAV.
3 Passuming that the model uses each concept correctly
11

---

Published in Transactions on Machine Learning Research (02/202))
layers.1
layers.2
layers.3
layers.4
layer3.4
layer3.5
layer4.0
layer4.1
0.20
0.15
0.20
0.15
0.10
0.05
0.05
Figure 6: Spatial norms reflect the spatial dependence of the probe dataset. Left: Mean spatial norms for red (top), red left (middle) and red right (bottom) for Elements. Right: Mean spatial norms across for striped (lop), striped edges (middle) and striped middle (bottom) for ImageNet.
If a CAV's spatial norm varies substantially across the (H, W) dimensions, it indicates that the CAV is spatially dependent (see Appendix 14.2 for an explanation). Visualising CAV's spatial norms shows which regions contribute most to the directional derivative and, consequently, to the TCAV score.
To create spatially dependent CAVs, we constructed spatially dependent probe datasets for Elements and ImageNet where we eit restricted the location of the concepts or greyed out parts of the image see fig. 2 for examples and Appendix 14.1 for further details.
When a spatially independent probe dataset is used to create CAVs, as in the top row of fig. 6 the spatial norms are uniform, suggesting the CAVs are not spatially dependent" However, when the probe dataset exhibits spatial dependence, do the resulting CAVs. The regions of near-zero norm indicate that the cor- responding spatial regions of the gradients do not contribute to the directional derivative and, consequently, to the TCAV score.
Next, we investigate the question does the model have a different conceptual sensitivity depending on the concept's 'ss location in the imput image? As CAVs operate the activation space of a specific layer, we can show that model is translation invariant if:
1. The model has ivtivation spatial dependence, iee. pixels in different locations affect the activations differently.
Each westh-wise slice of the activations, of shape (1, D), affects the logit output differently.
Both of these components affect the TCAV score. (1) influences Ve, and (2) influences Vh, (gi (gr(x)). For (1), fig. 6 demonstrates that the model has activation spatial dependence as the locations with the highest spatial norms approximately correspond to the location of the concept the image space.
To address (2), we compute the TCAV scores for different sets spatially dependent CAVs to determine if the sensitivity of the model changes depending on the concepts location. To investigate this, we created spatially dependent classes in the Elements dataset, where the class depends on what concepts are present and on where they are in the image, such as 'striped triangles to show that model is not translation invariant with respect lo striped or triangle in fig. 5b Here, we discuss the results for the class of 'striped triangles on the left'. The TCAV scores for striped, triangle, striped left and triangle left are high, indicating positive influence of these concepts on the class. However, the striped right and triangle right TCAV scores often do not differ significantly from the null scores, providing no evidence to suggest the model is sensitive these concepts. The difference bet the left and right biased TCAV scores indicates that the model is translation invariant with respect these concepts the model's sensitivity depends where the concept is present in the image input space.
the left'. We use spatially dependent CAVs
4the individual CAV' may still be spatially dependent, but this cancels across training runs. See Appendix details.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Overall, this suggests that we use CAVs to detect model translation invariance. See Appendix 14.6 for examples on ImageNet and in Appendix 14.7 even though we cannot use spatial norms to visualise it, some preliminary evidence that spatially dependent CAVs can exist for transformer-eet based architectures.
7 Practitioner Recommendations
Our results have shown that failure to appropriately consider consistency, entanglement, and spatial de- pendence may result in drawing incorrect conclusions when using TCAV. Therefore, we recommend the following:
Consistency: creating CAVs for multiple layers, rather than single one;
Entanglement: (1) verifying expected dependencies between related concepts, and (2) being mindful that a positive TCAV score may be due to concept entanglement;
Spatial Dependence: visualising concept vector spatial dependence using spatial norms.
In S provided example papers do not consider these properties but may be influenced by them. To provide a concrete example, we examine the use-case of Yan et al. (2023) which uses CAVs in the context of skin cancer diagnosis. Below we demonstrate how our recommendationss could have been used and how the analysis impacts the conclusions drawn.
The authors use CAVs on a single layer. As discussed in $ 6. and 12 different. layers can represent different aspects of the same concept. To have better understanding of the overall effect of the concept on the model, CAVs should be created for multiple layers.
Consistency
Entanglement There multiple concepts which have opposed meanings, for example regular streaks and irregular streaks, or regular vascular structures and irregular vascular structures. As such, we expect the cosine similarities between the CAVs to confirm that these concepts are negatively correlated (or less similar to each other than to other concepts).
Spatial Dependence Some the concepts have expected spatial dependencies, for example, dark borders and dark corners. Spatial norms could be used to confirm these spatial dependencies exist. Equally, for concepts such as the presence of ruler, the spatial norms could confirm the CAVs have no overall spatial dependence.
7.1 Experiment Setup
To further this example, we run illustrative experiments on similar dataset Yan al. (2023). The
dataset used in Yan et al. 2023 is not publicly available.
Model We finetune a ResNet50 (He al. 2016) pretrained on ImageNet (Deng et al. 2009 on the ISIC 2019 dataset (Tschandl al. 2018: Codella al. 2017 Combalia al. 2019) for the binary classification of melanoma. We use binary cross entropy loss and the Adam optimiser [Kingma & Ba 2015), training until convergence of validation loss to achieve an area under the receiver operating characteristic curve (AUC) of 0.91 on the validation split.
There are 12 clinical concepts which have been expertly labelled within the datasct. We hand labelled three additional concepts, which were used in Yan et al. (023), of dark corners, dark borders and ruler which are possible confounders for the model. We defined dark corners as any image with a circular aperture which left the corners of the image black, dark borders as any image containing rectangles of blacked out arcas and ruler as any image containing a ruler. For each of the medical concepts, there are three labels: typical/regular, atypical/irregular and absent. When training CAVs for these concepts, either used random images
CAVs
For the CAVs, in Yan et al.
2023
use the derm7pt dataset (Kawahara al. 2019
13

---

Published in Transactions on Machine Learning Research (02/2025)
images with the label of 'absent' the negative probe dataset. We used the 'absent' labels because using random images as the negative set gave low-quality CAVs, with accuracies of 50-65% (see fig. 7 It is unclear from Yan et al. (2023) what they used for negative sets. For training the CAVs we used 70 images per concept
and used 30 different random seeds for the random negative probe set to get 30 CAVs per concept. Yan
et al. )0223) do not. use TCAV; so this setup is different from the original paper.
7.2 Results
As shown fig. initial results with random CAVs gave poor performance for the medical concepts, so we used the "absent" category for each class as the negative set. This gave better performances with accuracy between 60-75% for the medical concepts but this is still far lower than for the confounders at 80-90%. We hypothesise that this is due to the simplicity of the confounding concepts. This is supported by the accuracy reported by Yan et al. (2023), where they also obtained lower CAV accuracy for the medical concepts. The accuracy for the confounding concepts of dark borders and dark corners drops in later layers. This is likely due to the spatial nature of the concepts an idea further supported by fig. 10 where the CAVs in later layers have reduced spatial distinction.
Below, we examine how the three CAV properties analysed in this paper affect the TCAV scores and the conclusions you can draw from them.
Doppica_ltgmenttte
0.70
0.70
Imeeuler, dar dotl and globuess inreudar: Mtrects
regular pigment network
0.65
0.65
regular voscular
0.55
0.55
regular dots and globales regular streaks
Imogulor www.:plmeeettettnnnneee
layer25
layont11 3
layard
aappeed
Uppe3
lipardd
layod.
lyyard
lapard
ayppond
1.000
1,000
dark corners dark bandess
0.950
0.925
0.925
Toxk
0.500
0,900
0.850
0.25
0.825
0,00
1yye24
byw33
lapa4.00 a
loyork11
lnyork?
sappom
0,000
topw34
leyer
ay
layeek.1
loyor 2
Figure 7: Mean CAV test accuracies for the melanoma use-case. Top: Medical concepts where random images (right) or images where the concept is labelled as absent (left) are used in the negative probe dataset. Bottom: Potential confounders where CAVs were trained with (right) and without (left) a flip augmentation.
Consistency The TCAV scores for many of the medical concepts (fig. Sa are consistent across layers, irrespective of if random images 'absent' images were used for the negative probe set. The consistent scores provide more confidence in using them to explain the model's behaviour as repeated significance tests are performed indicating the model has the same sensitivity to the concept. In terms of understanding the model, the scores provide some evidence that it operates similar to human experts, as the TCAV scores for the
medical concepts are high for the malignant class, as expected, and the confounding concepts are often not significant (prior to using flip augmentation see the spatial dependence section below for discussion), providing little evidence that the model is sensitive to the confounders.
14

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
0.8
0.6
0.6
TOW
02
0.2
0.0
regular, wigment. networr
reguler vascular structures
typical, pigmentation
reguiar dats and piobuies
irrepilas pigment network
regular, srreakd
imeguiae vasrular, structured
atypical, pipmentation
ineeuarr dois and globutes
imeqular streaks
regulas pigment neworkk
regular vasiddar, structured
typical nigmentation
reguler dots and giabules
iregular pigment network
regular, draats
imeggler vascular strictures
alypical pigmentation
imeeg dots and globuter
irrepuarr streal
(a) TCAV scores are not qualitatively different for CAVs of differing accuracy. TCAV scores for medical concepts where random images (left) or images where the coucept is labelled as absent (right) are used in the negative probe dataset.
0.8
TCAV Score
0.6
0.4
resnet.layer3.4
resnet.layer3.5
0.2
resnet.layer..0
resnet.layer4.3
dark corness
dark corners fipp
dark borders
dark borders flip
nuler
ruler flip
(b) CAVs trained with an augmentation were significant more often. TCAV scores for potential confounders where CAVs were trained with and without flip augmentation.
Figure 8: TCAV scores for the melanoma use-case. The standard deviation is black red for significant and insignificant results, respectively. The null for each layer is shown as horizontal black line.
Entanglement Interestingly, for the CAVs with 'absent' labelled images in their negative probe dataset (the right of fig. D), the CAVs with opposed meanings often appear to be the most similar to each other. For example, regular vascular structures has a negative or xero similarity with all concepts except irregular vascular structures (with a similarity of 0.11). We hypothesise that this is because the two concepts share the same negative set. This hypothesis is supported by Ramaswany et al. (202aa) where, while they did not discuss changing solely the negative set, they did show that CAVs are sensitive to the choice of probe dataset. In addition, if compared the similarities between CAVs trained using random images as the negative set (the left of fig. we see that this pattern disappears. The higher similarity between concepts which have opposite meanings suggests that the CAVs do not represent the concepts they are labelled for. Therefore, in this case, we do not believe the TCAV scores can be interpreted as we do not have confidence the CAVs represent their desired concept. This example highlights more general problem that the negative probe set can have a substantial affect on the resulting CAV, even though is the positive probe set that is designed to represent the concept.
Spatial Dependence The spatial norms in fig. 10 show clear spatial dependence in the center for each of the medical concepts across all layers. This aligns with expectations, as the datasct requires the skin lesion to be centered in the image and each concept is related to the appearance of the lesion. The dark corners and dark borders concepts, however, show deviations to this pattern in the earlier layers, with dark corners having high spatial norms in the corners and dark borders high spatial norms in the top. This is desirable, as these confounding concepts depend on features the edge of the image, away from the lesion. Along with the accuracies in fig. this suggests that the CAVs in earlier layers better represent these
15

---

Published in Transactions on Machine Learning Research (02/2025)
lasnet.layer..1
rosnetllayee
reguler_vesculer sessullr_etruccuurs
imeguker_ye yescular structures
4.02
0.4
205
4.01
0.75
imeguler vascular jttuctures 001
0.44
4.35
typical_ _pigmentation 2.
2:88
AAL
0.50
sppcal_iggmenttle 2.M4
611
423
505
120
0.50
atypical pigmentation
Similart
atypical pignenttaion
globules
400
0,00
imegular dots and bol.gllbel globules
4.80
023
1000
6:21
6.0
011
044
645
Irreuior do1s and globules 1404
6MM
-014
6.0
8.4
844
4.99
031
645
regular, streaks
-0.25
strearsstreals
-0.25
incgular_strann streaks
Inregurr_steace
ireguiat_ cerimmmttnntwwot
6.0
0.4
031
-A75
imguiln_jiamen__reeeeo
6.6
-0.75
regular_piggment www.ggmttgneeteeeeeee network
en
009
021
022
631
regull_pigmentteetee 400
regudar verccsier, urnctires
pregiaa yeectaar strictures
typical, aigmentatiion
atypical, www.enennennne atation
reguar dob and giobales imegular, dats and gusbates
regui streaks
praguis pig internt
imregriar. streenk
regaiar plgmmert neliwnrt
1.00
regulas yiec sirectures
bregilat yioction structures
typical aigmentation
etypicll plgmentaaton atton
regulic dots and globules ir dols, and glabales regular Streatss
ireesieer Digmentt netwrrk
irregulds streaks
regniter pigmertt netewart
11000
Figure 9: Cosine similarity matrix for CAVs of different concepts from derm7pt when random images (left) images where the concept is labelled absent (right) are used in the negative probe dataset.
layer3.4
dork_comers
darkk_corners_fii
dark_borden
dork borders_flly
ruler
ruler_flip
regular_streaks povicalpggmeetationn
Mebn Spottl Nom
0.10
0.00
layer3.5
dark
dark
ruler
regular typical pigmentation
Mean Spatial Norm
0100
0.775
a.550
layer4.0
dark_comer
dark www.oooonrsrimm flip
dark borders
dark borders, flip
ruler
ruler_flip
regular_strese typical pigmentation
Moon Spato Nomm
0.20
0.15
0.10
layem1.1
dark cormers
dark, corners, flip
dark borders
dark borders, flip
ruler
ruler, flip
reqular streaks lypical, pigmentation
Mean Spotta korm
0.20
0.15
-.10
Figure 10: Mean CAV spatial norms for selection of CAVs from the melanoma uso-case.
spatially dependent concepts. For dark borders, however, we hypothesise that the high spatial norms in just single direction suggest that the CAVs are not good representation of dark borders in locations other than the top. Therefore, we retrained CAVs for each of the confounding concepts but with an augmentation applied to the probe datasct to randomly flip the images in the horizontal and/or vertical direction. This removes any bias that there may be in the probe dataset for the concept to be in one particular direction. Figure 10 shows that the CAVs trained for dark borders with an augmentation (dark borders flip) had only minor improvements for their spatial norms, with some layers being slightly less uni-directional. Figure however, shows that the accuracy for each of the CAVs improved. This suggests that there was an issue with bias in the probe dataset and by using flip augmentation we create CAVs which better represent the
16

---

Published in Transactions on Machine Learning Research (02/2025)
concepts, but, surprisingly, the activations of the model encode the dark borders concept in spatially unsymmetrical manner.
The TCAV scores for CAVs trained with the augmentation (fig, 8bj obtain significance in more layers but the results are still fairly inconclusive. For example, the TCAV scores for ruler in layer3.4 and layer 4.2 are above/below the null, respectively. This means that layer3.4 4 suggests a positive influence, whereas layer4.2 suggests negative influence. With less than a 1% difference in accuracy between the CAVs of the two layers it is not clear which layer we might trust more and conclusive statements can be said about the influence of ruler malignancy predictions.
7.3 Summary
The poor accuracy of medical concepts when using random images in the negative probe set required the use of the 'absent' category instead. However, the similarity matrices in fig. suggest that the CAVs trained using the 'absent' category do not represent the desired concepts. Therefore, we do not think either set of medical CAVs can produce meaningful TCAV scores.
For the confounders, however, the CAVs had high accuracy and the spatial norms indicated spatial depen- dence in concepts we expect spatial dependence, providing evidence that these CAVs are suitable to use. However, the spatial dependence seemed directional, and so an augmentation was added to flip the probe images horizontally l Although this did not substantially change the spatial norms, improved CAV accuracy and increased the number of layers for which we had significant. TCAV scores. From these scores, it appears the model is sensitive to dark corners although the evidence is weak with large standard deviations in TCAV score and for ruler we found inconclusive results with inconsistent scores across layers.
This use-case demonstrates the importance of analysing consistency, entanglement and spatial dependence of CAVs, alongside more typical evaluations such as CAV accuracy and statistical significance, in order to understand CAV-based explanations and the conclusions you can draw from them. Our experiments provide an example for practitioners to follow in their own experiments with CAVs.
8 Conclusion and Future Work
In this work, explore three key properties that influence concept activation vectors (CAVs): consistency, entanglement and spatial dependence. First, we derive conditions under which CAVs in different layers are not consistent and substantiate our findings with empirical evidence. This sheds light on why CAV- based explanations methods can give conflicting conclusions across layers. Next, we introduce visualisations designed to facilitate the exploration of associations between concepts within a dataset and model. Lastly, we show that spatial dependence impacts CAVs, and introduce a method that can be used detect spatial dependence within models. We provided clear recommendations properties CaV-based explanations and demonstrated how to use those recommendations for medical imaging use-case. The CAV properties were explored using synthetic dataset, Flements, where custom probe datasets can easily be created to analyse properties of interest. We release this dataset to help further explore this problem space.
how to mitigate the impact of these
In the introduction, we cited several interpretability methods that employ vector representations to convey semantically meaningful concepts. Our study has illuminated certain properties and consequential outcomes arising from these vector-based approaches. In future research, the characteristics inherent in alternative forms of representation, such as clusters within activation space (Crabbé & van der Schaar 2022 should be investigated and the relative merits assessed.
Acknowledgments
We appreciate both the members of OATML and the Noble group for your support and discussions during the project, particular Andrew Jesson. We are also grateful to Been Kim for your thoughts and feodback on our work. Thank you to Ben Eastwood for our discussions on mathematical proofs. We also extend our thanks to the reviewers and the action editor of TMLR for their invaluable feedback and constructive
17

---

Published in Transactions on Machine Learning Research (02/2025)
comments which have enhanced the quality of the resulting paper. A. Nicolson is supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). L.S. is supported by the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems grant (EPP/S24450/1) and DeepMind. J.A. Noble acknowledges EPSRC grants EP BP/X040186/1 and EP/T028572/1.
References
Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From 'where' to 'what': Towards human-understandable explana- tions through concept relevance propagation, 2022.
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9525 1525-9536, 2018.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In- ternational Conference on Learning Representations, 2017.
Andrew Bai, Chih-Kuan Yeh, Pradeep Ravikumar, Neil Y. C. Lin, and Cho-Jui Hsieh. Concept gradient: Concept-based interpretation without linear assumption, 2022.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2017.
Valerio Biscione and Jeffrey S. Bowers. Convolutional neural networks are not invariant to translation, but they can learn to be. Journal of Machine Learning Research, 22(229):1 28, 2021.
Chris Burgess and Hyunjik Kim. 3d shapes dataset.
2018.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2:772 782, 2020.
Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward mclanoma detoction: challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). 2017.
Julien Colin, Thomas FEL, Remi Cadene, and Thomas Serre. What i cannot predict, do not understand: A human-centee centered evaluation framework for explainability methods. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Mare Combalia, Nocl C. F. Codella, Veronica Rotemberg, Brian Hclba, Veronica Vilaplana, Ofer Reiter, Allan C. Halpern, Susana Puig, and Josep Malvehy. Bcn20000: Dermoscopic lesions in the wild. 2019.
Jonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A generalized framework for concept- based explanations. In Advances in Neural Information Processing Systems, volume 35, pp. 2590- 2607, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In CVPR, pp. 248 255, 2009. doi: 10. CVPR.2009.520666
Finale Doshi-Velez and Been Kim. vrzi::1702.08608, 2017.
Towards a rigorous science of interpretable machine learning.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Golly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Represendations, 2021.
18

---

Published in Transactions on Machine Learning Research (02/2025)
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfiel-Doodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. 2023.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. CoRR, abs/ 1801.03454, 2018.
Christoph Fürböck, Matthias Perkonigg, Thomas Helbich, Katja Pinker, Valeria Romco, and Georg Langs. Identifying Phenolypic Concepts Discriminating Molecular Breast Cancer Sub-Types. In MICCAI, 2022.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explana- tions. In Advances in Neural Information Processing Systems, 2019.
Shantanu Ghosh, Ke Yu, Forough Arabshahi, and Kayhan Balmanghelich. Dividing and conquering a blackbox to mixture of interpretable models: Route, interpret, repeat. In JCML, 2023.
M. Graziani, V. Andrearczyk, Marchand Maillet S., and H. Müller. Concept attribution: Explaining CNN decisions to physicians. Computers in Biology and Medicine, 123, August 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lena Heidemann, Maureen Monnet, and Karsten Roscher. Concept Correlation and Its Effects on Concept- Based Models. In WACV, pp. 4769-4777 4777, 2023. doi: 10. 10.1109 10.1109/WACC WAACV6668.2023.00111
Katherine Hermann and Andrew Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. In Advances in Newral Information Processing Systems, volume 33, pp. 9995- 10006, 2020.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment Your Batch: Improving Generalization Through Instance Repetition. In IEEE/CVF Conference on Computer Vision and Paltern Recogrition (CVPR), pp. 8126 8135, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. 1669.0000.
arXiv Preprint 111:::112..00 068900 2016. doi:
Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, and Ghassan Hamarneh. Seven- checklist and skin lesion classification using multitask multimodal neural nets. IEEE Journal of Biomedical and Health Informatics, 23(2):538-544) 2019. doi:
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, el al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learning, pp. 2668-2677. PMLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Confer- ence on Learning Representations, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongic, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014.
19

---

Published in Transactions on Machine Learning Research (02/2025)
Adriano Lucieri, Muhammad Nasecr Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. In IJCNN, 2020.
Loic Matthey, Trina Higgins, Demis Hassabis, and Alexander Lerchner. diprites: Disentanglement testing 2017.
sprites dataset.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik, Acquisition of chess knowledge alphazero. Proceedings of the National Academy of Sciences, 119(47), November 2022. ISSN 1091-6490. doi: 10.1073 pnas. 2206625119.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. Google Research Blog. 2015. Retricved November 2017.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 754 762. ICCV, 2021.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):c7, 2017.
Jacob Pfau, Albert T. Young, Jerome Wei, Maria L. Wei, and Michael J. Keiser. Robust Semantic Inter- pretability: Revisiting Concept Activation Vectors. In ICML WHI, 2020.
Naveen Raman, Matco Espinosa Zarlenga, Juyeon Hco, and Mateja Jamnik. Do concept bottleneck models obey locality? In NewrIPS Workshop XAI in Action: Post, Present, and Future Applications, 2023.
Naveen Janaki Raman, Mateo Espinosa Zarlenga, and Mateja Jamnik. Understanding inter-concept rela- tionships in concept-based models. In JCML, 2024.
Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth C. Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. 2023 IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10932 10941, 2022a.
Vikram V. Ramaswamy, Sunnie S. Y. Kim, Nicole Meister, Ruth Fong, and Olga Russakovsky. Elude: Gencrating interpretable explanations via a decomposition into labelled and unlabelled features, 2022b.
Mauricio Reyes, Raphael Meier, Sérgio Pereira, Carlos A. Silva, Fried-Michael Dahlweid, Hendrik von Tengg- Kobligk, Ronald M. Summers, and Roland Wiest. On the Interpretability of Artificial Intelligence in Radiology: Challenges and Opportunities. Radiology: Artificial Intelligence, 2020. doi: 10.1148/ryai. 2020190043.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang, Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Casc Generalization. 2020.
Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Lorcaux, Ralph Blanes, James Wexler, Alan Karthikesalingam,, and Been Kim. Best of both worlds: local and global explanations with human- understandable concepts. ArXiv, abs/2106.00 086441, 2021.
Rahul Soni, Naresh Shah, Chua Tat Seng, and Jimmy D. Moore. Adversarial tcav - robust and effective interpretation of intermediate layers in neural networks. ArXio, abs/2002. 03549, 2020.
Sho Sonoda and Noboru Murata. Neural Network with Unbounded Activation Functions is Universal Ap- proximator. Applied and Computational Harmonic Analysis, 43(2):233-22) 228, 2017. ISSN 10635203. doi: a0..016/..acha.2115.11 12.00...0....
P. Tschandl, C. Rosendahl, and Kittler H. The ham10000 dataset, large collection of multi-source der- matoscopic images of common pigmented skin lesions. Sci. Data, 5:180161, 2018.
C. Wah, S. Branson, Welinder, P. Perona, and S. Belongie. The uest Computation Neural Systems Technical Report, 2011.
dataset.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints, 2020.
S. Yan, Yu, X. Zhang, D. Mahapatra, S. Chandra, M. Janda, P. Soyer, and Z. Ge. Towards trustable skin cancer diagnosis via rewriting model's decision. Tn CVPR, 2023.
Mengjiao Yang and Been Kim. BIM: towards quantitative evaluation of interpretability methods with ground truth. CoRR, 195/1007.09701, 09701, 2019.
Chil-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completenes--wwae concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554 20565, 2020.
Mert Yuksekgomul, Maggic Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR, 2023.
S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVE /CFF International Conference on Computer Vision (ICCV), pp.
2019.
Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards Robust Metrics for Concept Representation Evaluation. AAAI, 2023, doi: 10.48550/ A.Iv.2301.10367,
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learving Representations, 2018.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin I. P. Rubinstein. Invert- ible concept-based explanations for emn models with non-negative concept activation vectors. In AAAI Conference on Artificial Intelligence, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation, In Computer Vision ECCV 2018, pp. 122 138. Springer International Publishing, 2018.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Explaining Explainability: Recommendations for Effective Use of Concept Activation Vectors Supplementary Material
9 Consistency Proof
Let oni be the activation vector in layer I for the input 2; E X. Function f maps the activations in layer 4 to layer l2, where 1 < 12, i.c. f(an))) a'g,3- We assume that f is contimuuu and differentiable.
Let áz and ā be linearly perturbed activations in each of these layers:
âi, al + a
(17)
âi₃- = 012, + = f(an).)) +
(18)
where Rm and Rmag are vectors of non-zero and my, and mi, are the dimensions of layer I and 12, respectively.
(since CAVs directions in activation space)
For the two perturbations to have the same effect on the activations (and hence the model) it must hold that:
[(a;,) ā12 Maa + u) f(ai,,)) +
(19)
where we have substituted in Eq. 17 and Eq. 18 For and v to be consistent for all possible activations they must be constant with respect to at, ie. we can assumee that and v are not functions of at, If we rearrange Eq. 19 to obtain an equation for v, we obtain
f =
u) u) f(a).
(20)
where we have simplified the notation by writing ai as a. Let us differentiate Eq. 20
d d
_ (s(a + s(a))
(21)
da da
0 da f(a u) da f(a)
(22)
0 f' u) (a)
(23)
f' + u) (a)
(24)
which implies that the derivative of f, f", is periodic with period u. This is strong restriction on the form that f' can take. Let's integrate to find out what implications it has the form of f. First, let's split the periodic function f' into its mean value and its oscillatory part:
(aa)= +M
(25)
where M mR xma, is the value across interval for each component of f'(a) and g'(a) is periodic function with zero integral across a single period, i.e.
I. (a)dda 0.
(26)
If we integrate g' (a) from to + (using change of variables with t E Rmm)) we find

---

Published in Transactions on Machine Learning Research (02/2025)
a4+4 ((t) = g + +u) - g(a)
(27)
and by using Eq. 26 we find
g(a + + g(a) 0.
(28)
Therefore g is periodic with period u. If we now take the integral of l'(),, we find
f(a) = / f'(adda
(29)
f(a) / d'(a)da+ / MMa
(30)
f(a) g(a) + Ma b.
(31)
Hence, satisfies Eq. 24 (and therefore Eq. 20 if and only if is composed of a periodic function with period u and linear term. However, there further restrictions upon f. Let M so that is simply periodic function. In this case
f(a u) f(a) f(a) f(a)
(32)
(33)
v= 0
(34)
which contradicts our non-zero assumption the norm of v. Hence o and if and only if f is composed of periodic function with period u and non-zero linear term M. We provide no proof as to whether this form of f can occur in practice in a neural network, however our empirical results in the main paper and $ 12 suggest that it does not. Our proof holds generally, however, in the next three sections, we go into more detail for a linear function as it is a special case of Eq. 31 and for the ReLU and sigmoid functions as they are common activation functions used neural networks.
obtain layer consistent vectors
9.1 Special Case: Linear Function
One counter-example that at first look seems to contradict Eq. 31 is linear function. If f is linear function,
i.e. it conserves vector addition, then Eq. 20 trivially holds:
f(a u) f(a)
(35)
f(a) f(u) - f(a) f(u).
(36)
(37)
The general result (Eq. 31 requires that f be combination of periodic function, g(a), and linear function, Ma + b, but we have just shown that f = Ma + would also hold. This is because function that outputs some constant value is a special case of a periodic function, where there is no minimal period as all periods are valid. So, for the case where f is linear g(a) = c, where c e R Or for a more specific example, in the case f = Ma + b, e = O. Hence, the result in Eq. 3T generally holds and, for the case where linear, the only layer consistent vector a in layer for some vector in layer 4 that same vector projected into layer la by f, i.e. fuu).
2

---

Published in Transactions on Machine Learning Research (02/2025)
9.2 Example: ReLU Function
In a neural network, f often involves a rectified linear unit (ReLU), so below we find v when f f= ReLU, Let al,i.j bj and uy refer to the individual elements of ai, and u, respectively. By the dcfinition of ReLU activation:
a > 0
f(0n)5) = max(0, (1,
(38)
an, S
So, Eq. 20 becomes:
1/1 max(0, 0h.i uz)
max(0, (1,i,))
0n, + U at..
4,.i, + 4j > 0, al.i. > 0
0gg + Uj +0
01ii + +, > 0, ahاو S 0
"h,i. + 0j - Uhiij > 0
0
41,. + 4 s 0, anit 50
(39)
Uj
A1g.i = Uj > 0, (1.i. > 0
01,,i, + Uj
0lg. + Uj > 0, @lg.i.j S
0hi Uj < 0, 011i 0
0,,i, - 4 - 0, dl, S 0.
If U1,,, + uj 0, an.. S 0 ahi,y + uj S 0, 4...j > for any clement j then there does not exist a a such that Eq. 19 is true for all i, i.e, when either of these statements true, you cannol, have two vectors which have the same effect the activations across layers for all possible inputs. And if we assume that the elements of can take any value in practice then there exists two layer consistent vectors across a ReLU function.
9.3 Example: Sigmoid Function
In this sontion, we consider the sigmoid activation: f(x) Itexp(-x) For ease of notation, we drop i and as they do not change, but ar, and at, refer to an,is and abaij respectively. From Eq. 19 the concept vectors are consistent iff
f(an + +u) f(at,) +
(40)
exp(-ai u) 1 (xxx(a-aa)) +
(41)
Simplifying Eq.
41
for v, we get:
V
+ e - - u) 1 ++xp(-aa))
(1 (exp(-ar,) )) (1 exp(-an u))
(1 + expeexp-aaa (1 + exp( exp(-a) (aa - u))
exp(-a; exp( (aa, 01,
(1 + exp(-a; )1 exp (aan - u))
1 - expppx1
(exp(ar, 1)(cxp(a1,) exp(-u))
3

---

Published in Transactions on Machine Learning Research (02/2025)
This can be simplified further with partial fractions:
exp(-
v (exp(an, + (exp(()))) exp(- -u))
(42)
exp(-a0p
exp( aa - u)
(43)
(exp(an, + (exp(ar, + exp -u)
exp(-a,, (exp(ar, + (exp(a) + u) + exp(-u))
exp(-o1,)
(44)
For single to exist which is consistent for all af, it cannot depend on di Since the left half of Eq. 44 depends o the only way that does not depend at, is if the right hand side cancels out the left. This only occurs when = = 0. Since u and are directions in activation space, and hence have a non-zero norm, this is a contradiction. Therefore, for the sigmoid function, under no conditions does there exist layer consistent vectors.
10 Implementation Details
In this section, we provide general implementation details applicable the whole paper. For details relating to individual experiments and additional results, Sections 12
13 and 14
10.1 Concept Activation Vectors
Background In (Kim et al. 2018), a statistical test, TCAV, determines whether the model's sensitivity to concept is significant. The test compares a set of CAV scores found using a concept dataset with CAV scores found using random data. To do this, we must find multiple CAVs for each concept. In practice, each of these CAVs is trained with the same positive set, X but a different random set, X where E 1,2... R denotes the random index. A CAV corresponding to specific random index is labelled v',
Implementation Details In this work, we create multiple CAVs per training (30 unless otherwise stated), each using the same positive probe dataset but different random set. We label CAV trained with specific random set as v'2,, where r E 1,2.... ..... R denotes the random index. Random CAVs are generated from pairwise combinations of random data sets, and we conduct a two-sided Welch's t-test to test whether the means of concept and random TCAV scores are equal. If set of CAVs passes this test with a p value less than 0.01 we consider the concept meaningful. We refer to the mean TCAV score of the random CAVs as the null; it acts as the TCAV score all other CAVs should be compared against to understand their scnsitivity to the model. The mill is often very close to 0.5, simplifying the interpretation of the TCAV score to the concept having positive sensitivity when greater than 0.5 and negalive when less.
10.2 Elements
Classification Model The model architecture is a simple convolutional neural network with six layers: each layer contains a convolution, batch norm and ReLU, followed by an average pooling and fully connected layer to give the logit outputs. The first three convolutional layers utilisc max-pooling operation reduce dimensionality. We train the model using Adam (Kingma & Ba 2015) with a learning rale of 1e-3 until the training accuracy is greater than 99.99%, giving validation accuracy of 99.98% for the standard dataset. We use different number of channels for the models trained on different datascts. This allows us to provide more model capacity when needed. The number of channels per layer for each model model/datasel dataset is summarised in Table
The models for datasets E2 and E3 in section 6.2 are the same architecture as for the simple dataset (E)).
5 we use threshold of 0.01 to help reduce the false discovery rate
4

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: The number of each channels for the models trained on the simple, standard and spatial versions of the Elements dataset.
Model
Layer
Simple
Standard
Spatial
layers.0
64
64
64
layers. 1
64
64
64
layers.2
64
64
128
layers.3
64
128
256
layers.4
64
128
256
layers.5
64
128
256
Probe Dataset For Elements, the probe datasets are generated so that the positive examples for concept contain only objects with that concept, so, for example, a red concept image will contain four objects with random shapes and textures that occur within the dataset, but all of them will be red. The negative set consists of random samples from the dataset.
10.3 ImageNet
ImageNet used demonstrate the experiments on a real-world application.
Classification Model We use the default weights for a ResNet-50 (He al. 2016) in the TorchVision package in PyTorch, which used variety of data augmentation techniques including Mixup (Zhang et al. 2018). Cutmix (Yun et al. 2019), TrivialAugment (Müller & Hutter 2021 and Batch Augmentation (Hoffer et al. 2020
Probe Dataset Most probe dalasets used to train CAVs were collated from the Broden datasel [Bau et al. 2017), particularly focusing on textures such as striped, meshed or dotted, or objects such as car, sea or person. Some concepts were mamually curated, such as the anemone concept, which was collected from test images of the 'anemone fish' class from ImageNet that were used elsewhere the experiments. Examples of some of these concepts available in Figure

---

Published in Transactions on Machine Learning Research (02/2025)
striped
polka
meshed
anemone
sidewalk
bicycle
boat
person
water
waterfall
Figure 11: Example positive probe datasets for different concepts for ImageNet.

---

Published in Transactions on Machine Learning Research (02/2025)
10.4 Layer Selection
When creating CAVs, we need to choose model layer. For the small CNN used for Elements, we can create CAVs for all layers, but for larger models, such as the ResNet-50 for ImageNet, it is computationally infeasible. this paper, we focus on layers near the end of the model. The justification for this twofold. (1) From an information theory perspective, the activations earlier in the network may contain more irrelevant information, suggesting the activations closer the output may be more relevant the prediction (Xul ett al. 2020 McGrath et. al. 2022/- We aim to TCAV to explain the model output, therefore later layers may be more desirable. (2) The model representations may be more complex in later layers. This allows to create CAVs for more complex concepts. We find that the empirical evidence supports these hypotheses. Figures 12 and 13 show the accuracy of the linear classifiers used to create the CAVs on a held-out test set for each probe dataset. The accuracy for each concept tends to increase in later layers. This suggests the CAVs better capture the model representations in later layers. However, we observe variation across the concepts. For example, the colours Elements are easily classified in all layers, whereas the shapes/texte textures have lower performance in layers.1. As goal is to understand the behaviour of concept vectors (when the concept is represented), we focus on CAVs that obtain at least 90% test accuracy. Therefore, omit layers.1 in our analysis.
We do not create CAVs for the final convolutional layer in either the simple CNNs for Elements (layers.5) or the ResNet-50 for ImageNet (layer4.2) due to the gradient behaviour in these layers. In both cases, the network has no non-linearities after the layer. Therefore, the gradient of the logit with respect to the activations solely depends on the model weights, not the activations. TCAV relies on having a distribution of dircctional derivatives, which are then thresholded and averaged over different data points. For these layers, the gradient is the same for all inputs, and hence is the directional derivative. This means the TCAV score for individual CAV in these layers will be exactly 1 or O. As such, we do not perform TCAV layers after which there non- lon-linearisies.
1.00
0.95
0.90
CAV Test Accuracy
0.80
square
yellow
0.75
circle
green
triangle
cyan
solid
0.70
plus
blue
spots_polka _ll_ooakaa
cross
magenta
stripes_diagonal
2
3
1
3
4
1
2
4
Layer
Layer
Layer
Figure 12: Mean test accuracy for the linear classifiers from which the CAVs generated for all concepts in the standard Elements dataset (split by concept type).

---

Published in Transactions on Machine Learning Research (02/2025)
1.00
sidewalk
anemone
car
water
bicycle
waterfall
0.98
boat
sea
person
CAV Test Accuracy
0.96
0.94
0.92
striped
polka
meshed
3.2 3.3 3.4 Layer
3.5
3.0
3.4 Layer
3.5 4.0
3.0
3.1
3.2
3.3
3.5
4.0 4.1
Layer
Figure 13: Mcan test accuracy for the lincar classifiers from which the CAVs are generated for a sclection of concepts in ImageNet.
11 Elements Dataset
11.1 Benefits of the Elements Dataset
Configurable datasets The configurable nature of the dataset allow us to explore different properties of the model and of CAVs. For example, we can introduce an association between the red and striped concepts create a probe dataset of red
in the training set by requiring that all red elements striped; elements the right of the image to explore CAV spatial dependence.
Ground truth model behaviour The classes are configured as combinations of the elements' shape, colour and texture. Therefore, by construction, we have the ground truth relationship between each concept and class. As these relationships are within the dataset, knowing the ground truth relationship between each concept and class does not allow to explore model faithfulness. For that, need the ground truth inffuence of each concept on model predictions. By having class for each possible combination concepts, the model must learn linearly separable representations of the concepts in the representation space (before the final linear layer) of the NN to achieve a high accuracy. Therefore, we have the ground-truth relationship of how each concept influences the model's predictions and can explore the faithfulness of concept-based explanation methods.
11.2 Elements Configuration
In the Elements dataset, there various attributes vary. These attributes in two types image attributes and element attributes. The image attributes are: the number of clements per image and the size of the image. The element altributes are: colour, brightness, size, shape, texture, texture shift, and x and coordinates within the image. Most are explanarpy from their name, but texture shift requires more explanation.
a small change how the texture is applied the object so that, for example, spots are always in the same location with respect to the edge of the object. In this section, we describe the values that each of these attributes take for the different versions of Flements that are used in the paper.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Standard The default version contains four objects in each image and the allowed concepts are the primary colours and their pairwise combinations (red, green, blue, yellow, cyan, magenta), five shapes (square, circle, triangle, plus, cross) and three textures (solid, spots and diagonal stripes).
In some experiments, when stated, simpler version which contains fewer shapes and colours. This is to reduce the complexity of the figures. The standard and simple dataset configurations are in Table 2
Simple
Spatially Dependent For some experiments in section 6.3 standard Elements dataset. This has the same configuration but introduces spatially dependent classes. As in the standard dataset, there is a class for all combinations of two and three concepts, 'striped squares' or "spotted cyan triangles' However, there are additional classes which depend on where the element is present in the image. For all classes involving triangles, we add two new classes which depend on if the object is in the top or bottom half of the image. For example, the class 'blue triangles' will now have two additional classes related to of 'blue triangles on the top' and 'blue triangles on the bottom'. Similarly, we introduce two new classes for all classes involving squares, but for the left /fight halves of the image rather than the top/ bottom.
spatially dependent version of the
Entangled We use two alternative versions of the simple dataset in the Entanglement experiments: Eg and Egg, As described in the main text, these identical to the simple dataset, apart from the association between the red and triangle concepts. In each case, restrict some of the allowed combinations of concepts that an element can take. This also removes some classes from the dataset which we reflect in any trained models. Eg does not allow any shape apart from triangles to be red. This removes classes like 'red circles' or 'spotted red squares! Eg has this restriction and then places a further restriction that triangles have to red. This removes classes such as bluu triangles' or 'striped green triangles'.
Table 2: The configurations for the standard and simple versions of the elements dataset. Element size and image size are in pixels. Brightness the value of the pixels in the element.
Property
Dataset
Standard
Simple
Colours
red, green, blue, yellow, cyan, magenta square, circle, triangle, plus, cross spots, stripes, solid
red, green, blue
Shapes
square, circle, triangle, plus
Textures
spots, stripes, solid 153-255
Brightness
153-255
Element Size / pixels
48-80
48-80
No. Elements per image
4
4
Image Size / pixels
256
256
9

---

Published in Transactions on Machine Learning Research (02/2025)
11.3 Examples
No. Gusdes 15
No. dassess 25
No. dlasees 14
No. cusoe 15
No. Cussee 15
X
No. diaawu
classes
cioses
+
Figure 14: Example images from the standard elements clataset. The number of classes each image belongs to is displayed above
No. classuu
classes
classes
classes
+
No. ciesnes
No. dessee
ciasses
No. casses
clanses
Figure 15: Example images from the simple elements dataset. The number of classes each image belongs to is displayed above it.
12 Consistency Experiment Details
Figure 3 shows the consistency error for various types of CAV. In this section, we describe how we find the different types of CAV. In each case, Val, a CAV trained normal using probe dataset, whereas the creation method for Val, varies for each experiment and is described below:
10

---

Published in Transactions on Machine Learning Research (02/2025)
Optimised CAV We gradient descent to optimise Ve,12 to minimise the consistency error:
arg min [lf ve,,,) (a12 + ve,b))l (ll
(45)
Po.f
The starting point for each optimisation process is CAV in layer l2, viin trained different random probe dataset. The nearly identical errors for each optimisation process support the likelihood of global minimum being reached.
Concept CAV These are simply normal CAVs trained as described in Sections 2 and 10 the CAVs are trained using a probe dataset containing x and X The distribution is over 12 for different random probe datasets Xa,r for Vel22 where 72 # rı denotes the random set.
Projected CAV CAVs in layer I projected into layer la using f: f(Watt) The distribution over different CAVs in layer I1, Va4"
Random CAV CAVs trained using a random probe dataset for both the positive and negative sets:
X X oir
(46)
X X
72 + ri
Random Direction Each element of Vol2, is drawn from a uniform distribution between [-0.5, 0.5], and rescaled to be a unit vector. The distribution is over different random seeds for the random number generator.
12.1 Scaling perturbations
To ensure that a Vatt stays in-distribution, scale the perturbations follows:
ail
à a. + Ve.l
(47)
|e,ll
where y is a hyperparameter used for perturbation size (typically set to 0.01), ||-Il the Le norm of vector, and ||a/]] the average norm of aj We scale the perturbation by the mean activation norm to account for the difference in the norms between the activation and concept vector to have consistently sized perturbations across layers.
We performed experiments to explore the sensitivity of consistency error to the size of the perturbation, ? for various layers and concepts for the ImageNet and Elements datasets. In Figures T6 and 17 we show the results for a variety of concepts for both the ImageNet and Elements datasets. Similar patterns were observed across experiments. As we increase ly|, the consistency error scales linearly, causes larger difference between f(ār,) and مi The scale of the y axis on the left of Figures 16 and 17 is not particularly mcaningful without context, so we scale it by the norm of the perturbation in layer l2; [
larger perturbation
| ||, on the right of the same figures. Values greater than one mean that the difference between (ār;)) and at are larger than the perturbation made layer lz.
|لراه
In addition to scaling y for both perturbations (in layer and 12), explored fixing the size of the perturbation in layer 4 and varying 7 in l2. We fixed n, to be 0.01 and then varied Ni from 0 to 0.02 and mcasured the consistency error of the two perturbations. In Figure 18 we show these results for striped CAVs from a ResNet-50 trained on ImageNet. We repeated the experiment on two types of CAVs: (1) standard CAVs (2) optimised CAVs (as described in the previous section). In both cases having y fixed at 0.01 gives a larger error, but the minimum error is not substantially different. We observe a larger difference in error between M = 0.01 and the minimum error for the standard CAVs than for the optimised CAVs. This is to be expected, the directions of the standard CAVs are less aligned than the optimised CAVs.
11

---

Published in Transactions on Machine Learning Research (02/2025)
1.70
0.8
Ecoraatorcy / |lperturbation)
1.65
0.6
1.60
layers.1
1.55
layers.2
0.4
1.50
layers.3
0.2
layers.1
layers.2
1.40
0.0
0.010
-0.005
0.000
0.005
0.010
0.010
-0.005
0.000
0.005
0.010
Perturbation gamma
Perturbation gamma
||perturbationn]
layers.1
ayers.2
1.60
layers.a
0.6
Ecrussionyy
0.4
1.50
Ecersstenyy
0.2
layers.1
1.45
layers.2
1.40
layers.3
0.0
-0.010
-0.005
0.000
0.005
0.010
0.010
-0.005
0.000
0.005
0.010
Perturbation gamma
Perturbation gamma
[|perturbutioon]
1.8
0.8
0.6
1.7
layers.1
layers.2
0.4
1.6
layers.3
layers 1
0.2
layers 2
1.5
layers.3
-0.010
-0.00 00.055
0.000
0.005
0.010
0.010
-0.055
0.000
0.005
0.010
Perturbation gamma
Perturbation gamma
1.60
0.8
Ecoouterrcy [[peeturrationnl
layers.1
1.55
layers.2
layers.3
0.4
1.45
0.2
layers.1
1.40
layers.2
0.0
layers.3
1.35
-0.005
0.000
0.005
0.010
-0.010
-0.005
0.000
0.005
0.010
Perturbation gamma
Perturbation gamma
Figure 16: The mean consistency error for (from top to bottom) red, blue, triangle and striped CAVs across layers (left) scaled by the size the perturbation (right) for the Elements datasct.
12

---

Published in Transactions on Machine Learning Research (02/2025)
2.0
aayer3.0
layer3.0
3.3
Coveltercy [pporturatton
1eyer3.1
3.2
layer3.2
1.5
3.3
layor3.3
layer
1.0
3.5er3.5
layer3.5
bayor4.0
1.5
layar4.0
0.5
1.4
1.3
0.0
-0.00
-0.030
005
0.000
0.
0.005
0.010
-0.010
-0.005
0.010
Perturbation gemme
Perturbation garmma
2.0
layer3.0
layer3.0
3.3
Coneeee [perturratto)]
layer
1.5
3.2
layer3.2
byer3.3
1.6
layer3.3
3.4er3.4
1.yer3.4
1.0
1.5
layer4.0
layer4.0
0.5
1.3
1-2
-0.030
-0.005
0.000
0.005
0.010
-0.000
-0.005
0,000
3.005
0.010
Nerturbation
Perturbation gamma
2.0
1.yer3.0
Ecveeseccy [porturaattonn
layer3.0
3.3
layer3.1
1.5
aaye33.2
layer3.2
l.yer3.3
2.6-
layer3.3
1.0
layer3.5
1.5
layar3.5
layer1.o
layer4.0 0
1.4
0.5
0.0
0.000
Nerturbation gamma
Perturbation gamma
Figure 17: The mean consistency across 10 CAVs for striped (top), lined (middle) and dotted (bottom) CAVs across layers (left) scaled by the size of the perturbation (right) for a ResNet-50 trained on ImageNet.
13

---

Published in Transactions on Machine Learning Research (02/2025)
striped leyer3.0
striped leyer4. .
1.1
2.4
2.2
0.9
2.0
Nom
Nom
18-
0.7
14-
14-
0,0325
0.0050
2.075
0.0125
2.0150
00175
0.0200
0,0000
0.0075
0.0150 0.115
Gumma
Chemma
striped layor3.0
striped leyon1.0
1.3
2.8
1.2
2.4-
16m
Nom
2.4
0.9
18-
1.6
0,0000 0.0325
0.0500
2.075
0.0000
a0i2s 2015e
00175
0.2200
0.0000 0.0025
0,0050
0.0075
0.100 Cammaa
0.025
0.150
a.17s
8.0200
Gamma
Figure 18: The sensitivity of the mcan consistency error to scaling n, for optimised CAVs (top) and standard CAVs (bottom) where / =layer3.0 (left) and I =layer4.0 (right) of ResNet-50 trained on Net. = 0.01 is the standard perturbation made in our experiments and is marked by a grey dashed line and the which gives a minimal consistency error is marked by a red dashed line.
12.2 Additional results
In this section provide additional results for the different consistency experiments in fig. We include the results for multiple concepts and layers for both the ImageNet and Elements datascts. To allow for better comparison between layers, we normalise the consistency errors in some of the figures. For each layer, the consistency error is divided by the mean error for the optimised CAVs. For the normalised plots, a value of one can be seen as the lowest error possible for that layer. The relative ordering of layers approximately consistent across the different types of CAV. For example, in fig. 19 there is a downward trend between layer3.0 to layer3.5 and then increase for layer4.0.
14

---

Published in Transactions on Machine Learning Research (02/202))
Layer
layer3.0
layer3.1
layer3.2
1.75
1.yer333
layer3.4
1.50
leyer 3.5
1.25
layer4.0
1.30
0.75
0.50
Optimised CAV
Concept CAV
Projected CAV
Random CNV Random Direction
Layer
layar3.0
layer3.1
Tayer3.2
layer3.3
10
1.4
layer3.5
1.3
layerd.
1.1
1.0
Optimised CAV
Concept CAV
Projected CAV
Random CNV Rendom Direction
Figure 19: The distribution of consistency errors (top) and normalised consistency errors (bottom) for different Ve for striped in selection of layers from ResNet-50 trained on ImageNet. Optimised CAV: The lower bound - vector optimised to have the minimum error. Concept CAV: striped CAVs, trained as normal. Projected CAV: striped CAVs from layer 4 projected into layer l2, f(Ve),). Random CAV: CAVs with random images for the probe dataset. Random Direction: Random vectors drawn from a uniform distribution
1.2
1.1
Econuistency
1.0
0.9
0.8
0.7
6.6
Optimbted Concast Irsedied
Deuciaa
Figure 20: The distribution of consistency errors for different Velly for square, triangle, red, green, solid and stripes CAVs for 'layers.1', 'layers.2' and 'layers.3' Optimisod CAV: The lower bound a vector optimised to have the minimum error. Concept CAV: CAVs, trained normal. Projected CAV: striped CAVs from layer I projected into layer lN, f(v.))) Random CAV: CAVs with random images for the probe dataset. Random Direction: Random vectors drawn from uniform distribution.
CNN trained on the Elements dataset.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Layer
1.2
layers.1
layers.2
1.1
layers.3
1.0
0.9
0.8
0.7
0.6
Optimised CAV
Concept CAV
Projected CAV
Random CAV
Random Direction
Layer
layers.1
layers.2
layers.3
1.6
Narmalized
1.4
1.2
1.0
Optimised CAV Concept CAV
Projected CAV
Random CAV
Random Direction
The distribution of consistency errors (lop) and normalised consistency errors (bottom) for different Voda for square CAVs for variety of layers for the Elements dataset. Optimised CAV: The lower bound a vector optimised to have the minimum error. Concept CAV: CAVs, trained as normal. Projected CAV: striped CAVs from layer projected into layer l2, [(Vo,)) Random CAV: CAVs with random images for the probe dataset. Random Direction: Random vectors drawn from a uniform distribution.
Figure 21:
12.3 DeepDream
DeepDream [Mordvintsev al. 2015) a feature visualisation tool. It starts from an image, generated by sampling noise from random uniform distribution and then iteratively updates the input image to maximise the L2 norm of activations of particular layer. We use similar approach but instead maximise the dot product between the activations and a CAV: Va, dg. In Figures 22 and 23 we show these visualisations for selection of ImageNet CAVs for successive layers in a ResNet-50.
These visualisations offer qualitative evidence that the CAVs represent different components the same concept in different layers. For example, the car concopt in Figure 22 consists of many square box-like objects in earlier layers, but nothing recognisable a car, whereas, in later layers, whole car sections can be seen in the visualisations.
16

---

Published in Transactions on Machine Learning Research (02/2025)
striped
polka
meshed
anemone
sidewalk
car
layer3 0
layer3.1
layer3.2
layer3.4
layer3.5
layer4.0
layer4.1
Figure 22: CAV visualisations using DeepDream for selection of concepts from ImageNet. Each row corresponds to layer of ResNet-50 and each column different concept.
17

---

Published in Transactions on Machine Learning Research (02/2025)
bicycle
boat
person
water
waterfall
sea
layer3.5 0
layer3.1
layer3.2
layer3.3
Jayer3.4
layer3.5
layer4.0
layer4.1 1
Figure CAV visualisations using DeepDream for selection of concepts from ImageNet. Each row corresponds to layer of ResNet-50 and each column different concept.
18

---

Published in Transactions on Machine Learning Research (02/2025)
12.4 Inconsistent TCAV Scores
In this section, we display additional examples of inconsistent. TCAV scores across layers for ImageNet classes. In Figure 24 each subfigure contains at least one concept that has inconsistent TCAV scores. We include example images from those classes in fig. 25
layer3.0
layer3.1
0.6
layer3.2
layer3.3
TCAV
0.4
layer3.4
layer3.5
layer1.0
0.2
layer4.1
road
sidewalk
bicycle
person
(a) TCAV scores for selection of concepts for the 'car wheel' class in ImageNet.
1.0
0.8
layer3.0
layer3.1
0.6
layer3.2
layer3.3
TCAV
layer3.4
0.4
layer3.5
layer4.0
0.2
layer4.1
0.0
boat anemone water waterfall sea
(b) TCAV for selection of concepts for the 'dock' class in ImageNNt.
1.0
0.8
layer3.0
layer3.1
0.6
layer3.2
layer3.3
TCAV
layer3.4
0.4
layer3.5
layer1.0
0.2
layer4.1
0.0
sidewalk
waterfall
(c) TCAV scores for selection concepts for the 'sidewalk" class in ImageNet.
1.0
0.8
layer3.0
layer3.1
0.6
layer3.2
layer3.3
TCAV
0.4
layer3.4
layer3.5
layer4.0
0.2
layer4.1
0.0
person
(d) TCAV scores for selection of concepts for the 'lionfish' class ImageNet.
Figure 24: Inconsistent TCAV scores for a selection of concepts and classes in ImageNet. The standard deviation is shown in black for significant results and red for insignificant results. The mean TCAV score for random CAVs are shown as horizontal black lines.
19

---

Published in Transactions on Machine Learning Research (02/2025)
wheel
cocck
fourcain
lienfish
Figure 25: Example images from a selection of ImageNet classes.
12.5 TCAV Layer Consistency Score
At the end of S 6.1 we defined the TCAV layer consistency score as measure of how often significant TCAV scores disagree on the direction of sensitivity for a specific concept and class across set of layers. Here, we provide some example results for both the ImageNet and Elements datasets. each case we train 30 CAVs for each concept/layer pair to calculate the TCAV classes from the simple version of the datasct,
across. For Elements, when analysing all 69 of 0.841 across
obtaín mcan TCAV consistency all concepts and all layers of the simple NN. 38/690 sets of TCAV scores had TCAV layer consistency score of less than 0.2 and 158/690 less than 0.5. For ImageNet, we obtain a mean TCAV consistency score of 0.868 across the last 8 convolutional layers (from layer 3.0 to layer 4.1) for a range of concepts (striped, polka, meshed, road, sidewalk, car, bicycle, boat, person, anemone, water, waterfall, sea, sky, snow, tree) and classes (zebra, leopard, liger, car wheel, acoustic guitar, academic gown). 7/96 sets of TCAV scores had TCAV layer consistency score of less than 0.2 and 13/96 had a score less than 0.5. See Figure 26 for the full distributions.
70
400
50
Count
300
Count
40
200
20
100
10.
0.0
0.4
TCAV Layer Consistency Score
TCAV Layer Consistency Score
Figure 26: Distribution of TCAV layer consistency score for the simple Flements dataset (left) and selection of classes/concepp for ResNet-50 trained on ImageNet (right).
20

---

Published in Transactions on Machine Learning Research (02/2025)
13 Entanglement Experiment Details
We use cosine distance measure how similar two CAVs are. Assuming the CAVs, Vay, and vegds are unit vectors this simplifies the dot product of the two:
Veg. Veg.
Cosine Similarity || || |
(48)
-Vait Vo2.
In our visualisations (fig. 4nd scction 13.1 compare multiple CAVs for each concept, each with different random probe dataset, denoted by r. This allows us to see how similar CAVs for the same concept are
repeat training runs. Each value in the visualisation is the mean cosine similarity between the concepts on its corresponding x and y axis labels between all CAVs which do not have the same random probe dataset, i.e.:
R R 4o 2
Σ ΣE F1 r2!=11
(49)
1(
where 1 is the CAV corresponding to concept c1, layer and random probe datasct Xen
13.1
Additional Results
13.1.1
Elements
we show the cosine similarities for all concepts in the standard Elements dataset. The conclusions are similar to the visualisation for E, in fig. A but the negative associations between the mutually exclusive concepts are weaker. We hypothesise that this because there are more concepts within each group. This is empirically justified as the average cosine similarity of each group approximately corresponds to the number of concepts in the group. If this hypothesis truc, makes it unlikely that we will find similar groupings in real datasets containing natural images concepts are rarely partitioned as neatly or few possible combinations.
In fig. 27
21

---

Published in Transactions on Machine Learning Research (02/2025)
layers.4
1.00
square
0.51
0.06
-006
0.07
0.05
0.05 0.04 0.05 0.04 0.06 0.04
0.05
0.06 0.02
drcle
0.06
0.50
-0.09
0.03
0.05
0.05 0.04 0.03 0.06 0.06 0.05
0.04 0.04 0.01
triangle
-..05-0.09
0.53
0.03
0.03 0.06
0.03 0.01
plus
0.07-0..0 -0.05
0.52
0.07 0.05 0.07
0.05
0.05 0.04 0.07
0.06 0.05 0.03
0.50
0..0500.55 -0.05
-0.07
0.51
0.04 0.04
0.03
-0.01 0.01
0.06
0.01
0.02 -0.01
red
0.05
0.05
0.03
0.05
0.04
0.53
0.06
0.06
00.04 00.0300.33.000 9..0000005
0.03 0.03 0.03
yellow
0.04
0.04
-0.03
-0.01
0.03
green
0.05 0.03
0.04
0.05 0.03
00.06
0.05
0.52
0.04
0.02
0.03
0.03
0.07 0.04
0.04 0.06 0.03 0.05 00.00
0.00 0.03
0.04
0.55
0.00
0.005
0.05
0.04 0.03
blue
0.06 0.06 0.06 0.04 0.01 -0.03 0.01 0.02
0.08
0.53
0.06
0.04
0.05 0.02
magenta
0.05 0.06 0.07 -0.05 0..05
0.06
0.04
0.04
solid 0.05 0.04 0.06 0.06 0.01 0.03 0.05 0.03 0.05
0.04
0.02
0.51
0.16
spots_polka
0.06 0.04 0.03 0.05 0.02 0.03 0.03 0.07 0.04 0.05 0.04 -0.16 0.50 -0.16
cripes_diagona
0.03
square
aircle
triangle
plus
cross
red
yellow
green
cyan
blue
magenta
solid
spots polka
stripes diagonal
Figure 27: Mean pairwise cosine similarities between 30 CAVs for different concepts from the standard
Element dataset.
13.1.2 ImageNet
In Figure 2% we show the pairwise cosine similarities for selection of concopts for a ResNet-50 trained on ImageNel. Net. The associations between concepts are less clear-cut than for Elements, but qualitatively they make intuitive sense. For example, the concepts most similar to field are grass and earth, in the top of Figure [8 and the concopts most similar sidewalk are bicycle, road and hedge. The latter makes sense as many of the hedge exemplars in the probe dataset are next to a path or road. This emphasises the importance of designing your probe dataset to match the concept you want the CAV to represent.
22

---

Published in Transactions on Machine Learning Research (02/2025)
layer4.1
sky
0.35
0.13
0.07
0.10
0.11
0.14
0.17
0.18
0.14
0.07
0.13
tree
0.13
0.35
0.10
0.19
0.19
0.18
0.17
0.14
0.19
0.15
0.18
0.75
0.07
0.10
0.32
0.10
0.11
0.11
0.11
0.08
0.11
0.10
0.10
0.50
grass
0.10
0.19
0.10
0.36
0.23
0.16
0.17
0.14
0.27
0.15
0.15
0.25
earth
0.19
0.11
0.35
0.20
0.23
0.16
water
0.14
0.18
0.11
0.16
0.20
0.36
0.23
0.23
0.18
0.13
0.18
0.00
boat
0.17
0.17
0.11
0.17
0.18
0.23
0.34
0.21
0.18
0.15
0.19
-0.25
sea
0.18
0.14
0.08
0.18
0.23
0.21
0.35
0.18
0.08
0.17
-0.50
field
0.14
0.19
0.11
0.27
0.23
0.18
0.18
0.18
0.36
0.15
0.17
wood
0.07
0.15
0.10
0.15
0.16
0.13
0.15
0.08
0.15
0.30
0.14
-0.75
0.13
0.18
0.10
0.15
0.19
0.18
0.19
0.17
0.17
0.14
0.35
-1.00
sky
tree
person
grass
earth
water
boat
sea
field
wood
snow
layer4.1
person
0.32
0.13
0.14
0.11
0.11
0.08
0.11
0.10
0.75
bicycle
0.13
0.37
0.19
0.25
0.21
0.12
0.21
0.19
0.50
0.14
0.19
0.34
0.16
0.19
0.10
0.17
0.14
0.25
sidewalk
0.11
0.25
0.16
0.37
0.25
0.11
0.21
0.16
0.00
road
0.11
0.25
0.36
0.11
0.21
0.18
-0.25
waterfall
0.08
0.12
0.10
0.11
0.11
0.31
0.13
0.17
-0.50
hedge
0.11
0.21
0.17
0.21
0.21
0.13
0.34
0.23
-0.75
hill
0.10
0.19
0.14
0.16
0.18
0.17
0.23
0.34
-1.00
person
bicycle
car
sidewalk
mad
waterfall
hedge
hill
Figure 28: Mean pairwise cosine similarities bet ween 30 CAVs for different concepts from ImageNet.
23

---

Published in Transactions on Machine Learning Research (02/2025)
13.2 Polysemanticity
In Figures 4 and 27 we find that the vector representations of mutually exclusive concepts are anti-correlated with each other. Each concept vector does not just mean, for example, red. It means red, not green and not blue.
Elhage et al. (2022) also find evidence for polysemantic representations. However, they found individual neurons which were polysemantic, whereas here vectors, i.e,, groups of neurons, are polysemantic. In addition, the reasoning is different. The polysemanticity discussed in Elhage et al. (2022) is caused by sparse features being compressed into fewer neurons than there are features. Here, we do not have sparse features and have more neurons than features. Instead, the polysemanticity caused by associations between the concepts and the optimisation process favouring negatively correlated representations for mutually exclusive concepts.
13.3 Dot product distributions
The definition of entangled concepts eq. 6 uses the dot products of a CAV trained on concept e with the the activations of probe dataset for different concept C2. Figure fig. 29 shows the distribution of dot products for selection of CAVs, model training datasets and concept probe datasets. We use the same set of Elements datasets in section 6.2 E1, Ez, Ez, where E, has association between red and triangle and E, and E3 have succossively stronger associations between the concepts. This changing association is apparent in the dot product distributions. For E,, the dol products of the test probe datasets differing from the CAV concept do not differ significantly from the dot products for random images (the negative probe datasot). Whereas, for Eg and Eg, the random distribution is shifted lower than for either CAV, even the CAV labelled differently from the test dataset. This shows that the red and triangle CAVs are entangled for these datasets natasets/models.
24

---

Published in Transactions on Machine Learning Research (02/2025)
Train Datasct E. CAV red
Train Dotasct 6, CAV branngcc
Chunt
Train Dataset CAV red
Train Dataset
Ex
CAV briangle
Chunt
Test Dataset
Red
triangle
Random
Dataset red
briangle
Count
-15
-5
15
-15
-5
Dol Product
Dot Product
Figure 29: Distribution of dot products (Vey. amg.1) for the three versions of Elements with increasing association between red and triangle (E1, E, and Ez). The distribution of dot products are displayed for three different test sets containing in-distribution images for red, triangle and random images.
25

---

Published in Transactions on Machine Learning Research (02/2025)
14 Spatial Dependency Experiment Details
14.1 Spatially Dependent Probe Datasets
For Elements, the probe datasets contained elements that only appear in specified locations - example is shown in fig. 2 For ImageNet, we do not have direct control of where objects can appear. Therefore, we greyed out different regions of the image. For example, we created oppositely dependent concepts by either greying out the middle of the image, or the edges see Appendix 14.1 for examples.
+iml
(a) The positive sel.
+
(b) The negative set.
Figure 30: Example images for the positive and negative sets of the probe dataset for the red top in the simple elements dataset.
26

---

Published in Transactions on Machine Learning Research (02/2025)
(a) The positive set for striped middle.
The negative for striped middle concept.
(c) The positive set for striped edges concept.
(d) The negative set for striped edges.
Figure 31: Example images from spatially dependent probe datasets for ImageNet.
14.2 Spatial Norms Details
When finding CAVs, we use the activations layer in a convolutional neural network, which has shape H W x D, where H. W and D are the height, width and number of channels, respectively. When finding the CAV, these activations are flattened be vectors of length m W The value of each element in the activations depends its specific height, width and depth indices. As result, each corresponding
27

---

Published in Transactions on Machine Learning Research (02/2025)
element of the resultant CAV is also index-dependent. We are interested in spatial dependence, to visualise how CAV varies across the width and depth dimensions we calculate the T.2 norm of each depth-wise slice the CAV's spatial norms.
14.3 Individual Spatial Norms
In fig. 32 we present the spatial norms for striped in a ResNet trained on the ImageNet dataset. Each heatmap is for a different random probe dataset, denoted by r. The different. patterns in the heatmaps show that each individual vi has a spatial dependency which differs across However, when average the norms across multiple CAVs, Σ S obtain uniform distribution across the spatial dimensions. This uniformity suggests that the spatial dependencies of each individual CAV cancel out across multiple seeds, as depicted in the top rows of fig. 6 in the main text.
Spatial Norm
0.2
0.1
0.0
Figure 32: Individual spatial norms for striped, where each CAV was trained different negative probe set, for layer4.1 of a ResNet trained on ImageNet.
14.4 Additional Spatial Norms
layers.1 square bot
square
square_top top
square left
square_rigg right
red
red top
red bot
red left
red_right
0.06
Norm
0.04
solid_top
solid_boo
_eft
right
Figure 33: Mean CAV spatial norms across 30 CAVs for selection of concepts in the Element. dataset for the second convolutional layer.
28

---

Published in Transactions on Machine Learning Research (02/2025)
layers.4 square bot
square top
square left
square right
0.200
0.150
red
red
red bot
red left
red
0.100
0.075
top
right
0.025
Figure 34: Mean CAV spatial norms across 30 CAVs for selection of concepts in the Element dataset for the fifth convolutional layer.
14.5 Spatial Means
Instead of visualising the norm of cach depth-wise slice, we can visualise the mean. We default to showing the norm because you could have spatial mean of zero in region of activation space which has a large effect on the directional derivative. This could occur, for example, if half the elements of the CAV are large and positive and the corresponding gradients are positive, and half the elements the CAV are large and negative and the corresponding gradients are also negative. This would lead that spatial region having large contribution to the directional derivative, but the spatial mean would be close to zero. The spatial norm, however, would be large for this region. This makes the norm better measure of the effect of cach region, but the mean can still be useful to show the direction of that effect.
layers.1
square
square_top top
square bot
square left
square_ripe right
0.003
0.002
red
red top
red bot
red left
red_right
0.001
Mean
0.000
-0.001
bolid_oott
solid solid_left
right
-0.002
Figure 35: Mean CAV spatial across 30 CAVs for selection of concepts in the Flement dataset for the second convolutional layer.
29

---

Published in Transactions on Machine Learning Research (02/2025)
layers.4 square bot
square
square top
square left
square right
0.006
red
red
red bot
red left
red
0.002
0.000
solid top
solid right
-000004
-0.006
Figure 36: Mean CAV spatial means across 30 CAVs for selection of concepts in the Element dataset for the fifth convolutional layer.
14.6 Spatially Dependent TCAV Scores
In this section we provide example TCAV scores which differ across complementary spatially dependent CAVs, for at least one concept, the TCAV score is the opposite side of the null for the edges version of concept compared to the middle version (or the left/right and top bot/botoom versions).
TE0 Spoe 0.4
TOW Soure
byerd
kywon
aeconn Adges
water
wstes Adges
mster coidon
momontt eigios
aecovss Adges
westet moilde
dddges
unter roudte
anemant resges
(a) TCAV scores for selection of concepts for the (b) TCAV scores for a selection of concepts for the 'spiny 'anemone fish' class ImageNet.
lobster' class ImageNet.
Figure 37: Examples of spatially dependent TCAV scores in ImageNet. Each subfigure is a separate class. The standard deviation is shown in black for significant results and red for insignificant results. The mean TCAV score for random CAVs are shown horizontal black lines.
30

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
0.8
0.6
TCW Score
loyers
0.4
0.4
layers.3
0.0
green
green
oreen bet
Square
square top
Square b0C
0.0
123
red
square SQuare
square bot
sold
soldd 10p
5old box
top
199
002
top
green squares on the top'
(b) 'solid red squares on bottom'
1.0
0.8
1CAM State
0.6
leyers. leyers. 3
0.4
0.44
leyers: 4
0.2
0.0
blue
Diue
Blue
square
square
squara
stripes strpps strips top
0.0
red
red
red right
Eriangle
triangee
trinngle right
top
(c) 'striped blue squares'
(d) 'red triangles on the left'
1.0
Score
0.00
layers loyens layers 4
TCWW
0.4
0.2
0.2
1ed
1B0
red right
trinngle triangle triangle
5pots
spots
spots right
trangle
traangle
brongle
stripes
stripe
stries
right
right
right
(e) 'spotted red triangles the right"
(I) 'striped triangles the right'
Figure 38: Examples of spatially dependent TCAV scores in the spatially dependent version of Elements for Nes-et.50. Each subfigure is separate class. The standard deviation is shown in black for significant results and red for insignificant results. The mean TCAV score for random CAVs are shown as horizontal black lines.
14.7 Spatially Dependent CAVs in ViTs
Even though we cannot use spatial norms to visualise the spatial dependence of CAVs in a transformer based architecture, this section we have some preliminary results suggesting that that spatially dependent CAVs can still be crcated for a VIT-B16 (Dosovitskiy al. (021). As for the ResNet-50, we test the model on the spatially dependent Elements dataset and found TCAV scores that vary by location.
We finetuned VIT-B16 model pretrained on ImageNet for 50 epochs the spatially dependent version of Elements (i.c. there are some classes which depend on the location of the objects as well as which concepts are present). We used an exponentially decaying learning rale with initial learning rate of 0.0001 and of 0.95. Similarly to the previous section, we demonstrate that we obtain different TCAV scores for the left versions of concept than the right, although there substantially less variation in TCAV score than for the ResNet-50 (section 14.6p. Future work should perform more extensive experiments determine how the properties analysed in this paper affected by architecture.
31

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
0.8
TCAV Score
0.6
0.4
encoder layer
encoder layer
0.2
encoder layer 10
0.0
encoder layer 11
triangle
triangle left
triangle right
stripes diagonal
stripes diagonal left
stripes diagonal right
1.0
0.8
TCAV Score
0.6
0.4
encoder layer
encoder layer
0.2
encoder layer 10
0.0
encoder layer 11
triangle
triangle left
triangle_rightt right
stripes diagonal
stripes diagonal left
stripes, diagonal right
Figure 39: Spatially dependent TCAV scores in the spatially dependent version of Elements for a VIT-B16. The TCAV scores for the classes "striped triangles on the left" (top) and "striped triangles on the right" (bottom) are shown. The standard deviation is shown in black for significant results and red for insignificant results. The mean TCAV score for random CAVs are shown as horizontal black lincs.
14.8 Dot product distributions
The definition of concept vector spatial dependence in eq. compares CAV, datt with the activations of two positive probe datasets with different spatial dependencies, a and agt.ng by taking the dot product between them att,. Vet1- In fig. 40 we show the distribution of dot products for three concepts and three test probe datascts the spatially dependent version Elements. The scparation between the distributions for the stripes left and stripes right probe datasets (blue and green bars, respectively) for both the stripes left and stripes right CAVs (left and right plots, respectively) demonstrate that these CAVs are spatially dependent.
15 Further Related Work
Recent work highlighted problems with concept-basod explanation methods. Ramaswany et al. (002aa) showed that using different probe datasets to interpret the same model can lead to different. explanations for the same concept. Similarly, [Soni et al. (2020) showed these methods to be sensitive to the random soed used to sample images for the negative set. Our work complements this rescarch by investigating the underlying properties of concept vectors and how they may cause problems when interpreting concept- based explanations.
32

---

Published in Transactions on Machine Learning Research (02/2025)
CRC Sinpey Left
CAV Stipes
C4) So poo Nilht
Clas
Test Datacct
Stripee Lert
Stries
Stripen Right
-2
12
Dot Prosict
Dut Poouu
DotPhoduct
Figure 40: Distribution of dot products between spatially dependent CAVs and image activations (a (a (ee,)) for the spatially dependent Elements dataset. Each column is for different CAVs. From left to right these are: stripes left, stripes, stripes right. For each CAV we show the distribution for three positive probe datasets: stripes left (blue), stripes (orange), stripes right (green).
Extensions the original TCAV have been suggested, attempting to improve aspects of the original method. For instance, Ghorbani et al. 2019) automate concept discovery by using super-pixels and clustering, remov- ing the need to handcraft probe dataset. Zhang et al. (2020) and Schrouff et al. (2021) change how CAVs are created to produce local and global explanations. However, these methods still use vectors to represent concepts (Alain & Bengio 2017 As such, our work is still applicable to each of the extensions.
The properties analysed in this paper are generally applicable. To give insight into when the various prop- ertics may be relevant, we performed a review of papers which use CAVs in medical imaging and computer vision research. In each case, we checked if the authors had done any checks related to layer consistency, entanglement. spatial dependence of CAVs. Almost no papers evaluated the effect of these properties their results, and when they did it was by creating CAVs in multiple layers, providing some robustness to layer inconsistency. In Table 3 we provide a list of these papers, detailing any use-cases where our recommendations could have helped check the impact of CAV properties on results.
Table 3: Examples in computer vision and medical imaging rescarch, where consistency, entanglement and spatial dependence may impact analyses. We use the following abbreviations: skin cancer (SC), skin lesions
(SL), breast cancer (BC), histology (H) CIFAR-10/100 (CF) (Krizhevsky) 2009), COCO (CO) (Lin et al.
2014
CUB (CB) (aah et al. 2011). Places365 (PI) (Zhou et al. 2017), Waterbirds (WB) (Sgawa et al. ImageNet (Im) (Deng et al. 2009
2020
Property
Modicine
CV Research
Papers
SC BC
CF CO CB Wb.m
Consistency
4
4
Yan al. 2022Y
Ramaswany et al
0p92a Füchark at
0099 Chosh al:
Lucieri
al
202m
Yokomkgumi
2023
Entanglement
Pirbook
Spatial De-
20233
Rameswany
pendence
3050
2125
Vukrakgand
33


Reviews:
[{"review_id": "hhwsziKEc6", "reviewer": "Reviewer_yta8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\nS1. Good work discussing the properties of TCAV which indeed has not been well explored by related works.\nS2. Range of datasets considered is good and a good mix of real and synthetic data has been used to test out the proposed properties. \nS3. Appendix section is strong and exhaustive with results on a real-world ImageNet dataset included.\nS4. Synthetic Dataset Contribution: \"Elements\" dataset is a notable contribution and can be used further for controlled experiments to understand CAV properties better and provides a resource for the community to explore these issues further.\n\n\n## Weakness:\n\nW1. Lackluster properties discussion: The properties introduced and tested provide a good overview of CAVs, however in my opinion are not the most important considerations for TCAVs. Practitioners using such explainability methods are often more interested in learning about the robustness of the method itself [1], its behavior under distribution shift and adversarial attacks [2], etc. This weakens the motivation for the paper. In addition, it is not surprising that CAVs for different layers are not similar (information encoded in every layer is different) making the analysis simple and less useful. Properties 2 (entanglement) and 3 (spatial dependence) are more interesting for TCAV and unfortunately, the discussion on those is also not satisfactory (See W2 and W3). \n\nW2. Experiment design for entangled concept vectors: Analysis based on Equation-6 does not seem correct to me as it assumes that there is sufficient semantic gap between concepts. There can be multiple instances where $a^+ . v_{c,l}$ can be high if a significant portion of two concepts are related (for example vertical and horizontal stripes). Even though concepts are easy to isolate in Elements dataset, it might not be possible in more complex data. \n\nW3. Spatial dependence is not a 'concept-property': Even though Elements dataset captures the location of the objects pretty well and the concepts can be seen to be spatially correlated in the heatmaps, the choice of the dataset makes the property analysis too simplistic. As concepts are usually more abstract and cover multiple parts of the objects in the image themselves, it is not clear how this property will be satisfied for more complex, distributed concepts. For example, consider 'stripes' and the object in question is a Zebra, where stripes are all over the body and cannot be localized as in the Elements dataset (example from TCAV paper), this analysis does not make sense. I hope the authors will think deeply about such an example and perform experiments validating this. \n\nW4. TCAV for more vision backbones: In W1, I mentioned that spatial dependence is an interesting property. I would like to see how this changes when the backbone of the classifier itself is changed to architectures with different inductive biases than CNNs (like ViTs). If spatial dependence results are still valid, we can be sure of the property itself.\n\n[1] Fooling Explanations in Text Classifiers, Ivanky et al.\n[2] Interpretation of Neural Networks is Fragile, Ghorbani et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors analyze a popular post-hoc interpretability method - Testing using Concept Activation Vectors (TCAV). \n\nCnt1. Particularly,  this work analyzes 3 properties of CAVs - consistency across layers, the entanglement of CAVs among each other and that CAVs are spatially dependent. The authors propose experiments that demonstrate how the three aforementioned properties are satisfied/not satisfied. \n\nCnt2. The datasets utilized are both synthetic (Elements) and real-world (Lesions, ImageNet, etc.) making the analysis comprehensive. \n\nCnt3. Practical recommendations are given by the authors on how to best utilize TCAV for a real-world problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(major) C1. Incorporating ViT and similar backbones for a more thorough analysis (as mentioned in W4) \\\n(major) C2. Improve experiment design for entangled concept vectors (as mentioned in W2) \\\n(major) C3. The Elements dataset is too simplistic to test properties on spatial dependence. Please consider adding more complex datasets in the analysis. \\\n\n(minor) C4. Formulation of the experiment around entangled concept vectors: As mentioned in W2, please justify why the hypothesis in Eq 2 holds \\\n(minor) C5. Section 12.3 is unclear. I do not understand why DeepDream-type visualizations help in CAVs. Visualizations like these are not strictly related to CAVs and can be done using any sampling techniques (activations, PCA, etc.)", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X00hQZLO92", "reviewer": "Reviewer_rcyS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **Clarity and Presentation**: The paper is well-written with a clear presentation.\n- **Insightful Investigation**: It explores interesting problem of practical limitations of CAVs.\n- **Practical Application**: Interesting application to the model trained on the ISIC dataset.\n- **Recommendations for Practitioners**: Clear recommendations for the practitioners who to mitigate described limitations of CAVs.\n\n### Weaknesses\n\n- **Dependence on Underlying Data**: The CAV approach heavily depends on the dataset employed for probing. The paper focuses on the CAVs directly but approaches the stated problems/limitations only from the model's perspective, not considering that the problem might lie in the data, that is used for probing. It is possible that the entanglement or spatial dependency of CAVs arises from the data used to compute these concepts, and not in the model representations.\n\n- **Lack of Rigor in Hypotheses**:\n  - **Null Hypothesis 2 (NH2)**: The formulation lacks rigor and depends on the datasets employed. In the case of \"stripe\" and \"red\" concepts, the entanglement might arise if the \"stripes\" dataset includes images with red stripes, indicating a dataset issue rather than a problem with CAVs.\n  - **Null Hypothesis 3 (NH3)**: The discussion on spatial dependency lacks depth regarding what constitutes a \"concept.\" For example, this research question could be perceived not as discussion on spatial dependency of CAVs, but rather as discussion about that probing datasets used for CAVs do not have all the concepts, that model could have learned (i.e. that concept \"stripes\" should be not one, but more concepts \"stripes on the left/right/up/down\".\n\n- **Section 6.1**: - The impossibility of consistency is demonstrated only for functions $f$ that are a single ReLU or Sigmoid layer. The analysis does not consider more complex functions that involve multiple nonlinearities or non-linear mappings, that are more realistic examples, given that $f$ is introduced as a mapping between different layers of Deep Neural Network. For vectors $A \\in \\mathbb{R}^N$ and $B \\in \\mathbb{R}^K$, there exists a function $f: \\mathbb{R}^N \\rightarrow \\mathbb{R}^K$ that does not conserve vector addition (i.e.,  $f(a + b) \\neq f(a) + f(b)$ ) but satisfies $f(x + A) = f(x) + B$ for all $x \\in \\mathbb{R}^N$. Such function could be build as follows: assume $A = MB$, where $ M \\in \\mathbb{R}^{K \\times N}$.  $f$ could be defined as $f(x) = Mx + H(x)$, where $H: \\mathbb{R}^N \\rightarrow \\mathbb{R}^K $ s a periodic function with period $A$ (i.e., $H(x + A) = H(x)$). Then, $\\forall x \\in \\mathbb{R}^N: f(x + A) = M(x + A) + H(x + A) = Mx + MA + H(x) = f(x) + B$.\nThis suggests that theoretically non-linear mappings can produce consistent CAVs.\nThe paper does not address why, in practice, the mapping between layers cannot approximate such functions.\n- **Limited Quantitative Experiments**: The investigation significantly relies on qualitative demonstrations, such as the \"striped\" concept in Figure 3. Assessing the hypotheses across various models and a broader set of concepts would strengthen the conclusions. Without extensive quantitative analysis, the results may be anecdotal and not indicative of general trends. As a solution, a broader spectre of models and datasets could be employed, and results could be shown not on individual concepts, but rather on some qualitative general metrics.\n\n### Minor Weaknesses\n - Definition 1: in description layers are defined as $l_1, l_2$ but later actors write that $l_1 < l_2$ -- while conceptually it is clear what is meant there, I believe the inequality between layers is not appropriate way to state that one layer is preceding another\n- Definition 3: the  \\phi is not defined", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates three key properties of Concept Activation Vectors (CAVs): inconsistency across layers, entanglement with other concepts, and spatial dependency. The authors explore each property and apply their findings to a practical example involving a model fine-tuned on a medical dataset. Authors provide recommendations for practitioners on how to mitigate the impact of these properties when using CAVs for model interpretation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical to securing the recommendation for acceptance:\nAddressing the Weaknesses discussed in the Strengths And Weaknesses section.\n\n### Would strengthen the paper:\n1. Authors state research questions as Null Hypothesis, that naturally invoke the expectations that some statistical test would be employed to test the hypothesises -- I would propose to rename this for the Research Questions instead\n2. Hypothesis 3 is \"Concept activation vectors cannot be spatially dependent\" -- I would propose for a bit of a clarity to rename this to \"CAVs are spatially independent\"\n3. Before the Equation 9 there is an new line inserted could be deleted to save space and make the style consistent with other equations\n4. Missing discussion on the general limitations of the CAV approach -- the concepts are mapped *to* the model, therefore we can only find something we already know. Other methods in concept-based and mechanistic interpretability aim to explain features in the models, therefore in general being able to find some novel, previously unknown concepts.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t71Tpj89M2", "reviewer": "Reviewer_VsKT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thank you so much for submitting this work! I enjoyed reading this paper and appreciate the time taken to write it up in a very straightforward and clear manner. Below are what I believe are this paper’s main strengths, followed by what I believe are some of its weaknesses:\n\n### Strengths\n\n1. **[Significance] (Major)** I think this paper does a great job of introducing initial formalisms that capture several sorts of issues/problems that have been discussed for CAVs across the literature. In particular, I think this work’s take on studying CAVs is an excellent way to get a good practical understanding of how CAVs work and how you could use them. As such, I believe this paper has good potential to be useful/significant, particularly for practitioners (e.g., the skin cancer example is a good primer for using CAVs and analysing their scores).\n2. **[Significance] (Major)**  This work provides several contributions: analysis and formalisms for three key properties of CAVs, a new synthetic dataset (Elements), a set of useful, practical guidelines for using T-CAV, and a set of examples showcasing failure modes of T-CAV. I am marking this strength as “major” rather than “minor” because although all of these amount to a good set of contributions, independently, they may be seen as small contributions on top of previous existing/related works. See below for specific examples/questions about this.\n3. **[Quality] (Minor)** The plots, figures, and overall presentation are easy to follow and understand.\n4. **[Clarity] (Major)** The paper is well-written and easy to read. There are a few smaller typos (see below), but they are all minor, so the overall flow is unaffected.\n\n### Weaknesses\n\nIn contrast, I believe the following are some of this work’s limitations:\n\n1. **[Soundness] (Critical)** I have some serious concerns/doubts about the theoretical results introduced in this paper. They lack formalizing and may be **potentially wrong/unsound**. Please refer to my requested changes in Section 6.1 for specific comments on this matter.\n2. **[Significance/Originality] (Major)** Most of the results and ideas discussed in this work are not particularly counter-intuitive and are highly related to observations from previous works (some of which are, rightly so, cited by this paper as well). In particular, although I like how the null hypotheses are defined in Section 3, I would argue that all of these hypotheses are intuitively expected to be false from the way CAVs are defined. This does not mean this work is not useful (intuition, after all, can be very wrong), but it does diminish the potential impact and significance of the results discussed and introduced in this work. This is especially true since there are not a lot of other kinds of results (e.g., sound theoretical results) that could shine some light on the properties studied by this paper.\n3. **[Quality] (Major)** Some of the definitions and assumptions made by this work (particularly on the theoretical formalisms) could benefit from better justification (at least at an intuitive level and at best with a theoretical backing). See below for specific examples of elements exemplifying this concern.\n4. **[Quality/Clarity] (Minor)** A good part of the experimental section is spent making conjectures or claims about potential reasons for the observed results. While insightful, it is challenging from the reader’s point of view to understand which conclusions could be victims of confirmation bias unless more evidence is provided to back them up. This is particularly true in the case study, where there is a lot of subjective interpretation of CAV scores.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines three fundamental properties of Concept Activation Vectors (CAVs), a widely used post-hoc XAI approach that is the backbone of several modern interpretable architectures. In particular, this work studies whether CAVs are (1) consistent across layers, (2) entangled across related concepts, and (3) faithful to known spatial dependencies. This paper's main contributions are therefore:  (i) it provides a definition for CAV consistency, together with a metric to measure it, and shows that CAVs are rarely consistent and should therefore be observed from multiple layers within the model; (ii) it provides a formal definition for CAV entanglement and shows how cross-concept entanglement may lead to mistaken conclusions derived from T-CAV scores; (iii) it shows that CAVs can be spatially local (i.e., they may be localized to specific areas of the input) and that this locality could be detected via norm maps; and (iv) it introduces a new simple yet flexible concept-annotated dataset called Elements. These contributions are backed by empirical evidence on this work’s proposed dataset (Elements) and ImageNet. Moreover, using a real-world case study, this work showcases how all the considerations discussed in this paper can manifest themselves in a single practical scenario. Overall, this paper highlights some key limitations of concept-based post-hoc explanations while suggesting pragmatic considerations when deploying these methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below, I list some potential changes for each section and their importance in securing my recommendation for acceptance. If time is limited, please focus on all **critical** requests, followed by all **major** requests. If I misunderstood something at any point, just let me know, as this is always a possibility (apologies in advance if that’s the case!).\n\n### General Requested Changes\n\n- **(Major, Wording, Nitpicking)** When I read that one of the contributions was a “visualisation tool”, it might’ve accidentally biased me towards the wrong idea of the referenced contribution. I understand how visualising similarity matrices between CAVs can be seen as a \"visualisation tool\", however I would argue against using that term for it as it is potentially misleading and enlarges the extent of this particular contribution (visualising the cosine similarity matrices between vectors is a very common approach for analysis in ML, data science, DL, etc rather than an actual visualisation tool, in the sense of what “tool” is usually used for within these sorts of papers, e.g., a GUI).\n- **(Minor, Feel free to ignore)** Given how a significant contribution of this work is to provide guidelines on how to properly use T-CAV, it would be extremely helpful (and potentially reach-increasing) to make sure the title of this paper reflects this.\n- **(Minor, Nitpicking)** I would recommend ordering the appendices in the same order that they are referenced in the text and numbering their sections from a “conventional” starting point (e.g., §A.1 for the first appendix section rather than §9).\n\n### Section 1 (Introduction)\n\n- **(Major, Referencing)** The introduction makes several strong claims, all without any citations. At the very least, \"saliency\" should be defined for people not familiar with that term, and proper citations should be made to previous works arguing for the need for explainability/interpretability.\n- **(Major, Clarity)** The enumeration of properties in this section jumps directly to stating conclusions (e.g., \"They cannot be consistent across layers\") rather than stating first the research questions this paper is exploring. This leads to an enumeration that does not naturally follow the preamble (\"... we focus on understanding three properties of concept vectors:\") and that rather quickly jumps to conclusions before a problem has been presented.\n- (**Very Minor, Potential Typo)** In the caption of Figure 1, should “The top panel illustrates each …” be “The top *panels illustrate* each …” instead? This would also be consistent with the sentence that follows this one.\n- **(Very Minor, Nitpicking, Word Choice)** Should “… a substantial impact on the meaning of a CAV” be “… a substantial impact on the *interpretation* of a CAV” instead?\n\n### Section 2 (Background)\n\n- **(Major, Notation)** The use of $\\mathbb{X}_k$ and $\\mathbb{X}_c$ to mean very different sets (one partitions a set based on task labels, the other partitions a set based on a concept) opens up the room for confusing notation (consider $\\mathbb{X}_1$: is this the set representing all samples with label $y=1$ or with concept $c_1$?). I would suggest using different set names to represent these two different subsets.\n- **(Minor, Potential Typo)** “is” is missing before “defined as” before defining $S_{c, k, l}$.\n- **(Minor, Potential Typo)** “… output for class $k$ to the activation” should probably be “… output for class $k$ *with respect* to the activation”.\n- **(Very Minor, Nitpicking)** “NN” is used before it is defined as a \"Neural Network\". \n\n### Section 3 (CAV Hypotheses)\n\n- **(Critical, Clarification, Concern)** In definition 1 there is an implicit choice to make the definition a function of a **unit shift** in the direction of the CAV (i.e., we want $ f(\\mathbf{a}\\_{l\\_1} + \\mathbf{v}\\_{c, l\\_1}) = \\mathbf{a}\\_{l\\_2} + \\mathbf{v}\\_{c, l\\_2} $ rather than the more general formulation $f(\\mathbf{a}\\_{l\\_1} + \\lambda \\mathbf{v}\\_{c, l\\_1}) = \\mathbf{a}\\_{l\\_2} + \\lambda \\mathbf{v}_{c, l\\_2}, \\forall \\lambda \\in \\mathbb{R}$). Could you please clarify why this was done this way? And what is the argument to be made that a unit (or for that matter a $\\lambda$-scaled) shift in the direction of the CAV in layer $l\\_1$ should correspond to a shift of the same proportion in the CAV of layer 2? Intuitively, I can definitely see situations where a $\\lambda\\_1$-scaled shift in the direction of $\\mathbf{v}\\_{c, l\\_1}$ in layer $l_1$ (i.e., $\\mathbf{a}\\_{l\\_1} + \\lambda\\_1 \\mathbf{v}\\_{c, l\\_1}$) may correspond to a $\\lambda\\_2$-scaled shift in the direction of $\\mathbf{v}\\_{c, l\\_2}$ in layer $l\\_2$ (i.e., $\\mathbf{a}\\_{l\\_2} + \\lambda\\_2 \\mathbf{v}\\_{c, l\\_2}$), where $\\lambda\\_2$ is a function of $\\lambda\\_1$. That seems to me to be a fairly reasonable definition of consistency. Can you please clarify why the definition you use, where the shift is expected and assumed always to be a unit shift, is the appropriate definition for consistency?\n- **(Critical, Clarification, Concern)** After Definition 2, it is stated that “If concepts are entangled, it is not possible to separate the model’s sensitivity to one concept from its sensitivity to related concepts.” I can see the general intuition behind this; however, I am not entirely sure this is undesirable or unavoidable in real-world setups. Concepts are highly entangled as they are highly correlated in real-world situations. There are no assumptions or need for assumptions, stating that concept probes for CAVs should be trained on datasets containing completely independent concepts (if anything, part of their usefulness is that they allow this exact behaviour). Considering this, why should we ever expect that if CAVs satisfy definition 2, then it must not be true that “A CAV represents only the concept corresponding to the concept label of its probe dataset” as stated by NH2? Couldn’t it be that two CAVs are highly entangled just because the underlying concepts themselves are highly correlated, yet they still faithfully represent the concept they are intended to represent? In other words, **how can one claim that satisfying Definition 2 immediately leads to a violation of NH2?**\n- **(Critical, Clarification, Concern)** Definition 3 seems a bit too hand-wavy and ill-defined. This severely complicates understanding its usefulness and its implications. I would ask the authors to please clarify what the “location” of a concept $\\mu_i$ really means. How is this defined for concepts that are a function of multiple pixels (e.g., the example of $\\texttt{stripes}$ itself can be seen not at a specific pixel location but at a set of pixels whose locations can vary)? How is this defined for high-level concepts that may not be easily mapped to specific features (e.g., a bird’s $\\texttt{size}$ is a useful concept but one that does not correspond to a specific subset of pixels when working with bird images)? And how is this related to similar notions found in the literature (e.g., [9])? I hope these questions exemplify my concerns with the usefulness of this definition.\n- **(Very Minor, Potential Typo)** “… an infinitesimal change of the activations …” should probably be “… infinitesimal change *on* the activations …”.\n\n### Section 4 (Elements)\n\n- **(Major, Literature Relevance)** Would it possible to describe in this section what the proposed *Elements* dataset, which is claimed as a contribution in the introduction and abstract, offers that existing similar datasets such as dSprites [1] and 3D Shapes [2], also used in the concept XAI literature (and the disentanglement literature), do not already offer? I think this may help build a case for how one should use the newly proposed dataset over already existing ones.\n- **(Minor, Clarification)** Related to my question above I see that a similar comparison to the one requested above is made with CLEVR in Section 5. I can somewhat understand the efficiency argument made there. Yet, wouldn’t this be a one-time cost that is negligible overall, as once the dataset is generated, it doesn’t need to be generated again?\n\n### Section 5 (Related Work)\n\n- **(Minor, Missing Relevant Works)** **[Concept Correlation and Entanglement]** There are a couple of relevant works that could be helpful within this related works discussion. These works include those discussing the effect of concept correlations on concept representations [3, 4] and whether inter-concept relationships (which may imply correlations too) are captured by concept-based models (including T-CAV) [5]. I do believe there is a clear distinction between this paper and these other previous works, as this work focuses on T-CAV specifically, while the examples I included here (and others, e.g. [6, 7]) focus on studying the effect of concept correlation/entanglement on inherently interpretable concept-based architectures (e.g., CBMs).\n- **(Minor, Missing Relevant Works)** **[Spatial Dependence]** There are some highly relevant works that may be worth discussing here as they study the spatial locality of concepts. These include one of the first works casting doubt on the validity of a CBM’s explanations [8] and more recent works explicitly studying whether the explicit locality of concepts is captured by inherently interpretable concept-based models [9]. Again, this is a minor change request as these works relate more to inherently interpretable architectures than post-hoc methods such as T-CAV. As such, I do not believe they hamper the novelty of this work.\n\n### Section 6.1 (Consistent CAVs)\n\n- **(Very Critical, Potential Soundness Mistake)** I am not very convinced of the main theoretical result of this section. If I understood it correctly (based on §6.1 and Appendix 9), the crux of the impossibility for consistent CAVs relies on the claim that if $f: \\mathbb{R}^{m\\_{l\\_1}} \\rightarrow \\mathbb{R}^{m_{l\\_2}}$ is not a linear function, then there does not exist $\\mathbf{u} \\in \\mathbb{R}^{m\\_{l\\_1}}$ and $\\mathbf{v} \\in \\mathbb{R}^{m\\_{l\\_2}}$ such that $f(\\mathbf{x} + \\mathbf{u}) = f(\\mathbf{x}) + \\mathbf{v}$ for all $\\mathbf{x}$ in the domain of $f$. If this is indeed the main claim needed, then **I strongly believe this is not true**. As a pedantic yet trivial example, consider the case where $m\\_{l\\_1} = m\\_{l\\_2}$ and $\\mathbf{v} = \\mathbf{u} = \\mathbf{0}$. In that instance, it is always true that $f(\\mathbf{x} + \\mathbf{u}) = f(\\mathbf{x}) = f(\\mathbf{x}) + \\mathbf{v}$ for all $\\mathbf{x}$.\n\nAs a more complex counterexample, we can consider periodic functions or piece-wise functions (as all you care about is that the difference between $f(x + u)$ and $f(x)$ is constant for all $x$). For example, if $f(x) = \\text{sin}(x) + cx$ and $u = 2\\pi$, $v = 2\\pi c$ (for some $c \\in \\mathbb{R}$), then we have that $f(x + u) = f(x + 2\\pi) = \\text{sin}(x + 2\\pi) + c(x + 2\\pi) = (\\text{sin}(x) + cx) + c 2\\pi = f(x) + c 2\\pi = f(x) + v$ for all $x$. This shows that such $u$ and $v$ can indeed exist, **contrary to the claim made in this work**. I grant that this is a trickier example. However, my whole point is that claiming that a function does not exist just because it may be a function of another variable is not necessarily a valid or sound proof, and it is a potentially flawed argument (as these counterexamples show). Before moving onwards, especially since this is a key contribution of this work, I believe this result should be properly fixed, adjusted, and formally proved so that it reflects a true statement (this may likely involve making some stronger assumptions on the form $f$, $u$, and $v$ may have).\n- **(Very Critical, Potential Soundness Mistake)** Similarly, I have some concerns/hesitations about the ReLU and sigmoidal proofs (Appendix 9.1 and 9.2). First, when working with NN, the nonlinear function (e.g., sigmoid or ReLU) is applied to (usually) a linear transformation of the current layer’s input. Therefore, it will rarely be the case that the output of layer $l_2$ is just the relu or sigmoidal activation of the output of layer $l_1$ (this is the underlying assumption regarding those proofs). If the case that you are trying to build is that even in this simple instance, consistency is not possibly preserved (I have some doubts on this too tho), then you need to show that this implies that consistency is not possible anywhere downstream of layer $l_2$ (these statements by themselves say nothing as to whether consistency may be “recovered” down the line). More importantly, however, as in the proof in section 9, the key conclusion for both the ReLU and sigmoid proofs are derived from very hand-wavy arguments that may not necessarily be true (of a similar nature to the one I provided a counterexample above). As mentioned above, I would like to ask the authors to please make sure to properly formalize these statements and provide a sound proof once they have been formalized. All these proofs seem to depend on the false claim that for “a single $v$ to exist which is consistent for all $a_{l_1}$ it cannot depend on $a_{l_1}$”, something my counterexample above shows is not necessarily true. Because of this, I believe they require quite a lot of work before these theoretical results can be considered sound.\n- **(Major, Clarity)** I would appreciate it if any theoretical results (e.g., proofs) could be formally written and expressed by enumerating their assumptions and conclusions in the form of a theorem. Otherwise, capturing the essence of each proof is hard as it requires going over many separate pieces where assumptions and conclusions are stated throughout.\n- **(Major, Nitpicking)** Why is the conclusion “the projected CAVs have a nonzero error, indicating that vector addition is not preserved” unexpected when one considers that DNNs are by design non-linear?\n\n### Section 6.2 (Entanglement)\n\n- **(Very Minor, Potential Typo)** “… we investigate the effect of entangled concept vectors on TCAV score” should probably be “… we investigate the effect of entangled concept vectors on TCAV *scores*”.\n\n### Section 6.3 (Spatial Dependence)\n\n- **(Very Minor, Potential Typo)** “2. That each depth-wise slice…” should probably be “2. *Each* depth-wise slice…”.\n\n### Section 7.2 (Results)\n\n- **(Critical, Clarification, Potential Mistake)** At the beginning of Section 7.2, it is claimed that “as shown in fig. 7, initial results with random CAVs gave poor performance for the medical concepts, so we used the ‘absent’ category for each class as the negative set. This gave better performances …”. However, if one looks at Figure 7 and its caption, the top left pane (which the caption says are the CAVs trained with a random negative set, shows significantly more accurate CAVs than those in the top right pane (which the caption says are CAVs trained with “absent” samples in the negative set). These two appear to contradict each other directly. Am I misunderstanding something here? Or is this a potential labelling or plotting mistake?\n- **(Very Minor, Potential Typo)** In the “Entanglement” paragraph, the closing quotation for “absent” is backwards.\n- **(Major, Clarification)** In the “Entanglement” section, it is claimed that “We hypothesise that this is because the two concepts share the same negative set.” Is this something you could easily verify by changing the set and observing whether the similarity is still there? I am a bit puzzled by this result, as it appears to contradict things observed/discussed earlier in this paper when studying NH2.\n- **(Minor, Figure)** Some of the error bars on the figures are cut (e.g., Figures 5 and 8). I recommend extending the figure limits so that these bars are not cut.\n- **(Major, Clarification w.r.t. competing work)** This is roughly mentioned at a high level in the conclusion, but do you have a sense as to how the properties discussed in this paper may manifest or may be mitigated when getting rid of the linear-separator assumption of CAVs as Concept Activation Regions (CARs) [10] do?\n\n## References\n\n- [1] Matthey, Loic, et al. \"dSprites: Disentanglement testing sprites dataset.\" May 2017.\n- [2] Kim et al. \"Disentangling by factorising.\" ICML (2018).\n- [3] Heidemann et al. \"Concept correlation and its effects on concept-based models.\" WACV (2023).\n- [4] Espinosa Zarlenga et al. \"Towards robust metrics for concept representation evaluation.\" AAAI (2023).\n- [5] Raman et al. \"Understanding Inter-Concept Relationships in Concept-Based Models.\"  ICML (2024).\n- [6] Huang et al. \"On the Concept Trustworthiness in Concept Bottleneck Models.\" AAAI (2024)*.*\n- [7] Furby et al. \"Towards a Deeper Understanding of Concept Bottleneck Models Through End-to-End Explanation.\" arXiv preprint arXiv:2302.03578 (2023).\n- [8] Margeloiu et al. \"Do concept bottleneck models learn as intended?.\" ICLR Workshop on Responsible AI (2021).\n- [9] Raman, Naveen, et al. \"Do Concept Bottleneck Models Obey Locality?.\" NeurIPS Workshop “XAI in Action: Past, Present, and Future Applications” *(*2023)\n- [10] Crabbé et al. \"Concept activation regions: A generalized framework for concept-based explanations.\" NeurIPS (2022).", "broader_impact_concerns": "I do not believe there are any broader ethical implications from this work that would merit a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission investigates concept activation vectors (CAVs), which are used for neural network interpretability. Reviewers praised the clear and novel framing and the soundness of the investigations. The work even goes beyond negative results around CAVs to recommend mitigation strategies for undesirable properties of CAVs. One reviewer expressed remaining concerns surrounding the demonstration of the spatial dependence property, but did not elaborate. I assess that the authors have sufficiently responded to concerns raised in the initial review around this property. I recommend acceptance to TMLR.
Decision: Accept (poster)