Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence
Authors: Frederik Pahde
Maximilian Dreyer
Moritz Weckbecker
Leander Weber
Christopher J. Anders
Thomas Wiegand
Wojciech Samek
Sebastian Lapuschkin
Venue: ICLR 2025 Poster (2025)
Keywords: Explainable AI
Concept-based Explanations
Concept Activation Vectors
Primary area: interpretability and explainable AI
TL;DR: We introduce pattern-based CAVs, an alternative to widely used filter (e.g., SVM) CAVs, more robust to distractor patterns and thereby providing more accurate concept directions.

Abstract: With a growing interest in understanding neural network prediction strategies, Concept Activation Vectors (CAVs) have emerged as a popular tool for modeling human-understandable concepts in the latent space.
Commonly, CAVs are computed by leveraging linear classifiers optimizing the *separability* of latent representations of samples with and without a given concept. However, in this paper we show that such a separability-oriented computation leads to solutions, which may diverge from the actual goal of precisely modeling the concept direction.
This discrepancy can be attributed to the significant influence of distractor directions, i.e., signals unrelated to the concept, which are picked up by filters (i.e., weights) of linear models to optimize class-separability.
To address this, we introduce *pattern-based CAVs*, solely focussing on concept signals, thereby providing more accurate concept directions.
We evaluate various CAV methods in terms of their alignment with the true concept direction and their impact on CAV applications, including concept sensitivity testing and model correction for shortcut behavior caused by data artifacts. 
We demonstrate the benefits of pattern-based CAVs using the Pediatric Bone Age, ISIC2019, and FunnyBirds datasets with VGG, ResNet, ReXNet, EfficientNet, and Vision Transformer as model architectures.

Published as a conference paper at ICLR 2025
NAVIGATING NEURAL SPACE: REVISITING CONCEPT ACTIVATION VECTORS TO OVERCOME DIRECTIONAL DIVERGENCE
Frederik Pahde
Maximilian Dreyer
Moritz Weckbecker
Leander Weber
Christopher 2
Thomas Wiegand 12,33 Wojeiech Samek1.2.5,
Sebastian Lapuschkin
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute Department of Electrical Engineering and Computer Science, Technische Universität Berlin BIIOLDD - Berlin Institute for the Foundations of Learning and Data
'corresponding authors:
fwojciecch samek, sebastian, i Ahhhi @hhi. fraunhofer.de
ABSTRACT
With growing interest in understanding neural network prediction strategies, Concept Activation Vectors (CAVs) have emerged as a popular tool for model- ing human-understandable concepts in the latent space. Commonly, CAVs are computed by leveraging linear classifiers optimizing the separability of latent rep- resentations of samples with and without given concept. However, in this paper we show that such separability-orione may diverge from the actual goal of precisely modeling the concept direction. This discrepancy can be attributed to the significant influence of distractor directions, i.e., signals unrelated to the concept, which are picked up by filters (i.e., weights) of linear models to optimize saassaeparr.ility. To address this, we introduce pattern-based CAVs, solely focussing on concept signals, thereby providing more accurate concept directions. We evaluate various CAV methods in terms of their alignment with the true concept direction and their impact on CAV applications, including concept sensitivity testing and model correction for shortcut behavior caused by data artifacts. We demonstrate the benefits of pattern-based CAVs using the Pediatric Bone Age, ISIC2019, and FunnyBirds datasets with VGG, ResNet, ReXNet, EfficientNet, and Vision Transformer as model architectures
computation leads to solutions, which
INTRODUCTION
In recent years, eXplainable Artificial Intelligence (XAI) has gained increased interest, as Deep Neural Networks (DNNs) are ubiquitous in high-stake decision processes, such as medicine (Brinker
et al. 2019), finance (Rouf et al., (02T), and criminal justice (Završnik) 2021 Travaini et al. (022).. with black-box predictions being unacceptable. Whereas local explainability methods compute the relevance of input features for individual predictions, global XAI approaches aim at identifying global prediction strategies employed by the model, often to represented as human-unde understtandtt concepts. Backed by recent research, suggesting that DNNs encode concepts superpositions in latent space (Alain & Bengio 2017 Elhage et al.) 2022; Nanda al. 2023 Wang et al.) (023).. Concept Activation Vectors (CAVs), originally introduced for concept sensitivity testing (Kim et al.) 2018), model concepts DNNs by finding directions pointing from samples without the concept to samples with the concept. Commonly, the direction is estimated by taking the weight vector of a linear classifier (e.g., a linear Support Vector Machine (SVM)), representing the normal to the decision hyperplane separating the two sample sets. However, while linear classifiers optimize the separability of two classes, they might fail at precisely identifying the signal direction encoding the
"Code available at http:////////tthho/////rrrrraaaa

---

Published as a conference paper at ICLR 2025
Estimating Concept Direction via CAV
clean samples
band aid samples
Clean
Different Scaling
Rotated Distractor
distractre
<lstmetter
olber dreclios
filter
pottern Glue
potterm
pattern (ours)
pattern
fitter
fitce
(egg SVM, Insso,
Logissic, ridge)
direction
direction
4000
dinecton
Figure 1: CAVs obtained from filters, i.e. weight vectors from linear classifiers, are optimized for class separability, but fail at precisely estimating concept signal directions. Left: Different CAV computation strategies
employed to estimate the "band-aid" concept, confounding ar- tifact the ISIC2019 dataset. Right: Weaknesses of filter-based CAVs are apparent for simple transformations in a 2D toy experiment, where we scale concept features (x-axis) differently than other (e.g., distracting) features or rotate distracting directions. Only patternbbased CAVs pre- cisely estimate the concept signal direction, while filter-based CAVs diverge to optimize class separability. Animated visualizations for these and additional 2D experiments can be found here: https: /taes://gitie /githhb. con/ wwteerrikpahhee wwtpaaadeppatterr--Oee wwtpe//eaaattteeeeaaeeaeeeeeeeee maii.aiinnn. animatiimiinnn
concept. This can be attributed to the significant influence of distractor (i.e., non-signal) directions contained in the data, which are picked up by filters (i.e. weights) of linear models to optimize class-separability (Haufe et al. 2014). This decomposition filters into signal and distractor patterns has also been addressed in the context of local explainability methods (Kindermans et al. 2018). We follow their approach and introduce pattern-based -based CAVs for global explainability, disregarding distractors and thereby precisely estimating the concept signal direction (see Fig.
Despite directional divergence from the true concept signal, CAVs have been employed for plethora of tasks in recent years, such as concept sensitivity testing (Kim et al.) 2018), model correction for shortcut removal (Anders et al. [2022] Dreyer et al., (2024), knowledge discovery by investigation of internal model states (MeGrath al. (022)), and training of post-hoc concept bottleneck mod- els (Yuksckgonul et al. [2023). Many of these applications can be improved by more precise concept directions, as provided by patternCAAVs instead of optimized caass-seaarability, as provided by filters. To demonstrate the superiority of pattern-CAV using the Pediatric Bone Age, ISIC2019, and FunnyBirds datasets with VGG, ResNet, Reeeet, EfficientNet, and Vision Transformer architectures. Our main contributions include the following:
we run controlled and non-controlled experiments
1. We introduce Nattern-CAVs, more precisely estimating the concept signal direction and being less influenced by distractors.
2. We measure the alignment of CAVs with the true concept direction in controlled settings, confirming that pattern-CAVs align with the true concept direction, while the widely used filter-AAV diverge.
3. We measure the impact of directional shifts in popular CAV applications, including Testing with CAV (TCAV) and model correction with Class Artifact Compensation (ClArC) in controlled and real-world experiments, demonstrating benefits of pattern-CAV in both cases.
2 RELATED WORK
A variety of approaches has emerged to identify human coinman-uunderttnddbblle concepts in DNNs. Some works consider single neurons as concepts (Olah et al. 2017 Achtibat et al. 2023), while others focus on identifying interesting subspaces (Vielhaben et al. 2023 or linear directions Nanda et al., 2023). We follow the latter approach and encode concepts as linear combinations of neurons, also known as superposition (Elhage et al. 2022). These directions can be identified through unsupervised activation matrix factorization (Fel et al. (202)) or by the supervised training of CAVs, i.e., vectors pointing from samples without to samples with the concept. In the absence of concept labels, automated concept discovery approaches can further streamline this process (Ghorbani et al., 2019: Zhang et al., 2021).
Various methods leverage CAVs as latent concept representation. For instance, TCAV measures
2

---

Published as a conference paper at ICLR 2025
model's sensitivity towards specific concepts. CIArC aims to unlearn model shortcuts, Le., prediction strategies based on unintended correlations between target labels and data artifacts, represented by CAVs. Post-hoc concept bottleneck models project latent representations into a space spanned by CAVs to obtain an interpretable latent representation. Beyond these applications. CAVs have been employed understand the strategies learned by AlphaZero in playing chess (McGrath et al.) 2022) and to identify meaningful directions for manipulation (e.g. no-smile + smile) in diffusion autoencoders (Preechakul et al., 2022). Related works aim enhance CAV robustness by alleviating the linear separability assumption (Chen et al. 2020 Pfau et al. 202T), for example by representing concepts as regions (Crabbé van der Schaar 2022). In contrast, our approach adheres the linear
separability assumption but improves the precision of the modeled direction.
3 ESTIMATING SIGNAL OF CONCEPT DIRECTION
We view DNN function / X + V mapping input samples X' to target labels y y. With- out loss of generality, we assume that at any layer with m neurons, f can be split into a feature ex- tractor a X'' RT computing latent activations at layer I, and model head f. f m + V. mapping latent activations to target labels. We further assume binary concept labels & (-1, CAVs are intended to point from latent activations of samples without concept A fa(x)) E Rm f -1} activations of samples with concept A (a(x;) E Rm f, +13. The optimal choice of layer I depends on the type of concept, as simple concepts (e.g., color and edges) are learned earlier layers, while more abstract concepts (e.g., band-aid) are learned closer to the model output (Olah et al. 2017 Radford et al., 2017: Bau et al. 2020).
3.1 FILTER-BASED CAV COMPUTATION
Traditionally, a CAV h is identified as the weight vector w E Ih" from a linear classifier, describing
hyperplane separating latent activations of samples with the concept A from activations of samples without the concept A= Commonly (Kim et al., 2018 Yuksekgonul et al. 2023), linear SVMs
are used, minimizing the hinge loss with L.2 regularization (Cortes Vapnik 1995). Other options include Lasso (Tibshirani 1996). Logistic, or Ridge (Hoerl & Kennard 1970) regression.
Concretely, the classification task usually described as a linear regression problem. With concept labels as dependent variable and latent activations a(x) E IR" regressors, assume linear model funear (x) = a(x) + with weight vector (or filter) h E I and bias b E R. Using ridge regression example, the optimization task find filter-CAY h is then given by
h lilt
min h.
Ah- 1bll2 + Al|b|12,
where A _Rax is summarizing latent activations for all n samples in X' in matrix form, t E R is the vector with concept label t; as its hth element and 1 E R" is vector of 1s. The optimization objectives differ by the type of linear model (see Appendix B.I.
However, research from the neuroimaging realm suggests that filters from linear classifiers not only model the signal separating the two classes but also capture a distractor component (Haufe al.) 2014). This component can arise from noise, but also from unrelated features in the data, which are not directly related to the signal. In the context of CAVs, s, any information unrelated to the concept considered distractor. The filters are optimized to weigh all features to achieve optimal separability w.r.t. I. However, this optimization does not disentangle concept signals from distractor signals. As result, distractor pattern present in the training data influence the direction of -Iter.CAVs.
3.2 PATTERN-BASED CAV
We introduce a baated CAV, which is based on the assumption that we can model latent activations given the concept label via the linear function (i)eea()) th + b for vector h R and a bias vector b E RT.. The difference activations with and without the concept, h, can be obtained by optimizing the following objective (Haufe al. 2014):
3

---

Published as a conference paper at ICLR 2025
min |A th -1b I|F; b.b
(2)
where || || denotes the Frobenius norm. Contrary to Eq. ④ which finds an h maximizing the class-separability. Eq. ( finds pattern best explaining A w.r.t. concept label t. This is solved as linear regression task for each feature dimension, leading to
h
Σ (a(x) A) - X,EX
of 0f1
with mean latent activation A. mean concept label f and sample concept label variance o7. which is equal to the sample covariance between the latent activations a(x) and the concept labels t divided by the sample concept label variance. In contrast to filter-CAVs, the resulting pattern- C-ee is invariant
under feature scaling and more robust to noise, as further outlined in Appendix [8.2 Given binary concept labels, Eq. (3) simplifies to the difference of cluster means, as shown in Appendix [B.3
Note, that the computation of pattern regression manner as described in Eq. (3) allows to further incorporate prior knowledge, e.g., sparseness constraints (Haute et al. 2014).
3.3 2D TOY EXPERIMENTS
We demonstrate the difference between filter- and pattern-CAVs CAVs in a toy experiment inspired by Kindermans al. (2018). We simulate n activations A, E R2 split equally between the concept labels I, E [+1,- in the following manner: Each activation A, S; D; is decomposed into a deterministic signal part 8; and random (non-signal-, noise-) distractor part D;. The signal part 1t, (1 0) is aligned with the x-axis, the distractor part D, is modeled by identically distributed independent wwo-dimensional Gaussians of mean 0, variance o² each dimension and correlation between dimensions. The distractor contains true noise and signal related other concepts. Both are "noise" for the concept signal estimation. We experiment with two distractor patterns in Figure (right):
Scaling: We multiply values on the x-axis with scaling factor À = 103, such that the signal s; scaled proportionally and therefore signal features are on larger scale than distractor features. The filter-CAV diverges from the true concept direction (1 0) as the entry of the weight vector in direction of the signal scales pnti-propottionally to the scaling factor in logistic regression (see Appendix B.6 for the derivation). Feature normalization is commonly disregarded in CAV training.
We add another distractor term Drot r,&; with e, i.dd (O 1), which is oriented parallel to the vector F, = (sin cos cosT) This rotates the distractor direction based on т. Only the pattern-CAV hd obtained via Eq. (3) precisely identifies the concept direction, while the filter-CAV prefers diverging directions which increase the angle to rr, thus minimizing the variance of datapoints in direction of the weight vector (see Appendix B.7 for the mathematical derivation).
Noise Rotation:
Moreover, filter-based CAVs face further challenges, including sensitivity to regularization strength and random seeds, particularly in low data scenarios, demonstrated in Appendix B.4
EXPERIMENTS
After describing experimental setup (Section 4. concept directions 4.22). as well as the impact of CAVs on applications, including concept sensitivity testing with TCAV (Section 4.3.1) and CAV-based model correction (Section 4.3.2).
measure how precise CAVs represent true
4.1 EXPERIMENT DETAILS
We conduct experiments with three controlled and one real-word datasets. For the former, we insert artificial concepts into ISIC2019 (Codella et al. 2018 Tschandl et al. 2018 Combalia et al.) 2019), dermatologic dataset for skin cancer detection with images of benign and malignant lesions, and a Pediatric Bone Age dataset (Halabi et al., 2019), with the task to predict bone age based on hand radiographs. Specifically, we insert timestamps as text layover into 1 1% of samples of class
4

---

Published as a conference paper at ICLR 2025
(UBR)
(127)
ISIC3019
tronetin sog
PatternCCA
1101 12//51
214 (1.7%)
Figure 2:
Example for timestamp artifact inserted into ISIC2019 samples (left) and RelMax visualization for neurons (right) corresponding to the largest absolute values in filter- and pattern- CAVs, along with the Conv filter ID and the fraction of all (absolute) CAV values. While the filter-CAV picks up noisy neurons, the pattern-CAY uses neurons related to the relevant concept.
"Melanoma" of ISIC2019, encouraging the model to learn the timestamps shortcut. For the Bone Age dataset, we insert an unlocalizable concept by increasing the brightness (i.e., increase pixel values) of 20% of samples of only one class. We implemented bone age prediction task, with target ages binned into five equal-sized groups Lastly, we use FunnyBirds (Hesse et al. 2023). a synthetic dataset with part-level annotations, synthesize dataset with 10 classes of birds, where each category is defined by exactly one part (e.g.. wings, beak). Other parts are chosen randomly per sample, foreing the model use the class-defining part (i.e., concept) as the only valid feature. Detailed class definitions and examples for synthesized images are provided in Appendix C.T Further, we consider real data artifacts present in ISIC2019, including "band-aid", "ruler", and "skin marker". We finetune VGG16 (Simonyan & Zisserman 2015). ResNet1 8/50 (He et al. 2016) Wightman et al. 2021), ResNeXt50 (Xie et al. 2. ReXNet100 (Han et al. 2021). EfficientNet- BO (Tan & Le| 2019), EfffcientNeiV2-((Tan & Le| [022] 1) and Vision Transformer (Dosovitskiy et al.,
classification
2020 models pre- preitrained on ImageNet (Deng et al. 2009: Ridnik et al. 2021) for all datasets with
tramiing details given in Appendix C.2
4.2 PRECISENESS OF CONCEPT REPRESENTATION
The primary goal of Pattern-CAVs is the optimization of the precision of concept representations. Therefore, to assess how precisely CAVs represent true concept directions both qualitatively and quantitatively, we (1) visualize the key neurons associated with the CAV, and (2) quantify the alignment between CAVs and the ground truth direction.
How clean are CAVs qualitatively? We investigate the focus of CAV h fitted on layer by em- ploying feature visualization to neurons corresponding to the largest absolute, hence most impactful, values in h. Specifically, we use RelMax (Achtibat et al. (0023) to retrieve input samples maximizing the relevance, computed by feature attribution methods, for the neurons with the largest absolute values in h. We further use receptive field information to zoom into the most relevant region and mask out irrelevant information. Results for filter- and pattern- based CAVs for the timestamp artifact in ISIC2019 are shown in Fig. 2 Whereas the pattem-CAV leverages neurons focusing the desired concept, i.e., the timestamp, the filter-CAV is distracted by other features. In addition, we show the percentage for the value associated with the neuron of the entire CAV and higher values, as observed for Pattern-CAV. indicate a less uniform distribution over neurons and a larger focus on the corresponding top neurons. Additional neuron visualizations for different filter- and pattern-CAVs are shown in Appendix D.2
CAV Alignment with True Concept Direction Using our controlled datasets, we generate pairs of samples with and without the concept (x² and x; and compute the sample-wise true latent concept direction h = =(xx²) a(x; This definition aligns with TCAV's intuition, i.e. adding activations along the concept direction corresponds adding the concept in input space. To quantify the alignment between CAV h and h² per sample, use cosine similarity as the similarity function
II FunnyBirds, we remove concepts by randomizing the class-defining part, while keeping others identical.
5

---

Published as a conference paper at ICLR 2025
ISIC2019
Age
Coite Stetlants
0.5
0.8
11.5
Coelue Sinflcctt
0.6
0.5
0.4
logistic
Vklge
kvia
patlien
Aogistle
ridge
CA Type
Gio 19900)
pullin
centered
centered
mane-scclled
Dsteet
1802009
Pance Bank s...n Beer Age
Figure 3: Left: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability (bottom), using filter- (SVM) and pattern-CAV for all Conv layers of VGG16 trained on ISIC2019, Bone Age, and FunnyBirds. While expectedly filter-AVS have superior class- separability, pattern-CAVs have a better alignment with the true concept direction. Right: Cosine similarity between true concept direction h and CAVs with different feature pre- pre-processing methods fitted on the last Conv layer of VGGI6 trained on ISIC2019 and Bone Age. Compared to filter-CAVs, pattern-CAV has higher alignment with h and is invariant to feature rre-processing.
sim h.)). We calculate the overall aligmment ā by averaging the alignment scores for all samples:
= 1/1x1 Σ sim(h, h,")
Moreover, we measure the separability samples w.r.t concept label t by computing the AUC of h' ax). Fig. (left) presents the results, including standard errors, for both CAV alignment (top) and separability (bottom) across all 13 convolutional (Conv) layers in the VGG16 models for all three controlled datasets. We estimate standard errors of AUC scores using the Wilcoxon- Malloxxo-MMan---e Whitey statistic equivalence (Cortes & Mohri 2004). The alignment with h significantly higher for pattern-based CAVs across all layers for datasets, confirming a more precise estimation of the true concept direction. As expected, filter-based CAVs exhibit higher concept separability. Additional experiments in Appendix [D3] demonstrate the superior robustness pattern-CAV towards the reduction of concept sizes and concept labeling errors. Moreover, the experiments show that pattern-CAVs outperform concept directions found in unsupervised manner in terms of precision. We further investigate the relation between the distribution of noise in the activations and the divergence of estimated concept directions from the true concept direction in Appendix D.6
Moreover, we study the sensitivity of CAVs different pre-processing methods for latent activations, specifically centering, max-scaling, and their combination. Results for CAVs fitted on the last Conv layer of VGG16 for ISIC2019 and Bone Age are shown in Fig. (right). While filter-CAVs have better alignment with true concept directions when features are re- scaled, which is often overlooked practice, pattern-CAV consistently outperform filter-CAVs regardless activation pre-processing. This can be attributed to the fact that the covariance (see Eq. (3) is translation invariant, while scale invariance is proven in Appendix B.2 Another disadvantage of Elter-CAVs is their dependence on hyperparameters, e.g., regularization strength. In contrast, pattern- pattern-CAss do not require parameter tuning and are therefore more computationally efficient.
4.3 IMPACT OF DIRECTIONAL SHIFTS ON CAV APPLICATIONS
We measure the impact different CAVs on applications requiring precise concept directions, namely concept sensitivity testing with TCAV and concept-based model correction with ClArC.
4.3.1
TESTING WITH CAV
TCAV (Kim al. 2018) is technique to assess the sensitivity of DNNs' prediction w.r.t. given concept represented by CAV h. Specifically, given the directional derivative V va(())))), measure

---

Published as a conference paper at ICLR 2025
2D Toy Example
Rotating Distractor
Coucept Sensitivity over Distractor Rotation
class A (with concept)
filter CAV
TCAV
pattern CAV
filter (SVM)
pattern
=005
Dimoniion
45" distrnctor
0
0.5
clesa
wrogins
without comnept)
of
=0.5
x/2 Distractor Rotation
Class B
Dimnetsion
Figure 4: Left: 2D TCAV experiment with distractor rotated by r = 45 with samples from class A (purple with concept, blue without concept) and class B (green). The model classifies between classes A and B. CAVs are fitted on samples with and without concept from class A. The pattern-CAV aligns with the concept direction, while the filter-CAV diverges optimize class-separability. Right: TCAV sens for model f plotted over distractor rotation т. Positive and negative values indicate positive and negative influence of the concept direction and indicates insensitivity (TCAV 0.5).
the model's sensitivity towards the concept for sample
TCAV seeAseen(25 (x) = V Vaf(a(x)) h.
(5)
The TCAV score measures the fraction of the sample subset containing the concept x+ (x; E X +1] where the model shows positive sensitivity towards changes along the esti- mated concept direction h:
TCAV [x E x+ TCAV sens (x) > 0H
(6)
|
Hence, truthfully measure the model's sensitivity, a precise estimated concept direction h is required. A TCAV score = 0.5 indicates minimal influence of the concept on the model's decisions, while scores above and below 0.5 indicate positive and negative impacts. We show the effects of directional divergence by conducting experiments in 2D and with our controlled FunnyBirds dataset.
TCAV in 2D Toy Experiment Consider samples x E R² with class labels y E {+1, -1]. referred to as class A and B, perfectly separable by a linear model f with weights w (1 1) and bias b=0 O. We introduce data artifact in class A where some samples contain concept with concept direction c = (1 1)" perpendicular to w. As is insensitive to concept c, we expect a TCAV score of 0.5. Using the notation from Section 3.3 we rotate the distractor r, with E [0, π relative to CAVs are fitted to separate samples with and without c from class A, using concept labels t instead of class labels y. Results are shown in Fig. 4 For - //4- 45 (left), ha aligns with the concept direction c, whereas fll diverges significantly from the true concept direction. Plotting the models sensitivity towards c, here measured as TCAV sens w h, with
the gradient of w.r.t. x, over (right), we observe that he consistently achieves TCAV sens = O (corresponding to the expected TCAV score 0.5), while for hill the sensitivity towards e incorrectly depends on These results demonstrate that relying on the widely used SVM-CAVs (i.e., filter-CAV) may produce arbitrary TCAV scores, making the concept sensitivity testing procedure highly unreliable. In contrast, our proposed pattern- CAV is invariant to the distractors and leads to consistent TCAV scores.
Controlled Experiment with FunnyBirds The comparison of TCAV scores computed with different CAV methods requires ground truth information on the true concept sensitivity, which is commonly unavailable for DNNs. To address this, we use our FunnyBirds dataset designed to enforce certain concepts, as for each class all concepts but one are randomized per sample. For each class k define subset X; {xx X' И4 = k} and compute TCAV score w.r.t. to the class-defining concept (see Appendix C.I). These TCAV scores are expected to be 전 0.5, as the concepts are the only valid features. Fig. 5 (left) presents the results averaged across all 10 classes with patter-CAAs
7

---

Published as a conference paper at ICLR 2025
VGG16
EdieientNet-I Bo
0.50
somple
filter
patterrn
fllse
pattlern
CAV Type
0.45
heak-0t
Logette
coler yellor.
Rulge
wring
Pattenn
eolon rnd
6.30
oaker Mos
VCG16
ReaNo118
BliciessNetBB
Figure 5: Left: ATCAV (averaged over class-defining concepts) for different CAVs fitted on last Conv layers of VGGI6, ResNet18, and EfficientNet-Bl trained on BundySirds. As models must use these concepts by experimental design, high scores are better. In contrast to filter-CAVs, pattern- CAVs achieve best scores for all models. Right: Concept-sensitivig maps, measured as element- wise product V (axx)) @ h using filter- and pattern-CAVs for three concepts with VGG16 and EfficientNet-BOO Results are shown for the last Conv layer, upsampled to input space dimensions. While pattern-CAAs CAVs precisely localize the concepts, filter- Miter-CAVs lead to noisy sensitivity maps.
and different filter-CAVs computed for the last Conv layers of VGG16, ResNet18, and EfficientNet- BO. Additionally, we report the TCAV scores using the ground truth concept direction h (GT). The TCAV score reported ATCAV [TCAV 0.5], i.e. the delta from the score representing no sensitivity. Higher values reflect a stronger impact on the model's decision and are expected in this experiment. To evaluate statistical significance, we run a two-sided t-test and found that all ATCAV scores are significantly different from the random baseline score of O. We report the corresponding p-values, accuracies for filter-CAVs on the test set, and results for additional model architectures in Appendix D.5 While for VGG16 and EfficientNet-BB -BO, pattern- CAVs achieve perfect score of 0.5, the TCAV score for filter-based CAVs does not fully indicate the model's s dependence on the concept. Interestingly, all CAV variants achieve perfect score for ResNet18, which can be explained by not well localized concepts, as further qualitative investigations in Appendix D.4 indicate.
The above observations are supported by qualitative results in Fig. (right), where pattern-CAVs precisely localize concepts and measure positive concept sensitivity (red) correctly. In contrast, filter-CAVs produce noisy concept-sensitivity maps, negatively impacting the TCAV score. This because TCAV (x) for sample x is computed over all elements of the concept-sensitiiitt map va](((x)) @ h. For instance, for the "wing"-concep" -concept samples (2nd and 3rd row), the dominance of negative sensitivity (blue) caused by noise over positive sensitivity (red) in VGG16's filter-CAVs leads to an incorrect negative overall concept sensitivity.
4.3.2 CAV-BASED MODEL CORRECTION (CLARC)
The CIArC framework (Anders et al., 2022) uses CAVs model data artifacts in latent space to unlearn shortcuts, iee, prediction strategies based on artifacts present in the training data with unintended relation to the task. Specifically, Right Reason ClArC (RR-CIArC) (Dreyer al. 2024) is
a recent approach that finetunes the model with an additional loss term LRR (x) = (Vaf fa(x)())) h
This loss term penalizes the of latent features, measured via the gradient, pointing into the direction of CAV h. representing the data artifact. An accurate estimated concept direction is crucial to ensure that the intended direction penalized. Hence, we intentionally poison models by encouraging them to use our controllable concepts (timestamp and brightness) as shortcuts, followed by the application of ClArC unlearn these concepts. We further correct models trained on ISIC2019 w.r.t. the known artifacts "band-aid", "ruler", and "skin marker" with artifact-specifi CAVs. Training details are given Appendix D.7
Quantitative Evaluation We evaluate the effectiveness of model correction with different CAVs by studying the impact of data poisoning on the model's accuracy and its sensitivity to data artifacts. For the former, we measure the accuracy on a clean (artifact-free and biased test set, with the artifact inserted into all samples. For the real artifacts in ISIC2019, we automatically compute input localization masks (Pahde et al. 2023) to cut (localizable) artifacts from known artifact samples and paste them onto clean test samples. To probe the model's sensitivity the artifact, we measure the
8

---

Published as a conference paper at ICLR 2025
Table 1: Model correction results with RR-CIArC for VGGI6, ResNet50, EfffcientNet-BO, and VIT trained on Bone Age ISIC2019 (controlled) ISIC2019 (real). We report accuracy on clean and biased test set, the fraction of relevance on the region of localizable artifacts, and the TCAV score (as ATCAVE with the sample-wise ground-truth concept direction h measuring the models' sensitivity towards the artifacts after model correction. Stars indicate statistical significance according to z-tests with significance level 0.05, and arrows whether low (L) or high (1) are better.
model
CAV
Accuracy (clean)
Accuracy (biased) 1
Artifact relevance
ATCAV
Vanilla
0.78
0.82
0.83
0.50
0.28
0.75
0.62
0.51
0.29
U.14
0.10
VGG-16 16
lasso
0.77
0.82
0.82
0.55
0.30
0.76
0.60
0.49
0.25
0.13
0.12
logistic
0.72
0.82
0.82
0.63
0.37
0.78
0.54
0.43
0.25
0.07
0.09
ridge
0.71
0.82
0.82
0.61
0.31
0.76
0.59
0.50
0.24
0.13
0.12
SVM
0.69
0.81
0.82
0.70
0.36
0.78
0.55
0.46
0.24
0.10
0.11
Pattern (ours)
0.78
0.80
0.82
0.75
10.69"
0.79
[0.26"
0.31"
0.14"
0.10
|0.03"
Vanilla
0.77
0.85
0.87
0.48
0.51
0.82
0.46
0.34
0.14
0.04
0.27
RosNot50
lasso
0.77
0.84
0.87
0.53
0.58
0.82
0.44
0.30
0.03
0.02
0.05
logistic
0.77
0.85
0.87
0.55
0.69
0.83
0.39
0.24
0.04
0.05
0.05
ridge
0.77
0.84
0.87
0.52
0.58
0.82
0.45
0.30
0.13
0.03
0.05
SVM
0.77
0.855
0.87
0.55
0.68
0.83
0.40
0.26
0.04
0.04
0.05
Pattern (ours)
0.78
0.84
0.87
0.59*
0.71
0.83
|0.37"
0.22"
0.01
0.03
0.04
Vanilla
0.79
0.87
0.88
0.46
0.55
0.83
0.55
0.22
0.46
0.39
0.12
lasso
0.79
0.86
0.88
0.70
0.64
0.83
0.52
0.22
0.01
0.11
0.11
Liffiientt Net-BO
logistic
0.77
0.85
0.88
0.75
0.67
0.83
0.51
0.22
0.00
0.02
0.12
ridge
0.78
0.82
0.88
0.74
0.67
0.83
0.52
0.22
0.21
0.12
0.12
SVM
0.77
0.85
0.88
0.75
0.65
0.83
0.52
0.22
0.00
0.03
0.11
Pattern (ours)
0.77
0.85
0.88
0.75
[0.72"
0.83
10.48"
0.22
0.00
0.05
0.03"
Vanilla
0.73
0.88
0.89
0.38
0.67
0.82
0.15
0.10
0.47
0.25
0.05
lasso
0.74
0.88
0.89
0.39
0.67
0.82
0.16
0.10
0.42
0.25
0.06
logistic
0.73
0.88
0.89
0.62
0.72
0.82
0.16
0.10
0.02
0.25
0.04
ridge
0.74
0.88
0.89
0.48
0.66
0.82
0.16
0.10
0.12
0.25
0.06
SVM
0.73
0.87
0.89
0.48
0.61
0.82
0.22
0.10
0.46
0.50
0.04
Pattern
0.74
0.88
0.89
0.61
0.73
0.82
0.16
0.10
0.06
0.25
0.11
fraction of relevance, computed with Layer-wise Relevance Propagation (LRP) Bach et al. 2015) for convolutional architectures and SHapley Additive exPlanations (SHAP) (Lundberg & Lee 2017) for transformer-base based models, on the artifact region using our localization masks. No artifact relevance is reported for the brightness artifact, as it is considered unlocalizable. Moreover, we compute the TCAV score after model correction using the ground truth concept direction h." For real artifacts, the ground truth direction is computed for "attacked" samples x with artificially inserted artifacts, as h a(x²") a(x;). The model correction results for VGG16, ResNet50, EfficientNet-BOO and Vision Transformer (ViT) for ISIC2019 (timestamp artifact, controlled). Pediatric Bone Age (brightness, controlled), and ISIC2019 ("band-aid", real) are shown in Table [ We perform model correction on one of the last three Conv layers for the former three architectures, and on the last fully-connected linear layer for ViT. We use filter-based (lasso, logistic, ridge regression, and SVM) and nattern-based CAVs as introduced in Eq. 3 to represent the direction to be unlearned. The models are finetuned and compared a Vanilla model, which is trained without added loss term. Further training details are provided in Appendix C22 For VGG16, the accuracy on clean test sets remains largely unaffected, while pattern- Mintern-CCAs outperform other methods in terms of accuracy on the biased test set. Moreover, pattern-CAVs yield best results for reduced artifact sensitivity, measured through artifact relevance and ATCAV Similar artifact sensitivity results can be observed for the other architectures. Furthermore, pattern-CAVs achieve the highest accuracies sets in the controlled settings. For the "band-aid" artifact, all CAV's yield similar accuracy scores on both clean and biased test sets. This can be attributed the minimal impact of the artifact on EfficientNet-Bi -B0 and ResNet50, as indicated by the small accuracy difference between the two test sets. Detailed results with standard errors for additional model architectures, e.g., ResNeXt50, R0 and EfficientNetV2,, are shown in Appendix D.7
biased test
Qualitative Evaluation We compare attribution heatmaps for the Vanilla model with heatmaps for models corrected with RR-CIArC using filter- (SVM) and pattern-based CAVs w.r.t. the band-
9

---

Published as a conference paper at ICLR 2025
RR-CIAFC (Laeso)
RR-CIATC: (lngieti)
RI-CIAIC (Rilge)
RI-CIAPD (SVM)
RILCIAIC (Patlers)
Eapat
Vosifla
Bestm
Riffeennce
Hestmap
Viffeerune
Hensmp
nifirencce
Bestoup
Biffowooe
Hessmap
Tiffeence
Figure 6: Qualitative results for model correction w.r.t. real artifacts band-aid (top), ruler (middle), and skin marker (bottom) in ISIC2019 using VGG16. In addition attribution heatmaps for models corrected with filter- and pattern-CAVs, we show heatmaps highlighting the differences compared to the Vanilla model attribution heatmap, with red and blue indicating higher and lower relevance after correction, respectively. Whereas (Ater-CAVs have limited impact, Miteem-CAVs successfully increases the relevance on the mole and decreases the relevance on data artifacts.
aid, ruler, and skin marker artifacts using the VGGI6 model trained on ISIC2019 Fig. In addition to the attribution heatmap computed with LRP using the E2 ==*>-cooot (Kohlbrenner al. 2020) in zennit (Anders al. 202T), we show another heatmap highlighting the difference between the normalized relevance heatmaps of the corrected and the Vanilla model, with blue and red showing areas with lower and higher relevance after correction. of data artifacts after model correction significantly, while traditional SVM-CAVs have little impact. Additional examples are shown in Appendix D.7
reduce the relevance
5 CONCLUSION
While filters from linear classifiers can accurately predict the presence of concepts, they fall short in precisely modeling the direction of the concept signal. As many applications of CAVs, including TCAV and CIArC, heavily rely on accurate concept directions, we address this drawback by introduc- ing pattern-based CAVs, which disregard distractor signals and focus solely the concept signal. We provide both theoretical and empirical evidence to support the improved estimation of the true concept direction compared widely used filter-based CAVs. Furthermore, we demonstrate the positive impact applications leveraging CAVs, such as estimating the model's sensitivity towards concepts and correcting model shorteut behavior caused by data artifacts. Future research might explore the optimization of concept directions beyond binary labels, the incorporation of prior knowledge, semi-supervised concept discovery, and the disentanglement of correlated concept directions.
Limitations Our results confirm that pattern-CAVs exhibit superior alignment with ground truth concept directions compared to filter-CAvs. This has a positive impact on CAV applications heavily relying on precise concept directions, such concept sensitivity testing (TCAV) and model correction with ClArC. However, for CAV applications in which class-separability is more important, i.e., determining whether concept present given sample, filter-basse CAVs might be better choice. For instance, post-hoc hoc concept bottleneck models (Yuksekgonul et al.) 2023) project latent embeddings into an interpretable concept space spanned by CAVs and fit linear classifier in the resulting concept space. The linear classifier can handle directional divergence in CAVs and requires a precise decision hyperplane, making filter-based CAVs superior in such scenarios. Thus, the choice of CAV computation methods should be carefully considered based on the specific task at hand.
ACKNOWLEDGEMENT
This work was supported by the Federal Ministry of Education and Research (BMBF) grant BIFOLD (011S 1802AA, 01IS180371D; the German Research Foundation (DFG) as research unit DeSBi [KI-FOR& 5363] (459422098); the European Union's 's Horizon Europe research and innovation programme (EU Horizon Europe) as grant TEMA (101093003); and the European Union's Horizon 2020 research and innovation programme (EU Horizon 2020) as grant iToooSS (965221).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Reduan Achtibat, Maximilian Dreyer, Ionaa Eisenbraun, Sehastian Bosse, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin. From attribution maps to human-understandaale explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006-1019, 2023.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference Learning Representations, 2017.
Christopher J. Anders, David Neumann, Wojciech Samek, Klaus-Rodee Robert Müller, and Sebastian Lapuschkin. Software for dataset-wide xai: From local explanations to global insights with Zennit, CoRelAy, and ViRelAy, 2021.
Christopher J Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus-Rober Müller, and Sebastian Lapuschkin. Finding and removing clever hans: Using explanation methods debug and improve deep models. Information Fusion, 77:261-295, 2022.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)=e01301 300))00030140, 2015.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117449):0071-330777 2020.
Titus Brinker, Achim Hekler, Alexander Enk, Joachim Klode, Axel Hauschild, Carola Berking, Bastian Schilling, Sebastian Haferkamp, Dirk Schadendorf, Tim Holland-Letz, et al. Deep learning outperformed 136 of 157 dermatologists in head-to-heat dermoscopic melanoma image classification task. European Journal of Cancer, 113:47-54, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782. 2020.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In / 55th International Symposium Biomedical Imaging (ISBI 2018), pp. 168-172. IEEE, 2018
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, al. Ben20000: Dermoscopic lesions in the wild, 2019.
Corinna Cortes and Mehryar Mohri. Confidence intervals for the area under the roc curve. Advances in neural information processing systems, 17, 2004.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297, 1995.
Jonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A generalized framework for concept-based explanations. Advances in Neural Information Processing Systems, 35:2590- 2607, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, pp. 148.555.10 IEEE, 2009.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
11

---

Published as a conference paper at ICLR 2025
Maximilian Dreyer, Frederik Pahde, Christopher J Anders, Wojeiech Samek, and Sebastian La- puschkin. From hope to safety: Unlearning biases deep models via gradient penalization in latent space. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 21046-21054, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, et Toy models of superposition. arXiv preprint arXiv:2209. 20022200900062,, 2022.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In CVPR, pp. 27 1-2711, 2023.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in neural information processing systems, 32, 2019.
Safwan Halabi, Luciano M Prevedello, Jayashree Kalpathy-Crameet Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology, 290(2)449-500 2019.
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. In Proceedings of the IEEE/CVE conference Computer Vision and Puttern Recognition, pp. 732-741, 2021.
Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and Felix Bießmann. On the interpretation weight vectors of linear models multivariate neuroimaging. Neuroimage, 87:96-110, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Funnybirds: A synthetic vision dataset for part-based analysis of explainable ai methods. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3981 9991-3991, 2023.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1)555-67, 1970.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pp.
PMLR, 2018.
Picter Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Er- han, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternatrribution. In ICLR, 2018.
Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojeiech Samek, and Sebastian Lapuschkin. Towards best practice in explaining neural network decisions with Irp. In IJCNN, pp. 1-7. IEEE, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Scott M Lundberg and Su-In Lec. unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. https 2016.
//github. wtt.://tttttcccccccooto
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedingy of the National Academy of Sciences, 119647):e22662551 19, 2022.
12

---

Published as a conference paper at ICLR 2025
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, 2023.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin. Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models. In Medical Image Computing and Computer Assisted Intervention, 2023.
Jacob Pfau, Albert T Young, Jerome Wei, Maria L Wei, and Michael J Keiser. Robust semantic interpretability: Revisiting concept activation vectors, 2021.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffu- sion autoencoders: Toward meaningful and decodable representation. In CVPR, pp. 10619-10629, 2022.
Alec Radford, Rafal Jozefowicz. and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint 17X.041704.014444 2017.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Kennir-Manor. Imagenet-211 pretraining for the masses. arXiv preprint (xXi:22114410972. 2021.
Nusrat Rouf, Majid Bashir Malik, Tasleem Arif, Sparsh Sharma, Saurabh Singh, Satyabrata Aich, and Hee- Hee-Cheol Kim. Stock market prediction using machine learning techniques: a decade survey on methodologies, recent developments, and future directions. Electronics, 10(21):2717, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), ICLR 2015, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pp. 6105-611 61 14. PMLR, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning, pp. 10096-10106. PMLR, 2021.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Soocityy Series B: Statistical Methodology, 58(1):267-288, 1996.
Guido Vittorio Travaini, Federico Pacchioni, Silvia Bellumore. Marta Bosia, and Francesco De Micco. Machine learning and criminal justice: A systematic review of advanced methodology for recidi- vism risk prediction. International journal of environmental research and public health, 19(17): 10594, 2022.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 1000 dataset, a large collection of multi-source dermatoscopie images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (med): A unifying framework with completeness guarantees. 7MLRR 2023.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text- controlled generative models. In NewrIPS, 2023,
Ross Wightman. Pytorch image models. pytorch-image-moon 2019.
htttps://githbb... www.::/.tttt//a/m///
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2110.00476, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, In CVPR, pp. 1492-1500, 2017.
13

---

Published as a conference paper at ICLR 2025
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
Aleš Završnik. Algorithmic justice: Algorithms and big data in criminal justice settings. European Journal of criminology. 18(5)622--64, 2021.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept-based explanations for can models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11682 11690, 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A BROADER IMPACT
This work presents drawbacks of widely used filter-based concept activation vectors (CAV), specifi- cally their tendency to deviate from the true concept direction. To address this, our paper introduces robust pattern-based CAVs, providing more accurate concept directions. This advancement directly impacts safety-critical CAV applications like concept sensitivity testing and model debugging, thereby promoting the transparency, accountability, and understandability of deep neural networks. Ulti- mately, this work contributes to increasing the trustworthiness of Al and advancing the development of reliable and explainable Al systems, extending its impact societal dimensions.
B METHODS
In the following, we provide additional details and proof's related methods. Specifically, provide details for linear models considered for filter-hased CAVs in Section prove the robustness to noise and scaling of pattern-CAVs in Section [.2 and further prove that in case of binary target
labels the pattern is equivalent to the difference of cluster means in Section B.3 Moreover, we present additional 2D toy experiments in Section B.4 scale the toy experiment to high dimension in Section B.5 and provide proofs of divergence for Elter-based approaches in Section B.6 and Section B. for feature scaling and noise rotation, respectively.
B. DETAILS FOR FILTER- BASED CAV APPROACHES
We briefly summarize the optimization objectives for linear models for filter-based CAV approaches, including lasso, logistic, and ridge regression, as well as SVMs. All methods aim to find a hyperplane that separates dataset X C R of size into two sets, defined by their concept label t ++1, -1]. This is achieved by fitting a weight vector w E R" and bias E R such that the hyperplane consists of all x which satisfy w x+6 O.
Lasso regression Tibsshirani 1996) aims to minimize residuals r; = (4:-(w (w x; + +b) with L norm regularization, thereby encouraging sparse coefficients. The optimization problem is given
by
1 TU
min w.F
Royl
jE[m]
Similarly, ridge regression (Hoerl & Kennard 1970) fits a linear model which minimizes the residuals r, with L2-norm regularization by solving
Σ n
min w.b
(8)
(E(x)
JE(M)
The logistic regression model estimates probabilities via
ew" X++
(t. +1 +1|
1 e =o(wT -x x
where a denotes the sigmoid function (2)) Ites. The linear model is now fitted by maximizing the log likelihood of the observed data:
max ((w, b) w.b
max w.b
Σ
1(t, = +1) log
= +1 x.) Pw.b(t log
(E(m)
+ (44 = 11) log (1 P (1-
Pw, +1 xx)
Lastly, and most commonly used for CAVs, SVMs (Cortes & Vapnik 1995) fit a linear model by finding hyperplane that maximizes the margin between two classes using the hinge loss, defined
15

---

Published as a conference paper at ICLR 2025
4 max (0, 1 to (w
x; + b)) and Ly-norm regularization with the following optimization
objective:
Σ KE(M
min w,b
w
(10)
Th
ME
B.2 FEATURE SCALING AND NOISE ON PATTERN-CAV
In this section, we investigate the effect of feature scaling and additive noise on the resulting pattern- CAV. We start from the known solution for simple linear regression task provided Equation 3 resulting in pattern-CAV of
h
Σ (a(x) X
A)(t - 8).
(11)
Feature Scaling We start with the effect of feature scaling on h Specifically, we investigate the effect on the CAV, when we scale a specific dimension & of features a E IR (of dimension m) with a factor y, i.e.,
if k if i k.
a
(12)
70
Then, with Eq. (II). we get for the corresponding pattern-CAt
(K7),"²
Σ (x] (t) X.
021221
(13)
if Fiihh k if = k.
meaning that the CAV heat scales with the features (contrary to many classification-based CAVss..
Additive Noise We add random noise e with zero mean E[c] = 0, that is independent to the concept
labels to feature dimension k. Then in expectation
Elh pp0 =
1 B[(a ā; + St E)]
o
1
[[ at)(t -)] (tcx(t E)]l
(14)
of
1
B[(a: at)(t 2)] = 1!!T!
where used the independence of and i.e., E(ct] E[c]t 0.
B.3 AATTERN-CAV REDUCING TO THE DIFFERENCE OF MEANS
Assuming for easier notation have binary concept labels t {0, 1}. We start from Eq. (3) for the pattern-CAV, given by the known solution for a simple linear regression task given as
h pat
Σ X
(a(x) A)(t E)-
(15)
For the sample covariance term, we get
(a(x) (a(x)
A)(t -8) 1
Σ
(a(x)
-
-
Σ (a(x) A)E
(16)
H XXEX
X1EX+
X.EEX
16

---

Published as a conference paper at ICLR 2025
We have | positive (with concept) sample activations A+ and | negative (without concept) sample activations A We further introduce a= x [2 t {0, 13. Thus, can write
therefore, f = at and 1 - = a using
1 X
Σ A)(t-)
Σ
(a(x) - A) - -a Σ (a(x) - AA) 20)
X
X,EEX
X,
X,TEX
(17)
1
aa
x
1A+ A) a a (A
1 - -A)
x
where we used for the last step that
1
A+
Σ
a(x).
(18)
|
X
Finally, we receive
A)( f)= 11 (A--A-)
(a(x)
A-) =
-A)
(19)
X
1212
X,EE
and
h = A-
(20)
B.4 2D TOY EXAMPLES
In addition to the 2D experiments conducted in Section 3.3 in the main paper, we investigate two more scenarios in which we (1) increase the standard deviation and (2) vary the random seed. For the former, we randomly sample data points for class A from NN(O 1) ,) and for class B
and incrementally increase a. For the latter, we sample from (/5 1)T,E) E) with Σ= (0" 02)
from the same distributions with fixed = 1, but use different random seeds for each run. We fit both pattern- and filter-AAVs for both experiments. As filter, we use hard-margin SVMs. Fig. / presents results for additional runs for the settings discussed in the main paper, namely noise rotation (1 row) and feature scaling (2nd row), as well as the new 2D settings, including increased standard deviation (3"d row) and different random seeds (4th row). In addition to the observations discussed in the main paper, we can see that filter-CAVs from hard-margin SVMs diverge for increased values for a, samples are not perfectly separable anymore. Moreover, the filter-AAVs is sensitive to random seeds. In contrast, pattern-CAVs constantly point into the correct direction for all settings. Animated visualizations for all challenges discussed can be found here: https: /ttpss//githinn com/ freder www.errikaaeeeeeeeeeeereeeeee
cav treeetree///////// ma in/ an animat ions
B.5 HIGH-DIMENSIONAL TOY EXPERIMENT
We extended our 2D toy experiment dimension, noise rotation, and further 100 distractor signals in randomly selected dimensions. We measure the cosine similarity with the ground truth concept direction and report results in Fig. g The quality of pattern-CAV remains high, while SVM-CAVs are distracted by the rotated noise.
4224-dimessiona data with the concept signal in one
B.6 PROOF OF DIVERGENCE: SCALING
We consider the general case of logistic regression on a set activations A RT For a weight vector w E Rm and bias term b E IR logistic regression models the probability of an activation corresponding to concept label = +1 as
Tatb ew"
Pw. (t +1 a)
1 + atd =o(w] b),
(21)
17

---

Published as a conference paper at ICLR 2025
Rotation:
Rotation: 40
Rolation:
Rolaison:
2.0
doecinss
Noise Rotation
80-
Skaling 0.3
Scaliua
Sealiig
Stallan fustas
odhe daeeiaa
Feature Scaling
410
Alroctioo
Stradacd deviation:: 0.1
Studdar deviation: 0.0
Standand deviation:
Standard deviactoo:
Increasing Standard Devlation
obe dooeiias
vide destiuu
mocrepe dlvve km
coereyt diretioon
eoscepp
Random state:
Rendoca stater
Random state:
Random state: 3
14
Different Random
viler doeetiss
States
Figure 7: Multiple runs for 2D toy experiments. For noise rotation (1" row), the filter (magenta) diverges depending on the distractor direction, while pattern-CAV CCV (green) stays constant. When increasing the scale of the x-axis (2nd row), the filter CAV scales antiproportional. For increased standard deviation (3nd row), filter-CAVs (here: hard-margin SVM) diverge when the clusters are not perfectly separable anymore. Lastly, different random seeds for not perfectly separable clusters lead to varying directions for filter-based CAVs (4th row).
where denotes the sigmoid function a(z) It We predict
+1 for an activation if P. wt +1 a) 0.5 and -1 otherwise. To train unpenalized logistic regression classifier, we seek to maximize the log likelihood of our observed activations .4
L(w, A) = Σ 1(t + Pw.b(t +1 a,) + 1 1t2 = 1) log (1 Pw,b(t = +1 ai))
sE|n
(22)
Assume for our unscaled set A, we have found an optimal choice
w, E argmax L(w, b; A).
(23)
wER" bER
To introduce the scaling along an axis, for given vector by a² the vector which has the same entries as a except for the k-th entry, which has been replaced by yag. Further, let A (a" AA denote the set of scaled activations. Finally, for the weight
I²" and dimension k E [n]] we denote
18

---

Published as a conference paper at ICLR 2025
1.0
CAV Type
Cosine Simillrity
0.9
Pettern
SVM
0.6
0.5
360
Degree
Figure 8: Cosine similarity with ground truth direction over degree of noise rotation in experiment with 1024-dimensional generated toy data with true concept direction oriented along the first dimension, noise rotation, and 100 additive distractor signals in randomly sampled directions. Pattern-CA consistently points into the correct direction, while SVM-CAV is distracted by the rotated noise.
vector we introduce the equivalent notation w // for the vector in which only the k-th entry of w has been changed to Wk. Then we derive the following equality
(w//7) a + wiai +...+
(F (yax) +... + Wm +b= w a +bb (24)
which implies the equalities the predicted probabilities
Pwtrr,,f (t +1 a²) = = +1.o(t=+1|1 |Iaaa
(25)
and thus of the log likelihoods
(w+//,b;AA' = I(w, A).
(26)
Therefore it follows that the optimal solution to logistic regression the scaled dataset relates to original solution on the unscaled dataset via
W 1/7
argmax I(w,b; A7) wER. .BER
(27)
In conclusion, scaling the activations by a factor of by in one dimension leads the signal to also scale by factor of 7 in this dimension. The fiter-based CAV calculated as the weight vector of an unpenalized logistic regression, however, exhibits a scaling in the same dimension which is antiproportional to the scaling factor y. Such antipropotional scaling will misalign the filter-based CAV unless it is either perfectly aligned or perfectly orthogonal to the direction of scaling. This shows that even if the filter-based CAV theoretically lies aligned or orthogonal to the scaling dimension due to noise and constraints in machine precision, logistic regression may be hugely affected by the lack of feature scaling.
B.7 PROOF OF DIVERGENCE: NOISE ROTATION
With the additional rotational noise term and assuming the concept label t; be fixed, our activations A; are distributed according to independent multivariate normal distributions ((p,, Σ) with
( 1(t, = =1),
P4 =
(28)
02 + sin
sin COST
=
sin T cos
o²+ + cos2
From this formulation and orthogonal to the CAVs, unless т is a multiple of z in which case we only add noise parallel or orthogonal to the CAVs respectively. It thus follows that the random variable w A, + b has the following distribution:
that this adds noise which is correlated in the direction parallel
A + ind. N (wn 144 +1) + 8, VW ((T))
(29)
19

---

Published as a conference paper at ICLR 2025
Note that the choice of W2 does not affect the expected value of w TT b but may change its variance. To study this effect on the variance, for a given À & 0 define the family of weight vectors
1 A
Wx
W2 E R
(30)
W2
1
Theorem B.1. Define the vector W = A
where i
sin coS +-oos
Then W. is the unique
102
minimizer
WA- argmin V(wTA,) AA).
(31)
wEW,
Proof. The variance for WA is given by
V(w As) x2 w2)
222+ sin r CoST
1
sin COST a² + cos? T
102
(32)
= X² =0² + sin
200 sin CoS + wzo²
+ uz cos? }}
= x2 {0² + w20² + (sin + W2 cos ?)
Differentiating with respect to u2 gives the following expression
D-((T V(w T A.) = X2 {2wzo² + 2 cos Tsin + w2 cos })} 2x² {(o² + cos² F)w2 + sin T cos }
Own
(33)
This is set zero if and only if ū2
sin cos o² 40o0s
Furthermore, the second derivative
82
V(w A,) = 2X (o² + coss r) > 0,
(34)
Ow
so W2 indeed minimizes the variance.
The proof's of divergence for both logistic regression and SVMs are now analogous: Assuming there are two vectors w, for which w A; + has the same expected value but w yields a smaller variance, then the expected value of the objective function of the optimization problem of the model (the log likelihood for logistic regression or the size of the margin for SVMs respectively) will be larger for w. Together with Theorem B.] this proves that a vector of the form Wa maximizes the expected value the objective function and thus preferred as the weight vector over the true CAV (1 0) with non-zero probability.
B.7.1 LOGISTIC REGRESSION
For logistic regression, we intend to maximize the log likelihood of our observed data. We may express our log likelihood in terms of the random variables A, + by the formula
L(w, b; A)=
Σ KE[N]
I(t, = +1) log(o(w)) T A + b)) + L(t, = - 1) log(1 o(w A, bb)
(35)
Σ VE[N]
1(t, = +1) A, + b)) + 1(t, - 1)g(o w TTA, A, b)),
where a denotes the sigmoid function.
Theorem B.2. Let be two weight vectors with IE [w A. + +b] =E W A, + for all i [n]
and VW
W W
A, b). Then
E [[( b; A)]
E [[(( w b; A)]
(36)
Proof. To focus on the effect of the variance on the log likelihood, define independently distributed random variables Y; ~ A M( s²) with means μ4 and shared variance ş" > O. We allow the Y, to have different means as the mean of the random variables w A, b also differ depending on the concept label f,. We may now define the functions of and which both depend on ²² via
9 (V,) 1(: +1) (Y:)) (1, - log(o(-Y.))
(37)
20

---

Published as a conference paper at ICLR 2025
and
((s2) E
Σ 9i)) sElm
Σ E [x)() seimi
(38)
where we used the linearity of the expected value in the last step. After proving that the function f is strictly decreasing the claim follows from inserting w A, + for Y,. We prove first that log(o(=)) a strictly concave function by calculating the second derivative:
1 (())) - (1-o = - (x() < 0 for E R, (39)
(log(o(z)))) -
which holds as the image of the sigmoid function is the open interval (0, 1). Because the logarithm of the sigmoid is strictly concave function, so is log(o(-2)), hence each summand
9o (Y,) (t: log((((()))) 1 1(; - - log(o((-)))
(40)
strictly concave in Y, Now consider two variances xी < $2 and define independent random variables Y,' M( sf) and Y (, <2 - <t) such that their sum are independently distributed random variables Y Y! +Y ~ NN s2). Using the conditional version of Jensen's 's inequality on the strictly concave
functions gt. we derive
E (v(3)
E (Y,
YJ) [ + Y
[
E
B (Y, + 1 [g (Y, + E [TD) = E 99 (V7)),
(41)
E
gi
where the sccond-to-las step follows from the properties of conditional expectation for completely dependent and independent random variables. Summing over all finally proves the desired inequality
f(s = Σ E [g (Y:))
E 9 (Y?
f(2)).
(42)
se|n]
se [n]
B.7.2 SVMs
We inspect the behavior of linear hard-margin SVM, assuming that our data can be perfectly separated by a linear hyperplane. Then the optimization problem for this particular SVM is given by
2
max W.b
1w
(43)
subject to & (ww) A, b) 2 1 for all i [m]]
This states that we aim to maximize the margin which has length 2/||w// subject to every datapoint lying on the correct side of the margin. A fitted SVM will have at least one vector of each class, the so- called support vectors, on its margin, which can be equally formulated min4,=+1 (w A, + b) 1 and max;,=-1 (w" T A, + b) -1. We may use these quantities to reformulate the length of the margin as
min (w A, + +) max (w A + } 4-=-1
(44)
w
which is what we are trying to maximize in order to find the direction of our weight vector w.
Theorem B.3. Let W be two weight vectors with E [wT A, b
[WTA, A, b] for all i [n], V(w A, b) V(w A, + b) and ||w||2 < || 2- Then for sufficiently large sample size the expected margin size for the SVM with normal vector in direction of w is bigger than for the SVM with normal vector in direction of
21

---

Published as a conference paper at ICLR 2025
Proof. We denote M+ E w A, + μー = ===B A, +,
= = and <²
(w" A, b), and further define random variables N,, N as independent standard normal distribu- tions, We can now write the expected size of the margin as
E
min (w
A; + b)
max (w T A; + b) 4==-1
1
# - H + <E
min N₂ max NO
|
2
sE(n/22]
seln/2
1
M 11- sIR
max iE(n/2]
N; + max No. iE(n/2)
(45)
|w|l2
1
}
M-_ 11_ 2x
max N₁
|w||2
[iEn1/2]
1
{u+ - μ 2<m(n))
ww/2
where define m(n) E /m/x/2/0/2) NJ] and the step from the second third line follows by the symmetry of the standard normal distribution and the fact that min(S) max(() for symmetric sets Firstly, we show that the quantity m(n) grows unbounded. Let [max (N1, 0)]
<2=
Then
m(n) E max
max N, 0) (E(n/2]
E min max No., (E(n/2]
4M (P(m(N) 4M) E [max (N1, 0)]
4M {1 - P(N; < 4M for all i)}
(46)
4M {1 | < 4M ///2 M
{1 2 M = 2M M = M,
2 4M
where the last inequality holds for sufficiently large n. As M may be chosen arbitrarily large, this proves that mnn) unbounded.
Now for two weight vectors w. W with the same associated expected values μ4, variances
<² V (w" A, + b) V A, + b) 22 and ||wl|2 S [| follows by simple arithmetic that the inequality
1 w||12
{u+ ne_ 2mm(n)} > |w||2 {u+ - p_ - 25m(n))
(47)
equivalent to
m())
{ut
1-
(48)
|w|
W
l
|||2
||
Note, that since ç > [ and
||l|2
w 12. it follows that
5
> 0.
(49)
|w/2
So the left side of the inequality grows unbounded with n while the right side remains constant. Hence, for n sufficiently large, the inequality is fulfilledd and the expected margin of the SVM associated with w is greater than the expected margin for the vector w.
C EXPERIMENT DETAILS
We provide dataset details in Section C11 and training details in Section The former includes details for controlled "Clever Hans" datasets (Section C.I.I and the synthetic FunnyBirt dataset (Section C.12)..
22

---

Published as a conference paper at ICLR 2025
ISIC2018
Bone Age
insert
timestamp
brightness
Figure 9: Examples from controlled datasets with clean and attacked samples for ISIC2019 (leff) and Bone Age (right), with timestamp and brightness artifacts, respectively.
C.1 DATASETS
C.I.1 CONTROLLED "CLEVER HANS" DATASETS
Details for our controlled datasets with artificial "Clever Hans" artifacts, ie., shortcut features, are provided in Tab. Examples are shown in Fig. 9
Table 2: Details for our controlled "Clever Hans" datasets, including artifact type, number of samples, class names, the biased class, percentage of samples with artifact in the biased class (p-bias), and train/val/test split.
number samples
biased class
train / val test split
dataset
artifact
classes
p-bias
0-46, 47-91, 92-137,
Bone Age
brightness
12,611
92-137
20%
80% 1111110%//0/ 110%
188-182, 183-228 (months) MEL, NV, BCC, AK,
ISIC2019
timestamp
25,331
MEL
1%
80% 20fe/00%/10 10%
BKL, DE, VASC, SCC
C.1.2 FUNNYBIRDS DATASET
FunnyBirds (Hesse al. 2023) provides framework synthesize images of different classes of birds. Specifically, a bird is defined using 5 parts, for which the authors manually designed different types (4 beaks, 3 eyes, 4 feet, 9 tails, 6 wings). Further varying color, this leads to 2592 possible combinations, i.e., classes. We define a concept as a combination of part, type and color. For example, the concept :ea.:::bbakkOO:yelllooo
entails the beak shape beak 01 in color ye 1 1low. As outlined in Section 4..1in in the main paper, we construct new version of Funny Birds with 10 classes, with exactly one valid feature, i.e. concept, per class. While the class defininfing concept is identical for all samples per class, all other concepts
chosen randomly per sample. The class- class-defining concepts are listed in Tab. 3 When training models on this dataset, the class-defining property must be used by the model. We synthesize 500 training samples and 100 test samples per class, totaling to 5000 training and 1000 test samples. The training set is further split into training/validation splits (90%/10%). In order remove concepts, e.g., ., for the computation of sample-wise ground truth concept directions, we replace the class-defining property with another randomly chosen concept (e.g., beeek::baak-01:::ee
+ waak::beak-3::yyllllleee Examples for original and manipulated samples are shown in Fig. 10
while keeping other parts unchanged.
C.2 TRAINING DETAILS
Tab. provides training details for all models and datasets, including the source of the pre-trained model checkpoint, optimizer, learning rate (LR), number of epochs, and milestones, after which we divide the LR by 10. All models pre-trained ImageNet (Deng al. 2009 Ridnik al. 2021 with weights provided from eimm (Wightman) 2019) or torchvislon (maintanerrs & contributors 2016).
23

---

Published as a conference paper at ICLR 2025
Table 3: Class-defining concepts (part/shape/color) for all 10 classes in synthetic FunnyBirds dataset.
class-defining concept
class
part
shape
color
beak
beak-01
yellow
2
beak
beak-02
yellow
3
beak beak-03
yellow
beak beak-04
yellow
5
wing
wing-01
red
6
wing wing-02
red
7
wing wing-01
green
8
wing wing-02
green
wing wing-01
blue
10
wing
wing-02
blue
Clai Staee)
Clue (Clua)
(Clas)
Ubw (Clea))
Clarr (cimml
Clem Skonke Gawgy))
Clus (Reedam Ouenpt)
Cle 3 RRedees Gumpg))
Clue (Reedam Chanpt)
Cte Cheekes Sergli)
Cleir Sluekkes Guuuge)
Clue (Benion Ckemppt)
Clei /Reekkes Gavipe)
Clue (Raedom Chnmpt)
Cle (Randiz Conrept
Figure 10: Examples for samples from all 10 classes from our synthetic FunnyBirds dataset, including clean samples (top) and identical samples with class-defining concept randomized (bottom).
C.3 COMPUTATIONAL RESOURCES
We ran all model training and correction jobs on GPUs of type NVIDIA Ampere A100 with 40 GB RAM. Depending on the architecture and correction layer, model correction job including evaluation took between 20 minutes and 2 hours. Depending on the architecture, model training took 6-12h for ISIC2019, 1-4h for Bone Age, and 8-30mins for FunnyBirds.
C.4 LICENSES FOR EXISTING ASSETS
Existing used in this paper have the following licenses and terms of uses:
24

---

Published as a conference paper at ICLR 2025
Table 4: Model training details including the pre-trained checkpoint, optimizer, learning Rate (LR), number of epochs, and milestones, after which the learning rate is divided by 10.
epochs
dataset
model
pre-trained checkpoint
optimizer
LR
(milestones)
VGG16
torchvisions TMAG/NNTTAGGNGGGA KLV1
SGD
0.005
100 (50,80)
ResNet18
timm/resnet16
Adam
0.005
100 (50,80)
ResNer50
Ciam//reanttS-.al.iill
Adam
0.005
100 (50,80)
ResNeXt50
2imm/rresnext55-22xed... alh.iikk
Adam
0.001
100 (50,80)
Bone Age
ReXNet100
t timm/rexnet nav inlk
Adam
0.005
100 (50,80)
EfficientNet-0O
foxchvissoon/IMAEENETLL.
Adam
0.001
100 (50,80)
EllicientNet-22 V2-8
torchvvissoom/IVAGENN111..
Adam
0.001
100 (50,80)
timm/vit_base.
Vision Transtormer
SGD
0.0005
(50,80)
pacch16 a24.augreg in21k
timm/swir.base_
Swin Transformer
SGD
0.0005
100 (50,80)
pacchsd_...doo1122..m.....o
VGG
torchvissoo//TMAGRNN
SGD
0.005
300 (150,)50)
ResNet18
timm/tesnstii.all.nn.
Adam
0.0005
300 (150,250)
ResNet50
cimm/resnet50,al_inn1
Adam
0.0005
300 (150,250)
ResNeX150
Adam
0.0005
300 (150,250)
ISIC2019 (controlled)
ReXNet100
timm/rexnet nav inlk
Adam
0.0005
300 (150,250)
EfficientNet-Bl
Adam
0.0005
300 (150,250)
EfficientNet- V2-8
torchriisioo/IIMAGENTTTI
Adam
0.0005
300 (150,
Vision Transformer
gpoglee/vit.baaeepatchh 6.224
SGD
0.001
300 (150,250)
timm/swin.base
Swin Transformer
SGD
0.001
300 (150,250)
pat.hha_w.nmoox22224......
VGG16
SGD
0.005
150 (80,120)
ResNet18
timm/resnst16..ll
Adam
0.0005
300 (150,250)
ResNer50
Adam
0.0005
300 (150,250)
1SIC2019 (real)
ResNeXt50
2.smm/resnaxt55332xxdd...
Adam
0.0005
300 (150,250)
ReXNet100
1.sm/reexstt.100.navialkk
Adam
0.0005
300 (150,250)
EfficientNet-BB
torchivissoom//aaGzzez21333
Adam
0.0005
(150,250)
EfficientNet- V2-s
SGD
0.001
300 (150,250)
Vision Transformer
googla//tt.baae__acchhi222
SGD
0.0005
300 (150,250)
VGG16
torchwvission/11000NN11.1..
SGD
0.005
50 (30)
ResNer18
t
cimm/ressnet18.a11.mm
Adam
0.005
50 (30)
Funny Birds
ResNeXt50
timm//eesnaxt553332edd
32x4. inlk
Adam
(30)
ReXNet100
c:tm/rexxret.00.navv.inkk
(30)
EfficientNet-B
torchivissoom//AAGzzzzz3.3
Adam
(30)
EfficientNet-V V2-s
torchhvissoon/7AARREE7771111111 goggle//at.bbaessathhh..22.
Adam
0.001
50 (30)
Vision Transformer
SGD
0.005
50 (30)
ISIC2019: CC-BY-NC
Pediatric Bone Age:
The terms of use are
described
here:
Attribution Atribution.Final.caia www.auccceeennnne222200006666600
TunnyBirdss Apache License 2.0
timm model checkpoints: Apache License
torchvision checkpoints: BSD 3-Clause License
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 DETAILED CAV ALIGNMENT RESULTS
Additional CAV alignment results, including filter-LLasso, Logistic, Ridge, and SVM) and pattern- CAVs are shown in Figs 11
12 13 and 14 for ISIC2019, Figs 15 16 17 and 18 for Pediatric Bone Age, and Figs 19 20 and 21 for FumnyBirds for VGG16. ResNetl8, ResNet50, ResNeXt50, 100, EfficientNet. BO, and EfficientNet22 models. The results confirm the trends described in the main paper in Section 4.2 ie.. a higher alignment with the ground truth concept direction for pattern-CAVs and better concept separability for Clter-CAVs.
25

---

Published as a conference paper at ICLR 2025
Moreover, report the cosine similarities between CAVs obtained with different feature pre- processing methods (centering, max-scaling, and their combination) and the ground truth concept direction for ISIC2019 and Bone Age datasets the last Conv layers of ResNet18, ResNet50, ResNeXt50, ReXNer100, EfficientNer-BO, EfficientNeV22, Vision Transformer and Swin Transformer in Figs.
and 25
1.0
Costre Stedlante
0.8
0.0
1.0
AUC
0.8
Cunr
Ralge
Putten
Figure 11: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and for all Conv layers of VGGI6 trained on ISIC2019.
1S1
131
0.6
Coltae Suncuetty
06
6.1
0.4
0.2
1.00
1.00
0.93
0.90
0.90
0.55
Block
Block
Blcc
Block
Lust Blodk
Laso
Loglew:
Büdge
SVM
Pattern
Laso
Loglele
Bidge
SVM
Pictern
Figure 12:
Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each block of ResNet18 (left) and ResNet50 (left) trained on ISIC2019.
D.2 QUALITATIVE CAV RESULTS
Following-up the qualitative approach on Section 4.2 we present further RelMax visualizations for the most important neurons for different CAVs in Fig 27 In contrast to Fig. 2 in the main paper, include all our CAV approaches, namely 4 filter-based (lasso, logistic, ridge, and SVM) and the pattern-basee CAV. Again, all filter-CAVs include unrelated neurons, whereas the pattern-CAV mainly includes neurons focusing the concept of interest.
26

---

Published as a conference paper at ICLR 2025
3SIC2019
0.5
08-
Codaae Stulizity
Sunlartt
0.4
0.3
02
1.0
A00
0.06
0.5
0.90
0.7
10ack
Blook
Pultuu
Ralge
Putticu
Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each block of ResNeXt50 (left) and ReXNet100 (leff) trained on ISIC2019.
Figure 13:
I8IC2019
19EC2009
Costre Sunllstty
0.5
0.3
0.2
1.9
0.9
0.9
0.7
BRock
Blokk
Baak
Bhak 4
Block 5
Badk
Bhokk 7
Block
Laxe Block
BRock
Baak
Black
Block
Black
Blok
Back
Los Bllck
Leoi
Loginie
Falge
SV3
Patten
Logidie
Edige
SVM
Patten
Comparison of cosine similarity between CAV's and concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV. and after each block of EfficientNet-Bi (left) and EfficientNetV2 (left) trained on ISIC2019.
Figure 14:
D.3 REDUCTION OF SUPERVISION
In a further set experiments, we want to analyze the possibility to reduce the manual labeling efforts by (1) the unsupervised discovery of concept directions in Sec. D.3.1 and (2) the robustness of (supervised) CAV directions towards labeling errors and reduction of data size in Sec. D.3.2
D.3.1 ALIGNMENT OF UNSUPERVISED CAV DIRECTIONS
We do an unsupervised concept discovery in the penultimate layer of VGG16 trained on ISIC2019 (timestamp artifact) with CRAFT (Fel al. 2023) (via Non-negative Matrix Factorization) and compute the cosine similarity of each found concept direction with the ground truth direction and plot histogram of similarity scores in Fig. 28 It can be seen that the best CRAFT direction outperforms
27

---

Published as a conference paper at ICLR 2025
Brom Age
1.2
Contre Sunlartt
0.3
0.2
0.8
Cour
Rálge
Figure 15: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each Conv layer of VGG16 trained on the Pediatric Bone Age dataset.
Bone
Bone Age
0.8
Codbee Studhertt
008
Costoe Sinllortt
0.6
0.4
0.2
0.7
0.90
AT1C
0.90
0.90
0.85
Block
Block
Bluck
Poleen
Logisie
Figure 16: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV. and after each block of ResNet18 (left) and ResNet50 (right) trained on the Pediatric Bone Age dataset.
the SVM CAV, however, it is worse than pattern-CAV. Moreover, it is to note that unsupervised concept discovery comes with two drawbacks in practice: (1) requires manual inspection of found concepts decide which direction(s) represent the desired concept. (2) Matrix factorization will find statistical groupings without guidance, hence there is no guarantee that one direction will represent the desired concept.
D.3.2 ROBUSTNESS TOWARDS MISSING DATA AND LABELING ERRORS
As pattem--AAs more robust against noise in activations, they are more stable for low-data or mislabeled samples compared to filter-based CAVs. We verified this in additional experiments was activations from the penultimate layer of VGG16 trained on ISIC2019 (timestamp artifact) with results shown in Fig 29 (1) We gradually decreased the number of known artifact samples before CAV computation and found that Pat remains precise than filter-based CAVs with
28

---

Published as a conference paper at ICLR 2025
Contor Santiett
Codite Stmllattty
0.4
0.4
0.2
0.6
100
0.32
0.90
0.95
0.65
Block 1
Block
Block 3
Last Blokk
Block 13
Blode 14
Block 15
Blook 14
Lost Biodl
Laso
Laspesc
Ridke
SVM
Pattecu
Laso
Lauseee
Tidker
SVM
Partscu
Figure 17: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV. and after each block of ResNeXt50 (leff) and ReXNet100 (right) trained on the Pediatric Bone Age dataset.
Done Age
Base Age
Costue Santhetty
0.5
Ookiee Stnilbrtt
0.4
0.3
1.0
0.9
0.90
0.55
Black
Block
Blodk3
Black
Bleck
Black
Blcck
Block
Laxt Black
Blok
Blick
Block
Blakk
Bleck
Blok
Blick
Leaao
Logisk
Figure 18: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each block of EfficentNet-BB (left) and EfficientNetV2 (right) trained on the Pediatric Bone Age dataset.
reduced data (left). (2) We gradually increased artifact mislabeling rate p (false positive rate) and found that the quality of filter-CAVs decrease rapidly, while the quality of pattern-CAVs consistently remains high (right).
D.4 QUALITATIVE TCAV RESULTS
Extending qualitative TCAV results from Section 4.3.1 we show further sensitivity heatmaps for all considered CAV types, including four filter- (lasso, logistic, ridge, SVM) and our pattern-CAV in Fig. 31 We observe similar trends in the main paper. Specifically, filter-CAVs lead to noisy
29

---

Published as a conference paper at ICLR 2025
1.3
Funey
Flamo Biads
Conte: Sindlare
0.5
0.2
0.5
0.8
X00
0.5
Conr
Bork
Block
Ilook
Lost Nook
Leoi
Logisie
Rilge
Pattesn
Logisic
SVM
Putticu
Figure 19: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each Conv layer of VGG16 (leff) and ResNet18 (right) trained on FunnyBirds
DinnyDieds
Fiano Birds
0.200
0.20
Dormo Smmberr
0173
Smibwery
0.13
135
0.00
0.000
0109
0.7
0.85
0.1
0.80
0.5
3
Block 14
Block
Block
Blods
logiake
Rólge
Palicn
Logisie
Ridge
Figure 20: Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each block of ResNeXt50 (left) and ReXNet100 (right) trained on FunnyBirds.
sensitivity heatmaps, negatively impacting the TCAV score, while pattern-CA CAV precisely localizes the concept with positive sensitivity.
Note, that for ResNet18, instead of precisely localizing concepts, the sensitivity in the last Conv layer spreads over the entire sample (7 x pixels), as shown in Fig. [6) Therefore, TCAV scores for ResNet18 are less impacted by noisy concept sensitivity maps in irrelevant regions. Similar trends have been observed for ResNeXt50 and ReXNet100 models.
D.5 QUANTITATIVE TCAV RESULTS
In addition to the results for the controlled TCAV experiments with CunnyBirds shown in Fig. 5 in Sec. 1 we present results for additional model architectures in Fig, 31 including Re55eXt50. ReXNet100, EfficientNetV2. and Vision Transformer. Interestingly. ResNet18, Xt50 and ReXNet100 all share similar behavior, which is further discussed in the main paper. Note that due to
30

---

Published as a conference paper at ICLR 2025
BungeBinda
Funny Badd
3.000
Corte Samlanty
Coatre Simllerity
0.10
30000
0:05
0000
0:00
0.75
0.8
0.60
07-
0.60
0.50
0.-
Block 3
Block 5
Mhat
middk 7
Block
Lect Blrk
Blak
Black 5
Buck
Black g
Black 7
Las Mark
Toglek
Ring
SVM
Pattem
Loabetc
Bidge
Comparison of cosine similarity between CAVs and true concept direction (top) and concept separability as AUC (bottom), using filter- (lasso, logistic, ridge, and SVM) and pattern-CAV, and after each block of EfficientNet-Bi BO (left) and EfficientNetV2 (right) trained on FunnyBirds.
Figure 21:
ISIC2019
Bone Age
ISIC201
Bour
Cliie Siudlaie
0.6
0.5
0.4
0.1
leaatiic
calge
Iaso
Iagidie
2000
loga
paliena
logikic
siler
Y0aM
centered
oralled
centerel
certiree
rla eraled
Figure 22: Cosine similarity between true concept direction he and CAVs with different pre- processing methods fitted on the last Conv layer ResNet18 (left) and ResNet50 (right) trained on ISIC2019 and Bone Age. Compared filter-CAVs patternCAA has higher alignment with h and invariant to feature pre- pree-roccss.ng..
Age
3SCC2009
Hone Age
0.8
Chalta Sudlako
055
Qulaae Suilbalo
0.4
0.15
3.2
0.06
010
ou
0.00
Yeno
legaitie
150ge
paller
Tano
legitle
Hige
patlocs
Lamo
pabricn
Luno
Logotic
400
palser
1000
cantered
cestered
ecadidd
Figure 23: Cosine similarity between frue concept direction he and CAVs with different pre- processing methods fitted the last Conv layer of ResNeXt50 (left) and ReXNet100 (right) trained on ISIC2019 and Bone Age. Compared to filter-AVVs, pattern-CA has a higher aligmment with hat and is invariant to feature pre-processing.
the fact that the analyzed layer in Vision Transformers a fully-connecie linear layer instead of a convolutional layer, TCAV (x) in Eq. leads scalar per sample instead of per latent pixel. To test for statistical significance, following the original TCAV method, we ran a two-sided t-test for our controlled TCAV experiment with FunnyBirds conducted in Sec. 4.3.1 Specifically, computed
31

---

Published as a conference paper at ICLR 2025
0.0
0.5
0.6
0.0
Chilte Sqalarke
Chiite Susllcit
0.4
011
logatic
palara
ielge
policcs
logatic
LRlge
X7a1
pallens
Togakie
palicss
Figure 24: Cosine similarity between true concept direction he and CAVs with different pre- processing methods fitted the last Conv layer of EfficientNet-BO (left) and EfficientNetV2 (right) trained on ISIC2019 and Bone Age. Compared to ilter-CAVs pattern-Ca CAV has a higher alignment with h and is invariant to feature pre-prooccee pre-processing.
ISEC209
Bons Age
ISFC200
Bons Age
Chain Sustaro
055
0.5
0.4
0.8
0.2
012
<kgge
patters
Taso
Tegintle
Hegg
patters
3eto legette
rAgge
patters
Vnto
Teghtke
Hkgr
patters
cantere
centreed www.-ed----aaaaaaaaa maat-eraledd
1000
costere
exstrrnc
EDAD ecalind
Figure 25: Cosine similarity between true concept direction he and CAVs with different pre- processing methods fitted on the last Conv layer Vision Transformer (left) and Swin Transformer (right) trained ISIC2019 and Bone Age. Compared filter-CAVs, pattern- alignment with h and is invariant to feature pre-processing.
pattem-CAV CAV has higher
random
concept
sample
concept
filter
pattern
wing
shape: wing-12
color: rod
wing
shape: wing-01
color: blue
part: boak
shape: beaik-01
color: yellow
Figure 26: Visualization of concept sensitivity maps, measured as element-wiss product V ( using filter- (SVM) and pattern-CAVs for three concepts on the last Conv layer ResNet18. All CAV variants detect positive concept sensitivity (red) across all spatial locations (7 pixels). This makes ResNet18 18 less susceptible to noise. Similar trends have been observed for ResNeXt50 and ReXNet100 models.
each CAV 500 times with different, randomly drawn subsets. Using a significance level 5% and applying Bonferroni correction, all TCAV scores (for all CAV types, all 10 relevant concepts) are significantly different from the random baseline score 0.5 (corresponding to ATCAV = 0), except unseen test set
for few exceptions. Moreover, collected accuracies for filter-based CAVs and found that most CAVs achieve scores above 0.9. This confirms that most -tter-CAvs do not fail in fitting generalizable decision boundary. Note, that hyperparameters for filter-CAA CAVs have been tuned using an validation set. All TCAV scores, p- p-vluuss, and accuracies for filter-CAVs on an unseen test set shown in Tab.
32

---

Published as a conference paper at ICLR 2025
(1.15)
9969
Lasso-CA
Fitter: (1.4%)
(1.1%)
(1.158)
114
Logfzic-CAV
Conv Filter: 218 (0.9%)
(0.9%)
242 (0.8%)
114 10.5%)
464 (0.8%)
507 (6 (6.8%)
Ridge CAV
Cone Fikeer 242 (1.19)
114 EL. .%9
112 (1.0%)
180 194)
507 (0.0%)
576 (C (6.8%)
SVICCA
Pasteru-CCV
Piiher (2.2%)
19.
P.09%
214 (1.9)
108 (1.5%)
507
Figure 27: RelMax visualization for neurons corresponding the largest absolute values in different CAVs, including 4 filter- (lasso, logistic, ridge, and SVM) and the pattern-CAVs, along with the Conv filter ID and the fraction of all (absolute) CAV values. While the filter-CAV picks up noisy neurons, the pattern-CAl CAV uses neurons related to the relevant concept.
(best)
80
Pattern
SVM
60
Comnt
40
0.0
0.2
0.4
0.6
0.8
1.0
Cosine Simullseity
Figure 28: Histogram over CAV quality scores (cosine similarity with ground truth) for all CRAFT directions computed via non-negative matrix factorization on the last Conv layer of VGG16 for ISIC2019 with timestamp artifact with markers for the best CRAFT direction, SVM-CAV, and pattern- CAV. The best CRAFT direction has a higher quality than SVM-CAV, but lower than pattern-CAl CAV.
D.6 ANALYSIS OF NOISE DISTRIBUTION
We further investigate the relation between the discrepancies in CAV quality found in our controlled in experiments in Section 4.2 and the issues described in Sec. 3.3 namely feature scaling and rotated noise.
Feature scaling: We plot the absolute difference between CAVs and ground truth concept direction (how much does CAV diverge?) over the variance per dimension (how varying are feature scales?). The results for VGG16, ResNet50, and EfficientNet-BB BO with our controlled artifacts ISIC2019 and Pediatric Bone Age are shown Figs. 32 33 and 34 As expected, in experiments higher variance leads to higher divergence for filter-CAVs but not for pattern-CAVs.
Noise rotation: To analyze the impact of distractor directions, we Principal Component Analysis (PCA) to find the direction with the highest within-cluster variance for latent activations of negative samples (without concept). We then computed the cosine similarity between that direction and the CAV. Results for VGG16, ResNet50, and EfficientNet- BO models trained on ISIC2019 and Bone Age datasets shown in Tab. 6. For the filter-CAV, we get cosine similarity close to
33

---

Published as a conference paper at ICLR 2025
0.7
Type
CAV Type
0.0
Pattern
Pattem
Coatre Simllaity
SVM
0.5
0.2
0.0
0000
6.0
20
30
D.6
#AArtifiet Sumples
Mistsbolling ing. Patt
Figure 29: CAV quality (cosine similarity with ground truth) averaged over 10 random seeds plotted over number of known artifact samples (left) and mislabeling rate (right) for SVM- and pattern- CAVs trained on the penultimate layer of VGG16 for ISIC2019 with artificial timestamp artifact. In addition to the fact that pattern- CAVs represent the concept direction more precisely, the cosine similarity stays consistenlyy high even with less known artifact samples and high mislabeling rate.
0.50
CAV Type
0.45
Lasso
ΔTCAV
Logiesic
Ridge
0.40
SVM
0.35
GT
0.30
VGG16
ResNet18
RenNeXt50
RoXNer100
ElficiennNes BO
NBficieotNN1122
VIT
Figure 30: ATCAV (averaged over class-defining concepts) for different CAVs fitted last Conv layers of VGG16, ResNer18, ResNeXt50, ReXNet100, EfficientNet-BO, EfficientNetV2. and the last linear layer of Vision Transformer trained FunnyBirds. As models must these concepts by experimental design, high scores are better. In contrast to filter-AAVs, pattern Pattern-CAVs CAVs achieve best scores for all models.
Efficient.NetBD
concept
sample
Iasso
logistic
ridge
SVM
pattern
logistic
ridge
pattern
part: bosk shape: boaks 01 color
wing sloper winp-c0
color: red
part: wing chaper wing-00 biue
Figure 31: Visualization of sensitivity maps, measured as element- product V (a(()) O using different filter- (lasso, logistic, ridge, and SVM) and pattern-CAY for three concepts with VGGI6 (middle) and EfficientNet-BOO (right). Results are shown for the respective Conv layer, upsampled to input space dimensions. While pattern-CAy CAVss precisely localize the concepts. filter-CAV lead to noisy sensitivity maps.
0, meaning that it orients itself orthogonal to the (non-informative) distractor direction, while the pattern-Cal CAV does not show this behavior. Similar trends can be seen in Fig. 7 Appendix [B.4] where filter-AAVs tend to orient themselves orthogonal the distractor pattern.
34

---

Published as a conference paper at ICLR 2025
1STC
Pedictric Done Age
Palleca
Filee
Pitina
ditemme bers bensttre 1300
0.55-
2.33
0..00
U00
GTems
2:30
Victaan
Vartanoo
Vastenno
Figure 32: The absolute difference between CAVs and ground truth concept direction had on the last convolutional layer of VGG16 models trained ISIC2019 with timestamp artifact (left) and Pediatric Bone Age with brightness artifact (right) plotted over the variance per feature dimension. Higher variance leads larger difference from h for filter-CAVs but not for pattem--CA CAVs.
ISTC2019
Pecistrie Bonn Age
Filex
Pallo
Filte
dihereeeenn them trw EWN
0.05
CAY
11.0
2.33
To--iBeeaa eatterantii Eoee
320
0.2-
0.03
0:33
300
330
20C
200
Vacazee
Vastaan
Figure 33: The absolute difference between CAVs and ground truth concept direction he on the last convolutional layer of ResNet50 models trained on ISIC2019 with timestamp artifact (left) and Pediatric Bone Age with brightness artifact (right) plotted over the variance per feature dimension. Higher variance leads to a larger difference from h for filter-CAVs but not for pattern-CAt CAVs.
Peciadri Bour
Filter
Porter
Filler
Picten
0.15
0.15
0.00
Ale dfforce Doe tre CAY
6.10-
160
300
000
tusk-
Abx diferce
0.82
0.22
0.90
50
35
50
Varlacoe
Variane
Varlacee
Vic
Figure 34: The absolute difference between CAVs and ground truth concept direction h on the last convolutional layer of EfficientNet-BD BO models trained on ISIC2019 with timestamp artifact (left) and Pediatric Bone Age with brightness artifact (right) plotted over the variance per feature dimension. Higher variance leads larger difference from ha for filter-CAVs but not for pattern CAVs.
D.7 MODEL CORRECTION WITH RR-CLARC
Model correction is performed with RR-CIAIC for 10 epochs with the initial training learning rate (see Table 4 divided by 10. To balance between classification loss and the added loss term LRR, weigh the latter term with {10$, 10s 1020). The parameter is picked on the validation set and selected À values for all model correction experiments are shown in Tab. 7
The results for our controlled datasets (Bone Age and ISIC2019) including standard errors are shown in Table 8 Moreover, Tab. presents the model correction results for all artifacts ("band-aid", "ruler", and "skin marker"). Pattern-CAt consistently yield better scores for artifact sensitivity, i.e., low artifact relevance and ATCAVS after model correction.
35

---

Published as a conference paper at ICLR 2025
BR--CIACC (Euss))
BR-CCAAC (Logitie))
BR-CIAIC (Ridge)
RR-CIArC (8VM)
RR-CIA-C (Puicern)
Vasila
Differenee
Dilkoaace
Dileemeee
Differezee
Figure 35: Additional qualitative results for model correction w.r.t real artifacts band-aid (top swo), ruler (middle two), and skin marker (bottom two) in ISIC2019 using VGG16. In addition to attribution heatmaps for models corrected with filter- (lasso, logistic, ridge, and SVM) and patternCAAVs we show heatmaps highlighting the differences compared to the Vanilla model attribution heatmap, with red and blue indicating higher and lower relevance after correction, respectively. Whereas filte-CAVs have limited impact, pattem--AA -AVs successfully increases the relevance the mole and decreased the relevance on data artifacts.
Fig. 35 presents additional relevance heatmaps after model correction w.r.t. the real ISIC2019 artifacts for all CAV variants and their difference heatmap compared with the Vanilla model.
36

---

Published as a conference paper at ICLR 2025
Table 5: TCAV scores, p-values and accuracies for all CAV types on VGG16, ResNet18, ResNeX150, RoXNet100, EfficientNet-BO, EfficientNetV2 and Vision Transformer (ViT) models using the con- trolled FunnyBirds dataset. Bold p-values indicate that ATCAV scores are not significantly different from O using significance level of 5% and applying a Bonferroni correction.
Concept
GT
Lasso p-val
Logistic
Ridge
SVM
Signal TCAVV
TCAV
p-val
TCAV
ncc
TCAV
p-val
acc
TCAV
p-val
acc
TCAV
p-val
ncc
p-val
heuk01::yellow
0.25
0.066
0.96
0.0
0.96
0.48
0.96
0.50
0.0
0.96
bcak02::ycllow
0.5
0.0
0.35
0.001
0.98
0.50
0.0
0.97
0.25
0.192
0.98
0.49
0.0
0.98
0.5
0.D
beakf3::yellow
0.5
0.0
0.33
0.040
0.98
0.50
0.0
0.98
0.02
0,942
0.96
0.50
0.0
0.98
0.5
0.0
VOG16
heukO4:yelloow
0.45
0.99
0.457
0.98
0.38
0.98
0.015
wingOl
0.5
0.0
0.50
0.0
0.94
0.44
0.0
0.94
0.50
0.0
0.94
0.36
0.0
0.99
0.5
0.0
wingill: :green
0.5
0.49
0.97
0.0
0.97
0.50
0.98
0.49
0.0
0.5
0.0
wingfl rod
0.5
0.49
0.0
0.95
0.40
0.005
0.91
0.49
0.0
0.96
0.27
0.0
1.00
0.5
0.0
wingl12-bblue
0.5
0.0
0.50
0.0
0.97
0.48
0.0
0.99
0.50
0.0
0.98
0.50
0.0
0.99
0.5
0.0
wingi12: -green
0.5
0.0
0.50
0.0
0.96
0.50
0.0
0.98
0.50
0.98
0.50
0.0
0.99
0.5
0.0
wingt22-red
0.5
0.0
0.50
0.98
0.50
0.0
0.9%
0.49
0.0
0.98
0.45
0.0
0.99
0.5
0.0
byyel1:
0.5
0.0
0.5
0.0
0.48
0.5
0.0
0.90
0.5
0.0
0.78
0.5
0.0
0.97
0.5
0.0
bcak02::yclloww
0.5
0.98
0.0
0.93
0.5
0.98
0.5
beakfS:ycllow
0.5
0.0
0.5
0.0
0.98
0.5
0.0
0.9%
0.5
0.0
0.98
0.5
0.0
0.99
0.5
0.D
ResNetts
beakO4:yelloow
0.5
0.0
0.5
0.0
0.95
0.5
0.0
0.98
0.5
0.0
0.95
0.5
0.0
0.98
0.5
0.0
wing0l :blue
0.5
0.80
0.0
0.5
0.87
0.5
0.0
wingtl: ggreen
0.5
0.0
0.5
0.0
0.98
0.5
0.0
0.97
0.5
D.D
0.95
0.5
0.0
1:00
0.5
0.D
wing01 trod
0.5
0.0
0.5
0.0
0.98
0.5
0.0
0.94
0.5
0.0
0.98
0.5
0.0
0.99
0.5
0.0
wingli2-bbiae
0.5
0.94
0.0
0.93
0.5
0.95
0.0
wingU2: grreen
0.5
0.0
0.5
0.0
1.00
0.5
0.0
0.9%
0.5
D.D
1.00
0.5
0.0
1.000
0.5
0.D
wing02.-red
0.5
0.0
0.5
0.0
0.98
0.5
0.0
0.97
0.5
0.0
0.97
0.5
0.0
0.99
0.5
0.0
::yclo:
0.5
0.0
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
0.98
0.5
0.0
0.99
0.5
0.0
heskf22:yellow
0.5
0.0
0.5
0.0
0.97
0.5
0.0
1.00
0.5
0.0
0.97
0.5
0.0
1.00
0.5
0.0
byck03::yclloww
0.5
0.98
0.0
0.99
0.99
RosNoXX55
beuk04::yellow
0.5
0.0
0.5
0.0
0.93
0.5
0.0
0.96
0.5
0.0
0.92
0.5
0.0
0.99
0.5
0.0
wing0l
0.5
0.0
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
winglll-gream
0.5
0.0
0.98
0.0
wingOl::red
0.5
0.0
0.5
0.0
0.93
0.5
0.0
0.92
0.5
0.0
0.94
0.5
0.0
0.99
0.5
0.0
wing022-bluee
0.5
0.0
0.5
0.0
0.98
0.5
0.0
1.00
0.5
0.0
0.98
0.5
0.0
1.00
0.5
0.0
wingliZ- grezn
0.5
0.95
0.0
0.9%
0.96
0.0
wingt22:rod
0.5
0.0
0.5
0.0
0.90
0.5
0.0
0.99
0.5
0.0
0.98
0.5
0.0
1.00
0.5
0.0
1eyell:
0.5
0.0
0.5
0:98
0.5
0.0
0.90
0.5
0.0
0.98
0.5
0.0
0.99
0.5
0.0
bycl022
0.5
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.98
0.5
0.0
0.99
0.5
0.0
beyk03::yelloww
0.5
0.0
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
1oXNec100
beuk04::yellow
0.5
0.0
0.5
0.0
0:97
0.0
0.98
0.5
0.0
0.97
0.5
0.0
0.98
0.5
0.0
wing01 -bluue
0.5
0.5
0.99
0.5
0.0
0.95
0.5
0.99
0.5
0.0
1.00
0.5
0.0
winglll:green
0.5
0.0
0.5
0.0
1.00
0.5
0.0
0.99
0.5
0.0
1.00
0.5
0.0
1.00
0.5
0.0
wingil iire
0.5
0.0
0.5
0.0
0.99
0.0
0.99
0.5
0.0
1.00
0.5
0.0
0.99
0.5
0.0
bblue
0.5
0.99
0.0
0.99
0.5
0.99
0.5
0.0
0.99
wingd2:: ggrecn
0.5
0.0
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
0.99
0.5
0.0
1.00
0.5
0.0
winggl22rred
0.5
0.0
0.96
0.5
0.99
0.0
beyk01::yellow
0.5
0.0
0.43
0.0
0.92
0.44
0.0
0.86
0.31
0.0
0.93
0.44
0.0
0.97
0.5
0.D
bcakf2::ccllow
0.5
0.0
0.18
0.124
0.98
0.33
0.0
0.99
0.22
0.038
0.99
0.36
0.001
0.99
0.5
0.0
ElBiccnnNee- :Bo
beak03::yeloow
0.32
0.99
0.0
0.46
0.98
0.0
heuk04::elllow
0.5
0.0
0.48
0.0
0.98
0.49
0.0
0.99
0.44
0.0
0.97
0.49
0.0
0.99
0.5
0.0
wing0t -bluc
0.5
0.0
0.43
0.0
0.98
0.50
0.0
0.99
0.46
0.0
0.99
0.50
0.0
0.99
0.5
0.0
Logreen
0.50
0.50
0.0
0.99
0.48
0.99
winglil:rrd
0.5
0.0
0.47
0.0
0.99
0.50
0.0
0.99
0.48
0.0
0.99
0.50
0.0
0.99
0.5
0.0
wingd22 bblue
0.5
0.0
0.39
0.007
0.79
0.50
0.0
0.98
0.34
0.0
0.99
0.50
0.0
1.00
0.5
0.0
srregti2:grreen
0.49
0.71
0.0
0.99
0,50
0.96
0.50
0.0
0.99
wing(22..rod
0.5
0.0
0.50
0.0
0.49
0.50
0.0
0.98
0.38
0.0
0.97
0.50
0.0
1.00
0.5
0.0
bcak01 :yycloo
0.5
0.50
0.0
0.70
0.50
0.87
0.50
0.0
0.98
beak02::yellow
0.5
0.0
0.30
0.012
0.93
0.50
0.0
0.97
0.37
0.001
0.93
0.50
0.0
0.98
0.5
0.0
heakf3::yeloow
0.5
0.0
0.46
0.0
0.98
0.49
0.0
0.99
0.46
0.0
0.98
0.49
0.0
0.99
0.5
0.0
Emicienteeris
byyl04::ylllow
0.49
0.0
0.44
0.97
0.0
wingOl -blue
0.5
0.0
0.50
0.0
0.98
0.50
0.0
0.96
0.50
0.0
0.99
0.50
0.0
1.00
0.5
0.0
wingül::gres
0.5
0.0
0.45
0.0
1.00
0.50
0.0
1.00
0.48
0.0
1.00
0.50
0.0
1.00
0.5
0.0
wingf]l::ed :nrdd
0.50
0.99
0.0
0.50
0.99
wingl22-blue
0.5
0.0
0.48
0.0
0.93
0.50
0.0
0.94
0.50
0.0
0.92
0.50
0.0
1.00
0.5
0.0
wingU2: grren
0.5
0.0
0.50
0.0
0.51
0.50
0.0
0.99
0.47
0.0
0.99
0.50
0.0
0.99
0.5
0.0
wingt22:red
0.48
0.99
0.0
0.11
0.98
bcyk01::ccloow
0.5
0.0
0.5
0.0
0.50
0.5
0.0
0.92
0.5
0.0
0.81
0.5
0.0
0.95
0.5
0.0
beuk02:yellow
0.5
0.0
0.5
0.0
0.94
0.0
0.97
0.5
0.95
0.5
0.0
0.98
0.5
0.0
0.5
0.95
0.0
0.97
0.5
0.95
0.5
0.0
bcylo4:
0.5
0.0
0.5
0.0
0.97
0.5
0.0
0.96
0.5
0.0
0.98
0.5
0.0
0.99
0.5
0.0
wing01-blas
0.5
0.0
0.5
0.0
0.94
0.5
0.0
0.99
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
wingt1 Iggron
0.5
0.0
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
wing t-red
0.5
0.0
0.5
0.0
0.94
0.5
0.0
0.98
0.5
0.0
0.97
0.5
0.0
0.99
0.5
0.0
wingd2:
0.5
0.5
0.0
0.5
0.97
0.0
wingt2: groon
0.5
0.0
0.5
0.0
0.95
0.5
0.0
0.95
0.5
0.0
0.96
0.5
0.0
0.99
0.5
0.0
0.5
0.0
0.5
0.0
0.97
0.5
0.0
1.00
0.5
0.0
0.98
0.5
0.0
1.00
0.5
0.0
37

---

Published as a conference paper at ICLR 2025
Table 6: Cosine similarity between CAVs and -on-inffrmmttve) distractor direction computed as direction with highest within-cluster variance for activations of negative samples (without concept) using PCA. Filter-CAVs orient themselves orthogonal to the distractor direction (i.e., cosine similarity close to 0), while pattern-CAVs do not show this behavior.
Cosine Similarity
Dataset
Model
Filter-CAV
Pattern-CAV
VGG16
-0.036
0.536
ISIC (timestamp)
ResNet50
0.048
0.155
EfficientNet-B0
-0.008
0.041
VGG16
-0.003
0.169
Bone (brightness)
ResNet50
0.005
0.076
EfficientNet-BO
0.005
-0.057
38

---

Published as a conference paper at ICLR 2025
Table 7: Selected À values for model correction with RR-CIArC weight for the added loss for experiments with VGGI6, ResNet18, ResNet50, ResNeXt50, ReXNet100, Efficien(Net-BO. EfficiemtNetV2, and ViT with Bone Age (controlled), ISIC2019 (controlled), and ISIC2019 (real artifacts). For the latter, we run separate corrections w.r.t. the "band-aid" (BA), "ruler" (R), and "skin marker" (SM) artifacts.
Bone Age (controlled)
ISIC2019 (controlled)
ISIC2019 (BA (BA (|SM)
model
CAV
Lasso
102
107
10"110"110"
Logistic
10"
107
10 10"110"110"
VGG16
Ridge
10"
107
10 10"110"110"
SVM
10"
107
10" 10"100
Pattern
107
107
10
10"
Lasso
10a
107
10 10"110 110
Logistic
10
10a
100 10 10
ResNet18
Ridge
10
105
10 10%110%110
SVM
10
107
10
Pattern
10a
10
10
Lasso
10a
1020
10"
Logistic
10
10
10
ResNet50
Ridge
10
10a
10"110"100
SVM
10
10a
10"110"10"
Pattern
10
10a
10
100|10"|100
Lasso
10
10a
10 10
Logistic
10a
10a
10 10
ResNeXt50
Ridge
10a
10a
10 10
SVM
1040
108
10 10
Pattern
10
10a
10" 110
Lasso
108
109
10
Logistic
10
107
10"100"100
RcXNet100
Ridge
10
10
10"
SVM
10
10a
10
Pattern
10
107
10
1 10 110
Lasso
10a
10a
100
Logistic
10
1000
10s 10
Efficient Net-BO
Ridge
10a
10
10
10"110$110
SVM
10t0
10a
10"
10
Pattern
10
10a
10"110"
Lasso
107
10a
10"
Logistic
10a
107
10
EfficientNNet22
Ridge
10c
10a
10"110"110"
SVM
108
10a
10"
Pattern
10
10a
10
10
Lasso
10
10
10
10
Logistic
10a
10a
10
10 10a
ViT
Ridge
10a
107
10
10
SVM
10
10a
10
10 10
Pattern
104
10a
10 10
39

---

Published as a conference paper at ICLR 2025
Table 8: Results after model correction with RR-CIAIC for VGG16, ResNet18/50, ResNeXt50, ReXNet100, EfficientNet-BD EfficientNetV22, and ViT trained on Bone Age ISIC2019 (controlled) including standard errors. We report accuracy clean and biased test set, the fraction of relevance put onto the data artifact region for localizable artifacts, and the TCAV score (reported as ATCAVS) using the sample-wis ground-truth concept direction ha measuring the models' sensitivity towards the artifacts after model correction. Stars indicate statistical significance according to z-tests with significance level of 0.05, and arrows whether low (4) or high (f) are better.
model
CAV
Accurcy (clean)
Accuracy (biased)
Artifact relevance
ATCAV 4
Vinilla
0.01
0.82 0.01
0
0.28 + 0.01
0.62 + 0.01
0.29 + 0.00
0.14 0.02
VGG-16
lasso
0.00
0.82 1.2..000 0.01
0.55 0.001
0.30 100110000 1.0.1
0.60 0.01
0.25 1 0.00
0.13 1 0.02
logistic
0.01
0.00
0.00
0.37 0.01
0.54
0.25 +:
0.07 : 0.02"
ridge
0.71 0.01
0.82 0.01
0.61 0.01
0.31 0.01
0.59 0.01
0,21 0.00
0.13 0.02
0.00
0.55 0.01
0.24
Patter
0.78 0.01
0.80 + 0.01
0.75 0.01
0.69
0.01
0.26 0.01
0.14
0.10 0.02
Vanilla
0.75 0.01
0.84 0.01
0.46
0.43 0.01
0.31 0.31 0.01
0.50 1: 0.00
0.50 0.00
ResNet-18
lasso
0.76 0.01
0.83 0.01
0.55 0.00
0.51 1: 0.01
0.27::00. 0.01
0.08 1: 0.00
0.01 0.02
logistic
0.76 0.01
0.83 1:83::000 0.01
0.55 0.00
0.59 0.01
0.25 0.01
0.07 1: 0.00
0.01 0.02
ridge
1.00
0.83 1.001
0.57 0.00
0.51 1: 0.01
0.27
0.00 0.000
0.00 0.02
SVM
0.06
0.83 4.00 0.01
0.54
0.57 0.01
0.26
0.07 0.00
0.01 0.02
Pattern
0.01
0.01
0.00
0.66 0.01
0.02
0.22
0.05 0.02
Vanilla
0.77
0.85 0.00
0.00
0.51
0.01
0.46 0.01
0.14 0.00
0.04 0.02
ResNet50
lasso
0.77 0.01
0.01
0.53
0.58
0.01
0.11 0.01
0,03 = 0.00
0.002 0.01
logistic
0.77
0.85 0.00
0.55 0.001
0.69 0.01
0.399.0.00
0.04 1 0.00
0.05 0.02
ridge
0.00
0.84 0.00
0.52
0.58 0.00
0.45 0.01
0.13 + 0.00
0.03 0.01
SVM
0.77
0.85 0.00
0.55 0.001
0.68 0.00
0.04 1
0.04 0.02
Pattern
0.78 0.01
0.84 0.01
0.59 + 0.01
0.71 + 0.01
0.37 +: 0.01
0.01 + 0.00
0.03 0.02
Vanilla
0.08
0.86
0.50
0.15
0.01
0.00
0.12 0.00
RasNeXt00
lasso
0.80 0.01
0.84
0.57 0.01
0.64 + 0.01
0.56 1.56+00 0.01
0.01 = 0.00
0.01
0.01
logistic
0.80
0.00
0.73 0.00
0.52 0.01
0.03 0.00
0.08 0.02
ridge
0.00
0.81
0.57
0.63
0.56 1:66:00000
0.06 0.00
0.01 0.01
SVM
0.78 1.001
0.85
1. 0.01
0.54
0.69 0.01
0.54 0.01
0.06 0.00
0.09 0.02
Puttern
0.79 0.01
0.85
2 0.01
0.01
0.75
0.01
0.49 0.01
0.45 0.00
0,03 0,02
Vanilla
0.76 1: 0.01
0.88
0.01
0.47 147
0.71 1: 0.01
0.22 0.01
0.29 1: 0.01
0.50 0.00
RoXNec-100
0.77
0.88
0.00
0.73
0.23
logistic
0.77 0.00
0.88 0.00
0.47
0.74 0.01
0.001
0.24 1: 0.01
0.17 0.03
ridge
0.88 0.77
0.00
0.74
0.02
0.27 0.01
0.03
SVM
0.07
0.88 0.01
0.17
0.71
0.22
0.26 = 0.01
0.16 0.03
Pattern
0.76 0.01
0.88 0.01
0.57 0.01
0.78 0.01
0.20:0.00
0.22 0.01
0.05 0.04
Vanilla
0.79
0.87
0.01
0.46
0.53 0.01
0.00
Efficient
Net.0 B30
logistic
lasso
0.77 0.01
0.79 1. 0.01
0.86 0.01
0.85
0.70
0.64 0.01
0.51
0.52 1.52.00001
0.01 1 0.00
0.02
0.11 0.03
ridge
0.78 0.01
0.02
0.74
0.67 0.01
0.52 0.01
0.12 0.01
0.77
0.85
0.75 0.01
0.52
0.00 0.00
Pattern
0.77 0.01
0.80+
0.75 0.751
0.72 0.01
0.00
0.00 0.00
0.05
Vanilla
0.01
0.86 1:6::0000 0.01
0.46
0.51 0.01
0.36 1:66::0000
0.25 2 0.00
U.32 0.03
Eliicient
Na
logistic
lasso
0.78 0.01
0.78 0.01
0.85 0.01
0.01
0.06
0.73
0.67 1 0.01
0.31 0.01
0.35
0.05 0.00
0.03 0.00
0.04 0.02
ridge
0.85
0.00
SVM
0.78 0.01
0.85
0.01
0.001
0.66 0.01
0.34 0.01
0.00 + 0.00
0.04 0.02
Puttern
0.01
0.85
0.71
0.32
Vanilla
0.03
0.001
0.38
0.67
0.01
0.00
0.47 + 0.02
0.25 0.22
lasso
0.71 0.01
0.88 0.00
0.00
0.67 0.001
0.16::000
0.42 1
0.25 0.22
VIT
logistic
0.00
0.00
0.02
0.72 0.01
0.16
0.02
0.25 0.22
ridge
0.71 0.01
0.88 0.00
0.66 0.00
0.16
0.12 = 0.05
0.25 : 0.22
0.73
0.00
Puttern
0.74 0.01
0.88 1. 0.01 0.01
0.61 1 0.01
0.73 1.3.
0.16.10.00
0.06 L 0.06
0.25 0.22
40

---

Published as a conference paper at ICLR 2025
Table 9: Results for VGG16, ResNet18, ResNet50, ResNeXt50, ReXNet100, EfficientNet-BO. EfficientNetV2, and Vision Transformer trained on ISIC2019 after model correction with RR-CIAIC w.r.t. to the real artifacts "aandaaid"""uule"""" test set, the fraction of relevance put onto the data artifact region for localizable artifacts, and the TCAV score (reported as ATCAV using the sample-wise ground-truth concept direction h measuring the models' sensitivity towards the artifacts after model correction. Note, that we artificially insert artifacts using estimated localization masks to create biased test set and to compute hat Stars indicate statistical significance according to z-tests with significance level of 0.05, and arrows whether low (4) or high (f) are better.
marker". We report accuracy on clean and biased
model
CAV
Accuracy (clean)
Accuracy (biased) 1
Artifact relevance
ATCAV" 4
Vanilla lasso
0.83
0.83
0.83
U.75
0.72
0.75
0.51
0.32
0.23
0.10
U.07
0.04
VGG-16
0.81
0.82
0.82
0.77
0.77
0.75
0.48
0.25
0.22
0.12
0.07
0.05
logistic
0.82
0.81
0.81
0.78
0.75
0.75
0.46
0.28
0.22
0.09
0.07
0.05
ridge
0.81
0.82
0.82
0.76
0.77
0.75
0.49
0.26
0.22
0.12
0.07
0.05
SVM
0.82
0.82
0.81
0.78
0.74
0.75
0.46
0.28
0.22
0.11
0.07
0.05
Pattern
0.82
0.82
0.82
0.79
0.79"
0.75
0.31"
0.188'
0.18"
0.03'
0.06
0.05
Varilla
0.85
0.85
0.85
0.79
0.83
0.80
0.22
0.13
0.15
0.32
0.18
0.02
ResNer18
lasso
0.85
0.85
0.85
0.80
0.83
0.79
0.18
0.13
0.15
0.06
0.18
0.03
logistic
0.85
0.85
0.85
0.81
0.83
0.79
U.17
0.11
0.14
0.05
0.03
0.02
ridge
0.85
0.85
0.85
0.80
0.83
0.79
0.18
0.11
0.15
0.06
0.03
0.03
SVM
0.85
0.85
0.85
0.81
0.83
0.79
0.18
0.12
0.15
0.05
0.03
0.02
Pattern (ours)
0.85
0.84
0.84
0.81
0.83
0.79
0.16"
0.11
0.14
0.09
0.05
0.04
Vavilla
0.87
0.87
0.87
0.82
0.84
0.81
0.34
0.14
0.19
0.27
0.03
0.07
ResNet50
lasso
0.87
0.87
0.87
0.82
0.85
0.81
0.30
0.12
0.19
0.05
0.03
0.07
logistic
0.87
0187
0.87
0.83
0.85
0.81
0.24
0.13
0.18
0.05
0.03
0.07
ridge
0.87
0.87
0.87
0.82
0.85
0.81
0.30
0.12
0.18
0.05
0.03
0.02
SVM
0.87
0.87
0.87
0.83
0.85
0.81
0.26
0.13
0.18
0.05
0.03
0.08
Pattern (ours)
0.87
0.87
0.87
0.83
0.86
0.81
0.22
0.12
0.17
0.04
0.03
0.02
Vanilla
0.87
0.87
0.87
0.82
0.85
0.80
0.37
0.16
0.24
0.06
0.03
0.05
ResNeXt50
lasso
0.87
0.87
0.87
0.82
0.85
0.80
0.34
0.15
0.23
0.06
0.04
0,05
logistic
0.87
0.86
0.87
0.83
0.85
0.80
0.30
0.15
0.22
0.06
0.04
0.05
ridge
0.87
0..6
0.87
0.82
0.85
0.80
0.34
0.14
0.22
0.07
0.04
0.05
SVM
0.87
0.86
0.87
0.83
0.85
0.81
0.31
0.15
0.23
0.06
0.04
0.05
Pattern (ours)
0.87
0.86
0.87
0.83
0.86
0.81
0.27'
0.14
0.23
0.04
0.03
0.04
Vanilla
0.88
0.88
0.88
0.82
0.86
0.83
0.21
0.10
0.14
0.16
0.11
0.11
ReXNet100 00
lasso
0.88
0.88
0.88
0.82
0.86
0.83
0.21
0.10
0.14
0.33
0.11
0.01
logistic
0.88
0.88
0.88
0.82
0.86
0.83
0.21
0.10
0.14
0.34
0.11
0.01
ridge
0.88
0.88
0.88
0.82
0.86
0.83
0.21
0.10
0.14
0.33
0.11
0.01
SVM
0.88
0.88
0.88
0.82
0.86
0.83
0.21
0.10
0.14
0.33
0.11
0.01
Pattern (ours)
0.88
0.88
0.88
0.82
0.86
0.83
0.19*
10.08"
0.13
0.30
0.04'
0.01
Vanilla
0.88
0.88
0.88
0.83
0.86
0.83
0.22
0.08
0.11
0.12
0.04
0.02
Efficient Net-BO
lasso
0.88
0.88
0.88
0.83
0.86
0.83
0.22
0.08
0.11
0.11
0.03
0.02
logistic
0.88
0.88
0.88
0.83
0.86
0.83
0.22
0.08
0.11
0.12
0.03
0.02
ridge
0.88
0.88
0.88
0.83
0.86
0.83
0.22
0.08
0.11
0.12
0.04
0.02
SVM
0.88
0.88
0.88
0.83
0.86
0.83
0. 22
0.08
0.11
0.11
0.04
0.02
Pattern (ours)
0.88
0.88
0.88
0.83
0.86
0.82
0.22
0.08
0.11
0.03"
0.02
0.02
Varilla
0.89
0.89
0.89
0.85
0.86
0.83
0.22
0.08
0.12
0.12
0.16
0.10
lasso
0.89
0.89
0.89
0.85
0.86
0.84
0.22
0.08
0.12
0.12
0.14
0.09
Lificientt Ne
logistic
0.89
0.89
0.89
0.85
0.86
0.84
0.22
0.08
0.12
0.12
0.16
0.10
ridge
0.89
0.89
0.89
0.85
0.87
0.83
0.22
0.08
0.12
0.12
0.14
0.07
SVM
0.89
0.89
0.89
0.85
0.86
0.83
0.22
0.08
0.12
0.12
0.17
0.10
Pattern (ours)
0.89
0.89
0.80
0.85
0.86
0.83
0.22
0.08
0.12
0.09*
0.09
0.09
Varilla
0.89
0.89
0.89
0.78
0.83
0.82
0.22
0.11
0.10
0.18
0.14
0.05
lasso
0.89
0.89
0.89
0.78
0.84
0.82
0.22
0.10
0.10
0.18
0.12
0.06
VIT
logistic
0.89
0.89
0.89
0.78
0.85
0.82
0.22
0.09
0.10
0.15
0.07
0.04
ridge
0.89
0.89
0.89
0.78
0.84
0.82
0.22
0.10
0.10
0.19
0.11
0.06
SVM
0.89
0.89
0.89
0.78
0.83
0.82
0.25
0.11
0.10
0.25
0.15
0.04
Pattern (ours)
0.89
0.89
0.89
0.78
0.85
0.82
0.22
0.08
0.10
0.16
0.001
0.11
41

---

Published as a conference paper at ICLR 2025
D.8 ADDITIONAL BONE AGE EXPERIMENTS
Complementing the experiments with the artificial brightness, we considered two additional artifacts in the Bone Age dataset. Specifically, insert artificial (grayscale) timestamp artifact into 20% of samples of exactly one class during training. Moreover, we consider a real-world artifact occurring in the Bone Age dataset: Images are scaled such that all hands are of similar size, leading to larger "L"-marker for hands of younger children, because the images needed larger scaling factor due to smaller hands. In both settings, we train VGG16, ResNet1 8/50, EfficientNNe--B and EfficientNet- V2 models. In Tab. TO we report the accuracy on the clean test set and the artifact relevance in both settings, as well as the accuracy on biased test set and ATCAV in the controlled experiment. For the timestamp artifact, the Pattern-CAY outperforms filter-based CAVs both in terms of accuracy on the biased data and artifact relevance by a large margin across all architectures, while maintaining high accuracy the clean data. For the real-world artifact, the bias mitigation approach using Pattem-CAA successfully reduces the artifact relevance by a large margin for all architectures.
In addition, similar the qualitative analysis in Section most important neurons for different CAVs for VGG16 model trained on Bone Age. We use CAVs representing the real-world "L" ma"mmrrker artifact (Fig. (36) and the artificial timestamp artifact (Fig. 37). The same trends as with ISIC2019 (see Fig 27) cab be observed. Specifically, while Filter-CAVs have high values for irrelevant noisy neurons, Pattem-CAVs have less uniform distribution neurons, with top neurons focusing on the concept of interest, i.e., the timestamp and the "L"- -"-marerr
present RelMax visualizations for the
Table 10: Results for VGG16, ResNet18/50, and EfficientNet-B0/22 trained on Bone Age after model correction with RR-CIArC w.r.t. the artificial timestamp (left) and the "L"-marker artifact (right). We report accuracy on clean and biased test set, the fraction of relevance region, and the TCAV score (reported as ATCAVS!) using the sample-wise ground-truth concept direction h measuring the models' sensitivity to the artifact. Stars indicate statistical significance according to z-tests (significance level. 0.05), and arrows whether low (4) or high (f) are better.
onto the artifact
model
CAV
Accuracy (clean)
Accuracy (biased)
Artifact relevance +
ATCAVØ
Vanilla
0.79
0.79
0.33
0.72
0.17
0.26
lasso
0.77
0.79
0.34
0.65
0.17
0.33
VGG16
logistic
0.77
0.79
0.37
0.62
0.18
0.29
ridge
0.77
0.79
0.34
0.67
0.17
0.33
SVM
0.79
0.79
0.35
0.59
0.17
0.20
Pattern (ours)
0.78
0.78
0.70
0.30"
0.14"
0.11
Vanilla
0.77
0.76
0.44
0.36
0.19
0.03
ResNet18
lasso
0.78
0.76
0.48
0.32
0.18
0.06
logistic
0.78
0.77
0.57
0.28
ridge
0.77
0.54
0.30
0.17
SVM
0.77
0.76
0.59
0.28
0.18
0.02
Pattern (ours)
0.77
0.75
0.62*
0.25
0..55
0.08
Vanilla
0.79
0.78
0.49
0.48
0.24
0.03
ResNer50
lasso
0.80
0.78
0.59
0.41
0.24
0.02
logistic
0.80
0.79
0.70
0.35
0.23
0.01
ridge
0.80
0.78
0.66
0.38
0.24
0.02
SVM
0.80
0.79
0.68
0.37
0.24
0.01
Pattern (ours)
0.80
0.78
0.72
0.33
0.17'
0.02
Vanilla
0.78
0.79
0.39
0.61
0.36
0.24
lasso
0.78
0.79
0.10
0.61
0.36
0.24
Efficient
Net-B0 BO
logistic
0.78
0.78
0.47
0.58
0.36
0.21
ridge
0.78
0.79
0.40
0.61
0.36
0.24
SVM
0.78
0.46
0.59
0.36
0.21
Pattern (ours)
0.78
0.71
0.64
0.48
0.22*
0.05
Vanilla
0.80
0.80
0.43
0.44
0.18
Efficient NetV2
lasso
0.80
0.80
0.43
0.44
0.18
0.08
logistic
0.80
0.80
0.45
0.44
0.18
0.06
ridge
0.80
0.80
0.43
0.41
0.18
0.08
SVM
0.80
0.80
0.53
0.38
0.18
0.07
Pattern (ours)
0.81
0.77
0.62
0.35
0.18
0.27
42

---

Published as a conference paper at ICLR 2025
Corry Rlcce 47 10.9!
424 (0.95)
200 (0.8%)
334 (5.751
365 (0.7%]
275 10.786)
Lono CAV
Om
Conty Riter 49 11.0%!
301 10 995)
9G (0.7%4
207 6OL0%I
362 (0.0%)
275 10.635)
Lngeti--C. CAV
Conv Rieer 47 11.0%]
124 10 9%5
200 C3 0%%
3344 C5 7%61
365 (D. TN1
271 10.7%
HAAg CAY
0th 0
Corv Ribrr 68 10.8%)
101 10.8%)
142 C3. 8%6
211 (9.7%)
DC ID.7%5
(0.05
SVA-CIA
Come Filer: 133 (3.5%)
(2.3%)
233 (1.150
219 (3.0%)
160 13.0
124 12.0%%
ELC
IMJ
Figure 36: RelMax visualization for neurons corresponding to the largest absolute values in different CAVs, including 4 filter- (lasso, logistic, ridge, and SVM) and the pattern-CAs CAVs, along with the Conv filter ID and the fraction of all (absolute) CAV values for the real-world "L" "L"-marker artifact in the Bone Age dataset using VGG16 model. While the filter-CAV picks up noisy neurons, the Cnt uses neurons related to the relevant concept.
Conv Fiten 504 (3:8))
260 0.38)
424 22 8%)
17 12 265
507 12.59)
434 12:2853
Lawro
Conv Fittr: 507 11 591
327 (1.25)
(03(1.255
504 11.191
448 11.0%3
242 11.0%
Conv Tile:: 260 (1.78)
504 (3 (5.95)
17 (2.0%)
337 (3.9%)
403 (2.05)
10B (0.0%)
E4 CAV
Cary Nile:: 242 11.0%)
579 (0,255)
203 (3. 0%5
5C4 (3.9%5]
220 (A.8%)
348 (0.8%
SVARGAY
Corr Fitu:: 597 (338))
527 (3.256)
5o4 (2.4%)
448 (2.3%)
280 (2.18)
192 (1.8)
01166
Figure 37: RelMax visualization for neurons corresponding to the largest absolute values in different CAVs, including 4 filter- (lasso, logistic, ridge, and SVM) and the pattern-CAVs. along with the Conv filter ID and the fraction of all (absolute) CAV values for the artificial timestamp artifact in the Bone Age dataset using a VGG16 model. While the filter-CAV picks up noisy neurons, the pattern-CAI uses neurons related to the relevant concept.
43

---

Published as a conference paper at ICLR 2025
D.9 ADDITIONAL IMAGENET AND CELEBA EXPERIMENTS
We conducted additional bias mitigation experiments with a natural spurious correlation in CelebA (Liu al. 2015) and artificial artifact in ImageNet (Deng et al. 2009). For the for- mer, we study the negative correlation between the presence of ties and blonde hair for a hair color predictor, caused by the existence many For the latter, we insert an artificial timestamp into 50% of samples of class "tench" (n014 4 44076 64) and finetune pre-trained models for epochs. To amplify the impact of the artifact, we further insert the artifact into 0.5% of samples from other classes as a backdoor by flipping the label to "tench". On both datasets, we train VGG16, ResNet18, ResNet50, EfficientNet-BBO and EfficientNet-V2 models and report bias mitigation results with RR-CIArC in Tab. 11 For all architectures, Pattern-CAVs outperform Filter-CAt in terms accuracy on the biased test set for ImageNet. Moreover, Pattern- CAVs achieves superior artifact relevance and ATCAV for all architectures except for ResNet18. where results are similar to those for Filter-CAVs.
men wearing suits (with ties) in the dataset.
Table 11: Results for VGGI6, ResNet18/50, EfficientNet-BO and EfficientNNt12 after model cor- rection with RR-CIArC for ImageNet with the artificial timestamp artifact (leff) and CelebA with the real-world -art-fatt (right). We report accuracy clean and biased test set, the fraction of relevance put onto the data artifact region, and the TCAV score (reported as ATCAV using the sample-wiss ground-truth concept direction h mcasuring the models' sensitivity towards the artifacts after model correction. Stars indicate statistical significance according to z-tests with significance level of 0.05, and arrows whether low (4) high (f) are better.
model CAV
Accuracy (clean)
Accuracy (biased)
Artifact relevance |
ATCAV
Vanilla lasso
0.66
0.92
0.53
0.15
0.30
0.15
0.66
0.92
0.52
0.15
0.24
0.24
VGG16
logistic
0.65
0.92
0.58
0.13
0.21
0.13
ridge
0.67
0.92
0.53
0.13
0.22
0.22
SVM
0.65
0.91
0.58
0.14
0.23
0.16
Pattern (ours)
0.65
0.91
0.63
0.14' 0.09
0.11
Vanilla
0.66
0.93
0.55
0.12 0.25
0.32
ResNer18
lasso
0.63
0.92
0.60
0.10 0.26
0.02
logistic
0.64
0.93
0.63
0.08 0.26
0.03
ridge
0.64
0.92
0.62
0.09 0.26
0.02
SVM
0.64
0.62
0.08 0.26
0.01
Pattern (ours)
0.67
0.93
0.64
0.09
0.26
0.05
Vanilla
0.77
0.93
0.73
0.10 0.29
0.06
ResNet50
lasso
0.78
0.77
0.07 0.29
0.05
logistic
0.79
0.93
0.78
0.07 0.28
0.05
ridge
0.79
0.93
0.77
0.11
0.29
0.09
SVM
0.78
0.93
0.78
0.07
0.28
0.06
Pattern (ours)
0.78
0.93
0.78
0.05
0..27
0.01
Vanilla
0.74
0.92
0.52
0.18
0.27
0.48
lasso
0.74
0.92
0.51
0.18
0.27
0.47
Efficient
Net-B0 B0
logistic
0.75
0.92
0.55
0.17
0.27
0.47
ridge
0.74
0.92
0.51
0.18
0.27
0.48
SVM
0.75
0.92
0.53
0.18
0.27
0.48
Pattern (ours)
0.75
0.92
0.71
0.10
0.23*
0.42*
Vanilla
0.81
0.90
0.68
0.12
0.25
0.45
Efficient NetV2
lasso
0.81
0.90
0.69
0.11
0.25
0.44
logistic
0.81
0.72
0.11
0.25
0.43
ridge
0.81
0.90
0.69
0.11
0.25
0.44
SVM
0.81
0.90
0.72
0.11
0.25
0.44
Pattern (ours)
0.81
0.91
0.78
0.10
0.20"
0.23*
44


Reviews:
[{"review_id": "00cgC5MMtS", "reviewer": "Reviewer_Tafu", "summary": "In this paper, the authors propose a pattern-based concept activation vectors (CAV), which focuses on concept signal and provides accurate concept directions. Previous CAV methods are designed to compute by leveraging linear classiifers optimizing the separability of latent repsresentations of samples, which is harmful for accurately modeling the concept direction. The authors evaluate various CAV methods in terms of their alignment with the true concept direction and their impact on CAV applications.", "questions": "- Would it be possible to apply the proposed method on a large realworld dataset like ImageNet?\n- Would it be possible to add statistical significance of the results in Table 1?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to read. Related work includes several key papers.\n- CAVs is widely explored to improve the interpretability of model and this paper tackles an important issue of the current CAV studies and propose a new CAV method named pattern-based CAV.\n- The authors measure the alignment of CAVs with the true concept direction by setting controlled environment and the results show that the pattern-CAVs align with the true concept direction.\n- The authors show the impact of directional alignment/shift in testing with CAV and model correction.\n- Experimental design with real-world datasets look interesting.", "weaknesses": "- Some relevant papers are missing [R1]. It would be informative to clarify the contribution of the proposed method and discuss the pros and cons of the proposed method to better find better position of this paper.\n- The paper looks many overlaps with [R2]. It would be great to clarify the novel contribution of this study and add discussion with [R2].\n- The method is evaluated on BoneAge, ISIC2019 datasets. Evaluation on ImageNet/CelebA will make the paper more strong as in [R2]. \n\n\n[R1] Fel, Thomas, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. \"A holistic approach to unifying automatic concept extraction and concept importance estimation.\" Advances in Neural Information Processing Systems 36 (2023).\n[R2] Dreyer, Maximilian, Frederik Pahde, Christopher J. Anders, Wojciech Samek, and Sebastian Lapuschkin. \"From Hope to Safety: Unlearning Biases of Deep Models by Enforcing the Right Reasons in Latent Space.\" arXiv preprint arXiv:2308.09437 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0u4YWuA3te", "reviewer": "Reviewer_1Rm2", "summary": "Concept Activation Vectors (CAVs) is a tool in explainable AI for understanding neural network prediction strategies through human-understandable concepts in latent space. This paper argue that current CAV computation methods, which focus on class separability, often diverge from the actual goal of modeling the concept direction accurately. To address this, the authors propose \"pattern-based CAVs,\" which emphasize concept signals while disregarding distractors. They evaluate their method against various model architectures and datasets, demonstrating that pattern-based CAVs align more closely with the true concept direction and yield improved results in applications like concept sensitivity testing and model correction.", "questions": "1. how to address the unsupervised case? I see the descriptions in line 266 and Appendix, could you please provide more details, since the unsupervised is more important in real applications. \n2. In Table 1, why the results of Efficient Net-B0 are quite similar?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors clearly identify the limitations of traditional CAV methods, specifically the influence of distractors in filter-based CAVs.\n2. The authors conduct extensive experiments on multiple datasets and architectures, providing both quantitative and qualitative results that support their claims. The improved CAV method is demonstrated to benefit important applications, such as TCAV for concept sensitivity testing and ClArC for model correction.", "weaknesses": "1. My main concern is on the limited novelty, it seems that the key idea of pattern-CAV has been proposed, and the paper puts more efforts on evaluation. \n2. The pattern-based CAV method, while precise in modeling concept directions, lacks the boundary optimization and transformation capabilities of traditional linear classifiers (e.g., SVM), which may limit its robustness in complex or noisy datasets. Specifically, the absence of separation-focused optimization could lead to inconsistencies in concept direction in cases with overlapping or ambiguous boundaries.\n3. The authors focus on achieving high alignment with the true concept direction but acknowledge that in some applications (e.g., post-hoc concept bottleneck models), class-separability may be more critical. A clearer guideline on when to use pattern- vs. filter-based CAVs based on application needs would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWUkDOoa4t", "reviewer": "Reviewer_KXeP", "summary": "The paper proposed pattern-based CAVs based on Concept Activation Vectors (CAVs). The authors assume that the weight vector of the linear classifier may fail at precisely identifying the concept. To amend this, the proposed method slightly modifies the optimization objective from Eq. (1) to Eq. (2) called pattern-based CAV.", "questions": "On page 5, \"we generate pairs of samples with and without the concept\". Was this generation the same or different for different datasets? It could be clearer how a pair for a timestamped image or a bird image is generated (e.g., with examples).\n\nIn Figure 5, it seems the performance is heavily dependent on the network architecture, and the results of Logistic, SVM, and Patter, are all very similar to GT if the model is ResNet. Does this mean that if we stick to ResNet as the model, then previous methods like SVM and Logistic are already good enough?\n\nIs there any theoretical analysis of how close the vectors can be to the ground truth concepts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper performed a large number of experiments on both toy datasets and controlled datasets.", "weaknesses": "The writing and presentation make the paper hard to follow. It could be better to reorganize the paper to make it easier to understand. For example, it seems better to introduce the concept of TCAV in the Method rather than Experiment section.\n\nThe novelty of the paper seems to be somewhat limited. The main contribution Eq. (2) was proposed by a previous work. The authors made a commendable effort to design the dataset, but the technical contribution seems to be the slight modification to the CAV based on linear models.\n\nThe datasets used here mainly emphasize hand-crafted concepts (\"band-aid\", \"ruler\", \"bird\", etc.). Is it possible to evaluate the performance on real-world and more realistic datasets like ImageNet?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EFphPTYYSE", "reviewer": "Reviewer_LfDf", "summary": "The paper introduces pattern-based concept activation vectors (CAVs) that focus solely on concept signals to provide more accurate concept directions, addressing issues with distractor concept directions in filter-based CAVs. The proposed pattern-based CAVs are invariant to feature scaling and more robust to noise. \n\nThe method is evaluated across different datasets and model architectures through both controlled and uncontrolled real-world experiments.", "questions": "My primary concern is the effectiveness of pattern-based CAVs in real-world applications, including tasks with real artifacts and grayscale image classification, please refer to the weaknesses for more details.\n\nhere are some minor questions:\n\n1. A brief introduction to RelMax should be provided for better understanding.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work proposes a novel computation approach for CAVs, known as pattern-based CAVs, which are less sensitive to distractors compared to existing filter-based CAV methods.\n\nThe paper is well-written and well-structured, presenting multiple experimental results to validate the method’s effectiveness.\n\nThe authors objectively analyze the advantages, limitations, and suitable contexts for pattern-based CAVs, highlighting that while pattern-based CAVs offer more accurate concept signals, filter-based CAVs emphasize concept separability, which is important for tasks such as concept-based image classification.\n\nThis paper offers valuable insights by highlighting issues with existing filter-based CAVs in the presence of noise and feature rescaling, as well as potential challenges when using filter-based CAVs for TCAV and model explanation.", "weaknesses": "However, I have the following concerns:\n1. In Table 1, the performance improvements of pattern-based CAVs on the real-world ISIC2019 dataset are not significant, particularly in reducing artifact sensitivity. Given that artifacts (e.g., textual elements in radiology reports or image discrepancies caused by different imaging equipment across medical centers) are hard to avoid in medical contexts, I am concerned that pattern-CAVs may not outperform filter-based CAVs in practical applications.\n2. Many medical imaging modalities produce grayscale images, where lesions may be small, and inter-class visual features are often very similar (unlike in ISIC, where subtypes generally have distinct attributes like color or shape). It might be beneficial to include experiments on grayscale datasets, such as Bone Age or similar medical datasets, to better assess the effectiveness of pattern-based CAVs.\n3. In the RelMax visualization, samples with the highest neuron activation values are presented. Why do the neurons in the pattern-CAV method have higher activation values than in other CAV methods? Could the authors provide the distribution of neuron activation values across different methods and analyze it? Additionally, could they quantify each CAV method’s ability to identify samples with artifacts (e.g., showing that a certain percentage of neurons with the top activations can identify 95% of samples)?\n4. The experiments in the main text should include transformer-based architectures to evaluate the generalizability of the method across different model types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)