Revisiting Discover-then-Name Concept Bottleneck Models: A Reproducibility Study
Authors: Freek Byrman
Emma Kasteleyn
Bart Kuipers
Daniel Uyterlinde
Venue: Accepted by TMLR ()

Abstract: Concept Bottleneck Models (CBMs) (Koh et al., 2020) are a class of interpretable deep learning frameworks that improve transparency by mapping input data into human-understandable concepts. Recent advances, including the Discover-then-Name CBM proposed by Rao et al. (2024), eliminate reliance on external language models by automating concept discovery and naming using a CLIP feature extractor and sparse autoencoder. This study focuses on replicating the key findings reported by Rao et al. (2024). We conclude that the core conceptual ideas are reproducible, but not to the extent presented in the original work. Many representations of active neurons appear to be misaligned with their assigned concepts, indicating a lack of faithfulness of the DN-CBM’s explanations. To address this, we propose a model extension: an enhanced alignment method that we evaluate through a user study. Our extended model provides more interpretable concepts (with statistical significance), at the cost of a slight decrease in accuracy.

Published in Transactions on Machine Learning Research (06/2025)
Revisiting Discover-then-Name Concept Bottleneck Models: A Reproducibility Study
Freek Byrman University of Amstendam
freek. www.bb//wmaaansddddn.......
Emma Kasteleyn University of Amstendam
emw..kasttdeyn@Sttddett....uu
Bart Kuipers University of Amsteniam
hurt.Kuippers@sttuden.....uuu
Daniel Uyterlinde University of Amsterdam
dunied..ijerrinneee@tttdee uvo.n
Reviewed OpenReview: https: https://openrevieuu
openreview. net/ forum? id= 946cT3Jsq5
Abstract
Concept Bottleneck Models (CBMs) (Koh et al. 2020) are class of interpretable deep learning frameworks that improve transparency by mapping input data into human- understandable concepts. Recent advancos, including the Discover-then-Name CBM pro- posed by Rao et al. (2024). eliminate reliance on external language models by automating concept. discovery and naming using a CLIP feature extractor and sparse autoencoder. This study focuses on replicating the key findings reported by |ao et al. (2024). We conclude that the core conceptual ideas reproducible, but not to the extent presented in the original work. Many representations of active neurons appear to be misaligned with their assigned concepts, indicating lack of faithfulness of the DN-CBM's explanations. To address this, we propose a model extension: an enhanced aligument method that we evaluate through a user study, Our fine-tuned model provides more interpretable concepts (with statistical significance), at the cost of slight decrease accuracy. The implementation our work available in our GitHub repository
1 Introduction
Interpretable frameworks like Concept Bottleneck Models (CBMs) Koh et al.) 2020) have gained attention for their ability to enhance explainability in deep learning. CBMs accomplish this by mapping input data into human-understandalle concept space, which can then be used for downstream tasks, such as classification. This is achieved through a linear combination of concepts, which allows for explaining the predictions made by a classifier. Traditional CBMs require labeled attribute datasets, but recent CBMs (Oikarinen et al. 2023
Vang et al. (2023) use Large Language Models (LLMs)) (Brown et al.) 2020 and vision-language models (VLMs) (Radford et al. 2021) for attribute-labcl-froe concept learning. While effective, these methods have several limitations: They require creating concept sets for each new classification task, depend on prompt engineering, are inherently stochastic, and may fail to produce key concepts needed for accurate classification.
To address these limitations, Rao et al. (2024) propose the Discover-then-Name CBM (DN-CBM), which aulomates the discovery and naming of concepts without relying on external LLMs. It uses a CLIP-based feature extractor (Radford et al., 2021) and Sparse Autoencoder (SAE) to disentangle input embeddings into unmaa-ndderstaddalle concepts for classification. |Rao et al. (2024) report that the DN-CBM model not only provides interpretability but also achieves competitive performance in terms of classification accuracy. Notably, it perforrs state-of-the-art explainable baseline models, such as Yang et al. (2023). across most

---

Published in Transactions on Machine Learning Research (06/2025)
datasets, while remaining task-agnostic. This compelling result motivates reproduction to further explore its mechanisms and contribute to advancements in explainable artificial intelligence.
In this study, we reproduce and evaluate the findings presented by Rao et al. (0024) focusing on the performance and explainability the DN-CBM framework. Building on their findings, we investigate the impact of cosine similarity on concept explainability and introduce a loss function that promotes more interpretable concepts. This Joss function encourages alignment between the SAE neurons and their assigned concepts, which we then evaluate through user study.
2 Scope of reproducibility
We investigate the following (main) claims from Rao et al. (2024) and label them (C1-C3) for reference.
C1: Automated concept discovery. The DN-CBM framework can successfully discover latent concepts in the data without pre-selecting them. The SAE effectively identifies meaningful and human-understanddlle concepts directly from the CLIP feature space.
C2: Interpretability. The method demonstrates that the discovered dictionary vectors align well with text embeddings of the concepts they represent in CLIP space. This alignment enables intuitive naming of the concepts, facilitating model interpretability across different. tasks. As a result, the approach supports task-agnostic explanations of the model's decision process.
C3: Performance. The DN-CBM achieves competitive performance on classífication tasks across variety of downstream datasets, ensuring task-agnosticity.
3
Methodology
This section outlines the methods used in this study. Sections 3.1 and hyperparameter considerations, respectively. Section 3.4 details the experimental setup used to validate the claims, and Section 3.5 presents the motivation, theoretical foundation, and evaluation of our extension.
3.2 and 3.3 discuss the models, datascts,
3.1 Model descriptions
Concept discovery. We follow the SAE approach proposed by Bricken et al. (0023) to transform CLIP features into a more interpretable latent space. This is achieved using a linear encoder f(-) with weights WE E Rdxh, followed by a ReLU activation function Φ(-). The SAE is trained in a self-supervised manner by reconstructing the original features using linear decoder g(-) with weights WD E Axx The latent space dimension, denoted by h, is much larger than the CLIP embedding dimension, denoted by d. For a given embodding Rd that is produced by CLIP image encoder I, the loss function is defined as:
LSAE (a) = [[SAE(a) all + A || (f (a))
(1)
where A1 a hyperparameter that enforces sparsity in activations. The SAE typically trained on a large dataset, denoted as D Dextract, to extract wide range of concepts.
Concept naming. After training, we aulomatically assign names to individual feature dimensions in the SAE's hidden representation. To achieve this, vocabulary set V {fi...., is embedded using CLIP text encoder T. To enable effective gencralization and meaningful concept naming, the set V should be broad and ftexible, for example, a large collection of unigrams. Neurons in the SAE's latent space are labeled according to the highest cosine similarity between their dictionary (decoder) weights and the CLIP vocabulary representations. This is a natural choice as CLIP was trained to optimize cosine similarities between text and image embeddings. The dictionary weight vector Pe for neuron is defined as the tt row of the decoder weight matrix:
Pe WDle: E Rd.
(2)
The corresponding label se is then determined as:
Sc = argmax cos (< (pc. T(t)))
(3)
vEV
2

---

Published in Transactions on Machine Learning Research (06/2025)
We define vector representing alignment as distribution over all cosine similarities so that can refer to this later. Specifically, we introduce a vector IRA,, where the e element is defined as:
Ve cos (< (pc. T(s))).
(4)
We refer Ve as the cosine similarity score of concept throughout this work. It quantifies the alignment between the assigned vector in CLIP space and the dictionary vector of neuron
Constructing CBMs. A CBM is constructed by connecting a linear probe h(-) to the encoder's out- put for downstream classification tasks. The probe is trained on a separate dataset, denoted D prabe = [(11,1)), (22, ...,..., where represents the ground truth label of æ, The CBM t(-) is defined as:
((x,) = (h Φ f I) (m:).
During probe training, the SAE encoder layer is frozen, and the probe weights (w) are adjusted based the following loss function, where X2 is sparsity hyperparameter and CE denotes the cross-entropy loss:
probe (2:4) CE (I<z;), 21) + 22/w11-
(6)
3.2 Datasets
CC3M used for training and evaluating the SAE (Dextract), whereas ImageNet, Places365*, CIFARIO, CIFAR100, and Waterbirds-100 are used for training and evaluating the linear probe (Pprobe):
CC3M. CC3M consists of imago-caption pairs generated by extracting text from alt-texts of images on the web (Sharma et al.) 2018). Due to link rot, approximately 68% of the originally collected images were available in the final dataset. The dataset can downloaded here
ImageNet. ImageNet-1IK is a standard benchmark for image classification DDngg et al. 2009
Places365. We use 10% subset of the Places365 dataset (Zhou et. al. 2017). sampled to preserve the original class distribution, for downstream classification. We refer to this subset as Places365*.
CIFAR100. CIFAR100 contains 60,000 images evenly distributed across 100 classes Krizhevsky 2009
Waterbirds-100. The Waterbirds- 100 dataset (Petryk et al. 2022 Sagawa et al. 2020 features landbirds and waterbirds with spurious background correlations during training, but not in the test set.
3.3 Hyperparameters
We reproduce the experiments using the hyperparameters from Rao et al. (024)) reported in Table 1 with as the default for standard classification and for concept interventions, unless otherwise specified.
Table 1: Hyperparameters. We use the same hyperparameters Rao et al. (024)), though the original paper describes hyperparameter sweep without detailing the probe settings. For standard classification, we evaluate two linear probe configurations: U1 based on the GitHub README example for Places365, and U2, the default settings provided in the code for each probe dataset. A third configuration, va, is used specifically for concept intervention experiments. The Adam optimizer is applied with its default
hyperparameter settings (Kingına and Ba 2015).
General
SAE
Probe
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
text encoder (T)
CLIP ResNet-50
latent dim (h)
8192
learning rate
10-2
10-3
10
image encoder (Z)
ResNet-50
L sparsity (Az)
3 10-5
batch size epochs
512
512
512
vocabulary (V)
Google 20k
learning rate
200
200
vocabulary size ((
20000
epochs
200
Li sparsity (Aa)
0.1
1
10
embedding din (d)
1024
batch size
4096
optinizer
Adam
Adam
Adam
batch resample freq
10
top-A pruning
5
optimizer
Adam
For CIFARIO0 probe learning rate of 10-2 was used 12-
3

---

Published in Transactions on Machine Learning Research (06/2025)
3.4 Experimental setup and code for reproducibility
We used the publicly available codebase by Rao et al. (0224) to reproduce the key findings of the original study. This comprehensive Git repository includes all necessary scripts to generate the plots and the final results. The computational tasks were carried out using GPU resources provided by the Dutch national supercomputer, Snellius, with funding support from the University of Amsterdam. An Nvidia A100 Tensor Core GPU was used to run the experiments. The total computational expense for the reproduction equals 91.91 GPU hours. This has an estimated emissions of 10.87 kgCO2eq P
To investigate CI, we train an SAE with the same architecture and hyperparameters as the original paper. We then align the dictionary vectors (Equation 2 with the CLIP feature representations. We visualize the rankod cosine similarity scores for the DN-CBM from checkpoint in the GitHub repository and our reproduced DN-CBM. This will help us determine our reproduced latent space is as closely associated with CLIP features as in the original work. We use this same figure as an initial indication for C2, which asserts that the vectors should align well. To further investigate C2, we reproduce qualitative and quantitative analyses from Rao et al. (2024) related to interpretability.
The qualitative analyses that we reproduce for C2 comprise illustrating examples of named concepts along- side the top images that activate these concepts across four datasets, as well as explaining the decision of the DN-CBM by classifying random images from the Places365* dataset and reporting the top contributing concepts. While Rao al. (2024) include only concepts with high cosine similarity scores in their analysis, we extend this by also incorporating lower-aligned concepts to assess their representational quality. We further evaluate generalizability by providing similar results for additional datasets in Appendix [.1
We reproduce the two components of the quantitative analyses for C2. First, the human feedback survey is reproduced to validate the alignment between concepts and neurons across varying cosine similarity scores. Participants were asked to rate concept consistency and naming accuracy for 12 images under one concept. Similar to the qualitative analysis, these are the top images that activate the concept. High consistency is defined as a set of images with a consistent overarching theme. The survey employs a 1-5 rating scale for accuracy, where 1 indicates poor alignment and 5 indicates strong alignment between the concept and the images. This survey was conducted with 22 participants, following the exact structure of the original study by Rao et al. (2024).
Second, concept interventions are evaluated using the Waterbirds-100 100 dataset, which is specifically designed to test robustness to spurious correlations between bird type and background, evaluating C2. In this dataset, landbirds (waterbirds) are shown on land (water) backgrounds during training, while such correlations are absent in the test set. We replicate this setup by training on Waterbirds- 100 and applying two interventions to the DN-CBM model: (1) retaining only bird-related concepts, and (2) removing them. We assess classification accuracy before and after these interventions, focusing on their impact on 'worst-group' examples: Landbirds on water- and waterbirds on land backgrounds.
To reproduce C3, we follow the original DN-CBM model specification by attaching a linear probe to the SAE to classify images. We compute the classification accuracy on ImageNet, Places365*, CIFAR10, and CIFAR100, specified in Section 3.2 We compare our accuracies to the original accuracies.
We omit certain additional analyses from Rao et al. (2024), they not essential for verifying the main claims. One such analysis is quantitative evaluation using the SUNAuributes dataset (Patterson et al., 2014), where discovered concepts from the SAE are compared to ground truth labels by filtering and merging nodes based on cosine similarity with text embeddings. We did not reproduce this evaluation, as it constitutes a minor aspect of the original work and employs a filtering approach not used elsewhere in the study. We also omit the semantic consistency analysis using k-means clustering on concept activation vectors and the -Disset component of the user study. These analyses, while informative, are not critical for assessing the main contributions.
'he emissions were calculated using Machine Learning calculator Lacoste et al. 2019 The estimated carbon efficiency of the Netherlands was 0.473 kgCO2eq in January 2025 (Nwtricity 2025
4

---

Published in Transactions on Machine Learning Research (06/2025)
3.5 Extension to original work
In their novel method, Rao et al. (2024) propose post-hoc approach to explainable artificial intelligence. This approach is compelling for maintaining the model's accuracy, it does not constrain the model to be inherently explainable during training. However, the degree of explainability is theoretically limited as certain aspects a freely trained model cannot be fully captured by a single word or concept due to the constraints of human language. Rao et al. (2024) report good aligument between dictionary vectors and text emboddings CLIP space (high cosine similarity score). However, dcfining "good' alignment high- dimensional space is challenging, especially when the alignment varies signilicantly across the neurons. We investigate this alignment in C2 and propose model extension to improve the alignment of dictionary vectors and text embeddings.
3.5.1 Model extension
We introduce a loss function that encourages SAE neurons to align with more interpretable concepts. This loss function is designed for fine-tuning, as it depends on a set of pre-defined concepts learned during training. This loss function is defined as:
& (f(a))
LPSAE (a) [|SAE(a) all2 + 110 (f)
( (f(a))]l,.e) 0
where is the cosine penalty parameter that dictates the strength of our introduced penalty term, and is small positive constant added to prevent division by zero in cases with no activations. The term C encourages alignment between the latent space and explainable features using the cosine similarity scores of The conceptual idea behind our fine-tuning loss is shown in Figure !
Xintersection)
Mintersection)
rTStreet)
7SSrreet)
P1
CLIP feature
CLIP feature
715unlight)
([Sunlight)
(Pink)
(PPnk)
CLiP feature
CUP feature
(a) Dictionary vectors before fine-tuning.
(b) Dictionary vectors after fine-tuning.
Figure 1: Conceptual overview of the fine-tuning process, in CLIP embedding space. In this example, the vocabulary set is V (Street, Pink, Interscction, Sunlight), with both CLIP and latent dimen- sions h d = 2. Neuron 1 is assigned $1 {Intersection), and neuron 2 is assigned fine-tuning, the dictionary vectors are better aligned with the CLIP embeddings of their respective assigned names. In practice, the vocabulary set, CLIP embedding dimension, and number of neurons are much larger.
(Pink). After
A key advantage of scaling by <((()) is that the incentive for alignment. proportional to the activation magnitude. This prevents rarely activated neurons from arbitrarily aligning with CLIP embeddings without capturing meaningful representations. Consequently, the neurons involved in inference are more likely to exhibit a high cosine score. The normalization prevents the optimization routine from generating excessively large ivaitiionns to exploit the cosine loss.
Excessively large C values, however, introduce an issue. When C bocomes too large, the cosine similarity term dominates the loss function, causing the distribution of shift towards higher values. This reduces variation in similarity values. As more explainable neurons are incentivized to activate, their activations
5

---

Published in Transactions on Machine Learning Research (06/2025)
may lose meaning. Furthermore, because normalize C()(f(a))) v) by ( there is no incentive to increase the activation magnitude. This leads to the diffusion of neuron activations, meaning that more neurons become active but with lower activation values. This reduces the model's interpretability because when it relies on a vast number of weakly activated neurons, it becomes difficult to pinpoint specific ncurons as the primary contributors to the decision-making process. This creates a trade-ofT: increasing neuron explainability can sometimes come at the cost of making their activations less meaningful.
To determine appropriate value for C, experiment with different values and analyze their impact neuron interpretability, activations, and accuracy. We fine-tune our model seven times on CC3M, each time using a different value of the hyperparameter C' E [10-6, 10-5 .0......)..... We use the mcan cosine similarity score as & proxy for neuron interpretability, while activation levels are quantified by the average magnitude of nonzero elements in @ff(a)). Additionally, we measure the validation accuracy on the Places365* dataset. A balanced value of C' then selected based on a qualitative assessment of these factors. The code for our reproduction and extension publicly available in our GitHub repository
3.5.2 Evaluation model extension
To evaluate the impact of our method, we compared the reproduced DN-CBM model with our fine-tuned extension using the optimal C value. Wo this by conducting user study. For our user study, we selected all concept explanations for image classifications that appeared more than five times in a subset of the Places365* test set. Next, we ranked these concepts by cosine similarity and selected the bottom 40, middle 40, and top 40 aligning concepts for both models (resulting in a total of 6 groups). For each concept, we sampled five images without replacement corresponding to the classification explanation, resulting in a total of 240 concepts, each paired with five images. We asked each participant to rate two randomly sampled concepts from each group on a scale from 0 to 5, reflecting how many images aligned with the concept. This sampling approach ensured that each participant evaluated only 12 images while the entire dataset was thoroughly assessod across all participants through the randomized selection process. A total of 203 participants completed the questionnaire. Further details of the outline of our user study are given in Appendix A.T To assess whether the differences in user ratings between the two models are statistically significant, signed-rank test (Wilcoxon, 1945). This non- parametrct test is appropriate given the paired nature of our data and the lack of a normal distribution in the ratings. The details of the test and its implementation are further discussed in Appendix A.2
employ the Wilcoxon
Note that this user study, conducted to evaluate our model extension, the second user study in this work and differs from the first, which followed the exact structure of the study by |ao al. (2024). In their work, they analyzed neuron activations in response to concepts. We argue that classification explanations are more meaningful for our objective because they align with the model's primary task. For example, a neuron representing the concept of 'turquoise" may have list of top-activating images, such as car in CIFAR10. However, this does not nocessarily indicate that the neuron plays a key role in classifying the object (car).
We also include classification experiments on CIFARIO, CIFAR100, Places365*, and ImageNet, and assess concept intervention effectiveness Watcrbirds-100 100 to evaluate extension.
4 Results
This section begins by presenting findings on the reproducibility of C1 through C3. Subsequently, the results of extensions to DN-CBM are discussed. The findings confirm C1 and C3, while contradictory results are observed for C2.
4.1 Results reproducing original paper
To assess the reproducibility of C1 we visualize the ranked cosine scores for the original and reproduced DN-CBM in Figure 2 We observe that we successfully reproduce a latent space that maps concepts within a certain cosine similarity range. The distribution of values closely aligns with the original findings, demon- strating that our approach achieves comparable mapping. This validates the reproducibility of C1.
6

---

Published in Transactions on Machine Learning Research (06/2025)
04
0.3
0.3
0.2
0.0
2000
4000
5000
8000
0
2000
4000
5000
8020
Nodes ranked in similarlies with assigned concepts
Nodes ranked in similaniies with assigned concepts
(a) DN-CBM.
(b) Reproduced DN-CBM.
Figure 2: Ranked cosine similarity scores of the assigned concepts. Comparison of the cosine similarity scores from the original DN-CBM (Figure 2ah and the reproduced DN-CBM (Figure 2b
To validate C2, we extracted the top-activating images with the highest cosine similarity score, across the four datasets, as shown in Figure 3 Notably, the top-activating images strongly correspond to their respective concepts for ImageNet and Places365, indicating high degree of semantic consistency. This is in line with the results of Rao et al. (202)) and supports C2. For CIFAR10 and CIFAR100, the images correspond less with the concepts "plaid" and 'sweater', which may be attributed to the limited expressiveness of the dataset.
plaid
burgundy
turquoise
sweeler
ImageNot
CIFAR10
CIFAR100
Places365
Figure 3: Task-agnosticity of concept extraction. We present the concepts with the highest cosine similarity score, alongside their top-activating images from four datasets.
To further assess the reprodueibility of C2, we examine Figure 2 The cosine scores range broadly from approximately -0.01 to 0.42, indicating the presence of lower-aligned concepts. This raises concerns on the faithfulness of the interpretability of some notes. The original study focuses only on highly aligned concepts (Figure 3), which may not generalize to the explainability of all neurons. To assess C2 across the entire network, we examine Figure 4 which presents concepts with lower cosine similarity scores and their top- activating images across four datasets. These images demonstrate low consistency with the assigned concept names, which suggests that C2 is not fully supported by this figure, as the images should only activate for concepts they actually represent.
zealand
silhouette Indec 7008
commissioned Inder 0123
sauna
Inceo 7301
Inder ERG
ImageNet
CIFAR10
Places365
Figure 4: Lower aligned task-agnosticity of concept extraction. We present low-aligned concepts alongside their top-activating images from four datascts. The images associated with each concept demon- strate low consistency with the assigned concept name across datasets.
7

---

Published in Transactions on Machine Learning Research (06/2025)
To continue the qualitative analysis of C2, we display the top concepts contributing to the decision Docision-making process for two randomly selected samples from the Places365* dataset in Figure 5 Rao et al. (2024) show similar examples with concepts that, indeed, all describe aspects of the image's s theme. This qualitative anal- ysis supports their claim that concepts are associated with the predicted class, thus aiding interpretability. Upon examining Figure [ we find that not all displayed concepts contribute meaningfully to the decision- making process, which partially challenges C2. The left figure illustrates that the concepts are thematically consistent with the corresponding class, supporting the validity of the approach. The right figure raises concerns. Specifically, the presence of concepts such "kayaking", "dams", and 'trivium', do not clearly correspond to the class. Consequently, we find that the original results cannot be reproduced to the same extent. This qualitative analysis is extended to different datasets in Appendix [..] which demonstrates similar results. One final observation related to Figure 5 is that a local explanation, such as "greenhouse, directly mirrors the class label 'greenhouse'. Explaining a prediction with its own class label, however, offers limited insight; this highlights a general limitation of the DN-CBM framework.
sultway
kayylinng
contemperas 4441
stairs
gerden
Figure 5: Explaining decisions using the reproduced DN-CBM. We present randomly drawn amples of images from the Places365* dataset alongside the top concepts contributing to their classification.
To reproduce the quantitative analyses for C2, the survey and concept interventions results are prosented in Figure and Table 2 respectively. Figure 6a shows general decline in semantic consistency with decreasing concept aligmment. The intermediate alignment trend differs slightly from the original work, likely due to limited sample sizes. Figure 6bb reveals a positive relation between accuracy and consistency, though some low-accuracy cases still exhibit high consistency. Overall, the trends align with the original results, indicating reasonable reproducibility. The concept interventions Table 2 demonstrate effectiveness, consistent with the original findings.
Boxplot of User Ratinge by Alignment Level
Consistency vs. Accuracy
User Rating
Consistency
High
Intorediaae dl Alignmenk Level
Low
3 Accuracy
(a) Semantic consistency of concepts.
(b) Consistency vs. accuracy.
Figure 6: Reproduced user study on concept accuracy. Semantic consistency is plotted for nodes with high, intermediate, and low alignment to their assigned text embeddings of the reproduced SAE (Fig- ure 6a. In Figure 6b. the semantic consistency scores are plotted against. name accuracy. The survey has participants.
To assess C3, we report the classification accuracy for the reproduced model and the original work in Table 3 The classification accuracies of ImageNet, CIFAR10 and CIFAR100 are similar for the original and reproduced DN-CBM. The performance on Places365* is worse for the reproduced model. This could be attributed to the use of smaller version of this dataset. Overall, results in agreement with C3.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Concept interventions. Classification accuracy (%) is reported on the full test set ('Overall") and four specific groups before and after interventions. 'Reproducod" uses configuration Va while "Fine- tuned' uses 0g with A2 = 1 (vs. 10). Details and the intervened concepts are provided in Appendix C.1
Source
Model
Overall
Worst groups
Training groups
L.Bird&W
W. W.Bird@L
L.Bird@L.
W.Bird@W
Before intervention
84.24
76.30
58.27
97.83
91.37
Reproduced
Only bird concepts
88.07 (+3.8)
84.78 (+8.5)
82.73 (+24.55
93.28 (-4.6)
87.05 (-4.3)
Non-bird concepts
59.47 (-24.8)
28.70 (-47.6)
15.83 (-42.4)
97.40 (-0.4)
79.14 (-12.2)
Before intervention
67.39
42.39
19.42
99.57
91.37
Fine-tuned
Only bird concepts
83.24 (+11.8)
73.70 (4-31.3)
75.54
(+56.1)
92.19 (-7.4)
92.81 (+1.4)
Nou-bird concepts
65.14 (-2.3)
43.70 (+1.3)
4.32 (-15.1)
100.00 (+0.4)
81.29 (-10.1)
Table 3: Comparison of performance of the original and reproduced results. We report the classification accuracy (%) of the original paper and our reproduction using CLIP ResNet-50 on ImageNet, Places365*, CIFARIO and CIFAR100. "Fine-tuned' is our model suggestion, which incorporates a cosine loss function with C' = 10 (Equation 7 ** indicates the use of a smaller dataset compared to the original paper; 10% of Places365. Results show mean standard deviation over 3 probe training runs.
Model
ImageNet
Places365
CIFAR10
CIFARI00
DN-CBM (Rao et al.)
72.9
53.5
87.6
67.5
DN-CBM Reproduced
72.65 =0.02
49.96*
=0.04
86.71
#0.02
68.51
#0.04
DN-CBM Fine-tuned (Ours)
70.47 20.01
49.30*
-0.03
83.88 #0.02
64.39 20.16
4.2 Results beyond original paper
Figure 7 shows the results of our experiments with different cosine penalty parameters, C. After completing standard training, we fine-tuned for an additional 30 epochs using the loss function in Equation 7 which
was generally sufficient for convergence. We observe the theoretical trends discussed in Section 3.5 As C
increases, activations become more diffused, the average cosine similarity score rises, and some point, it comes at the cost of accuracy. Figure g shows the cosine score distributions for C 10-3 and C = 10-4
both of which are significantly higher average compared to Figure 2
The more diffused activations result
in sharper distribution, seen in Figure 8b Appendix provides visualization of the cosine score distribution across all tested C' values.
Moan (nenzero neurons)
Moon similarity
Validation accuracy Flacosd65
0.50
0.178
0.150
0.43
0.325
0.53
0.30
0.33
0.00
170
1110
1110
10
10
20
2227
30
12
30
3244
40
10
3011
307
30"
301
18
10
10
(a) Mean activation.
(b) Mean cosine similarity.
(c) Validation accuracy on Places365.
Figure 7: Impact evaluation of C. Comparison of mean activation across nonzero neurons, mean cosine score, and accuracy for different C' values. These plots are obtained with probe hyperparameters V1.
Unfortunately, we lack a straight.frward quantitative metric for determining the optimal value of C. We recommend using the theoretical considerations discussed earlier to guide its selection. For all remaining experiments, we use C 10-4, which offers a balanced tradc-off, substantially improving the average cosine similarity (from 0.146 to 0.540), while maintaining reasonable activation patterns and achieving the highest validation accuracy. Our goal is to provide proof of concept that demonstrates how promoting alignment
9

---

Published in Transactions on Machine Learning Research (06/2025)
C= 10
C-10
1.0
0.5
0.6
204
0.4
0.1
0.0
0.0
5000 Nodes rarked similarties assigned concepts
4000
ranked simileritie with sssigned concepts
(a) Fine-Luned DN-CBM (C = 10"*).
(b) Fine-luned DN-CBM (C 10 10-
Figure 8: Ranked cosine similarity scores of the assigned concepts after fine- Sinettnnigg Comparison of cosine score distributions after fine-tuning with penalties 10-3 (Figure Sa) and C 10-4 (Figure 8b
can lead to more faithful and interpretable model explanations. We encourage future work to develop more principled, quantitative approaches for selecting C.
Referring to Table 3 we compare the accuracy of our model to the original DN-CBM using the same hyper- parameters (u2). We observe that, under these hyperparanotong our fine-tuned model underperforms, with accuracy drops ranging from -0.7% to -4.1%, depending on the dataset. Although concept interventions remain effective for the fine-tuned model, a noticeably larger drop in accuracy is observed on the Waterbirds- 100 dataset, as shown in Table In the concept intervention setup from Rao et al. (2024), only the top five contributing concepts for a given class are used as predictive features, and interventions are performed on these. This setup inherently disadvantages our method. For example, concept labeled "duck" that repre- sents broad direction in the feature space associated with birds may serve more informative predictor than a neuron that is highly aligned specifically to 'duck'. Having addressed the accuracy component, we now turn to the interpretability aspect, which we evaluate through our user study.
The results of our user study are presented in Figure g Our findings indicate that, across all alignment levels, our model, on average, achieves higher user ratings. This effect particularly pronounced for low and intermediate-alignee concepts, where the Wilcoxon signed-rank -aaned-rank test produces p-values of 0.000. This indicates that, even under conservative significance threshold of 0.1%, the null hypothesis thatt there is no average difference in ratings between the reproduced and fine-tuned models would still be rejected. For the highly aligned concepts, the ratings are similar, and the statistical test yields p-value of 0.176. Notably, the intermediate-aligeee aligned concepts receive higher user ratings than the high-aligned concepts in our model. This occurs because enhancing ncuron explainability can diminish the meaningfulness of activations, as discussed in Section 3.5
To explain the observed improvements in the user study, we recreate the randomly selected image example from Figure 4 Jusing the fine-tuned SAE in Figure I0 Additionally, present several example classifications from the fine-tuned model, paired with their corresponding concept- based explanations in Figure
The concepts displayed in Figure 10 are much more semantically coherent with their top activating images in contrast to Figure | For instance, the concept 'zealand', likely referencing New Zealand, now clearly depicts natural landscapes such as mountains, hills, fjords, and lakes. The concept "silhouette" now reflects dark, shadowy objects, while "commissioncd", a broader term often related to commissioned artwork, includes images of jewelry, statues, tables, and paintings. The "sauna" concept is well-represented in the Places365* dataset but is less coherent in other datasets, which contain few no sauma-related images. Though not perfect, the results show substantial improvement in concept-image consistency.
Figure compares explanation from the original DN-CBM (left) and our fine-tuned model with C = 10-4 (right) on randomly selected images from Places365* and CIFAR-10. In the Places305* example, the original model Ila)) predicts 'Vegetable garden," supported by the concept "ivy," " but other concepts like "arnold', 'cosmos', 'labrador', and "eleven" lack clear relevance. In contrast, our model 11bb predicts "Field wild',
Note that these hyperparameters were explicitly optimized for the original DN-C placing our model at slight disad- vantage.
10

---

Published in Transactions on Machine Learning Research (06/2025)
User Ratings by Alignment Level
Reprediced
Pinclaned
Mean
User Rating
High
Intermediate
Alignment Level
Figure 9: User study results. User ratings for the reconstructed DN-CBM (orange) and our model with C = 10-1 and probe hyperparameters vi (green) on Places365*. The ratings are evaluated across three groups based on descending relative cosine similarity scores: high alignment (top 40 highest-aligning concepts of the model), intermediate alignment (40 concepts from the middle), and low alignment (bottom 40 concepts).
zealand
silhouctte
commissioned
ImageNet
CIFAR10
CIFAR100
Places365
Figure 10: Task-agnosticity of concept extraction in the fine-tuned model. We replicate Figure 4 using the fine-tuned SAE (C'= = 10- showing top-activating images for named concepts across four datasets.
backed by semantically coherent concepts such as 'meadow', 'fields", "flower', and 'crops". Similarly, for CIFAR-10, the original model [1cc) predicts 'Horse' but offers unrelated concepts like 'pelican", 'michigan', "aaliyah", 'busty", and 'elephants'. Our model (1dc instead yields meaningful concepts such "horses', 'equine", and "horseback'. These results suggest that fine-tuned model provides more semantically aligned explanations.
Note that if two latent neurons share the same CLIP label (i.e., their dictionary vectors map to the same label) and both rank among the top 5 contributors, they are merged into a single explanation, making our fine-Luned model (right) appear to produce fewer explanations. Additional examples on CIFAR-100 and ImageNet are shown in Appendix B While the explanations more accurately reflect the image content in Figure [1] they also tend to align more closely with the class labels, potentially undermining their explanatory value. A similar, though less pronounced, pattern is evident in Figure 5 This issue reflects fundamental limitation of task-agnostic methods, where predefined features may mirror the class names. We hypothesize that excluding concepts that closely resemble the class name during probing could help mitigate this effect. However, such an approach may come at the cost of reduced predictive accuracy, presenting a trade-off that warrants further investigation in future work.
A key difference between the original SAE and our fine-tuned version lies in how highly aligned concepts are represented. In our model, the concepts with the highest cosine similarity scores tend to be abstract and frequently activated with low magnitude. This likely occurs because they're not tied specific scene types and are more casily optimized by the loss function. Despite their high alignment, these concepts have low predictive power and rarely influence the final probe decision during inference. As a result, displaying a similar agnostiinsyyc plot, Figure [0] for the four most aligned concepts in our model isn't meaningful. These abstract concepts (such as "judaism") do not appear any dataset and these neurons have no impact on inference, thus are not relevant when rescarching model reasoning. Instead, we focus on the faithfulness of neurons that actually contribute to the model's reasoning.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Wed repiidee padde Naddend
Muil miunnh Dueiieiingg Crorgho
Phat Pire wid, Nackred
Mail siunggy cidiital
+133
Hower
copa
(a) Original model (Places365*)
(b) Fine-tuned model (Places365*)
Pradi CT Herse
Most Shringh Consibiting Concepts
Proft Homee CT Hore
Most Strenghy Contituu Conceppt
Feltrasch
(c) Original model (CIFARIO).
(d) Fine-tuned model (CIFAR10).
Figure 11: Explaining decisions using DN-CBM and our extension. The top row shows an example from Places365* with the predicted class, ground truth, and top contributing concepts for both the original DN-CBM (left) and our fine-tuned model with C 10-4 (right). The bottom row presents a similar comparison for CIFAR10.
5 Discussion
Our study successfully reproduces C1 and C3, demonstrating that the DN-CBM framework effectively uncover latent concepts in the data without pre-selecting them while maintaining competitive classification accuracy. Our results for C2 demonstrate that the authors' work is reproducible; however, the claim is not fully supported, as additional experiments, particularly those involving lower-aligned concepts, often fail to show meaningful contributions to the decision-making process. Fine-tuning the DN-CBM with an extended loss function that drives the dictionary vector of neurons towards explainable CLIP vectors enhances the interpretability of these neurons, as supported by both qualitative and user study analyses. However, this comes at the cost of classification accuracy, introdueing trade-off that can be controlled by adjusting the value of C, provided that C does not increase to the point where activations in the hidden state of the SAE diffuse excessively. Future research could involve conducting user studies across range of values for the cosine penalty parameter C.
The concept of an alignment loss through the C penalty term could also be generalized to incorporate different proximity measures. While cosine similarity is one approach assessing neuron-concepl aligmment, it may not always be the most effective metric. Determining similarity between vectors in high-dimensional spaces is inherently challenging (Aggarwal et al. 2001).
Lastly, we propose several general directions for future research that are not necessarily specific to our fine-tuned model: removing vague concepts from the vocabulary set, using larger dataset than CC3M to develop a more general off-the-shelf SAE, discouraging low-aligning and class name mirroring concepts from contributing to classifications through the probe, and introducing a threshold to exclude concepts with low similarity.
What was easy. Reproducing the study was relatively straight due to the authors' well-documented,, publicly available code. The GitHub repository included clear instructions setting up the environment, running experiments, and reproducing figures, making it casy to verify the majority of the original claims. We considered it unnecessary to contact the original authors.
What was difficult. Despite the code being well-documented, we encountered some minor issues, such as errors related storing results. These were managcable but required some extra debugging. Additionally, the paper did not clearly specify all hyperparameter settings, and while the GitHub code provided some guidance, inconsistencies in argument values added to the difficulty of replicating the results exactly.
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
Aggarwal, C. C., A. Hinneburg, and D. A. Keim (2001). On the Surprising Behavior of Distance Metrics in High Dimensional Space. In J. Van den Bussche and V. Vianu (Eds.), Database Theory - ICDT 2001, Berlin, Heidelberg, pp. 420-434 Springer Berlin Heidelberg.
Bricken, T., A. Templeton, J. Batson, Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, al. (2023). Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread 2.
Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell, et al. (2020). Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Eds.), NewrIPS, Volume 33, pp. 1877-1901 Curran Associates, Inc.
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei (2009). ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248 255.
Kingma, D. P. and J. Ba (2015). Adam: A Method for Stochastic Optimization. In ICLR.
Koh, P. W., T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang (2020, 13-18 Jul). Concept Bottleneck Models. In ICML, Volume 119, pp. 5338 5348. PMLR.
Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. Technical report, Computer Science Department, University of Toronto.
Lacoste, A., A. Luccioni, V. Schmidt, and Dandres (2019). Quantifying the Carbon Emissions of Machine Learning. In CoRR, Volume abs/1910.00 1910.09700.
Nowtricity (2025, 1). CO2 emissions per kWh in Netherlands.
Oikarinen, T., S. Das, L. M. Nguyen, and T.W. Weng (2023). Label-Free Concept Bottleneck Models.
Patterson, G., C. Xu, H. Su, and J. Hays (2014, 05). The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding. IJCV 118(,- 59-81.
Petryk, S., L. Dunlap, K. Nasseri, J. Gonzalez, T. Darrell, and A. Rohrbach (2022). On Guiding Visual Attention with Language Specification. In CVPR, pp. 18092-18102.
Radford, A., W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021). Learning Transferable Visual Models From Natural Language Supervision. In ICML, pp. 8748 8763.
Rao, S., S. Mahajan, M. Böhle, and B. Schicle (2024). Discover-then-name: Task-agnostic concept bottle- nocks via automated concept discovery. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Fds.), Computer Vision ECCV 2024, Cham, pp. 444-461. Springer Nature Switzerland.
Sagawa, S., P. W. Koh, T. B. Hashimoto, and P. Liang (2020). Distributionally Robust Neural Networks. In ICLR.
Sharma, P., N. Ding, S. Goodman, and R. Soricut (2018). Conceptual Captions: Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. ACL, pp. 2556 2565.
Wilcoxon, F. (1945). Individual Comparisons by Ranking Methods. Biometrics Bulletin 166), 80-83.
Yang, Y., A. Panagopoulou, S. Zhou, D. Jin, C. Callison-Burch, and M. Yatskar (2023). Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification.
Zhou, B., A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba (2017). Places: A 10 Million Image Database for Scene Recognition. In IEEE TPAMT.
13

---

Published in Transactions on Machine Learning Research (06/2025)
A Additional explanation
A.1 Survey method
In our user study, aimed to evaluate concept aligmment across different models. Below, provide additional details on participant selection and survey structure.
Participant recruitment and survey distribution. The survey was distributed to a diverse group of participants, including university students, colleagues, friends, and lamily. To ensure unbiased evaluation, participants were not informed which concepts originated from our model versus the reproduced DN-CBM model.
Survey structure and model parameters. The parameter C' in our model was set to 10 Before answering the main survey questions, participants were provided with example questions to familiarize them with the task (Figure 12 An example question from the study is included in Figure 13).
t
UNIVERSITEIT VAN AMSTERDAM
Example: we will ask you to judge whether the displayed images align well with the given concept. For each concept, you will see up to five images. Your task is to assign a score from to 5, where 0 means none of the images match the concept, and 5 means all the images match the concept, based on your judgment.
Answer: 1
Concept: Golfing
Answer: 5
Concept: Boxing
Figure 12: Examples presented the start of the survey. At the start of the survey, we included three sample questions (two of which shown in this figure) to help participants become familiar with the question format and response process.
14

---

Published in Transactions on Machine Learning Research (06/2025)
W
UNIVERSITIT AMSTEIDDM
Concept: Vertaling:
In how many pictures do you recognize the concept?
0
0
Quatrics
Example of question in the user study. We present five randomly selected images in which "Tomb" appears among the top five explaining nodes in the local explanation.
Figure 13:
A.2 Wilcoxon signed-rank test
To determine whether the differences in user ratings between the two models are statistically significant, we employed the Wilcoxon signed-rank test (Wilcoxon, 1945). This test is a non-parametric alternative to the paired t-test and particularly suitable when the assumption normality violated.
Let X; and Y, represent the average user rating given by participant i for the reproduced DN-CBM and our model respectively, specific alignment. level (high, intermediate, low). The difference in ratings for participant i is:
D; Y, X,,
where D; represents whether the participant preferred one model over the other for that alignment level.
For each alignment level, we test:
Null hypothesis (Ho).. The average difference in ratings across participants is zero (D 0). This implies that there is no significant preference for either model.
Alternative hypothesis (HA). The average difference is not zero DD * 0), meaning participants system- atically rate one model higher.
Assumptions. The Wilcoxon Signed-Ran Rane Test assumes that the two samples are dependent, meaning the data consists of paired samples. Morcover, the distribution of D, should be approximately symmetric around the median. Lastly, the test requires that the data is ordinal. These assumptions hold for our data, as the ratings range from 0 to 5, and each participant's ratings dependent.
15

---

Published in Transactions on Machine Learning Research (06/2025)
B Additional qualitative results
B.1 Generalization local explanation
This section provides an extra local explanation for the reproduced DN-CBM and examines the local expla- nation for Image Net and CIFARIO.
Podt Rutted grass OT. Rutled grese
Meer Strongl Contibuting Consapes
Prod: Gendsio, ST: Paccle
Mad Stoongyy Contritainn Concapts
+188
113
mush
dana
b
Sigs RBER SHOE
manket
akroodd
(c)
(d)
Figure 14: Explaining decisions using the reproduced DN-CBM. We present randomly drawn examples of images from the ImageNet dataset alongside the top concepts contributing their classification. Figures 14a and [4c are correctly classified, whereas Figures 14b and [4d deviate from the ground truth labels. While the predicted labels for Figure 14b appear reasonable, and most labels for Figure 14a are also interpretable-ent except for the label "lynx' the classifications for Figures 14c and 14d are less coherent.
Specifically, concepts such as "muller', "bookstores", "shortest", and "bakery" fail to provide meaningful rationale for the barbershop classification. Furthermore, in Figure [4d]] one the highest contributing concepts is "clic", which lacks a clear semantic interpretation, making it difficult to understand its role in the model's decision-making process.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Pre ls CT. Asionoblle
Meet Strongly Contibuting Concapts
Prod Cat, 07: Cad
Most Stoongy Contrilaring Concapts
harchoack
-641
corvete
Bemei
(b)
plans
binda
hellcoplery
Tyec
cregority
pars
cc)
(d)
Figure 15: Explaining decisions using the reproduced DN-CBM. We present randomly drawn ex- amples of images from the CIFAR10 dataset alongside the top concepts contributing to their classification. Our observations indicate that Figures T5a 15b.. and 15c are correctly classified, while Figure [5d] is mis- classified. The explanations for the classifications of "Attmmoblle", 'Airplane', and "Cat" appear reasonable, they include relevant concepts such 'car', "plane', and 'worldcat.' In the case of Figure [5d] which is misclassified as a 'Bird', the contribution of the 'birds" node can be interpreted as a plausible factor. However, the inclusion of concepts such as "snakes", "lynx", and "otter" is less intuitive and does nol provide a clear rationale for the model's prediction.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Post: Mouse GT Hameltrr
Meet Strongly Contibuting Concapta
Predt Bably OT: Soby
Most Strongyy Contrilaring Concapts
amvrdcons
+1:0
bed
tet
abanmn
b
Mox
stalnd
who
hopper
(c)
(d)
Figure 16: Explaining decisions using the reproduced DN-CBM. We present randomly drawn ex- amples of images from the CIFAR100 dataset alongside the top concepts contributing their classification. Our analysis reveals that Figures 16b and 16e are correctly classified, whereas Figures T6a and [6d are mis- classified. The primary contributing concepts for the classifications of 'Bicycle" and "Baby", such as "bikes" and 'infant", provide meaningful and interpretable justifications. However, in the case of the misclassification as 'Mouse," the concepts "albert', 'emoticons", and "pokemon" do not offer a coherent explanation for the model's decision. Similarly, for the misclassification as 'Turtle,' the contributing concepts 'staind' (a band), "Serbia", and 'hopper" lack clear semantic relevance, making the prediction difficult to interpret.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Predt
Strenply Sartrtbiing Guncapts
Poot: Strong Centrixuting Coreepts
Hnskox
tuddy
Fustiics
Tucky
(a) Original model (CIFARI00).
(b) Fine-tuned model (CIFAR100).
Pood Try tenter, GT: Tap tender
Meel Strongly Canstasting Concopes
Prod: Tay tariter OT:
Madt Strongly Contriturinn Concepts
dags
diggs
fnateriig
(c) Original model (ImageNet).
(d) Fine-tuned model (ImageNet).
Figure 17: Explaining decisions using DN-CBM and our extension. The top row shows an example from CIFAR100 with the predicted class, ground truth, and top contributing concepts for both the original DN-CBM (left) and our fine-tuned model with C = 10-4 (right). is observed that our model predicts the incorrect label 'wolf", yet the rationale behind this classification interpretable, with contributing concepts such as "husky" and 'huskics'. Given that CIFAR100 does not contain a "husky" class, "wolf" is identified the next closest match. In contrast, the original model correctly predicts the label, with explainable concepts such "firefox', "husky", and "huskies", although concepts like 'nhl' and "bei" are less interpretable. The bottom row compares similar examples for ImageNet. In this case, both models correctly classify the image as 'Toy terrier". Our model provides set of fully interpretable concepts, including "dogs", "ears', and 'dog'. The original model also identifies several explainable concepts such as "ears", "dogs', and "fostering', but includes less interpretable concepts like "pelican" and "triviun'.
19

---

Published in Transactions on Machine Learning Research (06/2025)
C Additional quantitative results
C.1 Concept interventions
For examining the effectiveness of concept interventions on both the original DN-CBM and our proposed fine-tuned method, we use the same methodology as |ao et al. (2024): We use the SAE for the CLIP ResNet-50 model, and train a linear probe for the binary classification task of the Waterbirds- 100 dataset. For increased sparsity, we prune the weights for cach class, leaving only the five largest weights and replacing the rest with zeroes.
For each class, then look at the five largest weights with their appropriate names, and classify them as 'Bird Concept' or 'Non-Bird Concept'. In Table | and 5 we show the five largest retained weights (concepts) per class (landbird or waterbird) for the reproduced model and our fine-tuned model respectively. For our fine-tuned model, we used a hyperparameter value of C 10-4 and removed duplicates in the five largest weights. Just like Rao et al. (2024), we perform two sets of interventions: (1) keeping only the bird concepts, and (2) removing only the bird concepts.
Class
Bird Concepts
Non-Bird Concepts
Landbird sparrow, parrot ibis, duck
branch, wetland, woodland pond, landing, beach
Waterbird
Table 4: Set of concepts in evaluation for performing interventions the original reproduced DN- CBM. For each class, manually classify each of the five concepts as being a bird or non-bird concept, for applying appropriate interventions.
Class
Bird Concepts
Non-Bird Concepts
Landbird
robin, sparrow swan, duck
woods, jungle, woodland ocean, beach, water
Waterbird
Table 5: Sct of concepts in our evaluation for performing interventions on our finc-tuned DN-CBM. For each class, we manually classify each of the five concepts as being a bird or non-bird concept, for applying appropriate interventions.
20

---

Published in Transactions on Machine Learning Research (06/2025)
C.2 Cosine similarity score distributions
C-0 0
C=10--
0.4
0.4
9.2
202
0.0
Nodes ranced
assgred
Nodes ranked
with
10
C 10
0.5
0.7
0.5
2.4
0.0
0.2
Nodes ranked simiarthes with greed
Nodes ninked sémilie lims with assigred concects
C- 10-3
C 10-2
0.6
0.5
0.4
Nodes ranced simil on fass with assignte concapts
Noces ranked siniliar lis With assigned concepts
C 10
C - 10
0.6
0.6
0.4
0.4
Nodes nanced aimil an with assggedd compapts
Nodes nanked sémille liss with lssigrad conceepss
Cosine similarity distribution for different paramcters C. The ranked cosine similarity values of the assigned concepts after fine-tuning with varying penalty parameters are presented. C 10 yields similar distríbution to the original cosine distribution of C = O. For C = 10-5, the shape of the distribution of cosine scores remains similar to that for 0, but with substantially larger range. When 10-4, a noticeable shift in the distribution emerges, with most similarity values becoming positive and a larger proportion of concepts exhibiting higher cosine similarity. For C = 10-3 and C = 10-2, the cosine similarity scores reach for the first 6,000 nodes before rapidly declining to 0.2 or lower. For = 10 and C 10", the cosine similarity score goes to 0.2 after 8000 nodes.
Figure 18:
21


Reviews:
[{"review_id": "LmSRlYOMV7", "reviewer": "Reviewer_UByM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thank you so much for submitting this work! I enjoyed reading this paper. Below are what I believe are this paper’s main strengths, followed by what I think are some of its weaknesses:\n\n### Strengths\n\n1. **[Significance and Originality] (Critical)** I appreciate the extensions on top of the reproducibility study. These include the extended qualitative analysis, the new alignment method, and the extended user study. All of these make this reproducibility study interesting and worth reading even if one is familiar with the original work.\n2. **[Quality] (Major)** The motivation and experimental design are all very clear. More importantly, I believe the key claims examined in this paper (i.e., C1 - C3) are well-motivated and followed by a series of appropriate experiments to evaluate them.\n3. **[Clarity] (Major)** The English and writing are almost typo-free, very clear, and extremely easy to read. I have some issues with the organization (as described below), but this is orthogonal to the quality of writing.\n\n### Weaknesses\n\nIn contrast, I believe the following are some of this work’s limitations:\n\n1. **[Quality] (Critical)** I believe some of the key claims studied in this work would’ve benefited from questioning some of the key assumptions behind the proposed methodology (e.g., the use of a fixed dictionary) and studying the reproducibility of the concept interventions results by Rao et al.. As I argue below, I do not particularly agree with the argument for not exploring the intervention results.\n2. **[Clarity] (Major)** Some of the sections could’ve benefited from better subsections/splits as they seem to be conglomerates of connected but not entirely related topics. For example, see Section 3.4.2 where methodological changes as well as model selection and user study design are all discussed in the same subsection of a “Methodology” wider section.\n3. **[Significance] (Minor)** This work would benefit from discussing, in a potential extended related work section, works that have come since the publication of Rao et al.. This may shed some light on how this method behaves and works, as well as on how one could potentially extend this approach. Moreover, such a section would help better place this study within the appropriate literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the replicability of Discover-then-Name CBMs (DN-CBMs) by Rao et al. (2024). In particular, it investigates whether DN-CBMs are (1) capturing meaningful concepts without any domain knowledge, (2) producing interpretable predictive models, and (3) producing accurate predictive models. By recreating most of the key experiments of Rao et al., as well as running a larger-scale user study than that done by Rao et al., this work concludes that most of the experimental claims in the work by Rao et al. do hold up to scrutiny. Nevertheless, this paper shows that, contrary to what is discussed in the work of Rao et al., there are still latent dimensions in DN-CBMs that are not fully aligned to meaningful concepts. To address this, this paper proposes a simple modification to the learning objective of Rao et al. that encourages better alignment for all discovered latent dimensions through a fine-tuning process.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below, I list some potential changes for each section and their importance in securing my recommendation for acceptance. If time is limited, please focus on all **critical** requests, followed by all **major** requests. If I misunderstood something at any point, just let me know, as this is always possible (apologies in advance if that’s the case!).\n\n### Section 1 (Introduction)\n\n- **(Minor, Attribution)** I would recommend properly citing a work the first time it is introduced. So, when first discussing CBMs in the introduction, please citer them in the same line.\n- **(Minor, Attribution)** As above, it would be beneficial if, when discussing “recent innovations” in the introduction, you provide valid evidence and citations of which “innovations” you are referring to.\n- **(Critical, Wrong Attribution)** The work of Margeloui et al. cited in the introduction does not, in any form of fashion, discuss issues of using LLMs for CBMs. They show some concerning results on CBMs in general (in their instance, they use ground-truth concept annotations rather than LLM-generated annotations). Using this work to say that “LLMs can lead to unfaithfulness to the model’s reasoning process” is misleading and not a proper attribution.\n\n### Section 3 (Methodology)\n\n- **(Major, Helpful Clarification)** When introducing the vocabulary $\\mathcal{V}$ in Section 3.1, it would be helpful if one may preemptively indicate how such vocabulary may be generated. In this reproducibility study, this is particularly helpful since this is one of those areas where domain knowledge may be required and, therefore, the task-agnosticism claim of the original work may be challenged.\n- **(Minor, Typesetting Nit)** For consistency and better typesetting, I would recommend using LaTeX’s \\text{…} guard for multi-character subscripts (e.g., $\\mathcal{D}_{extract}$).\n- **(Minor, Typesetting Nit)** For clarity, I would suggest making external hyperlinks (such as those used in the data of CC3M) blue and underlined.\n- **(Minor, Potential Typo)** “A Nvidia A100 …” should probably be “An Nvidia A100 …”\n- **(Major, Clarity)** When one says something like “The qualitative support for C2 includes a human feedback survey…” the assumption is that this is something done as part of this work. However, this is in reference to the original evaluation of the proposed method by the original authors. As such, I would strongly suggest that this is made clear by avoiding the use of the passive voice in statements that can be ambiguous (e.g., this could be rewritten as “Rao et al. provide qualitative support for C2 that includes a human feedback survey…”).\n- **(Minor, Accuracy)** Related to the point above, in Section 3.4.1 it is claimed that the user study by Rao et al. was replicated, however the used survey and study is different for this reproducibility study. I would recommend specifying this here as otherwise this seems contradictory to what one sees Section 3.4.3.\n- **(Major, Formulation)** Shouldn’t there be a stability $\\epsilon$ term in the denominator of the new term in Equation (7) as, in theory, this denominator can have zero norm due to the nature of the ReLU nonlinearity? Similarly, shouldn’t one normalize $\\mathbf{v}$ in this term as well to avoid loss reductions by a simple rescaling of weights?\n- **(Major, Clarity)** Could you please be more specific on how model selection is done for $C$ given the mean cosine similarity, the activation average magnitude, and the validation accuracy? In other words, given those three quantities, how was the “best” model selected (i.e., what is the cumulative metric one uses to select the best candidate)?\n\n### Section 4 (Results)\n\n- **(Critical, Claim)** I would disagree with the claim that because the lower-aligned concepts do not show consistent samples (Figure 4), then C2 cannot be “**reproduced**”. I would agree with a claim saying that C2 may not be as strong as a statement as put forth by the original authors, but claiming it is not reproduced implies the authors tried a similar experiment and generated very different results. This does not appear to be the case, as far as I am concerned. As such, I would recommend changing the way these (interesting) results are portrayed.\n- **(Critical, Error Bars)** Against common good practices, there are no errors reported in Table 2. I would strongly recommend these error bars to be included as part of this evaluation to better understand the sensibility and variance of the method being studied.\n- **(Critical, Fair Evaluation)** Why does Figure 9 show less concepts for the extended model than the reproduced? This seems rather odd and potentially unfair. I would appreciate it if it is possible to show the same number of most-relevant concepts for all models as otherwise this selection may bias the analysis.\n\n### General Questions\n\n- **(Major, Question)** I understand computational constraints do limit a lot of the things one can run and attempt; however, for a reproducibility study, it would be ideal to use the same setup and evaluation as used in the original paper. As such, could you please comment on why using 10% of the Places365 data does not violate the spirit of checking reproducibility on this task by following all the original steps of the work in question? Is there an alternative to using just 10% of the data?\n- **(Critical, Question)** I am a bit at odds with the decision not to evaluate concept interventions. I would strongly disagree that intervention analysis only measures “human-driven interventions.” In fact, interventions are a key way of verifying quantitatively the interpretability of concepts by confirming that when a concept is modified in a certain direction, the output is updated as expected. Considering this, could the authors please elaborate on why the evaluation of interventions would not help explore and verify claim C2? As this is a core claim in both the paper and this study, I would therefore recommend that interventions, in some manner at least, are considered for this reproducibility study.\n- **(Critical, Question)** Is it the case that when looking at the lower-aligned neurons in the fine-tuned model, do these make sense? Or are they still completely unaligned? In other words, does the proposed fine-tuning process also lead to better alignments in the same evaluation setup used for Figure 4?", "broader_impact_concerns": "I do not believe there are any broader ethical implications from this work that would merit a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fMErAoCicm", "reviewer": "Reviewer_RjVw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ **S1.** The paper carefully designs reproducing experiments and evaluates DN-CBM from various perspectives, including human evaluation.\n+ **S1.** The paper points out issues with the interpretability of DN-CBM through case studies and user studies. Since DN-CBM forcibly maps SAE dictionary vectors to text concepts, it is reasonable that the interpretability of concepts with low alignment is not good.\n+ **S1.** The paper proposes fine-tuning the SAE dictionary vectors by regularization to align them with specific text concepts in order to improve the interpretability of DN-CBM. This strengthens the mapping between dictionary vectors and specific text concepts, improving interpretability but degrading performance.\n### Weaknesses\n- **W1.** The paper does not show strong motivation for the need to reproduce the DN-CBM experiments. The DN-CBM is one of many CBMs that are still being developed, and it is unclear why the DN-CBM in particular should be the focus of attention. The goals of the verification addressed by the paper and the new knowledge advancement it provides to the community are unclear.\n- **W2.** The experimental facts provided by the paper are mostly trivial, and the differences are small. The paper discusses the fact that the claim (ii) cannot be partially reproduced. However, it seems to point out the issues that arise when re-evaluating DN-CBM from a different perspective (e.g., low alignment concepts or large sample size) rather than the original paper is not reproducible. In addition, the fact that the SAE dictionary vectors are forcibly mapped to a finite number of concepts is a clear limitation of the DN-CBM design, and it is theoretically obvious that this can not provide a perfect interpretation. From this perspective, the proposed extension method emphasizes this limitation, suggesting that DN-CBM, which has a trade-off between interpretability and performance, is impractical.\n- **W3.** The paper arbitrarily restricts the content of the reproducing experiments. In particular, human intervention is an important aspect when evaluating CBM, and it is unnatural not to re-evaluate this. The justification for excluding a re-evaluation of human intervention is lacking because the verification goal addressed by the paper is unclear as mentioned in W1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reports on a reproducing experiment of DN-CBM, which was presented at ECCV 2024. The paper investigates DN-CBM in three perspectives: (i) the ability of DN-CBM to automatically discover concepts, (ii) the interpretability, and (iii) the performance. While the paper confirms the reproducibility of (i) and (iii), it reports potential issues related to interpretability that were not discussed in the original paper. Specifically, it points out that the automatically discovered concept dictionary vectors extracted by sparse auto-encoder (SAE) cannot accurately provide interpretability for input images when the cosine similarity with the original textual concept vectors is low. To solve this issue, the paper proposes fine-tuning the SAE dictionary vectors by regularizing them to align with specific text concepts. The proposed method confirms that even concepts with relatively low alignment have high interpretability for humans.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please clarify the motivation for re-evaluating DN-CBM (see W1).\n- DN-CBM limits the number of explainable concepts to the dimension $h$ of the SAE dictionary matrix. The paper should add a discussion of what impact scaling $h$ has on interpretability and performance.\n- Please evaluate human intervention. In particular, what changes do the proposed methods make to the intervention results?\n- Ultimately, DN-CBM may be able to achieve full interpretability by replacing the SAE dictionary vectors with the corresponding text concept vectors. Is it possible that static replacement may produce better interpretability results than applying the proposed method? I would recommend adding this discussion.", "broader_impact_concerns": "Nothing to report.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kA8y8T4S1U", "reviewer": "Reviewer_XTw2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts\nThe writing style and presentation is clear. All experiments seem to be technically well performed and the code is provided via a GitHub repository. Remarkably, the authors have also independently verified the results of a user study presented in  Rao et al. (2024). Overall, the paper gives a good overview of the capability of the DN-CBM method. \n\n# Weaknesses\nThe problems around (C2, interpretability) are not further discussed. This is a missed opportunity since faithful interpretability is core for any method in XAI. In particular also the proposed extension does not address this point successfully (see discussion below). \nThe paper closely resembles the original paper. While this is necessary for a reproducibility study, some well-designed additional investigation of DN-CBM would help to judge the robustness of the original findings. In particular, regarding the interpretability claim C2. \n\nThe paper nicely highlights the problems around low-aligned concepts in Fig. 4. However, the authors lack to discuss this phenomenon, its consequences and where it originates from. More details how the problem of incoherent concepts originates from the underlying methodology would be helpful for the community. For example, why is it not possible to introduce a threshold and exclude concepts with low similarity from the final CBM model? \n\nComplementary, I spotted concepts names which are very close to the predicted class (Fig. 9d, GT: Horse with concept horses&horseback) or nearly identical (Fig. 15, eg. Huskies&husky or dogs & dog). This seems to be a general issue of the DN-CBM methodology as it is also present in the extended method. This has the potential to severely limit the usefulness in practice. Identifying a concept with the same name as the predicted class does not provide new information about the model reasoning. Thus, this deserves further investigated. One possible idea would be to modulate the size/capacity of the concept vocabulary excluding too similar names. \n\nLastly, the proposed extension (based on regularization) seems to be overly restrictive. It has significant impact on the performance of the CBM model even though this is major selling point of the original methodology. Also the resulting concepts are very narrow and closely related to the original class. This raises questions regarding the usefulness of this extension in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reproduces the results of Rao et al. (2024) along the lines of three central claims: (C1) automated concept discovery, (C2) interpretability and (C3) performance. The findings of the paper suggest that (C1) and (C3) can be verified whereas some questions regarding (C2) arise. Further, the authors propose an extension by explicitly regularizing the concept embedding space based on pre-defined concepts. They presents results showing that this improves the concept coherence at the price of reduced predictive accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More details around C2 as outlined above. A fully conclusive solution to this problem is out of scope but some effort should be put into improving the manuscript in this regard. In particular, this central findings should more prominently mentioned and the resulting limitations for DN-CBM further discussed. For example consider mentioning this finding also in the abstract. \n\nThey present a distribution of concept similarity for the proposed extension in appendix Fig. 16. It might be worth moving sub-figure C=10^-4 into Fig. 2 in the main part. This would enable to compare it to the original methodology and provide more intuition on the effect of the regularization. I leave this up to the authors to decide.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers are satisfied with the paper and the authors' rebuttals.
Decision: Accept (poster)