Selective Concept Bottleneck Models Without Predefined Concepts
Authors: Simon Schrodi
Julian Schur
Max Argus
Thomas Brox
Venue: Accepted by TMLR ()

Abstract: Concept-based models like Concept Bottleneck Models (CBMs) have garnered significant interest for improving model interpretability by first predicting human-understandable concepts before mapping them to the output classes. Early approaches required costly concept annotations. To alleviate this, recent methods utilized large language models to automatically generate class-specific concept descriptions and learned mappings from a pretrained black-box model’s raw features to these concepts using vision-language models. However, these approaches assume prior knowledge of which concepts the black-box model has learned. In this work, we discover the concepts encoded by the model through unsupervised concept discovery techniques instead. We further leverage a simple input-dependent concept selection mechanism that dynamically retains a sparse set of relevant concepts of each input, enhancing both sparsity and interpretability. Our approach not only improves downstream performance, but also needs significantly fewer concepts for accurate classification. Lastly, we show how large vision-language models can guide the editing of our models' weights to correct model errors.

Published in Transactions on Machine Learning Research (05/2025)
Selective Concept Bottleneck Models Without Predefined Concepts
Simon Schrodi University of Freihurg
schrodiGcs.unis wni-freibbug. de
Julian Schur"
Mnllon.echurrsstudee edu
University of Freibury, Korlsruhe Institute of Technology
Max Argus University of Freiburg
sanimmmessuunfffiiuryy de
Thomas Brox University of Freibury
broz watissuuii-ffrfrrrg de
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- PHO30TLI41
Abstract
Concept-based models like Concept Bottleneck Models (CBMs) have garnered significant interest for improving model interpretability by first predicting human--ndderstandabll con- cepts before mapping them to the output classes. Early approaches required costly concept annotations. To alleviate this, recent methods utilized large language models to automat- ically generate class-specifie concept descriptions and learned mappings from & pretrained black-box model's features to these concepts using vision-language models. However, these approaches assunee prior knowledge of which concepts the black-box model has learned. In this work, we discover the concepts encoded by the model through unsupervised concept discovery techniques instead. We further leverage a simple input-dependent concept selec- tion mechanism that dynamically retains a sparse set of relevant concepts of each input, enhancing both sparsity and interpretability. Our approach not only improves downstream performance, but also needs significantly fewer concepts for accurate classification. Lastly, show how large vision-language models can guide the editing of our models' weights to correct model errors.
1 Introduction
Deep neural networks have achieved tremendous success in a variety of tasks on various input modalitios. However, they are black-boz models, making it difficult for humans to understand and comprehend their decisions. Thus, there has been considerable recent interest in developing interpretable models. One popular framework is Concept Bottleneck Models (CBMs) (Koh et al., 2020), i.e., models that first predict human- understandable concepts and then use these concepts predict the classes (Lampert al., 2009; Kumar et al., 2009). Initial CBMs are trained in an end-to-end fashion through supervision on both the concepts and classes. However, the need for human-annotated concepts during model training requires the time-consuming and expensive collection of such.
To address this limitation initial CBMs, recent work (Yuksekgomul et al., 2023; Oikarinen et al., 2023; Menon & Vondrick, 2023; Laguna et al., 2024; Dominici et al., 2024) has proposed converting pretrained black-box models into CBMs in post-hoc fashion. To avoid the need for annotations, they leveraged large language models (e.g., GPT-3 (Brown al., 2020)) to generate class-specific language descriptions and learned mapping from the black-box model's uninterpretable features to these concepts using vision- language models (e.g., CLIP (Radford et al., 2021)). However, this raises a crucial question:
"Equal contribution.

---

Published in Transactions on Machine Learning Research (05/2025)
Raw bottlemeck features
Pretrained model
Unsupervised concept discovery
Concept set & visualizations
(Section 2.1)
Datasct
Alignment scores
Raw bottleneck features
Ireput- dependent
Pretrained model
Sparse
Prediction
0.20
0.00
0.01
concept
0.00
linear
Class O: 0.100
0.60
selection
0.57
classifier
Class 1: 0.002
0.09
(Section 2.2)
0.00
Class 2: 0.700
Input
0.50
0.34
sample
Figure Overview of Unsupervised Concept Bottleneck Models (UCBMs). Top: We propose to extract concepts from raw bottleneck features of pretrained black-box model using an unsupervised concept discovery method (Section 2.1). Bottom: We compute the alignment between the bottleneck's features and previously discovered concepts (middle). Finally, we train an interpretable classifier consisting of our proposed input-dependent concept selection mechanism and sparse linear classifier (middle to right, Section 2.2).
How con we know a priori which concepts a pretrained black-boz boz model has learned?
Instead of defining the concepts in advance, we propose to discover concepts that accurately decompose the features learned by the black-box model. To do so, we draw from the rich literature on unsupervised concept discovery (Ghorbani et al., 2019; Zhang et al., 2021; Zou et al., 2023; Fel et al., 2023b; Vielhaben et al., 2023; Fel et al., 2023a; Huben et al., 2024; Stein et al., 2024). We chose CRAFT (Fel et al., 2023b) for our experiments because has been shown to yield human mamaan-nddrsttadable concepts (Fel et al., 2023a), but other techniques
also possible. CRAFT employs non-negative matrix factorization (Lee & Seung, 1999) to decompose each feature activation into sparse lincar combination of concept vectors. The set of shared concept vectors forms a dictionary matrix. After learning this dictionary matrix, we compute the alignment between the raw bottleneck features and the concept vectors to measure concept's presence or absence.
Subsequently, we train an interpretable lincar classifier on the concepts' alignment scores, linking the align- ment scores to the predictions. Previous work (Yuksekgonul et al., 2024) has shown that sparsity penalty on the linear classifier's weights ensures that each class relies only sparse set of concepts. However, they did not examine the per-sample number of concepts that affect the classification across all classes. That is, while individual classes rely on sparse sets of concepts, the overall model depends substantially more. Empirically, we found that typically 90% of the avail- able concepts up to ca. 4200 concepts (see Table 1) affect the classification per input. As result, it complicates the interpretation of the model's classification.
al., 2023; Oikarinen et al., 2023; Srivastava
To address these challenges, we propose an input-dependent concept selection mechanism that ensures that only a sparse set of concepts relevant for the classification of an individual input sample is dynamically retained. We achieve this by applying non-linear function before the sparse linear classifier to filter out (i.e., zero out) concepts. We enforce the filtering either by forcing the output of the non-linear function to be sparse or by directly controlling its sparsity through its hyperparamcter. In our experiments, the TopK function (Makhzani & Frey, 2014) performed best. This mechanism allows the concepts that are retained or removed to vary between inputs, making it input-dependent. Importantly, it also preserves the interpretability of CBMs, as the predictions remain linear w.r.t. the retained concepts. Finally, we show that also effectively controls information leakage; a common problem of CBMs (Mahinpei et al., 2021; Yan et al., 2023; Srivastava et al., 2024).
2

---

Published in Transactions on Machine Learning Research (05/2025)
In summary, our contributions are follows:
We propose a new type of CBM called Unsupervised Concept Bottleneck Models (UCBMs)'; see Figure 1 for an overview. UCBMs convert pretrained, black-box models into a CBM by discovering and using the concepts that the black-box box model has learned.
We propose an input-dependent concept selection mechanism that dynamically retains a sparse set of concepts relevant to classification. For example, as few as ca. 1.4% of the available concepts are used per imput (Table 1).
We show that UCBMs improve performance while having a substantially higher degree of sparsity com- pared to previous work (Figure 3) and effectively controls information leakage (Figure 5).
We show that UCBMs are interpretable qualitatively and through a user study (Section 3.2), and show that large vision-language models can help us lo intervene on UCBMs' weights to fix errors (Section 3.3).
2 Unsupervised Concept Bottleneck Models with input-dependen concept selection
In this section, we introduce Unsupervised Concept Bottleneck Models (UCBMs), a novel CBM that uses concepts that are automatically discovered and most accurately decompose the features learned by a black- box model (Section 2.1), dynamically only retains the concepts most relevant classification of each input, and finally classifies the input with sparse linear model (Section 2.2). Figure 1 provides an overview of our method, and the above steps are described in detail below.
Notations. Let f : X'' + Rp be a pretrained, black-box model's feature extractor that maps from an input space X' C Rd to the bottleneck feature space of a size of p. Further, let X E Ndd be the input data matrix where the 7th row is the input x; E X and let = s(X) E RNx be the bottleneck feature activations. Lastly, let y denote the class label space.
2.1 Discovery of concepts learned by the black-box model
Previous post-hoc CBMs have either used human-annotater concepts (Yuksekgomul et al., 2023; Laguna et al., 2024; Dominici al., 2024) or aligned the black-box modcl's features with procomputed text fea- tures from vision-language models, using natural language descriptions, such as those generated by a large language model (Yuksekgomul et al., 2023; Oikarinen et al., 2023; Menon & Vondrick, 2023; Laguna et al., 2024). Importantly, both approaches rely on predefined set of concepts-cither through concept annota- tions or language descriptions thereof-implicite assuming which concepts the black-box model has learned. However, the concepts are typically unknown in advance.
Discovering the concepts that the black-box model has learned. To address this, we propose using unsupervised concept discovery techniques for UCBMs. These enable us to discover the concepts that the black-box model has actually Icarned, and do not roquire defining the concepts in advance.
Formally, the goal of unsupervised concept discovery to extract a small set of interpretable concepts C that most faithfully reconstruct the feature activations A. Assuning linearity of concepts, as per the superposition hypothesis (Kim et al., 2018; Elhage et al., 2022), unsupervised discovery methods can be understood as an instance of a dictionary learning problem (Dumitrescu & Irofti, 2018):
(U*, C*) arg
aig min || UC U,C
(1)
where U E N.CC (sparse coefficient matrix) represents the activations A = s(X) E RNX w.r.t. new basis spanned by the set of |C| concept activation vectors C E [gC[Xp (dictionary matrix), and ||-||r denotes the Frobenius norm. Intuitively, we learn a sparse linear decomposition of the feature activations of each input in Equation 1, where we weigh the shared concept vectors by the input-specific sparse coefficients. Fel et al. (2023a) showed that previous methods, such as K-Means (Ghorbani et al., 2019), PCA (Zhang et al., 2021; Zou al., 2023), non-negative matrix factorization (Lee & Seung, 1999; Olah et al., 2018; Zhang al.,
Code is available at bttps://githhe cottuuub.oommmmmma www./oooo//raaooeooeeooeeeeeeeee caabb-ffrrbuurguetmmm
3

---

(a:ee) is the cosine |a:12-|| Published in Transactions on Machine Learning Research (05/2025)
2021: McGrath et al., 2022; Fel et al., 2023b), sparse autoencoders (Makhzani & Frey, 2014; Huben et al., 2024), only differ in their constraints on U, C in Equation 1.
In this work, we chose non-negative matrix factorization (i.e., CRAFT (Fel et al., 2023b)) for UCBMs, as it has been shown to discover human- undert concepts (Fel et al., 2023a). However, we emphasize that UCBMs will benefit from future unsupervised concept discovery methods.
2.2 Learning the classifier with input-dependent concept selection
In the previous subsection, we discovered concept vectors Cj that most accurately decompose the uninter- pretable features of black-box model. Next, we compute the alignment scores between cach concept vector and the model's features, denoted sxm [-1,,]|C|, where simc(x:); similarity between the feature activations f(x)) = a, of input x; and concept сソ E C. Then, we dynamically select the most relevant concepts and subsequently classify the input with a sparse linear model (Wong al., 2021). Both are described in detail below.
Sparse linear classifier. Following Yuksekgonul et al. (2023); Oikarinen et al. (2023); Srivastava et al. (2024), we learn a sparse lincar classifier by enforcing sparsity on its weight matrix (Wong et al., 2021):
min W.b
((Wsimc(X.) + +b,y)) + Aw Ra (
(2)
CW sparsity
where We E D[x[C] are the weights, b E RD b is the bias, y1 E E is the target class for input X,, repre- sents the task-specific loss function cettosyyntropy loss throughout this work), Au controls the regularization strength on W, and R. (W): := (1 a) | || + all W|],, denotes the elastic net regularization (Zou & Hastie, 2005). Note that (x; is normalized and frozen during optimization. Importantly, the sparsity aims to make the linear model's classifications sparse and Yuksekgonul et al.; Oikarinen et al. & Srivastava et al. have shown that an individual class indeed relies on only a sparse set of concepts.
The main limitations with only applying sparsity on the weights W are that it fails to produce globally sparse classifications and is input-indeennent. This lack of (global) sparsity limits interpretability and makes it challenging to comprehend prediction. Specifically, we found that even when concept is non-visible, it either for the predicted class or any other class (Table 1). We consider concept to be actively contributing if it has a non-zero influence on the output (see Equation 7 for details). The reason that the concepts are non-zero and, consequently, influence classification is that the cosine similaritics between the black-box model's activations and concepts are generally non-zero."
impacts classification ieite
Input-dependent concept selection mechanism. To ensure that only few concepts affect classification per imput without significant performance sacrifices, we propose a simple yet effective dependen concept selection mechanism. Specifically, we introduce a concept selector IR CI - RICI, which takes the alignment scores (x;; as input and outputs a sparse set of non-zero (i.c., active) scores and zeroes out the others. We enforce sparsity through a penalty term on concept. selector's output: C || Intuitively, the sparsity penalty Capariity drives the concept selector to only retain a sparse set of concepts which are important for classifying the input x,, as signaled by the task-specific loss L in Equation 2.
We considered three candidates for the implementation of the input-dependent concept selection mechanism (please refer to Appendix C for further technical details):
ReLU: We define the concept selector using the ReLU activation function as:
* (xx) max(0, sim o) with trainable offset parameter o E R
(3)
We apply clastic net regularization on the selector's output: Caparsity = (a(T(())).
While the classifier could technically "turn off" concept cj by setting its associated column vector to the null vector (W., = 0), this would effectively reduce the number of concepts and degrades performance, e.g., see Figure 4. Consequently, the sparse linear classifier unlikely learn many of such mull vectors.
4

---

Published in Transactions on Machine Learning Research (05/2025)
JumpReLU;
We use JumpReLU activation function (Erichson et al., 2019) for concept selection with trainable offset parameter o E i and the Heaviside step function H. We define the concept selector as:
(x.) == sx.) - ((sim((x.) -x) -
0. sx,c simc(x,) >
simc(x.) S
(4)
Following Rajamanoharan et al. (2024), we compute the gradients of the expected loss using straight-throuul--ssitttii (Bengio et al., 2013). We use the following sparsity penalty spaarity Σ/e H (x.m - -o)). Note that Sipariity directly optimizes LO.
TopK: The TopK activation function (Makhzani & Frey, 2014) only keeps the h < the largest alignment scores and zeroes out the remaining concepts:
|C| concepts with
π (x,) == TopK, (simc(x,) o) with trainable offset parameter o E IR+
(5)
Note that the sparsity can be directly controlled by k and, thus, saprriity = 0.
Final interpretable classifier. We obtain the final interpretable classifier by plugging Equation 3, 4, or 5 into Equation 2 logether with the respective implementation of a and sparalty
min W.b.o
Σ ((Wa(x))+ + Aw C Sparsiitty Xa Cy sparsity
(6)
where ) (or & for TopK) controls the regularization strength of Caparsity Appendix C provides a detailed overview of all variants. It is important to note that the selection of concepts is learned in an unsupervised manner, and that the prediction remains linear w.r.t. the active concepts (T(x,) * 0).
Concept dropout. During initial experiments, found that models became overly reliant on a single concept. To reduce this reliance, we added a dropout layer (Srivastava et al., 2014) after concept selection. As dropout is applied per concept, it encourages the model to spread its classification decisions across more concepts. Interestingly, we found that this could also improve performance.
3 Experiments
We evaluated UCBM on diverse image classification tasks and compared it to relevant basclines. We show that UCBMs outperform prior work and narrow the gap to their black-box counterparts, while relying on substantially fewer concepts globally interpretability qualitatively as well as through a user study (Section 3.2). Lastly, we showcase how large vision-language models can be leveraged to intervene on UCBMs by informing weight editing in order to fix model errors (Section 3.3). Appendix N provides further analysis on the out-of-distribution robustness, fairness, and shape vs. texture bias of UCBMs.
their classification (Section 3.1). Then, we demonstrate the
Datasets & black-box feature backbones. The CBMs are evaluated on ImageNet (Deng et al., 2009) with a pretrained ResNet-50 V2 (He et al., 2016), CUB (Wah et al., 2011) with ResNet-18 pretrained on CUB, and Places-365 (Zhou et al., 2017) with ResNet-18 pretrained on Placcs-365. These datasets cover a diverse set of lasks from standard image classification (emageNet), fine-grained classification (CUB), to scene recognition (Places-365). Experiments with Inception and transformer feature backbone are done Appendix F. We find that UCBMs achieve performance close to the original black-box models, consistent with the results observed for the ResNet feature backbones in Table 2.
Implementation details. We trained our UCBMs with Adam (Kingma & Ba, 2015) and cosine anncaling learning rate scheduling (Loshchilov & Hutter, 2017) for 20 epochs. We used learning rate of 0.001 on ImageNet and Places-365, and 0.01 on CUB; except for the JumpReT.U for which we set it to 0.08 on CUB. We set = 0.99 for the elastic net regularization for all variants. We tuned the other hyperparameters (Ax or k, Aw. and dropout rate) to yield a good trade-off between performance, sparsity, and fair comparability. Refer to Appendix D for the hyperparameterss and to Figure 6 and Appendix G for their effect.
Models are provided https://gittaei httpttttww...o/pporr/itttsttoooe (ImageNet), https:// /tttt wwwtttuu..oo/eeeeeeeee con/osmr com/osmtringcceeo ingelemob (CUB).
and htap::://gtthan comub..com/eee won./rustworthy-ttt.... Lab-Laaaaaaaaa--aaaaaaaaaaaaaala CBM (Places-365).
5

---

Published in Transactions on Machine Learning Research (05/2025)
Concept id
1985
Concept id
1283
1283
551
1766
1766
1985
981
1722
0.00
0.25
0.00
0.25
Cosine similarity
Cosine similarity
Figure 2: The discovered concepts exhibit faithful behavior. Removing the saw blade (right) from the original image (left) shrinks the aligument score of the respective concept 1985 (blue). Concepts are represented by their most activating crops. Additional results are provided in Appendix A.
Experimental setup. Since the number of concepts |C| substantially influence downstream performance (see Figure 4), we set |C] proportional to the number of classes with various (expansion) factors (0.5, 3,3,). All models were trained on single NVIDIA RTX 2080 GPU and a full training run took from few minutes to a maximum 1-2 days depending on dataset size and number of concepts |C]. We report top-1 accuracy on the standard holdout sets throughout our experiments.
Baselines. We compared our UCBMs to Post-hoc CBM (Yuksekgomul et al., 2023), Label-free CBM (Oikarinen al., 2023), and VLG-CBM (with NEC 5) (Srivastava et al., 2024), as they are the most related to our work and the latter is the current state-of-the-an CBM. Note that Post-hoc CBM requires concept annotations and is therefore not applicable ImageNet and Places-365. Finally, we compared our concept selectors with the binary (latent) indicator concept selector proposed by Panousis et al. (2023). We reproduced the baseline results using their respective original codebases.
Quality of the discovered concepts. Before evaluated UCBMs, we verified that the discovered concepts behave faithfully. For this, we analyzed the change in cosine similarities between feature activations and concepts after the removal of relevant image parts of a certain concept; see Figure 2 and Appendix A. For example, as we remove the saw blade (concept 1985), the cosine similarity of the aforementioned concept decreases from ca. 0.5 to around 0.25 (Figure 2). We also manually verified that concepts are semantically consistent and numan-undeestandablee The quality can be exemplarily seen in the top activating crops (i.e., the top-n crops are selected based the cosine similarity of their bottleneck feature activations to the concept) throughout this paper. In addition, evaluated the degree of polysemantictty in Appendix B.
3.1 Sparsity and performance results
How sparse are UCBMs' decisions? Previous work evaluated sparsity based on (the average per- class) number of non-zero weights in W (Oikarinen et al., 2023; Srivastava et al., 2024). However, these approaches fail to consider two important factors: (1) how many concepts influence classification across all inputs (globally), and (2) that certain concepts can be inactive for specific inputs, e.g,, their value is zero.
To account for the aforementioned, we propose to compute the average number of concepts that actively influence the classification decision for each input x,. Formally, we consider concept сj active for the classification of input x; if
Cj Txx) 4
A 3 (1, 131}} for which W wi #
la the concept c, active? dloes the concept have an effect on any class?
Tables 1 and 5 show that UCBMs with concept selection use substantially fewer concepts than UCBM without concept selection and the other baselines. For example, on ImageNet, UCBM with TopK concept selector uses an average of 42.0 concepts per input, while Label-free CBM, VLG-CBM, UCBM with binary indicator concept selection, and UCBM without concept selection use averages of 4238.0, 3018.97, 1995.7, or 3000.0, respectively. We find similar differences for CUB and Places-365.
How good is the performance of UCBMs? Table 2 shows that UCBMs mostly outperform the basc- line methods across all datasets, while being substantially sparser (Tables and and Figure 3). The
6

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: The concept selection mechanism leads to substantially fewer concepts being used in the classification. We report the mean percentage number of active concepts according Equation w.r.t. to the total number of concepts Cl. Absolute numbers are provided in Table 5 in Appendix Label- free CBM, VLG-CBM, UCBM without concept sclection, and UCBM with binary indicator use many concepts than our UCBM variants with concept selection.
Mean number of active concepts/ CC (4)
Method
ImageNet
CUB
Places-365
Post-hoc CBM (Yuksekgonul et al., 2023)
n/a
100%
n/a.
Label-free CBM (Oikarinen al., 2023)
93.74%
99.95%
90.64%
VLG-CBM (Srivastava et al., 2024)
70.21%
98.66%
63.27%
UCBM w/o concept selection
100%
100%
100%
UCBM with binary indicator (Panousis et al., 2023)
66.52%
100%
49.28%
UCBM with ReLU concept selector
1.59%
30.5%
8.9%
UCBM with JumpReLU concept selector
1.43%
31.15%
9.11%
UCBM with TopK concept selector
1.4%
32.1%
8.88%
Table 2: UCBMs mostly outperform the baselines and reduce the gap to the original, black-box
model. We report mean top-1 accuracy with standard deviation across three training runs (we kept the discovered concepts fixed). Note that the methods use different levels of sparsity (see Table 1) and refer Figure 3 that plots sparsity against performance.
Top-1 lest accuracy (f)
Method
Sparse?
ImageNet
CUB
Places-365
Original, black-box model
X
80.9
76.7
53.69
Post-hoc CBM (Yuksekgonul et al., 2023)
C3333
SS 333
n/a
60.10
n/a
Label-free CBM (Oikarinen et al., 2023)
78.09
74.38
50.67
VLG-CBM (Srivustava et al., 2024)
78.78
75.44
51.67
UCBM w/o concept selection
79.80 + 0.027
75.15 + 0.037
52.41 + 0.028 50.91 0.105
UCBM with binary indicator (Panousis al., 2023)
77.42 + 0.056
74.93 1: 0.309
UCBM with ReLU concept selector
79.07 +-2 0.029
74.61 + 0.128
50.86 + 0.021
UCBM with JumpReLL concept selector UCBM with TopK concept selector
79.49 0.016
74.57
0.290
51.24 0.019
79.32 + 0.009
74.96 0.083
51.20 + 0.050
performance-sparsiiy trade-off is visualized in Figure 3, where we control the number of active concepts (ac- cording to Equation 7) by varying the hyperparameter for UCBM with TopK concept selector, or X for UCBM with ReLU or JumpReT.U concept selector. We find that (o.g., UCBMs without concept sclection) unsurprisingly outperform our UCBM variants with concept seloc- tion. However, the UCBM variants with concept selection are substantially sparser and typically achieve at least competitive, but mostly superior, task performance to the baselines. For example, our UCBM variants outperform all basclines on ImageNet and all but VLG-CBM on CUB and Places-365. Besides that, Figure 3 shows that one can control the sparsity-perffrmance trade-off through the respective hyperparameters. This allows practitioners to set these hyperparameters according to their desired balance between sparsity (and better interpretability) and performance, based on the requirements of their application.
models that allow for more concepts
Efffec of the total number of concepts C. We found that performance is strongly influenced by the total number of concepts [C] used. In Figure 4, we varied the number of concepts to assess this and, as expected, find that increasing |C] improves performance. Note that UCBMs achieve competitive but mostly superior performance (Table and Figure 3) while using smaller number of concopts |C].

---

Published in Transactions on Machine Learning Research (05/2025)
ImageNet
CUB
Places-365
UCBM w/ ReLU
80.0
50
UCBM w/ JumpReLU
test acc
UCBM w/ TopK
77.5
UCBM wlo concept selector
X
Post-boc CBM
Label-frce CBM
10a
10a
10a
60
10a
10a
10a
111
10a
10
VLG-CBM
UCHM w/ binary indicator
active concepts
active concepts
active concepts
Figure 3: Trade-off curves between sparsity and performance. We plot the mean number of active concepts per imput according to Equation 7 we decrease k (for TopK) or increase A (for the others). For UCBMs we plot the Pareto-curves. The UCBMs are substantially sparser than the baselines (see also Table 1). Our UCBMs Pareto-dominate all baselines ImageNet and Places-365, while only being outper- formed by VLG-CBM on CUB (though it has substantially higher sparsity).
ImtgeNet
CUB
Places-365
80
32.5
test ace
UCBM w/ ReLU
30.0
UCBM w/ JumpRoLU
UCCB w/ TopK
72
UCCM wio concept selection
70
2000
number of concepts CC]
number of concepts |C]
number of concepts |C]
Figure 4: The more concepts [C], the better UCBMs' performance. We varied the total number of available concepts [C As expected, the more available concepts |C), the better the performance.
ImageNet
CUB
Places-365
Test acc
50
w/ discovered concepts
w/ random concepts
10
10a
10
10'
10a
active concepts
active concepts
active concepts
The TopK concept selector effectively controls information leakage. Previous work showed that even using random concepts could yield strong CBMs, suggesting information leakage. However, performance with random concepts declines quickly, whereas it remains consistently high when using the discovered concepts when varying k the TopK concept selector.
Figure 5:
Concept selection effectively controls information leakage. Recent work showed that CBMs' con- cept prediction may encode unintended class information (Margeloiu et al., 2021). For example, even many random concepts can achieve strong downstream performance (Yan et al., 2023; Midavaine et al., 2024; Sri- vastava et al., 2024). Figure 5 shows that & effectively controls information leakage, as the performance of random concepts quickly drops when using smaller (fewer active concepts).
Sensitivity analysis. We varied An (Figure 6a), k (Figure 6b), and dropout rate (Figure 6c) to analyze their impact on sparsity and performance. We find that only controls sparsity (Equation 7) in TopK, whereas for the other concept selectors, all hyperparameters affect sparsity (see Appendix G). We consider this is as an advantage of TopK, as disentangles the effect of the hyperparamoters. This is discussed in more detail in Appendix G. For performance, we find that larger A and smaller k lead to worse performance. For dropout rate, there typically seems to be a sweet spot.
3.2 Interpretability of UCBM
Explainable sample-wise decisions. Figure shows qualitative examples of the most contributing con- cepts with their contribution strength (contribution of concept cj to class yı: (xy.)=(x)))l)) We find that
8

---

Published in Transactions on Machine Learning Research (05/2025)
42.5
80
42.5
avg active concepts
avg active concepts
avg active concepts
test ace
test
42.0
test
78.5
42.0
78.0
78.0
111
10
10
10
10a
0.00
0.25
Auo
k
dropout rate
(a) Performance vs, Aw.
(b) Performance vs. k.
(c) Performance vs. dropout.
Figure 6: Sensitivity analysis over Au (a), k (b), and dropout (c) on ImageNet. Larger Au and smaller & worsen performance, though smallor & increases sparsity. There is no clear relation for dropout (also across other datasets). Results for the other datasets and concept selectors are provided in Appendix G.
Concept id
1012
Concept id
15
193
1189
198
2336
others
others
5
0.0
2.5
Concept contribution
Concept contribution
(a) 'tiger' (etageNet)), conf.: 93.68%.
(b) Ameriiaa goldfinch' (CUB), conf: 92.51%.
Figure 7: Decisions of UCBM with TopK concept selector rely on a few reasonable and diverse concepts. Results on ImageNet (a) and CUB (b). Additional examples are provided in Appendix H.
the most contributing concepts are relevant to both the input and prediction, while also being diverse. For example, UCBM with TopK concept selector focuses concepts such 'tiger striped fur', 'whiskers' big cats' snouts' for the tiger in Figure 7a, the 'bright yellow plumage' of the American goldfinch in Figure 7b.
Figure 8 compares the explanation of our UCBM with TopK concept selector, Label-free CBM, and VLG- CBM (more examples are provided in Appendix H). We find that UCBM relies on fewer concepts, that are present in the image and relevant to the predicted class. In contrast, Label-free CBM and VLG-CBM often rely on concepts that are correlated with the predicted class but absent the image. This is especially pronounced for misclassifications (Figures 20f 20i in Appendix H).
User study on explainable sample-wise decisions. To corrobo- rate the qualitative results above, we conducted user study assess the interpretability of UCBM with TopK concept selector compared to Label-free CBM (we omitted VLG-CBM due to its qualitative similarity to Label-free CBM, see Figure 8 and Appendix H). Specifically, we evalu- ated the comprehensibilityy of the explanations. Note that the approaches present their concepts differently: UCBM and Label-free CBM use visual or textual concept representations, respectively. Thus, for fair compari- son, we labeled concepts or retrieved images using SigLIP SoViT-400m (Zhai et al., 2023; Alabdulmohsin et al., 2023). Further details the user study design provided in Appendix
Which model is more comprehensible?
UCBM
Late froe CBN
100
Percentage of evaluations
Figure 9: Users strongly prefer UCBM. From clearly UCBM (blue) to clearly Label- free CBM (red).
Figure 9 shows that users strongly preferred UCBM over Label-free CBM, corroborating the qualitative results shown in Figures 7 and 8 and Appendix H. Further analysis is provided Appendix T.
Explainable class-level decision rules. To derive class-level decision rules, we computed the average contribution of each concept for a class. Figure 10 shows the top-3 concepts for two classes. We find that UCBM with TopK concept selector focuses reasonable, human-understandable concepts relevant. to

---

Published in Transactions on Machine Learning Research (05/2025)
672
4210
2223
Concept id
2283
443
2018
809
(needed rE2 ddgs
2907
1810
1963
2918
4061
2350
2652
others
UEae WisSOws
0.0
2.5
0.0
2.5
top
Concept contribution
Concept contribution
Concept contribution
13
35
34
Concept id
2253
NOT 2907
9:00
Einegale daye
1880
weafty perso
edegeen Easier
others
others
others
Beg
10
sacurity
0
10
Concept contribution
Concept contribution
Concept contribution
Figure 8: The decisions of UCBM with TopK concept selector (left) are more comprehensible than those of Label-free CBM (middle) and VLG-CBM (right). Our approach relies on concepts that are present in the image and relevant to the prediction, whereas Label-free CBM and VLG-CBM tend to use concepts that are not even present, which is particularly pronounced for misclassification. Appendix H provides additional examples. Best. viewed digitally and with zoom.
Concept id
2746
Concept id
113
187
187
1878
129
others
40
0
20
40
60
Avg concept contribution in (a) 'pineapple' (ImageNet).
Avg concept contribution in %
(b) 'tree sparrow' (CUB).
Figure 10: UCBM with TopK concept selector identifies class-relevant concepts (represented by the most activating crops). Results for ImageNet (a) and CUB (b). Additional examples are provided in Appendix J.
each class. For example, Figure 10a shows that UCBM bases its classification of pineapples on the typical 'pincapple's texture' or its 'laaves..
3.3 Case study: Correcting errors using multi-modal LLM
In this subsection, we show how a multi-modal LLM (GPT-4o (Achiam et al., 2023)) can guide us to correct errors in UCBMs (specifically, a UCBM with TopK concept selector trained on ImageNet). We prompted the model asking to adjust the weights of the sparse lincar classifier W UCBMs (Equation 6) correct an error without affecting the classification of other inputs. The prompt included the misclassified input image, the top-5 concepts, and their contributions for both the misclassified and correct class. For example of the prompt, see Appendix M. During initial experiments, we found that the suggested changes, AW, were sometimes too strong, leading to errors of previously correct inputs. To address this, we ran grid search on the training set of ImageNet to find optimal weighing factors 3 E [0, 1] for each proposed change AW;
Figure 11 shows three examples that were correctly classified after applying the weight adjustments proposed by the LLM. This demonstrates the intervenability of UCBMs and illustrates the potential use case of multi- modal LLMs to automatically identify and correct the traceable causes of errors of UCBMs.
10

---

Published in Transactions on Machine Learning Research (05/2025)
GT: loggerhead turtle
AW. (3.)
loggerhead turtle
concept
volcano
LVLM
+0.056 (0.75)
+ classification information
0.01 (0)
GT: tench
AW; (8.)
LVLM
concept
tench
eel
+0.125 (0.5)
classification information
GT: flamingo
AW: (8.)
concept
flamingo
goose
LVLM
+0.06 (0.5)
0.01 (0)
classification
information
example
prodiction, before after edit
test accuracy, before after edit
volcano + loggerhead sea turtle
79.322 + 79.322
eel tench
79.322 + 79.328
goose flamingo
79.322 + 79.326
Figure 11: UCBMs are intervenable. We used a multi-modal LLM to help us to correct errors by guiding the odits the weights of UCBM with TopK concopt selector (I: 42) that was trained on ImageNet.
4 Related works
Concept-based models. Concept Bottleneck Models (CBMs) (Koh al., 2020) are trained directly leverage concepts in their classifications (Lampert et al., 2009; Kumar et al., 2009). Many works highlighted (and partially addressed) the limitations of them (Margeloiu et al., 2021; Mahinpei et al., 2021; Havasi et al., 2022; Marconato et al., 2022; Raman et al., 2024). Other work improved the trade-off (Espinosa Zarlenga et al., 2022; Yang et al., 2023) or extended them beyond image classification (Tsmail et al., 2023; Zarlenga et al., 2023) (see Appendix how UCBM can be also extended to such).
The most related methods to our work convert a pretrained black-box model into a CBM post-hoc (Yuk- sekgomul al., 2023; Oikarinen al., 2023; Menon & Vondrick, 2023; Laguna et al., 2024; Dominici al., 2024). These approaches alleviate the need for costly concept annotations by leveraging language models, like GPT-3 (Brown et al., 2020), automatically generate class-specific descriptions and vision-language models, like CLIP (Radford el al., 2021), to learn mapping from a black-box model's uninterpretable lea- tures to these concepts. In contrast to these, we do not presume which concepts the black- box model has learned, but find the ones that most accurately decompose the black-box model's features in an unsuper- vised manner. Concurrently, Rao et al. (2024) discovered concepts with sparse autoencoders to transform CLIP into a concept- classifier. In contrast to the aooee works, we also introduced novel input-dependent concept selection mechanism that dynamically retains only sparse set of concepts per input.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Concept discovery. Early work searched for neuron-aligned concepts (Bau et al., 2017; Olah et al., 2017), while later works, inspired by the superposition hypothesis (Kim et al., 2018: Elhage et al., 2022), went beyond this to (linear) vector (Kim et al., 2018; Zhou et al., 2018; Olah et al., 2018; Ghorbani et al., 2019; Zhang et al., 2021; McGrath et al., 2022; Zou et al., 2023; Fel et al., 2023b; Huben et al., 2024; Stein et al., 2024), linear subspace (Vielhaben et al., 2023), or density-based (Vielhaben el al., 2024) concept representations. Early work nceded costly annotated datasets to find concepts through supervision. Later work overcame this bottleneck by formulating concept discovery as a dictionary learning problem (Fel al., 2023a).
Model editing. Model editing aims to modify model's weights remove bias or correct errors. Previ- ous work edited knowledge in large language models (Zhu al., 2020; Meng et al., 2022), generative image models (Bau et al., 2020; Oldfield et al., 2023; Gandikota et al., 2023), modified classifier's predic- tion rules (Santurkar et al., 2021; Oikarinen et al., 2023). These works relied on, e.g., human intervention, factorization, or hypernetworks, whereas we leverage large vision-language models to inform model editing.
5 Limitations & future work
The main limitation (or advantage) of our approach is that discovered concepts are only represented visually, not textually. While images may be more informative, texts aid faster and easier interpretability. To obtain textual descriptions of concepts, we could manually label concepts. However, this does not scale to large amounts of concepts. Thus, we also experimented with automatic concept labeling through large vision-language models (GPT-40 (Achiam et al., 2023)), see Appendix K for details. While we found it to yield overall good concept descriptions, we also found many instances with poor descriptions; especially for non-object-centric or more abstract concepts. Thus, we reviewed and edited, or manually crafted concept descriptions as needed.
Another limitation of our approach is that we only extract concepts from the bottleneck layer of black-box models. We conjecture that the use of concepts throughout the feature hierarchy of these models may be beneficial for concept-based models in terms of performance and/or interpretability, as such a hierarchy is also learnod by black-box modcls (Zeiler & Fergus, 2014). For instance, carly layer could find concepts for 'windows', 'car body', or 'wheels', while later layer assembles them to 'car' concept (Olah el. al., 2020).
Lastly, UCBMs inherit the limitations of unsupervised concept discovery methods, such as not fully resolved polysemanticity (Graziani et al., 2024) (see Appendix B for an empirical ovaluation) or concepts possibly not encoding the intended semantics (Mahinpei et al., 2021; Marconato et al., 2023; Bortolotti el al., 2024). Future work could explore discovery methods that are guaranteed (under certain conditions) identify the true concepts encoded by black-box model (Leemann et al., 2023).
6 Conclusion
We presented UCBMs, which convert pretrained black-box models into interpretable concept-based models by discovering the concepts that the model has learned through unsupervised concept discovery. We further introduced an input-dependent concept selection that effectively only retains the concepts most relevant for classifications of each input. Our experiments show that UCBMs outperform previous methods, while being substantially sparser globally. Finally, we qualitatively and quantitatively validated the interpretability of UCBMs, and showcased how molti-modal LLMs can guide the editing of UCBMs to correct its errors.
Broader impact statement
There are many potential positive as well negative societal impacts of our work. However, we do not see any particular impact specific to our work that does not apply to the general impact of advancing the field of concept-based models, a subfield of machine learning.
Acknowledgments
This research was funded by the German Research Foundation (DFG) under grant numbers 4 417962828 and 499552394.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. ar 2023.
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design. In NewrIPS, 2023.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. In CVPR, 2017.
David Bau, Steven Liu, Tongzhou Wang, Jun- Yan Zhu, and Antonio Torralba. Rewriting a Deep Generative
Model. In ECCV, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv, 2013.
Samuele Bortolotti, Emanuele Marconato, Tommaso Carraro, Paolo Morettin, Emile van Krieken, Antonio Vergari, Stefano Teso, and Andrea Passerini. A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts. In NewrIPS, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Noc- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NewrIPS, 2020.
Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Classes Are Not Equal: An Empirical Study on Image Recognition Fairness. In CVPR, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Gabriele Dominici, Pietro Barbiero, Francesco Giannini, Martin Gjoreski, and Marc Langhenirich. Any- CBMs: How to turn any black box into concept bottleneck model. arXiv, 2024.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
Bogdan Dumitrescu and Paul Irofti. Dictionory Learning Algorithms and Applications. Springer, 2018.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy Models of Superposition. arXiv, 2022.
N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney, JumpReLU: A Retrofit Defense Strategy for Adversarial Attacks. arXiv, 2019.
Mateo Espinosa Zarlenga, Pictro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Con- cept Embedding Models: Beyond the Accuracy-Explainabiie Trade-Off. In NearIPS, 2022.
Thomas Fel, Victor Boutin, Louis Béthune, Rémi Cadène, Mazda Moayeri, Léo Andóol, Mathieu Chalvi- dal, and Thomas Serre. A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. In NeurIPS, 2023a.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. CRAFT: Concept Recursive Activation Fac PacTorization for Explainability. In CVPR, 2023b.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Brasing Concepts from Diffusion Models. In ICCV, 2023.
Leo Gao, Tom Dupre Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Πly Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
Robert Gcirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A Wichmann, and Wicland Brendel. ImageNet--rrinee CNNs are biased towards lexture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision.
In NewrIPS, 2021.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards Automatic Concept-based Expla- nations. In NewrIPS, 2019.
Mara Graziani, Laura O' Mahony, An-Phi Nguyen, Henning Müller, and Vincent Andrearczyk. Uncovering Unique Concept Vectors through Latent Space Decomposition. TMLR, 2024.
Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez, Addressing leakage in concept bottleneck models. In NewrIPS, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse Autoen- coders Find Highly Interpretable Features in Language Models. In TCLR, 2024.
Aya Abdelsalam Ismail, Julius Adebayo, Hector Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In TCLR, 2023.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- protability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICML, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept Bottleneck Models. In ICML, 2020.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and Simile Classifiers for Face Verification. In ICCV, 2009.
Sonia Laguna, Ričards Marcinkevičs, Moritz Vandenhirtz, and Julia Vogt. Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable? In NeurIPS, 2024.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning To Detect Unseen Object Classes by Between- Class Attribute Transfer. In CVPR, 2009.
Daniel D Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 1999.
Tobias Leemann, Michael Kirchhof, Yao Rong, Enkelejda Kasneci, and Gjergii Kasneci. When are Post-hoc Conceptual Explanations Identifiable? In UAI, 2023.
Tly Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In ICLR, 2017.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and Pitfalls of Black-Box Concept. Learning Models. Workshop@CCMI, 2021.
Alireza Makhzani and Brendan Frey. K-Sparse Autoencoders. In ICLR, 2014.
Emanuele Marconato, Andrea Passerini, and Stefano Teso. GlanceNets: Interpretabile, Leak- proof Concept-
based Models. In NeurIPS, 2022.
Emanuele Marconato, Andrea Passerini, and Stefano Teso. Interpretability is in the Mind of the Beholder: A Cansal Framcwork for iimaan-nterpretable Representation Learning. Entropy, 2023.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do Concept Bottleneck Models Learn as Intended? Norkshop@ICLR, 2021.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašov, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in AlphaZero. PNAS, 2022.
Kevin Meng, David Ban, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In NewrIPS, 2022.
Sachit Menon and Carl Vondrick. Visual Classification via Description from Large Language Models. In ICLR, 2023.
Nesta Midavaine, Gregory Hok Tjoan Go, Diego Canez, Ioana Simion, and Satchit Chatterji. [Re] On the Reproducibility of Post-Hoc Concept Bottleneck Models. TMLR, 2024.
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free Concept Bottleneck Models. In ICLR, 2023.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2017.
Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The Building Blocks of Interpretability. Distill, 2018.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction Circuits. Distill, 2020.
James Oldfield, Christos Tzelepis, Yannis Panagakis, Mihalis A Nicolaou, and Ioannis Patras. PandA: Unsupervised Learning of Parts and Appearances in the Feature Maps of GANs. In ICLR, 2023.
Konstantinos Panagiotis Panousis, Dino Ienco, and Diego Marcos. Sparse Linear Concept Discovery Models. In Workshop@ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transforable Visual Models From Natural Language Supervision. In ICML, 2021.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autocncoders. arXiu, 2024.
Naveen Raman, Mateo Espinosa Zarlenga, Juyeon Heo, and Mateja Jamnik. Do Concept Bottleneck Models Obey Locality? Workshop@NeurIPS, 2024.
Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schicle. Discover-then-Name: Task-Agnostc Concopt Bottlenecks via Automated Concept Discovery. ECCV, 2024.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing Classifier by Rewriting Tts Prediction Rules. In NeurIPS, 2021.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Divyansh Srivastava, Ge Yan, and Tsui- Weng. VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance. In Newr/PS, 2024.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Πyy Sutskever, and Ruslan Salakhutdinov. Dropout:
A Simple Way to Prevent Neural Networks from Overfitting. JMLR, 2014.
Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards Compositionality in Concept Learning. In JCML, 2024.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper with Convolutions. In CVPR, 2015.
Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-diimensional concept discovery (MCD): A unifying framework with completeness guarantees. TMLR, 2023.
Johanna Vielhaben, Dilyara Bareeva, Jim Berend, Wojciech Samek, and Nils Strodthoff. Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers. arXiu, 2024.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Dataset, 2011.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging Sparse Linear Layers for Debuggable Deep Networks. In ICML, 2021.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley, Learning Concise and Descriptive Attributes for Visual Recognition, In ICCV, 2023.
Yue Yang, Arlemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. In CVPR, 2023.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc Concept Bottleneck Models. In ICLR, 2023.
Mateo Espinosa Zarlenga, Zohreh Shams, Michael Edward Nelson, Been Kim, and Mateja Jamnik. TabCBM: Concept-based Interpretable Neural Networks for Tabular Data. TMLR, 2023.
Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. In ECCV, 2014.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer Sigmoid Loss for Language Image Pre-Training. In ICCV, 2023.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible Concept-based Explanations for CNN Models with Non-negative Concopt Activation Vectors. In AAAI, 2021.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 Million Image Database for Scene Recognition. IEEE TPAMI, 2017.
Bolci Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable Basis Decomposition for Visual Explanation. In ECCV, 2018.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying Memories in Transformer Models. arXiv, 2020.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks. Representation Engineering: A Top-Down Approach to AI Transparency. arXiv, 2023.
Hui Zou and Trevor Hastic. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Slatistical Methodology, 2005.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Concept id
606
606
920
Concept
920
1908
548
93
1843
1590
0.0
0.5
0.0
0.5
Cosine similarity
Cosine similarity
(a) Removing the head and neck of an ostrich makes concepts 654 (green), 549 (red), and 1843 (purple) disappear from the top-5 cosine similarities.
Concept id
1569
1569
302
Concept
302
1693
472
1779
568
0.0
Cosine similarity
Cosine similarity
(b) Removing the ears an angora rabbit makes concept 1698 (green) disappear from the top-5 cosine similarities.
Concept id
493
ConcepL id
493
2043
1513
1513
2043
2975
2777
2932
2701
0.00
0.25
0.00
0.25
Cosine similarity
Cosine similarity
(c) Removing the neck an acoustic guitar makes concept 2975 (red) disappear from the top-5 cosine similarities.
Figure 12: Concepts discovered in unsupervised manner exhibit faithful behavior. Concepts are represented by their most activating image crops. From the original image (left), manually removed image parts (right) and computed the concept-activation cosine similarities for an ostrich (a), angora rabbit (b), and acoustic guitar (c). We find that cosine similarity scores reduce, as we remove an image part where that concept or these concepts were previously present.
A Additional results for the faithfulness of discovered concepts
Figure 12 provides additional results for the faithfulness of the discovered concepts. In Figure 12a removing the head and neck of the ostrich the imput image makes concepts 654 (groen), 549 (rod), and 1843 (purple) disappear from the top-5 cosine similarities. Since concepts 654, 549 and 1843 represent parts of an ostrich's 's head neck, this demonstrates the faithfulness of the discovered concepts. Figures 12b and 12c show similar behavior for rabbit's 's cars and guitar's neck, respectively.
B Evaluation of concept polysemanticity
The goal of unsupervised discovery methods is disentangle the concepts from the original feature space. That is, typically find that individual neurons represent multiple different concepts. Although concept dis- covery reduces the degree of polysemanticity, recent work shows that some level of polysemanticity remains, albeit significantly reduced (Graziani et al., 2024).
To evaluate the level of polysemanticity, we visually investigated whether the top-9 most activating image crops for cach concept appear to be mono- or polysemantic (sce Figure 13 for examples). Specifically, we randomly selected 100 concepts discovered our ImageNet experiment. We found that 12 of these concepts exhibited polysemanticity, while the remaining 88 appear to be monosemantie. This indicates that most
17

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Monosemantic concepts.
(b) Polysemantic concepts.
Figure 13: Examples of mono- and polysemantic concepts.
Table 3: Overview of interpretable classifiers. In the equations below, let 5(x)) == (x; denote the normalized cosine similarity between activations f(x) = a, for input and the concepts C, W RDiI|C and E RDl are the weights and bias of the linear classifier, Vi y' denotes the target class of imput x; for total of |}] classes, L denotes the task-specific loss function (cross-entropy loss throughout this work), Ra is the elastic net regularization penalty (Zou & Hastie, 2005), Aw, An govern the regularization strengths, H denotes the Heaviside step function, and TopK denotes the TopK activation function (Makhzani & Frey, 2014). Note that 8(x;) is frozen during optimization. Further, note that the TopK concept selector does not need a sparsity penalty since sparsity can be controlled directly using the hyperparameter
Hx( is trainable offset parameter,
name
concept selector π
interpretable classifier
ReLU
Txx;)= = max(0. (0, 8(x;) o)
min W.b.o
L (WT((x; + b. v1) + Au Ro (W) + A (=(x))))
C
JumpReLU
<xx;) = 8(x)) (8((x:)) -
min W.b.o
[(Wx(xx)) + b, R.. (W) + Am Σ (x()(x))) o,)
TopK
T(x)) = (8(x.) o)
min Σ ((WT(x)) + b, - A,, R. W.b.o
concepts discovered by the unsupervised discovery are monosemantic, but a number of concepts still retain polysemantic characteristics.
C Further details on the interpretable classifiers
Table 3 provides the complete overview of the interpretable classifiers for all of our UCBM variants from Section 2.2. Below, we provide further details for the JumpReLU and TopK concept selectors.
JumpReLU concept selector. The JumpReLU activation function (Erichson et al., 2019) is defined as follows:
(x) x H(x { 0, x x>o
(8)
where H is the Heaviside step function. Note that we cannot directly train our offset parameter o. Thus, (Bengio et al., 2013) to make o trainable. Specifically, we adopted the pscudo-derivates from Rajamanoharan et al. (2024):
following Rajamanoharan et al. (2024), we used
(umpReLU,(x)
K(
(9)
d0
18

---

Published in Transactions on Machine Learning Research (05/2025)
Table Hyperparameter settings for all UCBMs variants on ImageNet
|
CUB
Places-365.
Am
k
Aw
dropout rate
UCBM w/o concept selection
n
n/a
UCBM with ReLU concept selector
2e-5
1e-4
2e-5
n/a
1e.4
8e-4 4e-4
0.1 0.2
0.2
UCBM with JumpReLU concept selector UCBM with TopK concept selector
lo-5
4c-7
4c-7
n/a
n/a
42
66
| 162
and
Ô H(x o) 1
-
(10)
do
where ô denotes the psoudo-derivative, K is a kernel (following Rajamanoharan et al. (2024) we used the rectangle function: rect(x): == H(x ( H(x - 2)), and can be seen as the KDE bandwidth.
TopK concept selector. The TopK activation function (Makhzani & Frey, 2014) is defined as follows:
TopK,(x); = ={0- X, if x; E top-k(x), otherwise,
(11)
We used the TopK implementation of Gao et al. (2025) who internally apply a non-linearity (we used ReLU) after the actual TopK function. Note that we can directly control the sparsity through the hyperparamcter k and the TopK concept selector becomes equivalent the identity function k = -Cl.
Why do we add trainable offsct paramcter o? We introduce the additional trainable offset param- eter o TR. C allow the classifier adapt to different ranges of alignment scores for each concept. The reasons for this is that the distribution of scores can vary between concepts. For example, for one concept, the scores may be more uniformly distributed, indicating more ambiguous presence of the concept. For another concept, the scores might follow bimodal distribution, indicating two distinct modes that indicate the object is present or absent. The offsct paramcter helps the classificr in such cases to account for such different distributions.
D Hyperparameter settings
Table 4 provides the hyperparameters (Az, k, Aws dropout rate) for all our UCBMs variants. We chose those hyperparameters such that they yielded a good trade-off between performance, sparsity, and fair comparability (see Figure and Appendices and G). is important to note that we first optimized A, for the ReLU and JumpReLU concept selectors and then set i accordingly, as we found that its relationship sparsity (c.f., Equation 7) is straightforward.
E Number of concepts used per class prediction
Table provides the absolute numbers for Table UCBM variants with concept selector use substantially fewer concepts than prior CBMs and UCBM without concept selection or binary indicator (Panousis et. al., 2023).
Beyond the sparsity measurements in Tables and 5, we computed how many concepts the models need to explain their prediction of a class. For this, we computed the mean number of concepts that are required to explain 95% (or 90%) of a model's prediction per sample:
Σ
W (xxit(())ell
oEC
C. where min |C s.t. CCC
2 95%
(12)
N
Σ
W (x)ce
cEfL.....CC)
19

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: The concept selection mechanism leads to substantially fewer concepts being used in the classification. We report the mean number of active concepts with standard deviation according Equation 7. Parentheses show the total number of concepts |C]. Label-free CBM, VLG-CBM, UCBM without concept selection, and UCBM with binary indicator use many more concepts than our UCBM variants with concept selection.
Mean number of active concepts (c.f. Equation 7)
Method
ImageNet
CUB
Places-365
Post-hoc CBM (Yuksekgonul et al., 2023)
n/a
112.0 (112)
n/a
Label-free CBM (Oikarinen et al., 2023)
4238.0 (4521)
211.9 (212)
1820.0 (2008)
VLG-CBM (Srivastava et al., 2024)
3018.97 (4300)
661.99 (671)
1382.99 (2186)
UCBM w/ w/o concept selection
3000.0 (3000)
200.0 (200)
1825.0 (1825)
UCBM with binary indicator (Panousis et al., 2023)
1995.7 (3000)
200.0 (200)
899.3 (1825)
UCBM with ReLU concept. selector
47.8 (3000)
61.0 (200)
162.4 (1825)
UCBM with JumpReLU concept selector
42.8 (3000)
62.3 (200)
166.2 (1825)
UCBM with TopK concept selector
42.0 (3000)
64.2 (200)
162.0 (1825)
Table 6: UCBM with TopK concept selector requires less concepts to explain a prediction. We
report the mean and the standard deviation of the number of concepts that are required to explain 95% of the prediction (see Equation 12 for more details).
#concepts to explain 95% of the prediction (Equation 12)
Approach
ImageNet
CUB
Places-365
UCBM w/o concept selection
8.79 + 8.093
5.79 + 1.774
46.1 + 11.594
UCBM with ReLU concept selector
3.83 + 2.323
4.7 + 1.586
15.72 + 4.032
UCBM with JumpReLU concept selector
5.05 + 3.334
4.53 + 1.679
25.05 + 8.068
UCBM with TopK concept selector
4.95 +: 2.933
5.25 + 1.747
24.72 + 8.04
where き denotes the model's prediction of input x,.
Table 6 shows that UCBMs with concept selector rely on fewer concepts than UCBM without concept selection. Note that relying on fewer concepts makes easier for users to comprehend prediction since they do not need to inspect lot of concepts.
Figure 14 provides detailed per-class analysis of the mean number of concepts required to predict cach class, ie., to explain 90% or 95% of the prediction (Equation 12). While some classes only rely on a few or even only a single concept, others require substantially more. For example, on ImageNet, certain classes such as 'goldfish', 'great grey owl', 'trilobite', 'quail', 'hornbill', 'abacus", 'bell or wind chime', 'harp', jigssaw puzzle', 'marimba', 'maze', 'graduation cap', 'mousctrap', 'piggy bank', 'pinwheel', 'pool table', 'solar thermal collector', 'umbrella', 'water tower', 'crossword', 'jackfruit", and hoose chestnut seed'- typically only use single concept. In contrast, other classes such as 'Redbone Coonhound', 'Tibetan Terrier', "Golden Retriever', 'patas monkcy', 'titi monkcy', and 'monastery'-rqquie substantially more concepts on average (13).
F Other feature encoder choices
Table 7 shows the results for InceptionV3 (Szegedy et al., 2015) and ViT-B/16 (Dosovitskiy et al., 2021) feature encoders, both pro-trained on ImageNet." Consistent with the findings in Section 3.1, UCBM with TopK concept selector achieves performance close to the original, black-bor models. While we maintainod
Both (black-box) models are provided http:///ggtuub.ccmpppttrraah
20

---

Published in Transactions on Machine Learning Research (05/2025)
ImageNet
CUB
Places-365
Cum. concept contribution
1.0
1.0
0.8
0.8
0.8
0.6
0.4
0.1
0.2
0.2
0.2
10
Top-10 10 concepts per class (sorted)
Top-10 concepts per class (sorted)
Top- 10 concepts per class (sorted)
#classes explain 90%
200
60
40
150
20
10
10
10
15
20
25
30
Number of concepts
Number of concepts
Number of concepts
#classes explain 95%
150
30
100
20
50
10
2
10
12
2
10
20
30
40
Number of concepts
Number of concepts
Number of concepts
Figure 14: Analysis of number of concepts used per class. Top panel: Cumulative contribution of the top 10 concepts per class. Middle and bottom panels: Number of concepts required to reach 90% 95% cunulative contribution per class, respectively. While some classes rely only few concepts, others rely more.
Table 7: UCBM also performs well with InceptionV3 and VIT backbone on ImageNet.
Method
ImageNet top-1 test accuracy
Original InceptionV3 UCBM w/ TopK
77.29
73.23
Original ViT-B/16 UCBM w/ TopK
81.01
77.94
the same sparsity level for UCBM when using InceptionV3 as the feature encoder, applying the same level to ViT led to performance drop, necessitating a reduced level of sparsity. We hypothesize that non-negative matrix factorization may not be the most effective approach for extracting concepts from ViT's non-negative feature space. Exploring alternative concept discovery methods, such as sparse autoencoders, could allow to restore higher levels of sparsity with less performance compromises.
21

---

Published in Transactions on Machine Learning Research (05/2025)
64.5
162.5
1000
400
1est 800
75.0
2vg active
concegis
test acc
avg active
concepts
concepts
test ace
aWg active conceptl
64.0
182.0
30
200
74.8
aog
50.0
63.5
161.5
10a
10"
Axa
Au
80
(a) Performance vs. Au-
(b) Performance vs.
64.5
162.5
avg active concepts
avg active concepts
test acc
test aoc
7.4
64.0
51.0
162.0
63.5
161.5
0.0
0.2
0.4
0.00
dropout rate
dropout rate
(c) Performance vs. dropout.
Figure 15: Scnsitivity analysis for UCBM with TopK concept selector over Aw (a), &o (b), and the dropout rate (c) for CUB (left) and Places-365 (right).
ace
79.0
60
avg uctive concepts
avg conce pts
lest ace
822
160
DVg active 1o09939
test
78.5
74.6
78.0
Aue
(a) Performance vs, Age.
80.0
100
test noc
200
avg active concepts
avg active
concepts
test nce
400
77.5
74
200
10-6
10
10-3
Am
Av
(b) Performance vs. Av.
test acc
79.0
nVg active concepts
avg concepts
test ace
avg active concepts
60.6
78.8
150
712
0.00
0.25
0.0
01.2
0.4
0.00
0.25
dropout rate
dropout rate
dropout rute
(c) Performance dropout.
Figure 16: Sensitivity analysis for UCBM with ReLU concept selector over Xu (a), A (b), and the dropout rate (c) for ImageNet (left), CUB (middle), and Places-365 (right).
G Additional sensitivity analysis results
Figure 15 provides the results for the sensitivity analysis for UCBM with TopK concept selector CUB and Places-365. Figures 16 and 17 provide the results for UCBM with ReLU or JumpReLU concept selector, respectively.
We find that the hyperparameters (for TopK) or Ag (for ReLU and JumpReLU) control the trade-off between performance and sparsity (sce also Figure 3). Regarding the other hyperparameters, A and dropout rate, is important to observe that they have less influence on the sparsity for the TopK concept selector than for the other concept selectors. We consider this advantage of the TopK concept selector, it
22

---

Published in Transactions on Machine Learning Research (05/2025)
76
Test ncc
DVg active concepts epts
avg active
conneept
test ace
62.5
500
DVg activve concepts
80
60.0
47.5
Aw
(a)
Performance vs. Au.
100
test aoe
500
avg active concepts
75
active
concepts
test ace
80
500
av gr active concepts
17
75
40
78.5
73
30
10
10
Am
A
(b) Performance vS. An-
79.5
test ncc
avg active concepts
80
V active
contosts
test acc
avg active concepts
100
151
79.0
60
200
0.00
0.20
0.0
0.22
0.4
0.0
12.2
01.
dropout rate
dropout rate
dropout rate
(c) Performance vs. dropout.
Figure 17: Sensitivity analysis for UCBM with JumpReLU concept selector over Au (a), A (b), and the dropout rate (c) for ImageNet (left), CUB (middle), and Places-365 (right).
reduces the interaction between hyperparameters. This makes hyperparameter tuning simpler and simplifies the interpretation: & governs the average number of active concepts per sample, Aw governs the number of concepts used per class, and the dropout rate influences whether the classifier relies on broader or narrower set of concepts.
For Aw; we find that increasing typically leads to worse performance and a smaller average mumber of active concepts per sample. Interestingly, for the UCBMs with ReLU concept selector trained ImageNet and Places-365, we observe the opposite behavior. For the dropout rate, higher dropout rate results in more active concepts per sample, though its relationship with performance is less clear.
Additional examples of explainable decisions
Additional examples for sample-wise explanations. Figure 18 provides more examples of explainable decision of UCBM with TopK concept selector ImageNet, CUB, and Places-365. We typically find that our method relies on a small set of concepts that are present in the images, human-comprehensiln class-relevant. For instance, for the viaduct in Figure 18a, UCBM uses class-relevant concepts (e.g., 'arches', 'stones', or 'walkway'). For the 'railroad track' Figure 18c, it uses concepts such as 'tracks' or 'train'. Interestingly, it also uses the concept 'large window' that is also related to, e.g., buses. This indicates that UCBMs first assess if concepts are present or absent and then based on that evidence predict the class that is most likely given that.
and
Understanding misclassifications of UCBMs. Figure 19 shows that we can comprehend why UCBMs made a misclassification. For example, Figure 19a shows that the UCBM incorrectly predicted 'car wheel' instead of 'station wagon'. However, the image shows such station wagon mirrored in a car wheel. Looking at the most contributing concepts reveals that UCBM focused on concepts that are related to the car wheel, it is the most salient in the image.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Concept id
413
Concept id
171
1595
42
469
125
701
53
1974
87
others
others
5
0.0
2.5
Concept contribution
Concept contribution
(a) 'viaduct" (InaggNNet), conf.: 98.39%.
(b) "parakeet auklet' (CUB), conf.: 99.13%.
Concept id
717
Concept id
1648
428
197
651
297
1339
464
1588
148
others
others
0.0
2.5
Concept contribution
Concept contribution
(c) railroad track' (Places-365), conf.: 76.15%.
(d) 'art gallery' (Places-365), conf.: 72.55%.
Figure 18: Explainable decisions by UCBM with TopK concept selector ImageNet (a), CUB (b), and
Places-365 (c, d) classes. The model's prediction comprehensible and typically rely only few concepts.
Additional examples for the comparison of UCBM Label-free CBM and VLG-CBM. Fig- ure 20 compares the explanations of UCBM with TopK concept selector, Label-free CBM (Oikarinen el al., 2023), and VLG-CBM (Srivastava al., 2024). We find that approach provides more comprehensible ex- planations: UCBM relies intuitive concepts that are present in the image and relevant to the prediction. In contrast, Label-free CBM and VLG-CBM tend to rely on concepts that are correlated to the predicted class but may not be present in the image, e.g., the concepts 'graduation markings' or 'graduation ceremony' for the prediction "graduation cap' Figure 20d.6 6 We quantified the usage non-visible concepts in the predictions of each method in Table 8. Note that such reliance on prediction-clat
correlated but absent concepts is particularly pronounced for misclassifications (Figures 20f to 20i and Table 8). For example in Figure 20h showing broom nearby lake, Label-free CBM relies the concepts 'mellow, flute-like sound', 'wind instrument', of rope or string', or 'Mexican food'. None of these concepts are present in the image. We believe relying on such non-visible concepts not helpful to understand the decision of a concept-based model.
'bagpipe'. Similarly, VLG-CBM relies the concepts 'tangled twisted shape', 'made
Further details on the user study
In the user study, we studied whether users consider the explanations of the decisions of UCBM to be comprchensible. To do so, compared the explanations of UCBM with TopK concept selector with Label- free CBM (Oikarinen et al., 2023). Both were trained on ImageNet.
Task. We asked users to assess which model provides a more comprehensible explanation from a scale from 'Model A clearly more' 'Model clearly more'. Further, we asked for the reasons why they think one model more comprehensible than the other.
"These qualitative findings are further corroborated in the user study in Section 3.2 and Appendix
We suspect the reason for this are shortcomings of the vision language models used in both approaches. For instance, the concept 'graduated cylinder' is unrelated the prediction of 'graduation cap' in Figure 20d. However, the word 'graduated is related to 'graduation! Indoed, when compute the cosine similarity of text features (we considered the following: 'graduated cylinder', 'graduation ccremony', 'graduation markings' 'graduation', 'university", 'dog', 'house'), found that concepts related graduation have higher similarities with the graduated cylinder than the unrelated concepts. We leave further investigations future work.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Concept id
1373
Concept id
2410
1474
2968
341
1556
1940
1121
608
2740
others
others
Concept contribution
Concept contribution
(a) GT: "station wagon', pred.: 'car wheel..
(b) GT: felt', pred.: 'bottle cap?.
2880
Concept id
2169
Concept
1940
2218
1963
809
464
2482
others
others
0
2
Concept contribution
Concept contribution
(c) GT: 'granny smith apple', pred.: 'goblet?.
(d) GT: 'car wheel', pred.: 'sports car'.
ImageNet of UCBM with TopK concept selector. The image shows station wagon mirrored in wheel. Most of the top-5 concepts are related to car wheels, which explains that the model only focuses the car wheel itself instead of the mirrored station wagon. This clearly explains why the model predicts 'car wheel' instead of 'station wagon'. b: The image shows an eft next to a bottle cap. The concepts show that the model used concepts related to bottle caps, which is the object at the center of the image. c: The image shows two granny smith apples next to goblet that was predicted by the model. The concepts reveal that the model focuses on concepts related to the goblet at the center of the image. d: The image shows sports car, including one of its front wheels. The most important concept is related to sports cars. The other concepts also focus more general car concepts than on the wheels.
Figure 19: The most contributing concepts explain the misclassifications
User study data. We showed sample-wise (local) explanations based which concepts contributed the most to the decision of each model, akin to Figures 8, 10, 18 and 19. Importantly, 20% of samples showed misclassifications of both models (for the other 80% both model predicted correctly). 7 We include misclassificationn to also understand how comprehensible models are under errors. We believe this is an important aspect to study, as users will also interact with models that make errors practice. For sake of this user study, we simplified the explanations by removing the concept contributions and only showed the names and top-activating image crops of the five most contributing concepts and corresponding concept description.
Note that UCBM and Label-free CBM represent their concepts differently: UCBMs show visual representa- tions, whereas Label-free CBM shows concept descriptions. To ensure fair comparison, we labeled the most activating image crops of UCBM's concepts and retrieved images using SigLIP SoViT-400m (Zhai et al., 2023; Alabdulmohsin et al., 2023) for Label-free CBM's concepts.
Setup. We implemented the user study in a lightweight. Python GUT so that users could run the study locally on their machines. Users were provided with the task description (Figure 21) and an example (Figure 22). After the instruction, users interacted with our user study interface (Figure 23).
We asked ten users to rate a total of 200 samples (20 per user). Users participated voluntarily and without payment. They have strong background machine learning and related fields. However, none of them is working concept-based models had seen explanations of UCBM before.
No sample for which model correct and the other was incorrect was shown the study.
25

---

Published in Transactions on Machine Learning Research (05/2025)
352
3520
628
Concept id
949
2685
3962
15
NOT-4483
1736
2,002
3828
429
NOT 2178
Iom
Concept contribution
Concept contribution
Concept contribution
(a) GT: 'cougar', UCBM: 'cougar' (99.99%), Label-free CBM: 'cougar' (95.97%)
VLG-CBM: 'cougar' (97.40%
1513
Concept id
2975
2446
deivie
2715
lee billow bods
2043
3948
493
1444
1837
1811
others
Neggeiigt bover
others
kep Tedee body
others
Concept contrihution
Concept contribution
Concept contribution
(b) GT: 'acoustic guitar', "acoustic guitar' (68. 18%)
UCBM: 'acoustic guitar'
(99.93%), Label-free CBM: 'acoustic guitar' (42.35%). VLG-CBM:
680
3182
1930
Concept id
2568
4744
1145
1809
1785
655
2724
13
1258
1815
4245
NOT
4250
Inte soer odeer
0.0
2.5
10
Concept contribution
Concept contribution
Concept contribution
(c) GT: PPtti dish', UCBM: 'Petri dish' (85.84%). Label-free CBM: Petti dish
VLG-CC%) Petri dish
(89.22)) (2%)
1142
1620
811
Concept id
2940
91.5
3605
peunsiion voter
grwburoon marting
Nase Sieer
others
others
others
head
0.0
2.5
aduaad Syancer
padage
Concept contribution
Concept contribution
Concept contribution
(d) GT: "graduation cap', UCBM: 'graduation cap' (88.92%). Label-free CBM: 'graduation cap' (70.78%), VLG-CBM: "graduation cap' (79.5%%)
Concept id
277
Hat Will might reloe spob.
3958
stadiakk
1865
NOT 3317
545
71
2399
2846
2756
Jinsy Hark
NOT 943
1482
others
others
others
Concept contribution
Concept contribution
Concept contribution
(e) GT: 'spotted salamander', UCBM: 'spotted salamander' (90.35%), Label-free (92.77%), VLG-CBM: 'spotted salamander' (77.39%)
CBM:
'spotted salamander'
Further analysis. Complementary to the results presented in Section 3.2, we conducted further analysis on the results of the user study. Figure 24 shows that users strongly preferred our UCBM with Topk concept selector over Label-free CBM in ca. 65-70% of evaluations (Label-free CBMs are only preferred ca. 15%). Users' preference was similar for correct or incorrect predictions.
Users based their preference decisions mostly on relevance to the prediction (selected in 66.5% of the evalu- ations). However, relevance to the image (55%) and informativeness (55%) closely followed it.
26

---

Published in Transactions on Machine Learning Research (05/2025)
1807
2995
4165
Concept id
2510
3415
1175
1194
4708
214
1353
973
297
3171
others
Concept contribution
Concept contribution
Concept contribution
GT: 'tent', screen' (39.38%)
UCBM: 'sleeping bag' (97.70%), Label-free CBM: 'window screen'
(59.72%), VLG-CBM: 'window
2,099
3098
919
Concept id
2983
390
1301
1480
probction perr
ure regonee aeater
otbers
others
others
0.0
2.5
0.0
2.5
Concept contrihution
Concept contribution
Concept contribution
(g) GT: (32.57%)
'umbrella',
UCBM: 'vespa' (54.38%),
Label-free CBM: 'triumphal arch'
(23. 743.7
VLG-CBM: 'palace'
1558
Concept id
4706
753
2291
144
19
2653
meed, verica oope
63
13
others
others
others
Concept contribution
Concept contribution
Concept contribution
(h) GT: 'broom',
UCBM: 'shovel' (52.98%), Label-free CBM: 'flute' (26.59%). VLG-CBM: 'knot' (48. 52%%
2818
1597
995
Concept id
2021
4341
1890
1712
2831
rapart
yant
others
others
others
0.0
2.5
Concept contribution
Concept contribution
Concept contribution
(i) GT: 'pier", UCBM: 'boathouse' (32.64%), Label-free CBM: 'military uniform' (6.46%), VLG-CBM: 'gas mask' (19.59%)
Figure 20: Comparison of explainable decisions of UCBM with TopK concept selector (left) vs. Label-Free CBM (middle) vs. VLG-CBM (right). Subfigures and f-i show correct or incorrect predictions of the CBMs, respectively. Our UCBM with TopK concept selector provides more comprehensible explanations, while Label-free CBM and VLG-CBM often rely on concepts that are not even visible in the image (this is especially pronounced for misclassifications).
J Additional examples of explainable decision rules
Figure 25 provides more examples of explainable decision rules of UCBM. The examples show that UCBM uses reasonable human-interpretalle concepts to build the score of specific class.
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 8: Number of concepts used in the predictions that are actually visible the image. We report the numbers concepts that are actually visible in the image by inspecting Figures and 20. Our UCBM reliably uses concepts that are actually visible within in the image. In contrast, Label-freo CBM and VLG-CBM frequently use concepts that, while relevant to the predicted class, are not actually present in the given image.
Correct predictions (Figures 8 and 20a to 20e; total of 7)
i-th most important concept
UCBM
Label-free CBM
VL.G-BMM
1st
7/7
5/7
3/7
2nd
6/7
3/7
4/7
3rd
6/7
0/7
4/7
4th
7/7
2/7
3/7
5th
5/7
2/7
1/7
Incorrect predictions (Figures 20 to 20i; total of 4)
i-th most important concept.
UCBM
Label-free CBM
VLG-CBM
1st
4/4
1/4
1/4
2nd
3/4
0/4
0/4
3rd
4/4
0/4
0/4
4th
3/4
0/4
0/4
5th
4/4
1/4
0/4
LGG Club
INSTRUCTIONS
Task: Evaluate and juatfy which model provides more comprehens ble explanetion for it prediction. You navigane using bunons the bottom of the window. After completing the user soudy, please send us the results.jon joon file, which will be saved locally in this folder.
DETAILED INSTRUCTIONS
The purpose this task is acsess which madel betier explains Its predi predictions based on the key concepts ('eatures or attribtts) usos. For each exampte, you will sec: the imaee that causseed, the connect label (GT) above that image, two Al model predictions about the main object the image and the tive key concepts (ordered by imporance) that each model relied on represented both visuslly and and task campana explanations given hon incomeet prediction, may
is compeshwnsible comprehennitie explanation should help you underssand why modell correcs concept represontatinee are imperfact.
prediction that you are not evaluste correcteess predictions, Further, concepts may be abstract, goo-general, speciic.
case which one model
zxplanations.
predictions as visual
any
explanation
up using dictionary search angine.
belleve comprehensible explanation should the following propentes:
BEREEEEAA EVANT TO THE IMAGE
canoopt should be visible the Image and not just correizted to the class. For axample, the prediction "volcano", cencopt like "lav"' relevant the image and only it also visible.
2) REL WELEVVAN TO THE FREDICTION:
A concept should be helpf.l to determine why the image belongs to the eredicted class. For exampe, the prediction "volcano" a concept like "lave" relevant, whereas "turtle" not.
3) INFORMATIVE:
concept should be informative and reveal some intormttion about how the decision was made. For example. it the prodiction "valcana", the concept "valcano" rssett is not informative because simply restatee cless, instead, concepts lice "lav"" or "zaaer" are more informat mativeeeee Similrly, exptenations that use variety distrnct concegts are more informaive than those that repeat similar concepts.
assess
explonations
you beleve more comprehensirre. Flease provide justificaioo for your choice.
completing user study.
Next
Figure 21: Instruction text..
28

---

Published in Transactions on Machine Learning Research (05/2025)
Upe
INSTRUCTION EXAMPLE
GE volcumo
predicts
predicts volcann image concepis importacc)::
image importanc)::
soceepts
entyting volcnno
volcano
tak cloal
lanu
concepts exammple, model explanation more compaehensiii, g. "Pompaii caste").
image (e.g. cloud"
they hightight concepes prodiction (c.g i. "ash "mountain") while (c.g. "uutic")
being (e-g., "erupting dboud)
dlass).
Previous
Figure 22: Instruction example.
SAMPLE 1/20
GT: king peognin
Middl king pangain the imsge occeppts
image pengein
concepts (ordered impurtance):
cing pengsin testars
other pengrims
prngiin
Madehhtti centr
setentic
bird herd
banan sintle corvunre
pespula chick
Which explemetion is comprehensaille?
Why? that apply
Bisroo riensti
Dilcce sbove They ve eq cou
Previous
Figure 23: User study sample.
29

---

Published in Transactions on Machine Learning Research (05/2025)
Which model comprchensible?
Which model comprchensible?
free CBB
CBM
UCBM
UCBM
Labe
Labe
25
50
75
100
25
50
75
100
Percentage evaluations
Percentage of evaluations
(a) Correct predictions only.
(b) Incorrect predictions only.
Figure 24: Users strongly preferred UCBM with TopK concept selector over Label-free CBM
for correct as well as incorrect predictions.
Concept id
2255
Concept id
81
1054
219
103
others
others
Avg concept contribution in
Avg concept contribution in
(a) 'rooster'
(ImageNett.
(b) bluu jay" (CUB).
1311
1271
Concept
558
Concept
1179
993
others
others
20
40
20
40
60
Avg concept contribution in
Avg concept contribution in
(c) 'windmill' (Places-365).
(d) 'bowling alley' (Places-365).
Figure 25: Visualization of decision rules learned by UCBM with TopK concept selector
ImageNet (a), CUB (b) and Places-365 (c, d).
30

---

Published in Transactions on Machine Learning Research (05/2025)
K Concept labeling with a large vision-language model
As an alternative providing the top-activating image crops and manual concept labelling, we also experi- mented with large vision-language models (GPT-4o (Achiam et al., 2023)) to automatically label concepts. We prompted it with the lop-9 image crops and task description:
The nine pictures within the image matching specific concept. Can you describe the concept with very few words (en. 1-3)?
Figure 26 shows the outputted concept labels for twelve, diverse concepts. Overall, we found that concept labels are mostly matching to the top image crops, e.g,, Figures 26a, 26d, 26e and 26k. However, there are also concepts that may not be correctly labelled. For example, the large vision-language model outputs "motorcycle racing" for the image crops in Figure 26b. While this matches well with most of the image crops, it does not for the baseball player (bottom middle) and cyclist (bottom right). We suspect that the concept is representing a more general concept for "safcty equipment" instead. For another example, Figure 26h, the large vision-language model labelled the concept as "ocean textures". However, the image crops more likely resemble starry sky rather than some ocean textures due to the point structure.
metal fencing/
moloreycle racing
(c) leuce fencing
(d) white poodles
wire meshh
moka pot
(f) chains and links
exercise equipment
(h) ocean textures
SPA
BAVS
restaurant table/ dining experience
(i) lighthouses
(k) lifeboat
(I) tree bark/ wood textures
Figure 26: Labeling of concepts using large langunel models. The subfigures' captions are the labeling/descrittinng descriptions that the large vision-language model (GPT-4o (Achiam et al., 2023)) assigned to the provided concept visualizations.
31

---

Published in Transactions on Machine Learning Research (05/2025)
L Applications of UCBMs beyond image classification
Recent work applicd concept-based models to tabular data (Zarlenga et al., 2023) and language models (Ismail et al., 2023). Although our primary focus is image classification the domain where concept-based models have been studied most extensively- UUCMS are also applicable to other domains. Specifically, we first need to find the concepts again. For example, sparse autoencoders have become a popular method for uncovering human--nderstardable concepts in LLMs. Once we have found these concepts, we can train an interpretable classifier, as described in Section 2.2.
M Example prompt to the large vision-language model
Figure 27 shows an example prompt to the large vision- -lonuag model for the misclassification from the lower, left subfigure in Figure 11. Figure 28 shows the corresponding output from the large vision-language model.
N Robustness, fairness, and shape vs. texture bias of UCBMs
In the following analysis of robustness, fairness, and shape texture bias of UCBMs, we focus the UCBM model with the TopK concept selector trained ImageNet.
How robust are UCBMs? We evaluated the dittr--idittiitttinn robustness of UCBMs using the dataset provided in the mooel--vshumaan toolbox (Geirhos et al., 2021), with the corresponding codebase available This dataset includes twelve parametric image dis- tortions, such as uniform noise, rotations, etc. Figure 29 shows that UCBMs exhibit robustness comparable that of the original black-box model. This is expected, as UCBMs likely inherit the biases encoded in the frozen bottleneck features of the black-box model.
at https://github com bettheeabbmoodeleeeeeeee
How fair are UCBMs? Recent work has demonstrated significant disparities in class-wis wise accuracy referred to as "image recognition unfairnes"-even on balanced datasets like ImageNet (Cui et al., 2024). As shown by Figure 30, this unfairness is evident in both black-box models and also our UCBM. Specifically, UCBM achieves a test accuracy of 100% for the best-performing class ('ostrich') and only 20% for the worst-performing class (laatoo computer'). The black-box model (ResNet V2) shows a similar pattern, with 100% accuracy for the best-performing class ('ostrich') and just 16% for the worst-performing class ('laptop computer'). These results consistent with Cui al.'s hypothesis that the underlying representations (frozen bottleneck fcatures of the black-box model), rather than the classifier itsclf, are the primary source of this unfairness.
Are UCBMs more shape or texture biased? To investigate shape vs. texture bias, we used the shape- urx cue conflict dataset introduced by Geirhos et al. (2019), employing the associated codebase available Figure 31 shows that UCBMs exhibit a texture bias similar to that of the original black-box model. This again expected, as UCBMs likely inherit the biases present in the frozen bottleneck features of the black-box model.
at https: /
32

---

Published in Transactions on Machine Learning Research (05/2025)
The image from above leads the following output if we putting into trained concept bottleneck model.
Ground truth: tench Prediction: eel
Top-5 model prediction sorted by the logits:
Class eel: 7.71
Class tench: 7.05
Class silver salmon: 6.76
Class snoek fish: 6,08
Class sturgeon: 5.93
Largest five concept contributions (by absolute value) for ground truth class tench, sorted by size (and their weight in the final linear layer):
Concept 2088: 5.27 (0.34)
Concept 2987: 1.48 (0.20)
Concept 162: 0.17 (0.01)
Concept 2177: 0.01 (0.00)
Concept 2479: 0.01 (0.00)
Largest five concept contributions (by absolute value) for predicted class eel sorted by size (and their weight in the final linear layer):
Concept 333: 2.94 (0.34)
Concept 1367: 2.26 (0.20)
Concept 760: 0.72 (0.11)
Concept 2088: 0.68 (0.04)
Concept 2987: 0.51 (0.07)
Furthermore, the layer from the concept similarities the prediction linear layer. Which weights (concept and class) should be adjusted by how much in order to get correct classification for this image, while not changing any other classifications? Please give answer in case of 'increase/decrean weight of class and concept by x'.
Figure 27: Input prompt to the large vision- vision-language model to edit the linear classifier's weights to correct error.
33

---

Published in Transactions on Machine Learning Research (05/2025)
To get the model correctly classify the image as "tench" " instead of "eel," " we'll need adjust the weights in the final linear layer between the relevant concepts and the classes. Here's the approach identifying which weights should be adjusted and by how much:
Step-by-step Approach:
Identify Misclassification and Concept Contributions:
The model currently predicts "eel" with logit of 7.71, while "tench" has logit of 7.05.
To change the classification to "tench," we need the logit for "tench" to be higher than the logit for "eel."
2. Calculate Difference Needed:
Currently, the difference between "eel" and "tench" logits is: 7.71 - 7.05 = 0.66.
To correctly classify the image as "tench," the logit for "tench" must exceed "eel." Therefore, we need adjust the weights such that the logit for "tench" increases by at least 0.66.
Determine Which Concept Weights to Adjust:
For "tench," " the largest contributing concept is 2088 with a contribution value of 5.27 and weight of 0.34.
For "eel," concept 2088 is also present with smaller contribution (0.68) and lower weight (0.04).
Adjusting concept 2088's weight for "tench" might be the most straightforward way change the output because it has significant contribution value.
Calculate Weight Adjustments:
To increase the "tench" " logit by 0.66, we need find an increase in the weight of concept 2088 for "tench":
Increase in logit = concept contribution x weight increase.
Here, the concept contribution for 2088 5.27. So, we need to find the weight increase (Aw) such that:
5.27 Aw 0.66.
Solving for Aw:
0.66
Aw
0.125
5.27
Adjust the Weight:
Increase the weight between "tench" and concept 2088 by approximately 0.125.
Final Answer:
Increase the weight of class tench and concept 2088 by 0.125.
This targeted adjustment will help correct the classification for this specific image without affecting other classi- fications significantly, as focuses on the concept that most strongly contributes to the "tench" " logit.
Figure 28: Output of the large vision-language model to edit the lincar classifier's weights to correct the error.
34

---

Published in Transactions on Machine Learning Research (05/2025)
10
1.0-
1.0
10
Gassiaaten socurayy
0.8
Cassitcaton secumey
0.8
acuraay
00.8-
Cassiftattin accuracy
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.
0.2
0.2
0.0
0.0
0.0
greyscale
opponent
origial
Powerspectrum
Rotation anole
(a)
Color vs. greyscale.
(b)
True vs. false color.
(c) Power equalization.
(d) Rotation.
Carsitcataon accurady
0.0
Clesvitcation aecurety
0.0
Carsitiation accuracy
0..
0.8-
0.6
0.6
0.6
Carsifiction
0.4
0.4
0.4
0.4
0.61
0.0
.3
05
3
6
9
100
50
30
15
10 percint
10 15
40
16f 3.0
.55 45 4 Aittor standard deviation
1.5 1.0
Uniform noise width
Contrast
Filter stancard deviation
(e) Uniform noise.
(I) Contrast.
Low-pass.
(l) High-pass.
1.0
1.0
1.0
1.0
Clasfitation acuursty
0.8
Closslicatinn C
0.88
0.8
Clssiiication accuracy
0.8
0.6
0.6
0.2
0.2
0.2
0.0
90
120
0.0
7
0.0
5
6
Phase width
teach
'each
'reach
(i) Phase noise.
(i) Eidolon
(k) Eidolon II.
Eidolon III.
Figure 29: Out-of-distr accuracies for UCBM, the original black-box model (ResNet and human observers. UCBM behaves similar to the original, black-box model.
100
UCBM
test nce
75
ResNe150V2
0
200
400
600
800
1000
sorted class ids
Figure 30: Class-wise ImageNet test accuracy UCBM and ResNet50V2. UCBM cxhibits significant disparities in class-wise accuracy, indicating fairness issues similar to those of the original, black- black-box model. Class indices are sorted by the test. accuracies of ResNet
35

---

Published in Transactions on Machine Learning Research (05/2025)
Fraction of 'shape' decisions
0.9
0.8
0.7
0.5
0.4
0.3
0.2
0.1
Shape categories
0.2
0.3
0.5
0.6
0.7
0.8
0.9
Fraction of 'texture' decisions
Figure 31: UCBMs (blue circles) exhibit texture bias similar to that of the original black-box model (ResNet orange crosses). In contrast, humans (green diamonds) are more shape biased.
36


Reviews:
[{"review_id": "oxaAbxDtOw", "reviewer": "Reviewer_cisK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Users prefer the proposed method over Label-free Concept bottlenecked baseline.\n- Very few concepts are effect the classification logits because of the top-k selection.\n- The proposed model is further bridging the gap between interpretable methods and the non-interpretable baseline.\n\n\n### Weaknesses\n- Lots of coefficients in the loss function, namely $\\lambda_w, \\alpha$, and $K$ for top-K, make this somewhat tricky to use in practice. One also needs to do concept dropout on top to avoid the degenerate scenario of using a single concept.\n- It is unclear how this technique would be applicable to models that do object detection, point tracking, multimodal LLMs etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript presents a method for converting an image classification model into a concept bottlenecked image classification model. The latter maps from the final image embedding into a concept space and then uses these concept activations to predict class labels.\n\n- Concepts are discovered using non-negative matrix factorization (CRAFT Fel et al., 2023b).\n- A sparse subset of the active concepts are selected as an extra step before computing class logits.\n- Strong performance is reported for a very few active concepts per image.\n- GPT-4o is used to update concept specific weights to correct misclassifications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify the following:\n- How exactly are top patches per discovered concept identified? The mapping from image representation to concept activations is done after the global average pooling, so there shouldn't be a patch level assignment if I understood correctly.\n- The plots are very small making them hard to read. The x-axis label could be removed to make the plots taller perhaps?", "broader_impact_concerns": "The broader impact section claims no particular impact specific to the manuscript's methods beyond the general impact of advancing machine learning, but does not provide any evidence in this regard.\n- Are UCBMs more or less robust than the black-box baseline?\n- How do UCBMs do with regards to fairness metrics?\n- Do UCBMs have texture or shape bias?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TH45rdqzLT", "reviewer": "Reviewer_3BPc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## **Strenghts**\n\nThe paper is well-written and provides a clear presentation of key components of UCBMs. The experimental analysis covers several settings, including a comparison to state-of-the-art label-free CBMs, ablations on concept selection, a user study on interpretability of model explanations, and error correction with LLMs. UCBMs are a natural baseline when coming to lack of supervision on concepts, which is of interest in the field of weakly/language-supervised CBMs. \n\nSome interesting aspects that the authors contribute to spot: \n1) Label-free CBMs and VLG-CBMs tend to rely on a multitude of concepts when coming to predictions. This depends on the linear layer in use, which, despite being optimized to be sparse, does not make input-label predictions depend on those few relevant concepts. As a consequence, concept-based explanations depend on many concepts and are hard to understand. The idea of the selector is interesting and also addresses a form of concept leakage. This augments existing studies in that direction [Havasi et al. (2022), Marconato et al. (2022), Zhang et al. (2025)].\n2) Concept-based explanations of UCBMs remain competitive to Label-free CBMs and are typically preferred in the user study. Moreover, it seems that current language-supervised CBMs suffer from poor concept semantics, associating irrelevant concepts to the decision making. E.g., in Figure 8, the prediction for \"sport car\" highly depends on concepts like \"room for 7 or 8 passengers\" for LF-CBMs, and \"special occasion\" for VLG-CBMs. \n\nThese two aspects combined reveal that UCBMs remain competitive to these state-of-the-art language-supervised CBMs. \n\n## **Weaknesses**\n\nUCBMs inherit the limitations of leveraging concepts obtained through unsupervised post-hoc methods. I agree with the authors that UCBMs can benefit \"from future unsupervised concept discovery methods\", but, whether these concepts are even interpretable is not clear and depends primarily on the model used to obtain such concepts. \n\n1) The authors should be more explicit about this aspect instead of sidelining it by saying \"assuming linearity of concepts, as per the superposition hypothesis\" (Section 2.1). \nAuthors should be more precise on this point, as interpretability is the main focus of the work, and connect to guarantees of discovery [Leemann et al. (2023)] and failures [Bortolotti et al. (2024)]. \n \n2) It must also be clear how concept-naming is obtained, e.g. through CRAFT and featuring in Figure 8, as I believe it is possible to attribute a name to some of the concepts but not all in general. Indeed, unsupervised concepts are in the order of thousands in Imagenet and Places-365. This is a critical point that should be highlighted.\n\n3) Other techniques are also connected to UCBMs like Anycbms [Dominici et al. (2024)] for CBM distillation and concept-interventions in black-box models [Laguna et al. (2024)] and deserve a discussion. \n\n### **References**\n\n[Havasi et al. (2022)] Addressing Leakage in Concept Bottleneck Models, NeurIPS \\\n[Marconato et al. (2022)] GlanceNets: Interpretable, Leak-proof Concept-based Models \\\n[Zhang et al. (2025)] The Decoupling Concept Bottleneck Model, IEEE \\\n[Leemann et al. (2023)] When are Post-hoc Conceptual Explanations Identifiable?, UAI \\\n[Bortolotti et al. (2024)] A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts, NeurIPS \\\n[Dominici et al. (2024)] AnyCBMs: How to Turn Any Black Box into a Concept Bottleneck Model, xAI \\\n[Laguna et al. (2024)] Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?, NeurIPS", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Unsupervised Concept Bottleneck Models (UCBMs), a new class of concept bottleneck models (CBMs) that first extracts concepts in an unsupervised manner, via sparse dictionary learning with CRAFT, and uses a selector to linearly combine the concepts and provide label predictions. The authors thoroughly compare UCBMs against CBMs that leverage supervision coming from visual-language models, like CLIP or Grounding DINO. Overall, this paper goes in the direction of understanding how to create more transparent methods based on concept-level predictions when supervision on concepts is not available.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the authors have conducted a complete experimental verification on UCBMs compared to other models. \n\nThe interpretability of the proposed method heavily relies on (1) models in use and (2) post-hoc extraction method. The authors should clarify this point and address the weaknesses. It would be interesting to investigate if the guarantees in [Leemann et al. (2023)] are satisfied and concept discovery is successful.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G4ZdXfK5Vr", "reviewer": "Reviewer_7RW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the paper and I find it interesting but I consider its novelty incremental. The main contribution comes from the combination of existing and interesting approaches with a sparsity constraint. This demonstrates that CBMs can be built also when no concepts are known a priori, solving a practical issue. \n\nStrengths: \n- It solves a practical problem demonstrating the applicability of CBMs to a wider spectrum of problems, which could have high impact. \n- The evaluation is performed on multiple computer vision datasets with consistent results\n- The results are positive, showing the efficacy of the method as opposed to existing baselines\n\nWeaknesses: \n- One main weakness I see with the approach is that it is hard to claim full interpretability because there seem to be concepts that are non-visible but that still influence the prediction for the predicted class and other classes. This phenomenon could have been analysed more in depth to characterise the limitations of the approach correctly.\n- The paper structure could be improved, together with the exposition of some concepts and figures. For these points, see the requested changes. \n- An important lack is the application of the approach to transformers models. One example of it is only briefly discussed in the appendix. I find it misleading to not have the same discussion in the paper. Similarly, not testing on other convolutional architectures than ResNet is a limitation. The argument would be much stronger with results provided on ResNet, Inception and ViT within the same manuscript. \n- No mention is given on polisemanticity, and on the possibility that some directions may represent more than one concept. This is observed for individual neurons (https://arxiv.org/abs/2210.01892) but also for directions obtained through linear decomposition (https://arxiv.org/pdf/2307.06913). \n- The related work is also lacking of other important works that have explored concept-based explainability and alternative ways of finding concepts, e.g. https://arxiv.org/pdf/2412.06639.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper advances the training of concept bottleneck models (CBM) by removing the need of knowing the concepts a priori, building on top of recent work in unsupervised concept discovery based on matrix factorisation. \nThe main novelty is in the introduction of non-negative matrix factorisation combined with a sparsity constraint to create a bottleneck layer that is then used to create the explanations. \n\nThe authors validate their approach against other CBMs approaches on diverse computer vision datasets and convolutional architectures (ResNet 18 and 50).  The evaluation is based on the model performance against other CBM approaches and on several insights on the discovered concepts, their relevance and understandability. \nA user study is also included in the results. \nThe results show that the loss of performance given by their approach is minimal when it happens, and that in most cases their method grants higher accuracy than other approaches, and higher robustness of the discovered concepts. \nFinally, they prove that their method also benefits from the other perks of CBMs such as generating model interventions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some changes on the structure:\n- I find it confusing to have a high-level explanation of the method in the introduction, because it is hard to follow at the first read of the paper. I would suggest reducing the level of detail on the methodology and focus on contextualising the work, adding details on the motivation and main challenges of the approach as opposed to the potential impact. \n- Figure 1 could be improved by highlighting the \"Alignments score\" and \"input-dependent concept selection\" blocks as the main introduced novelty\n- Figures 3, 4 and 5 should be improved. They are way too small and the values barely comparable. The top of Figure 3 is cut and some points barely visible. \n- Figure 8 has a similar issue, the concepts are barely visible. Better to show y6ygless or increase the image size. \n- Table 1 values should be reported as a percentage, otherwise the comparison is hard. \n- Out of the 42 concepts found for Imagenet in Table 1 (and the 64 for CUB and 162 for Places365), what is the impact of each concept to the final prediction? An ablation study of the concepts would create a distribution of concept importance that could be interesting to see. \n- It is not clear what is K in Table 3. Please report it for each result if it differs, otherwise just specify the value. \n-The related work section should be largely improved. The paper https://arxiv.org/abs/2111.09259 should be in the references because it is the one that originated the actual idea of unsupervised concept discovery through matrix factorisation, so this should be at least mentioned. Similarly, recent work on identifying concepts in transformers architectures should be referenced (e.g.  https://arxiv.org/pdf/2412.06639). I would also expand the use of factorisation as a means of controlling model output, see for example https://arxiv.org/pdf/2206.00048. A discussion on directions that may be found by craft that represent more than one concept could be interesting to see, both in the experiments and also in the related work by mentioning polysemanticity.", "broader_impact_concerns": "No need for broader impact statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this is a solid paper; all complaints are reasonably minor.  Some have not been explicitly addressed in the rebuttal (see reviewer cisK) and I encourage the authors to update the paper according to **all** feedback.

Major reasons for acceptance include novelty; improvements over the state of the art, supported by a rather extensive experimental analysis and a user study; and significance, considering the prominence of concept-bottleneck models among current deep learning and XAI trends.
Decision: Accept (poster)