DCBM: Data-Efficient Visual Concept Bottleneck Models
Authors: Katharina Prasse
Patrick Knab
Sascha Marton
Christian Bartelt
Margret Keuper
Venue: ICML 2025 poster (2025)
Keywords: Explainable AI
concept bottleneck models
foundation models
Primary area: applications->computer_vision
TL;DR: Data-efficient CBMs (DCBMs) generate concepts from image regions detected by segmentation or detection models.

Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of neural networks by basing predictions on human-understandable concepts. However, current CBMs typically rely on concept sets extracted from large language models or extensive image corpora, limiting their effectiveness in data-sparse scenarios. We propose Data-efficient CBMs (DCBMs), which reduce the need for large sample sizes during concept generation while preserving interpretability. DCBMs define concepts as image regions detected by segmentation or detection foundation models, allowing each image to generate multiple concepts across different granularities. Exclusively containing dataset-specific concepts, DCBMs are well suited for fine-grained classification and out-of-distribution tasks. Attribution analysis using Grad-CAM demonstrates that DCBMs deliver visual concepts that can be localized in test images. By leveraging dataset-specific concepts instead
of predefined or general ones, DCBMs enhance adaptability to new domains. The code is available at: https://github.com/KathPra/DCBM.

DCBM: Data-Efficient Visual Concept Bottleneck Models
Katharina Prasse"
Patrick Knab
Sascha Marton Christian Bartelt
Margret Keuper'
Abstract
Concept Bottleneck Models (CBMs) enhance the interpretability of neural networks by basing predictions on human-underssandabll concepts. However, current CBMs typically rely on con- cept sets extracted from large language models or extensive image corpora, limiting their effec- tiveness in data-sparse scenarios. We propose Data-efficient CBMs (DCBMs), which reduce the need for large sample sizes during concept gener- ation while preserving interpretability. DCBMs define concepts as image regions detected by seg- mentation or detection foundation models, allow- ing each image generate multiple concepts across different granularities. Exclusively con- taining dataset-specific concepts, DCBMs are well suited for fine-grained classification and out- di-distribution tasks. Attribution analysis using Grad-CAM demonstrates that DCBMs deliver vi- sual concepts that can be localized in test images. By leveraging dataset-specific concepts instead of predefined general ones, DCBMs enhance adaptability new domains. The code is avail- able at: https: //gtt//b//btbbbttt com Ccom/Kathln Kathhrataarr hPra/ DCBM,
Crop
CBM
Boat-talled Grackle
becouse
Training samples
Visual concepts
Prediction
Figure DCBMs extract image regions as concepts. Using vision foundation models, we crop image regions concepts CBM training. Based on few concept samples (50 imgs class). DCBMs offer interpretability even for fine-grained classification.
tions weighted combination of uuman-unndestanddblle concepts. While early CBMs employ manually crafted con- cepts (Koh et al., 2020), recent advances in vision-language models have enabled text-aligned CBMs, which leverage textual descriptions for concept selection and interpretabil- ity (Menon & Vondrick, 2023; Oikarinen et al., 2023; Yang et al., 2023). Given that precise class descriptions are avail- able, text alignment effectively guide the concept se- lection towards representative semantic concepts. However, recent works highlight that CBMs benefit from concepts extracted in the visual domain (Kowal et al., 2024; Rao et al., 2024; Sun et al., 2024; Wang et al., 2023; Zhang et al., 2024a). This also increases their faithfulness against the reported image-text misalignment in the CLIP embedding space (Liang al., 2022; Roth al., 2023). Learning con- cepts (Rao et al., 2024) from large-scale datasets performs well for ImageNet classification, but is less suitable for fine- grained classification, where concepts must be defined at a specific granularity. Ideally, concepts can be recognized under domain shifts in order to increase their applicability in real-life use cases.
1. Introduction
Deep neural networks (DNNs) achieve state-of-the-art per- formance across various domains, yet their opaque nature limits trust, particularly in safety-critical applications. To ad- dress this, Explainable Artificial Intelligence (XAI) strives to make model decisions more transparent and interpretable. Concept Bottleneck Models (CBMs) (Koh et al., 2020) are an inherently interpretable framework, they base predic-
"Equal contribution 'Data and Web Science Group, Univer- sity of Mannheim, Mannheim, Germany 'Clausthal University of Technology, Clausthal, Germany Max-Planck-IInttius for Infor- matics, Saarland Informatics Campus. Correspondence Katha- rina Prasse < <katharina, wwwrarrinnnppasseeeeee muii-manneiimde.... Patrick Knab <patrick. knab tu-clulll clausthal.de>.
We propose Data-eflicient Visual CBMs (DCBM) im- prove interpretability in data-scarce domains. DCBMs use segmentation and detection foundation models to extract image regions as visual concept proposals. This approach generates multiple concepts at different levels of granular- ity from a single image, allowing to create a meaningful concept bank even data-scarce settings. Our design en-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Data-Efficient Visual Concept Bottleneck Models
ables flexible adaptation novel datasets while exclusively containing dataset-specific concepts. For wide applicabil- ity, DCBMs leverage the popular CLIP embedding space e.g. (Rao et al., 2024; Yang et al., 2023). By extracting concepts in the visual domain, we avoid bridging the gap between the modalities (Liang et al., 2022), while being able to name our generated concepts using the image-text alignment. DCBMs enable fine-ge aune-griied concept differen- tiation and maintain interpretability in out-of-distribution (OOD) settings.
Menon & Vondrick (2023) generate GPT-3 descriptions of class-specific concepts, providing insights into the visual attributes associated with each category (DCLIP). Similarly, Oikarinen et al. (2023) generate concept sets for CBMs using GPT-3 for training their label-free CBM (LF-CBM).. Panousis et al. (2023) extend LF-CBM by incorporating a concept discovery module (CDM) that encourages spar- sity by learning which subset of concepts is relevant for given image. Language-guider CBM (LaBo) employs GPT-3 for concept creation, but it stands out by using a submodular function to select concepts from candidate sets with the goal of retaining the most discriminative and least overlapping concepts (Yang et al., 2023). All aforemen- tioned methods incorporate text in their concept discovery process. We argue that multi-modality concept discovery adds nuances to image classification interpretability, which are not inherently there, in pure image classification mod- els. Given that images and texts occupy distinct areas of the CLIP embedding space (Liang et al., 2022), we argue that only image-level concepts allow for true interpretabil- ity. Following the same line of argumentation, Rao et al. (2024) introduce Discover-then-Nanee (DN-CBM), which employs sparse autoencoders for image-level concept gener- ation. These concepts mapped to names derived from corpus of broad textual descriptions, and the resulting mapped concepts are used to train a CBM. Parallel work by Schrodi et al. (2024) understands concepts as low-rank approximations of model activations, which they learn us- ing non-negan non-negative matrix factorization on image embeddings.
Our main contributions can be summarized as follows:
With DCBMs, we introduce a simple and data-efficient framework that can easily be adapted to novel datasets.
We create concepts using segmentation and detection models which are inherently dataset-specific and offer multiple concept granularities. We generate concepts in the visual domain avoid modality gaps and to be independent of textual descriptions.
We extensively evaluate the DCBM framework across concept proposal generation methods and datasets. To ensure its usefulness for real-life applications, we as- sess the transferability of DCBMs to out-of-domain (OOD) settings and verify the localization of the im- portant concepts.
2. Related work
Moreover, self-supervision can be used to learn concepts from data (Alvarez Melis & Jaakkola, 2018; Wang et al., 2023). Following the same motivation, we and parallel works have employed foundation models to propose visual concepts from data (Kowal et al., 2024; Sun et al., 2024; Zhu et al., 2024). We argue that concepts should have high granularity and be dataset-specific to allow adaptation to varying target domains. We use CBMs in their original form single linear layer while there exist variants learning residuals (Zabounidis et al., 2023), investigating locality (Raman et al., 2023), and assessing concept correlations (Heidemann et al., 2023).
Explainable artificial intelligence offers numerous ap- proaches to making model decisions interpretable. Post hoc methods create an explanation on top of an existing model based on the given model prediction. By leveraging activations (Selvaraju et al., 2019), the relevant image area is highlighted or model embeddings (Yuksekgonul al., 2023) used to show concept activations. Their interpretability is limited, one cannot be sure that their explanations are inherent to the model. Ante hoc methods, however, are inherently interpretable and provide predictions along with explanations, e.g. (Chefer et al., 2021; Menon & Vondrick, 2023; Oikarinen et al., 2023; Rao et al., 2024; Yang et al., 2023). They include single-layer linear neural networks (CBMs), e.g. (Koh et al., 2020), decision trees, e.g. (Mah- booba et al., 2021), and self explainng networks, e.g. DEAL (Böhle et al., 2022), and
2.2. Visual concept generation
CBMs typically contains autoencoders (Huben et al., 2024; Rao et al., 2024), non non-negative matrix factorization (Fel et al., 2023; Schrodi et al., 2024; Zhang et al., 2021), K- Means (Ghorbani et al., 2019), or PCA (Zhang et al., 2021; Zou et al., 2023). Panousis al. (2024) share our convic- tion that multi-level concepts enhance the expressiveness of CBMs. In contrast to their 2-1evel approach, our DCBM can detect multi-level concepts. Following other XAI methods, we generate concept proposals in an unsupervised manner using segmentation foundation models (Kowal et al., 2024;
(Li et al., 2025), VLMs such as LLaVA-NeXT (Li et al., 2024b). While other data-efficient methods exist, e.g., prompt-tuned CLIP, we focus exclusively on interpretable models.
2.1. Concept bottleneck models
This family of models predicts image classes based on weighted linear combinations of concepts (Koh et al., 2020).
CBMs mainly differ in the way they extract concepts.
2

---

Data-Efficient Visual Concept Bottleneck Models
Sun al., 2024; Zhu et al., 2024). Specifically, we extract image crops using segmentation (Carion al., 2020; He et al., 2017; Kirillov et al., 2023; Li et al., 2024a; Ravi et al., 2025) and detection foundation models (Liu et al., 2024), with hyperparameters tuned to capture both part-level and instance-level concepts. We can steer promptable detection methods more than generic models and expect to return fewer and more class-relevant concepts. To minimize re- dundancy, we cluster the generated concept proposals. In contrast to BotCL (Wang et al., 2023), we create visual concepts instead of learning them. We evaluate the cor- rect localization of concepts using the Grid Pointing Game (Bohle et al., 2021). In comparing concept activations of the corresponding class to activations of three random other classes, we evaluate whether the concepts are class-specific.
using segmentation or detection foundation models. The initial concept set is clustered to reduce redundancies. In line with the main idea of CBMs, we select simple methods (k- (e-mens)) to achieve maximum interpretability. DCBMs can be inspected at every step, (1) foundation model concept proposals, (2) cluster centroids, and (3) CBM explanations. On top of that, attribution methods visualize inference-time concept activations to provide interpretations grounded in the image plane. Figure 2 illustrates the framework that de- rives visual concepts, which are subsequently named using the CLIP space.
3.1. Concept proposal generation
We use foundation models for segmentation and detection to facilitate data-efficient concept generation and extract multiple concepts from the same image. Our choices are the foundation models Segment Anything & (Kirillov et al., 2023; Ravi et al., 2025), GroundingDino (Liu et al., 2024), MaskRCNN (He et al., 2017), and DETR (Carion et al., 2020). We compare the segmentation models, which require input, to the open-set detection method Ground- ingDino, in which we use various concept sets from the literature as input. We construct a concept proposal subset by selecting as few as 50 random training images per class. This has the goal of mitigating the risk of overfitting while increasing efficiency since not all training images are used for concept generation. Drastically reducing the number of images needed for concept generation makes DCBM highly data-efficient, i.e., for ImageNet, DCBM uses (-96%) of training images for concept generation compared to other data-driven CBMs. Further discussions on data-efficiency can be found in Appendix A.1.
2.3. Image-level interpretability
Besides CBMs, part-based classifiers (Chen et al., 2019;
Donnelly et al., 2022; Nauta al., 2021; Pham al.. 2024: Wang et al., 2021; Xue et al., 2024) and attribution meth- ods (Bohle et al., 2021; Knab et al., 2025; Ribeiro et al., 2016; Selvaraju et al., 2019; Zhang et al., 2018) offer inter- pretability in the visual domain. In contrast to CBMs, part- based classifiers are often trained end-to-end (Chen et al., 2019). While their prototypes have visual similarities with DCBM concepts and their explanations share similarities with CBMs, the fact that prototypes are learned in a super- vised manner, e.g. (Chen et al., 2019; Pham et al., 2024), sets them apart. BotCL. (Wang et al., 2023) learns concepts in self- sulf-supe manner and requires the number of con- cepts to learn as hyperparameter. Given that prototypes are generated during training time, attribution methods can be used to highlight relevant image areas during inference (Chen et al., 2019).
Selecting 50 images is a hyperparameter, for CUB, which contains fewer samples, we include all available images in the concept generation set D. Especially for large datasets such as ImageNet and Places365, this reduces the number of images considered during concept extraction by 96% and 99%, respectively, compared to other methods (Schrodi et al., 2024). Task-agnostic pre-training (Rao et al., 2024) requires large-scale datasets such as CC3M (Sharma et al., 2018). Efficiency can be increased by using fewer train- ing samples at the cost of slight performance degradations, as shown in Appendix E. After receiving the foundation model segmentation or detection results, we crop the images in D according to the bounding box and add the resulting sub-image to the concept proposal set S. For each image in D, multiple sub-images can be created, thus [D] S (S]. The removal of background information is ablated in Ap- pendix D. To refine the concept proposals, concepts that are below or above a given size threshold can be excluded from S, as evaluated in Appendix E.2.
Attribution methods trace model decisions back to the image plane, e.g. (Bohle et al., 2021; Knab et al., 2025; Ribeiro et al., 2016; Selvaraju et al., 2019). They determine the relevant image region by using, e.g., gradients, activations, occlusions, or perturbations. Grad-CAM (Selvaraju et al., 2019), for instance, visualizes the activations of the net- work's final layer, while Local Interpretable Model-agnostic Explanations (LIME) (Knab et al., 2025; Ribeiro et al., 2016) uses perturbation-basee based surrogates to approximate interpretable models. For DCBM, we evaluate the localiza- tion of concepts using the Grid Pointing Game to validate the concepts in the test images, following the same intuition as Wang et al. (2023).
3. Data-efficient concepts for CBMs (DCBM)
The design of DCBM is driven by the focus on data- efficiency, simplicity, and step-wise inspectability. To this end, we define concepts as image regions that we generate
3

---

Data-E Efficient Visual Concept Bottleneck Models
Clustering
Texts
Segmenta tion model (generic promptable)
Segmenn
Vision Encoder
Concegt sropesals
Text Enooder
Step 1: Image Segmentation
Step 2: Concept Generation
Concepts
(clustered segments)
Prediction and Explanation
Linear Leyer
Similarity
Vision Encoder
Pagori
Building Sandeiine Frunstation
Step.4; Concept Naming
Step Concept Bottleneck Model
Figure DCBM framework. The DCBM framework generates concept proposals through foundation models (Step 1). These proposals then clustered, each represented by its centroid (Step 2). Finally, the unique concepts utilized train sparse CBM, effectively (Step 3). We leverage the image-text alignment to map the visual concept to the corresponding textual concept (Step 4). We can remove undesired concepts after Step 2.
3.2. Concept generation
3.3. Concept bottleneck model
embed the concept proposal set S using a frozen im- age encoder with backbone architecture such that San f(S) (f())) Rdxmm where denotes the embed- ding space's dimensionality and d the number of concept proposals in S. To reduce redundancy while maintaining high variance of concepts, we cluster the concept proposal embeddings SoL.- In Appendix D.3, we ablate the choice of clustering algorithm by comparing centroid-based K- Means with agglomerative clustering. Additionally, Ap- pendix D.5, analyze the impact of varying the number of clusters in K-Means. We select rather large number of clusters, since we aim to create concise concepts. For each cluster, we define its centroid Cj as concept, thus obtaining c= [c E 4xm in the embedding space. We ablate two approaches for determining the centroid сj of cluster j: the mean and the median of the clustered embeddings, as reported in Appendix D.4. The concept set C constitutes the final concept set employed for training the CBM model.
The CBM learns a linear mapping function h(-) that trans- forms the concept activations into corresponding class label predictions such that
((x,) = h(((xx))) =xx(x))
(I)
where t(x;) denotes the predicted label for input x,, a(x; R represents the concepts associated with x,, and w E RR are the parameters of the linear model. To construct a(x;), we compute the projection of the embedding f(x,) onto each cluster centroid cj, normalized by its squared Lg-norm done by Yuksekgonul et al. (2023).
a(x,) (J(x)),G) ||c311
for b..., (2)
train the model using the cross-entropy loss, line with previous work (Rao al., 2024). Additionally, we apply regularization term the parameters promote sparsity in the learned weights,
Concept Removal. DCBM allows a targeted removal of specific, undesired concepts after the concept generation phase (Step 2 in Figure 2). Leveraging CLIP's multimodal capabilities, we specify the concept to removed using a textual prompt. For example, to exclude the concept "stone". compute its text embedding and remove all visual con- cepts with cosine similarity above the threshold. This capa- bility allows for fine-grained control to exclude concepts, allowing users to suppress spurious correlations or explic- itly highlight desired causal factors (see Appendix G.2 for further details).
Limin (x:) CC(((x,). + A||w||
(3)
where CE represents the cross-entropy loss, À is the sparsity hyperparameter that controls the influence of the regulariza- tion term, and w are the parameters of the linear mapping.
3.4. Concept naming
An additional feature of DCBM is the naming of the con- cept set C. Given that the backbone architecture f is text- image aligned, we can embed corpus of texts T into the

---

Pata-Efficient Visual Concept Bottleneck Models
same embedding space as the images. Specifically, we de- fine Tell fiex(() (fre (Ti)) RI'm, where I represents the number of textual concepts within T. It is important to note that 1 may differ from d, because we uti- lize an open text corpus for this task. Each visual concept is assigned to the nearest textual feature based on cosine similarity. In line with previous work (Rao et al., 2024), we select T as common English words (Kaufıaan 2012). The DCBM framework is further detailed in Algorithm 1 in Appendix A.
(Xian et al., 2018), CelebA (Liu al., 2015), and a subset of ImageNet (i.e. first 200 classes) compare against other CBMs and to exemplify its applicability to diverse datasets. We further evaluate it on two novel datasets for the XAI community, the social-media dataset Climate'TV (Prasse et al., 2023) and MiT-States (Isola et al., 2015), as inspired by (Yun et al., 2023). We also evaluate on ImageNet-R (Hendrycks et al., 2021) to assess CBM performance under complete domain shifts. See Appendix B for detailed dataset overview.
Hyperparameters. We ablate all hyperparameters on a held-out validation set. In case this does not exist, we construct it to comprise 10% randomly-selected, class- balanced training samples. Using this setting, we ab- late on the CUB, ImageNette and ImageWoof datasets to find suitable hyperparameters. The search space com- prises learning rate 1, = (1e-1, 1,,ee_,,ee" 1)_ sparsity- parameter À = {1e-1, le le 1e-2) number of clusters k {128, 256, 512, 1024, 2048, 4096), and concept pro- posal models. For all datasets and concept proposal genera- tion methods, 4, = le-d and sparsity parameter À = 1e-4 were found to be optimal. The number of clusters k de- pends on the size of the concept proposal set S, but we observed tendency regarding greater values of l., and stick to k = 2048 if not mentioned otherwise. We train each DCBM model for 200 epochs with a batch size of 512 (Places365 & ImageNet) or 32 (all remaining). Concept clustering and CBM training takes five minutes for small datasets and up to two hours for large datasets on a single RTX A6000 (more in Appendix 1)
4. Evaluation
DCBMs exclusively incorporate dataset-intrinsic visual con- cepts, with additional benefits in data-efficiency and appli- cability to fine-grained classification tasks. We first outline the experimental setup (Section 4.1), followed by a quantita- tive performance assessment (Section 4.2), and qualitative assessment of DCBM-generatee explanations in Section 4.3. Lastly, we evaluate concept quality based on localization within the test images using Grid Pointing Game (Bohle et al., 2021) in Section 4.4.
4.1. Experimental setup
Backbones. The evaluation is conducted in the CLIP (Rad- ford et al., 2021) embedding space with ResNet-50, ViT B/16, and ViT L/14 backbones. We compare different concept proposal methods: generic segmentation models, SAM2 (Ravi et al., 2025), SAM (Kirillov et al., 2023), DETR (Carion et al., 2020), Mask-RCNN (He et al., 2017), and Semantic-SAM (Li et al., 2024a), which segment the entire image, including background objects, and return con- cept proposals of the whole scene. Additionally, we employ a promptable detection model, i.e., GroundingDINO (Liu steer the concept generation to-
4.2. DCBM performance
We benchmark DCBM on standard CBM datasets and assess the generalization capabilities of its visual concepts on the -utto--distribution dataset ImageNet-R. Moreover, we eval- uate it on two uncommon datasets for XAI, MiT-States and Climate and the less common awa2 and CelebA datasets. Performance differences between datasets are marginal, ex- emplifying that DCBMs can be adapted to novel datasets.
et al., 2024), in which ward relevant image regions. To this end, we use prominent part labels from the literature as detection prompts. This includes the CUB labels for bird parts (Wah et al., 2011), the sun attributes (Patterson et al., 2014), the animal at- tribute labels employed in AWA (Lampert et al., 2009), part- Imagenet labels (He al., 2022), and Pascal parts labels (Chen al., 2014). We ablate the prompts in combination with all datasets Appendix D.
1.2.1. COMPARISON ON BENCHMARK DATASETS
In Table 1, we report quantitative evaluation of DCBM's performance (top-1 I accuracy) compared to recent CBM approaches, all of which leverage the text-image aligned backbones of CLIP. For each experiment, we include linear probe and zero-shot accuracy, with an asterisk (*) indicat- ing performances reported in prior literature, and a dash (-) indicating no reported accuracies. As a standard, we choose 2048 concepts; ablations have shown that 4096 clus- ters improve ImageNet accuracy while CIFAR-10 accuracy increases with fewer clusters (256). This indicates that the number of concepts and the number of classes are related.
Datasets. We evaluate DCBMs on the five commonly used datasets in the CBM community, For general image clas- sification, we employ 10, CIFAR- 100 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009), as they offer a wide range of classes. For Nomain-specifff tasks, use CUB (Wah et al., 2011) and Places365 (Zhou al., 2017), which provide targeted, domain-specifer categories. Additionally, we ablate DCBMs on ImageNette (Howard, 2019a) and the fine- -rainnd dog classification dataset Im- Woof (Howard, 2019b). Moreover, we evaluate on awa2
5

---

Data-Efficient Visual Concept Bottleneck Models
Table CBM benchmark. Top-1 accuracy comparison across CBM models (CLIP VIT L/14). The highest accuracies are bolt. DNCB excels in ine-grained classification, while rext-aligned and large-scale vision CBMs prevail on general datasets.
Table 2. Ood performance. Error rate changes compared between visual CBMs (CLIP VIT-L/14) on ImageNet-R classes (and base- line). DCBM loses less performance when moving from iid to ood evaluations, compared to the task-agnostic DN-CBM.
Model
CLIP L/14
IN IN-200
IN-R
Gap(%)
IMN
Places
CUB
CR10
Cir100
DN-CBM (Rao et al., 2024) +
16.4
55.2
38.8
Lincur Probe
83,9*
55.4
85.7
98.0*
87.5
Zero-Shot
75.3*
40.0
62.2
96.2*
77.9*
DCBM-SAM2 (Ours) 4
21.1
48.5
27.4
LF-CBM (Oikarinen et ol., 2023)1
49.4
80.1
97.2
83.9
DCBM-GDINO (Ours) 1.
22.6
47.2
24.6
LaBo (Yang ot al., 2023)
84,0*
97.8*
86.0*
DCBM-MASK-RONN (Ours) 4
22.2
44.6
22.4
CDM (Panousis al., 2023)
83.4*
55.2*
95.9
82.2
DCLIP (Menon& & Vondrick, 2023)
75.0*
40.5%
63.5
DN-CBM (Rao et al., 2024) f
83.6*
98.1
86.0
DCBM-SAM2 (Qurs)
77.9
52.1
81.8
97.7
85.4
using GradCAM.
DCBM-GDINO (Ours) 1
77.4
52.2
81.3
97.5
85.3
DCBM MASK- MASK-RCNN (Ouns) 1
77.8
97.7
Table 3. Adaptation to other domains. Top-1 accuracy reported for MiT-States and ClimateTV (CLIP VIT L/14). DCBMs perform superior to the lincar probe when classifying objects in diffcrent states. Social media classification performance differs between DCBM concept generation methods.
DCBMs perform within 6% of the linear probe for all datasets, with superior performance on the fine- -rne-grainddd bird classification dataset CUB (Wah et al., 2011). While CIFAR- 10 and CIFAR-100 performance is comparable between CBM approachss, DCBM performance on the large-scale dataset ImageNet and Places365 slightly subpar com- pared other methods. For all concept proposal generation DCBM's methods (SAM2, GDINO, and MASK- performance is comparable, indicating that the foundation model can be chosen in accordance with the given applica- tion. Appendix F contains results for other CLIP backbones.
Method
MiT-States
ClimateTVV
Linear Probe f Zero-Shot
37.3
84.5
48.1
69.7
DCBM-SAM2 (Ours)
42.8
85.6
OCBM-GDING (Ours) 1
43.3
81.8
DCBM-MASK-RONY (Ours)
43.2
87.9
We have designed DCBMs to be data efficient and evalu- ate their performance novel dataset. In Section 4.1, we introduced MiT-States (Isola et al., 2015) and ClimateTVV (Prasse et al., 2023), on which we evaluate the generaliza- tion of DCBMs. Table presents the top-1 accuracy of the DCBM prediction, linear probe, and zero-shot classifica- tion, all trained on the CLIP ViT L/14 backbone. On both datasets, DCBMs outperform the linear probes. Surpris- ingly, the zero-shot classification on MiT-States achieves the highest accuracy. The prediction of classes independent of their state appears to be challenging task. While the dif- ferent DCBM approaches perform similarly on MiT-States, their performance differs greatly on ClimateTVV, with seg- mentation models' concepts achieving the highest accuracy. Results for awa2 and CelebA can be found in Appendix F.2 along with results for BotCL's subset of ImageNet. We demonstrate that DCBM can capture diverse and complex visual and semantic relationships, showing that it can effec- tively be applied to other datasets.
4.2.2. GENERALIZATION OF DCBM CONCEPTS
We evaluate trained CBM models to understand its gener- domains (ood). To this end, we evaluate CBM models, ours and DN-CBM (Rao et al., 2024), trained on ImageNet on ImageNet-R (Hendrycks et al., 2021) which contains 200 ImageNet classes in various renditions (e.g., embroidery, painting, comic). To ensure a fair comparison, we report the error rates on the same classes within ImageNet (IN-200). Ta- ble 2 shows superior ood generalization capabilities of the DCBM model compared to DN-CBM. In contrast to stan- dard IN-R evaluations (Hendrycks et al., 2021), which train the model exclusively on the classes evaluated in IN-R, we re-use the DCBM trained on all 1,000 ImageNet classes. DCBMs trained exclusively on concepts of the 200 IN-R classes are expected to perform even better, given that the generated concepts would better match the target classes and avoid confusion with the 600 classes not contained in ImageNet-R.
alization capabilities in
4.3. DCBM interpretability
Our ood evaluation shows that DCBM exhibits consistently lower error rates on IN-R and a lower gap between IID and OOD in comparison to the task-agnostic DN-CBM (Rao et al., 2024). In Section 4.4, we further validate DCBM's i in- terpretability by analyzing concept activations in test images
We designed DCBMs to provide visual insights into model decision-making In this section, we qualitatively analyze and interpret the various explanations generated for Ima- geNet, Places365, and CUB predictions. These evaluations
6

---

Bata-Effiiciet Visual Concept Bottleneck Models
Rocks
Gull
Clicking
Knot
Blu
ImageNet
Ambulance
Med
Police
Kit
Injection
laring
RrOoneess
Caso Ambuiseon
Places365
Epoxy
ImageNet
ilook
Wardrobe
Keyboard
ImageNet
Carrier
MSC
Shield
Airfare
Mariner
Top Connact
Prod. Doskog Computu
Figure 4. DCBM ambiguous concepts/classes. Complex visual classes with ambiguous concepts ambiguous class labels (SAM2 and CLIP ViT L/14).
TapaGrn
Aecort
Figure 3. DCBM correct predictions. Concepts provide insights into the model's embedding space, as returns visually and/or semantically aligned concepts (SAM2 and CLIP ViT L/14).
that it is the second most important concept. In contrast, while the visual concepts for the second instance can be traced back the input, the predicted class differs from the ground truth, highlighting the inherent ambiguity of the class definition. Further qualitative examples can be found in Appendix G.
are conducted the CLIP VIT L/14 backbone, with SAM2 concept proposals. For each instance, we show the input image along with the five most important concepts. Each concept displayed alongside its name(as detailed in Sec- tion 3.4). We divide this investigation into two parts: in Figure 3, analyze correct predictions, and in Figure 4, examine challenging predictions.
4.4. DCBM concept investigation
4.4.1. CONCEPT CLUSTERING
We evaluate whether concept proposals form cohesive clus- ters to validate cluster centroid as visual concept. As shown Figure 5, different visual representations of the same concept are consistently grouped together. The sample images illustrate strong semantic coherence within clusters across all three datasets, reinforcing the interpretability of the learned concepts. Additionally, the plot illustrates the diversity of concepts captured within each dataset; for in- stance, ImageNet spans wide range of object-centered categories, from animals to objects. Depending on the ob- ject's complexity, these objects appear as whole instances or parts. In contrast, Places365's scenes are more cluttered and less object-centric. Hence, its concepts include mountain range formations and vegetation. The highly specific CUB dataset contains sub-parts (legs) well as complex clusters (bird on tree trunk, vertically). This confirms that founda- tion models can generate dataset-specific concept proposals suitable for CBM training.
The examples demonstrate the correct predictions and show- case concepts from each dataset that closely align with the semantics of the input image (see Section 3.3). The ma- jority of concepts are visually and semantically aligned, as the hockey (Place365) image's most important concepts are ice, (hockey) stick, sk(a)tes, the same holds for the CUB and ImageNet example. Overall, the visual concepts provide suf- ficient interpretability for the trained CBM's classification, making it applicable to domains that lack textual concepts or rely solely on a vision encoder. Moreover, these expla- nations highlight the quality of concept naming when in- corporated into our framework. For example, Places365, the concept name thirds corresponds hockey, where the game is played in three periods of 20 minutes. Given that DCBMs use visual concepts that have been named, the im- age should always be attributed more weight than the text, as the image-text alignment may be imperfect.
Figure illustrates two examples of challenges arising when using DCBM. The first image correctly classified using concepts not contained in the image, whereas the second one is incorrectly classified even though the important concepts are contained in the image. For instance, in the case of the ambulance image from ImageNet, concepts such as med, police, kit, or injection are not explicitly visible in the input image. This provides insights into the model's 's embedding space, where stethoscope so similar to an ambulance
4.4.2. CONCEPT LOCALIZATION
To validate DCBM concepts, we measure how well concepts in explanation can be traced back to the input image using Grad-CAM (Selvaraju et al., 2019) attributions. We employ the Grid Pointing Game (GPG) (Bohle et al., 2021) to as- sess whether a given class corresponds to the original image when placed in a grid with randomly selected images of different, other classes. Specifically, we include the original

---

Visual Concept Bottleneck Models
ImageNet
Places365
CUB
Figure Clustered proposals concepts. show four concept clusters ImageNet, Places365, CUB, illustrating clustering concept proposal embeddings produce semantically coherent clusters across datasets. The center image (orange border) represents the final concept (median image) for each cluster of concept proposals (SAM2 and CLIP VIT L/14).
Visuel Concept
Inputx
Pointing Game
Table 4, Grid pointing game metrics. Mcan values for different metrics are reported Places365 and CUB (SAM2 and CLIP RN-50).
Metric
Places365
CUB
Gini
0,4701
0.3608
Percentage t
0.6519
0.3402
Figure 6. Grid pointing game. GPG applied to ImageNet-R instance using a visual concept from ImageNet to demonstrate the effectiveness of our approach in ood (SAM2 and ResNet-50).
Abs f
0.6551
0.3445
tion alignment (c.g.. Gini: 0.4701 vs. 0.3608, Abs: 0.7028 vs. 0.4988), CUB also demonstrates better than random coherence. Overall, the lower numbers in CUB, dataset for fine-grained classification, are also to be expected since several concepts are usually shared between classes (e.g.. different bird species have similar beaks) and are thus acti- vated along with the other class. The consistent performance across metrics validates our method's ability to map visual concepts to relevant image regions.
test image in a 2x2 grid with three randomly chosen images from the same dataset (see Figure 6). We then select the top five visual concepts classes for Grad-CAM and apply them to each visual concept within the composite 2x2 grid image, analyzing whether activation maps align with the original image's quadrant. This evaluation is conducted on CUB and Places365 test sets using the ResNet-50 backbone. In Table 4, we compute the mean over the entire test set for each of the three metries: (1) Gini Index, which measures the attribution score dispersion across the grid, (2) relative percentage, which measures how much normalized attribu- tion overlaps with the correct quadrant, and (3) maximum score, which measures whether the highest attribution score in the correct quadrant. For all metrics, higher values indicate a better localization within the specific test image. The GINI index is defined within [0.1], where 0 stands for equal attributions in the entire group and 1 indicates per- fect localizability. Percentage and absolute scores are lower bounded at 0.25, where the attributions would be randomly divided between the 4 images in the grid.
5. Discussion
With DCBMs, we create a simple and data-efficient frame- work for model interpretability, which can easily be adapted to new datasets. Using segmentation and detection founda- tion models, we retrieve image-inherent concepts the vi- sual domain. For any prediction, we can trace their concepts' activations back to the test image. DCBM's accuracy in all experiments within 5% of the linear probe, indicating that its interpretable predictions are comparable between common and uncommon CBM datasets. Based on these findings, we expect to see similar behavior for other datasets. DCBM's applicability is limited by two factors, the suitability of a segmentation foundation model and the expressiveness of
The results validate that DCBM concepts can be localized within test images. While Places365 shows higher attribu-
8

---

Data-Efficient Visual Concept Bottleneck Models
the CLIP embedding space for a given use case. Further advances in vision-language models, segmentation, and de- tection models can enhance results, new methods can easily be integrated into the DCBM framework.
features to concepts. Besides image-concept similarity, we argue for integrating the GPG results into the mapping to ensure having the most suitable concept for each object in the image. In all cases, we strongly argue for employing a single model at a time to maintain model-specific inter- pretability. Moreover, we believe it would be interesting to extend DCBMs or CBMs in general to tasks beyond classification. Interpretable regression could be learned by integrating GPG into the prediction pipeline, assuming that the concept represents the largest extent of a given concept and that concept activations would be less when the extent is less, One use case for interpretable regression would be severity prediction for skin lesions.
In comparison to other CBMs, we achieve the highest perfor- mance on CUB (Wah et al., 2011). The sask-specific nature of our concepts appears well-suited for fine- mne-griined classifi- cation. The task-agnostic DN-CBM (Rao al., 2024) does not report their CUB performance, but their outlook pro- poses pre-training on larger datasets to enhance fine-grained classification. As of now, we steer the concept genera- tion process solely by clustering similar concept proposals into a single cluster. Our ablations of removing small or large concepts show a small influence on CBM performance. Moreover, concept proposal steering using the promptable GroundingDino results in lower number of concept pro- posals. Given that accuracy is comparable to steering-free methods such as SAM2, we can assume that the steered concept set contains equally relevant concepts. Additionally, DCBMs allow the removal of undesired concepts using a text interface, making it adjustable to spurious correlations within a dataset's context.
Lastly, efficiency is a strength of DCBMs since they do not require general (Rao et al., 2024) or dataset-specific pre-training (Schrodi et al., 2024). Moreover, the concept proposal set is created using only 50 images per class; fewer images may be used as a trade-off for accuracy.
6. Conclusion
In this work, we present DCBM, novel approach to CBMs that uses data-efficient concepts entirely in the visual do- main. DCBMs employ simple and intuitive approach to deriving concepts by using segmentation or detection foundation models. Clustering concept proposals, reduces the different representations of concepts to a single pro- totyical one, which is used for CBM training and evalu- ation. DCBM's efficient and unrestricted adaptability to other datasets highlights its potential as a CBM framework for any dataset. Our approach generalizes well across do- mains and datasets, demonstrating versatility with respect to concept proposal methods like SAM2, GroundingDINO, and MASK-RCNN, Given that no pre-training is required, DCBMs can provide interpretations for a new dataset in under one hour (depending on dataset size).
Qualitative concept analysis (Section 4.3) shows that DCBMs' concepts contain image- and part-level concepts of the dataset's s classes. DCBM's 's interpretations highlight spurious correlations within the CLIP embedding space by basing decisions on semantically close concepts that are not present in the test image. While spurious correlations can be observed in all CLIP CBMs (Menon & Vondrick, 2023;
Oikarinen et al., 2023; Panousis et al., 2023; Rao et al., 2024; Yang et al., 2023), DCBMs allow for the investigation of visual spurious correlation within dataset-specifec concepts. Attribution methods such as GradCAM can visualize these semantic similarities between visually different concepts. We utilize DCBM as an inspection tool and can exclude undesired concepts from CBM training. Ensuring concept grounding using slot attention as in(Wang et al., 2023) or an image tagging model such as RAM (Zhang et al., 2024b) can be an automated measure against spurious correlations.
Impact Statement
This paper presents research that aims to ease the access to classification interpretability. Moreover, we want to high- light that misleading classifications may reduce trust in Al applications rather than increase it. These impacts can be observed within any CBM and are not specific to DCBM.
The current concept set contains the clustered crops of seg- ments or object detections, a simple and efficient approach. Regularization the concept selection can incorporate con- cept diversity concerning hierarchy. Previous works have defined threshold for image coverage (Zhu et al., 2024) to determine the number of concepts. We believe that detailed investigations can reduce the number of clusters without re- stricting their diversity and granularity. Additionally, other embedding spaces can be evaluated to discuss inter- model differences. CBM sparsity can be further increased to in- crease model interpretability (Schrodi et al., 2024).
A Acknowledgements
This work is partially supported by the BMBF (Federal Min- istry of Education and Research) project 16DKWN0R Cli- mate Visions and by the German Federal Ministry for Eco- nomic Affairs and Climate Action of Germany (BMWK). All experiments were run on University of Mannheim's server.
A possible next step can be the extension of DCBMs to have post-hoc interpretability by learning mapping from model
9

---

Data-Efficient Visual Concept Bottleneck Models
References
Gildenblat, J. and contributors. Pytorch library for methods. https:/ //gss:///iii gig.ub................. com/ jacobgil/ pytorch grad- cam, 2021.
Alvarez Melis, D. and Jaakkola, T. Towards robust in- terpretability with self-explaining neural networks. Ad- vances in neural information processing systems, 31, 2018.
He, J., Yang. S., Yang, S., Kortylewski, A., Yuan, X., Chen, J.-N., Liu, S., Yang. C., Yu, Q., and Yuille, A. Partima- genet: A large, high-quality dataset of parts. In European Conference on Computer Vision, 2022.
Bohle, M., Fritz, M., and Schiele, B. Convolutional dynamic aligmment networks for interpretable classifications. In Conference on Computer Vision and Pattern Recognition, 2021.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In International Conference on Computer Vision, 2017.
Böhle, M., Fritz, M., and Schiele, B. B-cos networks: Align- ment is all we need for interpretability. Conference on Computer Vision and Pattern Recognition, 2022.
Heidemann, L., Monnet, M., and Roscher, K. Concept correlation and its effects on concept-based models. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pp. 4780-4788, 2023.
Carion, N., Massa, F., Synnaeve. G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. European Conference on Computer Vi- sion, 2020.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In International Conference on Computer Vision, 2021.
Chefer, H., Gur, S., and Wolf, I.. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In International Conference on Computer Vision, October 2021.
Howard, Imagenette: A smaller subset of 10 easily classi- fied classes from imagenet. https: /tt////ttttt //gtttb......ggg com/ fastai/ www.imaaageeeeeeeeeeeee March 2019a.
Chen, C.,, Li, O., Tao, D., Barnett, A., Rudin, C., and Su, K. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems, 2019.
Howard, J. Imagewoof: a subset of 10 classes from ima- genet that aren't It so easy to classify. https: https://gttth::h /ttp//gtttttaabt com/ Tatttaiiiaageeeeeeeeeeeeeoeeeeee immggwoof, March 2019b.
Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., and Yuille, A. Detect what you can: Detecting and repre- senting objects using holistic models and body parts. In Conference Computer Vision and Puttern Recognition, 2014.
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, I.. Sparse autoencoders find highly inter- pretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://p//// /oppnrevieeee net/ forum? i id=F7 6bwRSLeK.
Deng, J., Dong, W., Socher, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recogni- tion. Icee, 2009,
Isola, P., Lim, J. J., and Adelson, E. H. Discovering states and transformations in image collections. Conference on Computer Vision and Pattern Recognition. 2015.
Donnelly, J., Barnett, A. J., and Chen, C. Deformable protopnet: An interpretable image classifier using de- formable prototypes. In Conference on Computer Vision and Pattern Recognition, 2022.
Kaufman, J. googlee11000-enngissh 2012. URL
Accessed: 2024-11-12
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y,, et al. Segment anything. In International Confer- ence on Computer Vision 2023.
Fel, T., Picard, A., Bethune, I.., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Conference on Computer Vision and Puttern Recognition, 2023.
Knab, P., Marton, S., and Bartelt, C. Beyond pixels: En- hancing LIME with hierarchical features and segmen- tation foundation models. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025. URL https: / /poorrevv ew net fo Forum? d=JHe5p 6nPbG.
Ghorbani, A., Wexler, J., Zou, Y., and Kim, B. Towards automatic concept-batt based explanations. Advances in Neu- ral Information Processing Systems, 32, 2019.
10

---

Data-Efficient Visual Concept Bottleneck Models
Koh, W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E.,, Kim, B., and Liang, P. Concept bottleneck models. In International Conference Machine Learning, 2020.
Menon, S. and Vondrick, C. Visual classification via de- scription from large language models. In International Conference on Learning Representations, 2023.
Kowal, M., Wildes, R. P., and Derpanis, K. G. Visual concept connectome (vec): Open world concept discov- ery and their interlayer connections in deep models. In Conference on Compuler Vision and Puttern Recognition, 2024.
Nauta, M.. Van Bree, R., and Seifert, C. Neural prototype trees for interpretable fine aie-ggrined image recognition. In Conference on Computer Vision and Pattern Recognition, 2021.
Oikarinen, T., Das, S., Nguyen, I.., and Weng, L. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Panousis, K. P., Ienco, D., and Marcos, D. Sparse línear concept discovery models. In International Conference on Computer Vision, 2023.
Lampert, C. H., Nickisch, H., and Harmeling. S. Learning to detect unseen object classes by between-class attribute transfer. In Conference on Computer Vision and Pattern Recognition, 2009.
Panousis, K. P., Ienco, D., and Marcos, D. Hierarchical concept discovery models: A concept pyramid scheme, 2024. URL https: https:://pppeeeeeee http:///ppeeeeeeeeeee/// httpewwweeeeeeeeeeeeeeeeeeeeeee forum? i 1d=gM8X6RbXXY.
Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Li, C., Yang, J. Zhang, L., and Gao, J. Segment and rec- ognize anything at any granularity. In Computer Vi- sion ECCV 2024 18th European Conference, Mi- lan, Italy, September 29-October 4, 2024, Proceed- ings, Part XLVIII, pp. 467-484, Berlin, Heidelberg, 2024a. Springer- Verlag. ISBN 978-3-031-731944 doi: 10.10077977833301---115552222222 URL /tattps:/dde org/10.10011 0tgp://..1077791-----11---333323
Patterson, G., Xu, C., Su, H., and Hays, J.
The sun attribute database: Beyond categories for deeper scene understand- ing. International Journal of Computer Vision, 108(1-2), 2014. Pham, T. Chen, P., Nguyen, T., Yoon, S., Bui, T. and Nguyen, A. PEEB: Part-based image classi- fiers with an explainable and editable language bot- tleneck. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2018-2053, Mexico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. (0.18653/1122224.fmmg http:////22224..iinnn-a-a-aa 131. URL https: /tteee//lalaaae http///aaaaaaallceeee
Li, F. Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. dlvaa-nex-intterlea Tackling multi- image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024b.
Li, T., Ma. M., and Peng, X. Deal: Disentangle and lo- calize concept-level explanations for vlms. In European Conference on Computer Vision, 2025.
131/.
Prasse, K., Jung, S., Bravo, I. B., Walter, S., and Keuper, M. Towards understanding climate change perceptions: A so- cial media dataset. In NeurIPS Workshop on Tackling Cli- mate Change with Machine Learning. climatechange.ai 2023.
Liang, V. W., Zhang, Y., Kwon, Y., Yeung, S., and Zou, J. Y. Mind the gap: Understanding the modality gap in multi- modal contrastive representation learning. Advances in Neural Information Processing Systems, 35, 2022.
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, 2024.
Purushwalkam, S., Nickel, M., Gupta, A., and Ranzato, M. Task-driven modular networks for zero-shot composi- tional learning. In International Conference on Computer Vision, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8768. PmLR, 2021.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Mahbooba, B., Timilsina, M., Sahal, R., and Serrano, M. Explainable artificial intelligence (xai) to enhance trust management in intrusion detection systems using decision tree model. Complexity, 2023(1):,3,411, 2021.
Raman, N., Zarlenga, M. E.,, Heo, J., and Jamnik, M. Do concept bottleneck models obey locality? In NeurIPS Workshop: XAI in Action, 2023.

---

Data-Efficient Visual Concept Bottleneck Models
Rao, S., Mahajan, S., Böhle, M., and Schiele, B. Discover- then-name: Task-agnostic concept bottlenecks via auto- mated concept discovery. In European Conference Computer Vision, 2024. First 2 authors contribute equally.
Wang, J., Liu, H., Wang, X., and Jing, L. Interpretable image recognition by constructing transparent embedding space. In International Conference Computer Vision, 2021.
Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. Zero- shot learning a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern anal- ysis and machine intelligence, 41(9))2251-226 2018.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Gir- shick, R., Dollar, P., and Feichtenhofer, C. SAM Seg- ment anything images and videos. In The Thirteenth International Conference on Learning Representations, 2025. URL https: id=Ha 6RTeMMdD..
Xu, X., Qin, Y., Mi, L., Wang, H., and Li, X. Energy- based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In The Twelfth International Conference Learning Representations, 2024. URL https::////// /opp///eeeneeeeeeeeee net / htr/m//rrmmm/m wwweemidddiiiooooo
Ribeiro, M. T., Singh, S., and Guestrin, C. "why should trust you?" explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining, 2016.
Xue, M., Huang, Q. Zhang, H., Hu, J., Song, J., Song, M., and Jin, C. Protopformer: Concentrating pro- totypical parts in vision transformers for interpretable image recognition. In Larson, K. (ed.). Proceedings of the Thirty-Third International Joint Conference on Artifi- cial Intelligence, IJCAI-24. pp. 1516-1524. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/jicai..222////6 URL https: / / //doi.oorggg/// 10 24 2224449333222 2024 118.. Main
Roth, K., Kim, M., Koepke, A., Vinyals, O., Schmid, C., and Akata, Z. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15746-15757, 2023.
Schrodi. S., Schur, J., Argus, M., and Brox, T. Concept bottleneck models without predefined concepts. CoRR, 2024.
Track.
Yang. Y., Panagopoulou, A., Zhou, S., Jin, D., Callison- Burch, C.. and Yatskar, M. Language bottle: Lan- guage model guided concept bottlenecks for interpretable image classification. In Conference Computer Vision and Pattern Recognition, 2023.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In- ternational Journal of Computer Vision, 128(2), October 2019.
Yuksekgonul, M., Wang. M., and Zou, J. Post-hoc con- cept bottleneck models. In International Conference on Learning Representations, 2023.
Sharma, P., Ding, N., Goodman, S., and Sorieut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 2556 2565, 2018
Yun, T., Bhalla, U., Pavlick, E., and Sun, C. Do vision- language pretrained models learn composable primitive concepts? Transactions on Machine Learning Research, 2023.
Sun, A., Ma, P., Yuan, Y., and Wang, S. Explain any con- cept: Segment anything meets concept-based explanation. Advances in Neural Information Processing Systems, 36, 2024.
Zabounidis, R., Oguntola, I., Zhao, K., Campbell, J., Step- puttis, S., and Sycara, K. Benchmarking and enhanc- ing disentanglement in concept-residual models. arXiv preprint arXiv:2312.00192 2023.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. Caltech-ucsd birds 200. Technical Report CNS-TR- 2011-001, California Institute of Technology. 2011.
Zakka, K. A Playground for CLIP-like Models, 7 2021. URL httpt:ttt////tubbbb.bb http:///hhhub.ccoeeeeeeeaaaaaaee www.kewinkkkkkaaa/////oomm clip elip_playgroun
Wang, B., Li, L.., Nakashima, Y., and Nagahara, H. Learning bottleneck concepts in image classification. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10962-10971, June 2023.
Zhang. J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., and Sclaroff, S. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):0884-1103 2018.
12

---

Data-Efficient Visual Concept Bottleneck Models
Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Ru- binstein, B. I. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Conference on Artificial Intelligence, volume 35,13, 2021.
Zhang, R., Du, X., Yan, J., and Zhang,
S. The decoupling concept bottleneck model. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024a.
Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T. Li, Y., Liu, S., et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1724-1732, 2024b.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recog- nition. Transactions Pattern Analysis and Machine Intelligence, 40(6), 2017.
Zhu, L., Kong, X., Li, R., and Guo, G. Where you see is what you know: A disual-semantic conceptual explainer. In International Conference Multimedia in Asia, 2024.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A.. Yin, X., Mazeika, M., Dombrowski, A...., Goel, S., Li, N., Byun, M. J.,, Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engincering: A top-down approach to ai transparency, 2023.
13

---

Data-E Gata-Efficient Visual Concept Bottleneck Models
A. Framework
We describe the DCBM framework in the main paper. For better understanding, we provide the framework with pseudocode in Algorithm 1 and introduce notation this end.
An object O can be represented combination of concepts from global concept pool C. Let C₁ C denote individual concepts, and let each object be characterized by subset of these concepts with corresponding weights.
C W, Cr
W 20
where: O2 E C represents a concept from the global pool, wy denotes the contribution of concept C₁ to the image, C2 is located in the image J, R₂ C is the region of the image where concept C, is present,
Further, we assume visual concepts to be spatially localized in images and therefore conjecture that data efficient concept extraction should build upon image segments or regions rather than entire images. Therefore, we generate the global concept pool C by segmentation or detection foundation models, which are then cropped and used as concept proposals s;. All 8; E S are then clustered into the global concept set C.
Algorithm DCBM Framework
1: Input: 'D (Training dataset), T (Text corpus), n (Number of images per class), S (Segmentation model), f (Vision encoder), frert (Text encoder), & (Number of clusters), amin and Omax (Minimum and maximum area thresholds for segmentation).
2: 1. Concept Proposal Generation
3: Dn t- [randomly select n images from each class in D)
4: S + {M(x x } } (Apply segmentation to each image]
5: S + (crop(z, s E Dns si E S} (Extract bounding boxes based on segmentation]
6: S + {sf ESS aarr(ss) (t) [Amin, Amax]) (Filter segments by area}
7: 2. Concept Creation
8: sá + [(8) (Encode segmented concepts with vision encoder)
9: Ddt t f(D) (Encode entire dataset}
10: T t (T) (Encode text corpus}
11: ids + cluster( (S
Stnce k) {Cluster encoded segments into & clusters}
12: C + (centroid(S)e Sence ids = i) i i= [k} {Compute centroids for each cluster]
13: 3. Concept Bottleneck Model
14: A <- (Danet) for 1,....K l {A represents the activation of concepts in D) 15: DCBM + fit(A,y E DD (Train linear model with ground truth}
1e,
16: 4. Concept Naming (optional)
17: sim + siosssimmTlll C) [Compute similarity between textual and visual concepts}
18: + + reros([TTh)] (Initialize as a zero vector with the same length as C}
for сj E do
20: tg + arg max(sim;) (Get the index of the maximum similarity in the j-th row)
21: end for
22: Cramed <- (T(b)) E (1,2...... ... [T]}} (Named concepts with descriptions}
23: Output:
24:
DCBM, C, Cnamed
The underlying feature extraction method denoted by Q with the hyperparameter settings as hp. Possible choices are the foundation models Segment Anything & 2 (Kirillov al., 2023; Ravi et al., 2025), GroundingDino (Liu al., 2024), MaskRCNN (He et al., 2017), and DETR (Carion et al., 2020). We compare the segmentation models, which require no input, to the open-set detection method Groundingginno in which we use various concept sets from the literature as input. We construct a concept proposal subset, D, by selecting configurable number of random training images per class. This has the goal of mitigating the risk of overfitting while increasing efficiency since not all training images are used for training. We set n = 50 for the main paper, for CUB, which contains fewer samples, we include all available images in D. In Appendix A.I we diseuss DCBM's data-efficiency in detail.
14

---

Data-Efficient Visual Concept Bottleneck Models
Each image i in the subset D is segmented by Ω into concept proposals, denoted as s/. The number of concept proposals varies between images and is determined by the hyperparameter settings hp. We crop the concept proposal's bounding box and add the resulting sub-image to concept proposal set as S.The removal of background information is ablated in Appendix D.I. To refine the concept proposals, image parts that are below or above a given size threshold can be excluded from S, as evaluated in Appendix F.2.
A.1. Data-efficiency
DCBM is highly data-efficient in the concept generation phase and equally efficient as other models in the CBM training
phase.
In Table we show the differences to another data-driven CBM, i... DN-CBM (Rao et al., 2024). DCBM does not require general pre-training. but it can be directly employed for the dataset in question. This makes DCBM an effective tool in single-dataset settings.
DN-CBM
DCBM ImageNet
Dataset size
3,300k image-caption pairs (CC3M)
50k images (50 imgs/class) GB
Add. memory capacity
850 GB (assuming 256x256px)
No extra data required
X
Table CBM training preparation: DN-CBM (Rao et al., 2024) vs DCBM (ours).
We have set 50, i.e., selected only 50 images per class for the concept proposal generation. This subset is sufficiently large and diverse, as Table 6 shows. Using an alternative subset or the combination of the existing and new subset does not have an impact on the CBM accuracy. CUB is not included in the table as it has less than 50 images per class and thus the entire dataset is used for concept proposal generation.
# imgs/ / class
ImageNet
Places365
Cifar10
C0
sl (main paper)
50
77.4
52.2
97.5
85.3
(new)
50
77.5
52.2
97.6
85.4
s1+s2
100
77.1
52.1
97.7
85.5
Table 6. Performance evaluation of subset selection
Data-efficiency can be further increased by using fewer training samples at the cost of slight performance degradation, as shown in Appendix E. Moreover, reducing the number of training data points for CBMs feasible and results in only a modest performance degradation, as shown in Table 7. This relatively minor trade-off highlights the potential applicability of DCBMs in data-scarce environments.
# imgs / class
ImageNet
Places365
Cifar10
Cifar100
all
77.4
52,2
97.5
85.3
50 imgs/ class (SAM2)
75.5
46.1
91.2
79.1 1
50 imgs/ class (GDINO)
75.0
46.2
93.1
79.4
50 imgs/ class (MASK-RCN)
75.5
46.5
94.8
80.3
Table 7. Performance evaluation when reducing the number of training samples to 50 images per class (ViT-L/14).
15

---

Visual Concept Bottleneck Models
B. Dataset overview and details
B.I. Ablations
We ablate on three datasets: ImageNette (Howard, 2019a), ImageWoof (Howard, 2019b), and CUB (Wah et al., 2011). The first two datasets are both subsets of ImageNet (Deng et al., 2009). ImageNette contains easier-to- casier-to-classifyy categories like tench, English springer, and church. ImageWoof images focus dog breeds, curated for fine-grained classification tasks, The Caltech-UCSD Birds 200 dataset is designed identify fine-grained bird species with detailed annotations. For the datasets which do not have test split, we use the validation split for testing and create a new split, i.e., 10% of train set, for validation, Consequently, the train split comprises only 90% of the original train images (see Table 8).
Table 8. Ablation datasets. Overview of the datasets used for ablation (Image Woof, ImageNNette, and CUB-200-2011
Dataset
Classes
Images
%(rrai / val test)
ImageNette
10
13,000 (70 / 30 0)
ImageWoof
10
12,000 (70 / 30 0)
CUB-200-2011
200
11,788 (50 / 50 50/0)
B.2. Benchmark analysis
For the primary analysis, we compare five datasets: Imagenet (Deng et al., 2009), Places365 (Zhou et al., 2017), CUB (Wah al., 2011), cifar10 and c0far100 (Krizhevsky et al., 2009). For ImageNet, we report on the validation split. Thus, we use it as our test set. Again, we create small validation set to select the hyperparameters independent dataset in situations where we cannot access three labeled splits (see Table 9).
Table 9. Main datasets. Overview of the datasets used for benehmark experiments.
Dataset
Classes
Images
(train / val / test)
ImageNet
1,000
1,431,167 (90 / 3 77)
Places365
365
1,839,960 (98 / 2
CUB-200 LUB-200-2011
200
11,788 (50 / 50 / ()
CIFAR-10
10
60,000 (83 / / 17)
CIFAR-100
100
60,000 (83 / 17)
B.3. Additional datasets
the CBM research and summarize them in Table 10: MiT-States (Isola We use five datasets which are novel ImageNet-R (Hendrycks et al., 2021), awa2 (Xian et al., 2018), and CelebA et al., 2015), Climate-TV (Prasse et al., 2023), (Liu et al., 2015). MiT-States contains images of 245 object classes in combination with 115 attributes. This allows the assessment of concept recognition in previously unseen shapes or forms, as the test set contains 50% seen and 50% unseen versions of objects, e.g., ripe tomato, unripe tomato, moldy tomato. We evaluate the accuracy of the object class. Thus, we use this dataset to assess whether we can detect an object independent of its state. We use the train-val-test split (30k-10k- 13k) introduced by (Purushwalkam et al., 2019). ClimateTV contains social media images of climate change and includes a diverse range of images. We have created a balanced set for the animals-sue supealssspperllass. which contains 11 animal classes, including "no animals". This allows the assessment of real-life images, which are, by design, messier than curated datasets. We employ animals with attributes in the drop-in version 2 (Xian et al., 201 8) consisting of animal images labeled with 85 numeric attributes. We have run DCBM on AwA using the standard 50:50 train and test split. We created the val set by randomly selecting 10% of train samples. Lastly, we employ CelebA (Liu et al., 2015), which contains face photographs of 177 celebrities, in which 5 landmark locations and 40 binary attribute annotations are provided for each image. This dataset is designed for face landmark detection and is transformed as described by Zhang al. (2024a), using a 70 : 10: 20
16

---

Data-Efficient Visual Concept Bottleneck Models
(train:val:test) split. We randomly sample every 12th image as done by past research (Zhang al., 2024a). In order move towards classification datasets, the most balanced attributes taken and all possible combinations are used as class labels, thus resulting into 256 possible classes. The most balanced attributes are: Attractive, Mouth SaaihhSlllhhhppppse Smiling, Wearing Lipstick, High ChehhCheebbones, Male, Heavy MMakeup, and Wavy Hair. Given that some combinations of attributes are highly unlikely, we exclude classes with less than 10 samples, thus resulting in 110 actual classes.
Table 10. Additional datasets, Overview of the additional dataset (MiT-States, ClimateTV, and ImageNet-R),
Dataset
Classes
Images
% (train val / test)
MiT-States
245
53,743 (57 19 /9/24/ 24)
ClimateTV
11
660 (80 / 20)
AWA2
50
37,322 (16,789/ 1,863/ 18,670)
CclebA
110
16,651 (11.603 1,602 / 3,446)
ImageNet-R
200
30,000 (0/ 100 0)
ImageNet-R consists of 200 ImageNet classes and is generally used to evaluate ood performance of model trained on ImageNet. The "R" stands for renditions of which the dataset contains cartoons, graffiti, embroidery, graphics, origami, paintings, and many more (Hendrycks et al., 2021). This task is simplified by training the model exclusively on the 200 classes of ImageNet-R. our case, however, we report the model's performance when trained on all ImageNet classes.
C. Concept proposal generation
We provide the code for all segmentation methods employed and make publicly available. Table 11 provides an overview of the segmentation models' hyperparameters, which are unchanged for MaskRCNN and DETR compared to the pre-traince models. Hyperparameters were set to retrieve concept proposals that break the image into sub-parts. For GroundingDino, evaluated two thresholds to compare whether more, noisier or fewer, cleaner concept proposals performed better.
Table 11. Hyperparameters of segmentation models. We only report model hyperparameters if changed compared to standard setting.
Segmentation model
Hyperparameters
SAM
points.pee wwr.-perreeeeeeeeeeeeeeeeeeeeeee side = 64,
pred iou thresh 0.88,
stability score thresh 0.95.
box_nms_thresh = 0.5.
min heii_mass_reeionnoo arrea = 500
SAM2
points-per_side 64,
pred iou thresh 0.88,
stability cicllittyscore_tiee threshhrehhhhhhh 0.95,
box nms thresh = 0.5.
man_mmak_rrgionn.uu aaraa = 500
GDINO
box_ toxtthheshlld [0.35, 0.25).
text.thesshold 0.25
C.1. Prompts
For the promptable GroundingDINO model, we assess the efficacy of several prompts w.r.t. CBM performance. To this end, we use attribute or part labels, which have been published in other contexts, thus avoiding manual or LLM-based concept set generation. We use the part annotations for CUB (Wah et al., 201 1), the attributes for Animals with Attributes (GDINO Awa) (Lampert et al., 2009), and standard lart-labels from Imrgeetteeett (He et al., 2022) and GDINO (Chen et al., 2014).
17

---

Data-E Pata-Efficient Visual Concept Bottleneck Models
D. Ablations
conduct ablation studies to evaluate DCBM's performance under various hyperparameters and configurations. These studies are performed on the ImageNette, ImageWood and CUB identify settings that yield strong performance across all three datasets. The identified configurations are used for the main experiments presented in Section 4. Each DCBM was trained with 1024 clusters, using K-Means clustering with the median of each cluster as concept, a learning rate of le-4, and sparsity regularization set to le-4. We explore broader range of segmentation and object detection models beyond those discussed in the main paper, such as SAM and extended prompts for GDINO. The "Low" label indicates low threshoid for the corresponding object detection configuration. What differentiates these experiments from those in the main paper is the use PCA, reducing the dimensionality of concept proposals to 100. This approach accelerates clustering and enables more efficient experimentation. Additionally, we investigate the impact of PCA on the performance of DCBM.
D.1. Background removal
The background influences embedding the image regions used as concepts for the segmentation methods employed. To assess this effect, we examine accuracies in Table using three different CLIP embeddings: ResNet-50, ViT B/16, and ViT L/14 while excluding the background concept, based on ImageNette, ImageWoof, and CUB. We report SAM, SAM2, Mask R-CNN, and DETR results. Each model trained with learning rate of 1e-4, sparsity parameter À of le-4. 128 clusters using median and k-means, and 50 images per class within the DCBM framework.
Table 12. Background performance influence. datasets.
Performance comparison with (w/) and without (w/o) background on the ablation
CLIIP ResNet-50
CLIP VIT-B/16
CLIP VIT-/114
Model
ImageNette
ImogeWoof
CUB
ImageNette
ImageWoof
CUB
ImageNette
ImageWoof
CUB
SAM w/o
98.55
90.96
57.04
99.57
93.66
71.68
99.87
95.42
80.03
SAM w/
98.50
90.81
57.62
99.54
93.79
73.44
99.80
97.78
79.58
SAM2
98.73
91.70
59.89
94.38
73.61
99.87
95.75
80,98
SAM2
98.78
91.68
61.44
94.30
75.49
99.85
95.80
82.21
MaskRCNN w/o
98.65
91.93
64.62
99.59
94.38
77.60
99.80
95.70
83.60
MaskRCNN
98.68
91.89
65.46
99.59
94.60
77.61
99.82
95.65
83.32
DETR w/o
98.65
91.80
64.17
99.46
94.45
76.84
99.82
95.75
82.57
DETR w/
98.73
92.03
61.18
99.54
94.58
76.91
99.80
95.78
82.52
D.2. PCA
To assess the impact of different hyperparameter settings within the DCBM framework, we analyze clustering similarities across various values of k in K-means and compare these results across different CLIP backbone models in Figure 7. Our primary focus is on the CUB dataset, as its exclusive focus on bird images introduces ambiguity in clustering compared to other datasets. Additionally, Figure 8, we examine the effect of applying PCA with 100 components to the segment embeddings before clustering determine whether this transformation yields similar clustering outcomes. For this evaluation, we use the NMI (Normalized Mutual Information) metric, where an NMI score close similarity, meaning that the two clustering approaches capture comparable groupings structures. In contrast, score near 0 suggests minimal alignment, indicating substantially different clustering results.
I indicates high
18

---

Data-Efficient Visual Concept Bottleneck Models
Table 13. Impact PCA on CUB dataset. Accuracy comparison DCBM with and without PCA (100 components) evaluated on the
CUB dataset.
ResNet-50
CLIP ViT-B/16
CLIP ViT-L/14
DCBM-SAM2 (Ours)
61.2
75.1
81.9
DCBM-SAM2 (Ours) w/o
61.4
75.3
81.8
DCEM-GDINO (Ours)
58.9
73.9
81.1
DCBM-GIINO (Ours) w/o
59.0
74.1
81.3
DCBM-MASK-RCNG (Ours) w
64.6
76.8
82.
DCBM-MASK-RCNN (Ours) w/o
64.6
76.7
82,4
As shown in Figure 7, the B16, L14 combination consistently achieves the highest NMI scores. Additionally, as the number of clusters k increases, the NMI scores improve in all segmentation techniques and backbone combinations, indicating that larger cluster sizes capture more meaningful distinctions in the data. These results suggest concept clusters remain similar across different CLIP backbones, with minimal unique variation among specific backbone combinations.
NMI Scores for SAM2
NMI Scores for MaskRCNN
NMI Scores for GDINO
0.70
Performance
0.65
Performacc
0.60
Performance
0.55
Model Combinations
0.55
Model Combinations
0.50
Model Combinations
0.50
616,
0.45
616, RM50
B16, RN50
B16, RN50
0.45
L14, RN50
L14, RN50
L14, RNS0
256
912
2049
120
512
2048
512
1024
2040
Clusters
Clusters
Clusters
Figure 7. Backhone NMT scores for the CUB dataset. NMI scores for three CL.IP model combinations: B16, 1.14 (blue), B16, RN50 (orange), and I.14, RN50 (green), showing clustering performance across different backhones across cluster sizes (128, 256, 512, 1024, 2048).
Similarly, Figure illustrates the impact of PCA on cluster similarity. While PCA reduces dimensionality and speeds up the clustering process, the clusters formed remain broadly consistent with those generated without PCA. This suggests that PCA minimally affects the conceptual alignment within CBMs, preserving the overall clustering structure. Furthermore, as demonstrated in Table 13 for CUB, this limited influence extends to the overall performance of DCBMs.
NMI Scores (PCA) for SAM2
NMI Scores for MaskRCNN
NMI Scores for GDINO
0.77
0.78
0.76
Perforancce
Performance
Performance
0.74
0.74
Model Combinations
0.74
Model Combinations
Model Combinations
0.73
B16
0.72
B16
0.70
B16
RN50
RN50
RN50
0.66
125
512
3024
2048
512
2043
2046
Clusters
Clusters
Clusters
Figure 8. PCA NMI scores for the CUB dataset. NMI scores illustrate clustering consistency with and without PCA preprocessing (100 components) for each CLIP backbone across varying cluster sizes (128, 256, 512, 1024, 2048).
19

---

Visual Concept Bottleneck Models
D.3. Clustering algorithm
We evaluate clustering algorithms in the main ablation study: hierarchical clustering and K-means. The results, presented in Table 14, are based consistent configuration where the median centroid is used, and the number of clusters is fixed at 1024 across all datasets. While the performance scores of both methods are generally comparable, K-means slightly outperforms hierarchical clustering in most cases. Consequently, we use K-means for the main experiments.
Table 14, Clustering method comparison. Performance comparison of different clustering algorithms (Hierarchical and K-Means), both with l set 1024.
Chustering
CLIP RexNet-S0
CLIP VIT-L14
ImageNerte
Wool
ImtgeNNtte
ImageWoof
ImageNette
ImageWoot
CUB
SAM
Hierarchical
K-mnuns
98.52
90.86
37.62
99.57
93.74
73.44
99.82
95.90
80.03
SAM2
Hierarchical
98.75
92.11
61.15
99.62
94.45
75.39
99.85
95.80
82.12
Hierarchical
98.77
K-muuns
98.65
92.21
64.27
99.54
94,60
77.01
99.80
95.80
82.64
MiskRCNN
Hierarchical
98.68
92.03
65.41
99.57
94.35
77.53
99.85
95.78
83.21
GDINO Awa
Hierarchical
98.57
91.27
99.41
93.99
99.87
95.70
K-means
98.50
91.17
$7.97
99.16
94.01
73.63
99.89
95.60
$1.12
GDINO Awa Low
Hicrarchical
98 60
90.1
57,84
99.44
94.15
73.92
99.87
95.62
81.17
GDINO Partimagenct
Hicrarchical
98. 98.47
91.30
$9.30
99.39
94.12
74.30
95.54
81.14
K K-means
98.47
91.27
59.63
99.44
94.17
74.70
99.82
95.62
$1.52
GDINO Partimagenct Low
Hicranchical K-meuns
98.55
90.76
58.99
99.46
94.25
74.44
99.82
95.55
81.14
GDINO Poscl
Hicrarchical
90.81
58.77
99.44
94.15
73.66
99.87
95.62
80.74
K K-means
98.57
90.91
$8.80
99.41
93.17
73.23
99.87
95.67
$1.00
GDINO Poscal Low
Hicrarchical K-means
98.62
90.84
58.37
99.19
94,10
73.97
99.85
95.75
$1.03
GDINO
Hicrarchical K-means
98.09
83.10
58.85
99.16
88.90
73.70
99.62
93-10
$0.95
98.16
82.97
58.85
99.18
88.78
73.77
99.62
93.54
$1.00
GDINO Sun Low
Hicrarchical
98.60
90.94
58.91
99.17
93.79
74.12
99.82
95.88
$1.36
Additionally, we tested DBSCAN and HDBSCAN. However, these methods required significant hyperparameter tuning to achieve meaningful clusters. In contrast, K-means and hierarchical clustering required only the definition of k. We leave this analysis for future work.
20

---

Data-Efficien Visual Concept Bottleneck Models
D.4. Centroid selection
After clustering the concept proposals & into clusters, centroid is generated for each cluster to represent it. We evaluate the impact of using two different methods to compute centroids: the mean of the embeddings within cluster and the median embedding, as shown in Table 15. This analysis is applied to both integrated clustering techniques while keeping all other configurations consistent with the standard settings. Overall, the K-means configuration using the median centroid performs the best across the three datasets, While there are specific scenarios where other combinations yield better results, we adopt this standard setting for simplicity following this investigation.
Table Centroid selection: mean vs. median. This figure illustrates the comparative analysis of centroid selection methods mean and median- on hicrarchical and K-means clustering performance.
Clusering
Centrid
LP RosNa-50
VI--8/16
VIT-1114
ImageNotte
Woof
CUB
ImageNette
ImageWoof
CUB
ImageNeme
ImageWoof
CUB
Hicrandiccal
Hieranciica
Median
98.60
90.74
61.15
99.52
93.38
73.27
99.82
82.11
K-means
Mean
98.42
89.67
48.64
99.46
93.03
69.33
09.77
95.37
76.49
K means
Median
98.52
90.85
57,02
99.57
93.74
73.64
99.82
95.90
90.03
SAM2
98.73
Hicranchical
Median
98.25
92.11
61.15
99.62
94.45
75.39
99.85
95.00
K2.12
K-means
Mean
98.75
92.20
56.81
09
91:60
73.30
09.82
95.83
871 19
K means
Median
98.72
92,19
61.44
99.59
91.60
75.60
99.85
95.70
61.85
98.88
Hicranchical
Median
98.73
92.03
63.96
99.54
94.53
76.82
96.77
85.85
K2.31
K-means
Mean
98
19
63.27
99
94.52
76.21
10.80
95.80
81.00
K means
Median
98.65
92.21
64.27
99.54
94.60
77.01
99,80
95.80
82.64
MaiRCCNN
98.70
99.59
94.48
99.79
95.62
Hicranchical
Median
98.68
92.03
65.41
99.57
94.35
77.53
99.85
95.7H
K3.21
K-means
Mean
98.75
91.75
63.00
99.59
9463
77.13
99.82
95-62
81.22
X means
Median
98.25
91,86
65.46
99.60
94.00
77.70
99.85
95.65
83.33
Hicrandiical
98.57
9402
99.85
95.78
78.60
Hicranchical
Median
98.57
91.27
58.54
99.41
93.99
74.13
99.87
95.70
81.36
K-means
Mean
02
91.077
$3.00
99.41
93.97
70.92
99.87
95.70
70.00
X means
Median
98.50
91.17
$7.97
99.46
94.04
73.63
99.89
95.60
81.12
GDINO Awa Low
Dierarciical
Mean
98.60
90.61
50.31
99.44
93.87
89.95
99.87
95.33
78.25
Hicranchiaal
Median
93.60
90.81
57.84
99.44
9415
73.92
99.87
95.62
81.17
K-means
Mean
98.52
90.79
63.00
99.39
93.94
73.13
99.90
95577
87.22
X means
Median
98.57
90.91
65.43
99.41
94.17
77.70
99.87
95.67
83.09
GDINO Partimagenet
Dierarciical
Mean
98.44
90,99
53.57
99
94.17
71.35
99,90
95577
79.01
Hicranchiaal
Median
9X.47
91.30
59,30
99.39
94.12
74.30
99,90
95.54
81.14
K-means
Mean
98:47
11.19
$3.92
30
14
71.54
99.82
95.57
75.79
X means
Median
98.47
91.27
59,03
99.44
94.17
74.70
99.82
95.62
81.52
GDINO Partimagenet Low
Dieranciical
Mean
98.42
90.63
52,42
99.46
94,20
70,66
99.85
95.00
78.20
Hicranchical
Median
98.55
90.76
58.99
99
94.25
74.44
99.52
95.55
81.14
K-mrans
Mean
98.42
90.66
$1.97
99.49
94 94.90
70.56
99.84
95-62
75.11 12
X means
Median
98.52
90,99
59,39
99.52
94.17
74,20
99.85
95.83
81.22
GDINO Pascal
Dicranciical
Mean
98.60
90,60
$1,00
99.44
93,85
70.02
99.87
9582
71.99
Hicranchical
Median
93.60
50.81
59.77
99.44
9415
73,66
99.87
95.62
80.74
K-mrans
Mean
98.52
90.79
50.95
99.3
91.94
70.26
99.90
95.77
77.93
X means
Median
98.57
90.91
$8,90
99.41
93.17
73.73
99.87
95.67
81.00
GDINO Pascal Low
Dicranciical
Mean
98.62
90.83
50.17
99 99.46
93.84
89.66
99.87
95400
73.61
Hicrancicaal
Median
98.62
90.84
59.37
99.49
94.10
73.97
99.85
95.75
80,05
K-mrans
Mean
98
90.56
50.48
99.49
91.99
89.64
99.85
95.60
77.51
X means
Median
98.62
90.89
58.25
99.52
93.89
73.78
99.85
95.75
81.15
GDINO Sun
Iierarciccl
Mean
98.11
62.77
59,53
99.13
88.52
73.71
99.62
93.33
80.91
Hicranhicall
Median
93.09
83.10
59.85
99.36
85.90
73.70
99.62
93.40
80.55
K-mrans
Mean
98.09
63.13
59.73
99.13
88.65
73.66
99.57
93.54
80.84
K-mcans
Median
98.16
82.97
58.85
99.18
88.78
73.77
99.62
93.54
81.00
GDINO Sun Low
Ilerancicall
Mean
98.50
90,94
$4.09
99.49
93.71
72.39
99.82
79.43
Hicradiical
Median
93.60
90.94
59.94
99.47
93.79
74.42
99.82
05.38
81.36
K-mrans
Mean
98.55
90.91
$4,80
99.49
93.43
72.61
99.80
95.75
80.00
Kemzans
Median
98
90.94
59,00
99.47
94-46
74.54
99.80
95.88
81.51
21

---

Visual Concept Bottleneck Models
D.5. Number of clusters
As described in Section cluster the concept proposal S into & clusters. The choice of k determines the granularity of the concepts used in the DCBM method. To analyze this, test various values of k within the K-Means algorithm, as shown in Table 16, presenting ablation study of k. We do not impose the 50- iiages--pr--lanne experiments due to the generally smaller number of images available dataset and class. As the table illustrates, increasing the number of clusters leads to a corresponding improvement in accuracy scores, highlighting the benefit of finer-grained concept representations.
limitation in these
Table 16. Number of clusters. Here we ablate the impact of different values of for K-Means within DCBM.
Model
Clusters
CI IP ResNer-50
CLIP ViT:B/16
CLIP VIT-214
ImaggeNette
Image Wool
CUB
ImageNettie
ImageWooff
CUB
ImageNetic
ImageWWoof
CUB
SAM
128
97.73
82.29
37.40
99 13
89.41
59.82
99.54
93.05
71.35
256
98.27
86.77
46.58
99.31
92.08
68.61
99.80
95.06
76.61
72.11
79.94
1024
08.50
90.81
$7.62
99.9 54
93.79
73.44
99.80
97.78
79.58
2048
99.16
73.85
80,44
SAM2
128
98.
87,022
44.68
99.3 36
92.
65.29
99. 801
95. 35,50
75.56
512
98.62
91.14
57.21
99.35
94.35
74:70
99.80
95.78
81.55
1024
98.78
91.08
61.44
99.57
94,30
75.49
99.85
95.80
$2.21
2048
62.50 22.98
75.80
82.22
DETR
98.42
$2.07
99.23
93.06
70.62
99.25
95.42
78.03
90.81
34.30
74.59
95.62
512
98.58
91.81
63.48
99.47
94,50
76.41
99.77
95.67
82.24
98.73
92.03
99.54
94.58
76.91
99.80
95.78
82.52
76.67
MaskRCNN
256
98.
90.38
61. 866
99.41
94.
76.79
99. 811
95.70
82. 31
512
98.57
91,48
62.50
99.55 52
94.45
77.84
99.82
95.57
83.28
98. 68
95.65
2048
98.80
92.11
65.71
99.62
94.53
77.39
99.87
95.60
83.34
GDINO Awa.
128
98. 19
85,52
40.94
99.34
91.19
64,95
94,02
74,20
256
98.19
89.08
49.74
99.39
93.48
70.66
99,82
95.39
79.25
98.42
55.13
93.94
73.00
95.34
1024
91.17
60
94.04
73.63
95.72
$1,20
2048
60.18
74.68
$1.53
GDINO Awa Low
128
98.
84,37
39.23
99.34
90.81
62.43
99.72
94,40
72.37
256
98.19
47.76
99.29
92.90
69.55
95.01
78.48
98.
99
95.62
1024
98.50
91.22
$9.82
99.47
94.02
74.00
99.90
95.72
81.41
2048
59.58
74.32
81.62
GDINO Partimagenet
128
97.91
86.97
41.91
99. 36
91.09
66.32
99.75
94,42
75.20
512
98.39
90.38
56.42
99.43
93.39
73.80
99.82
95.47
80.43
1024
98.52
91.20
59.63
99.43
94.14
74.70
99.80
95.62
$1.51
GDINO Partimagenes
43.17
65.98
94.25
74.87
256
98.1
88.34
50.64
99 39
93.33
70.52
99.77
95.39
78.65
512
98.45
90.25
56.08
99.47
93.63
73.25
99,82
95.62
$0.41
1024
98.52
90.99
59.39
99.19
94.17
74.20
99.85
95.60
81.22
$1.72
GDINO Pascal
73.77
256
08.45
88.70
48.33
99.29
93.13
69.04
99.82
95.
77.91
512
98.4
90.48
54.35
99.41
93.82
72.54
99. 80
95.39
80.29
2048
59.37
74.20
81.38
GDINO Pascal Low
128
98.27
84.58
40.75
99.29
90.43
65.15
99.77
93.65
73.16
256
98.24
88.55
49.50
99.41
93.00
70.21
99,82
95.14
79.49
72.47
90.91
75.89
95.67
2048
59.84
74.04
$1.69
GDINO Stm
128
V8.11
$2.99
43.04
99 18
88.85
66.76
99.62
93.56
74.94
256
52.61
71.26
78.81
1024
58.66
75.75
$1.00
GDINO Sun Low
128
97.99
84.81
42.3 37
99.23
90.91
66.17
99.72
93.92
75.06
256
98.
88.34
49.6
99.42
92.84
71.73
99.75
95.09
79.0K
1024
98.60
90.99
58.94
99.49
93.48
74.59
99,80
95.85
$1.34
2048
60.03
75.08
$1.58
22

---

Data-Efficient Visual Concept Bottleneck Models
D.6. Sparsity parameter À and learning rate
Table 17. Part learning rates and sparsity. Performance across CLIP ResNet-50, ViT-B/16, and ViT-L/14 the ImageNette, ImageWooff and CUB datasets with varying learning rates and sparsity parameters for segmentation models.
Configaration
Model
Learning Rate
Sparsity
ImageNNette
ImageWoof
ImageNette
ImageWoof
Image
ImageWoof
SAM
1o-1
le-4
98.50
90.51
57.71
99.72
93.74
73.14
99.87
95.67
79.73
1o-3
1o-4
98. 24
89:41
43.25
99.31
92.54
66.38
99.75
95.19
74.68
1o-4
le-3
98.57
91.09
$4.59
99:44
93.87
69.38
99.77
95.72
76.46
1e-2
47.38
63.005
1o-3
le-3
98.14
89.18
38.06
99.29
92.52
58.50
99.72
91.99
68.43
1e-2
16-2
97.25
68541
3.37
88
82.4 44
6.77
99.54
50.22
10,44
SAM2
1e-1
le-4
98.75
91.50
61.44
99.51
94.32
75:49
99.90
95.75
S2.21
1e-4
98.55
49
93.84
70.33
95,72
77.55
1e-2
1e4
97.76
85.01
6.99
99.44
90.38
14.017
99.69
93.94
21.4 .45
1e-4
le-2
98.62
90.02
$1.19
99.49
93.00
81
99.8 83
96.60
72.04
1o:3
le-3
98.50
90.23
43.94
99.57
93.71
62.05
80
95.57
70.54
16-2
77.07
66
99.70
91.19
15.74
DETR
1e-1
le-4
98.75
92.03
63.20
99.52
94.58
76.91
99.79
95,80
82.64
1e:3
16-
96.55
91.11
56.32
99.46
94.19
72.92
99.77
95.72
78.63
91.70
1c-4
le-3
91.75
91.75
60.04
99.49
94.25
72.66
99. 801
95.75
79.50
le-2
54.16
1e-3
le-3
98.62
90.48
50.41
99.39
93.88
75.74
99.77
95.55
72.71
1e-2
1p-2
97.27
83.69
6.51
98.77
26. 92
10.08
99.44
92.44
13.36
MaskRCNN
1e-4
le-4
98.73
91.93
65.46
99.61
94.60
77.61
99.85
95.72
83.33
90.96
58.72
94.25
99.77
95.65
le-2
le-4
97.91
85.39
13.69
99.26
91.70
31.05
99.69
95.43
50.38
1e-4
le-2
98.57
89.92
55.75
99.46
93.26
68.52
99.89 85.85
92.19
75,80
16.3
le3
98.50
90.25
52.19
99.41
93.84
67.83
99.77
95.39
74.94
99.57
In Table 17 and Table 18, we analyze the impact of different hyperparameters for sparsity (A) in Equation (3) and the learning rate. To provide more transparent overview, we present the results in two separate tables: Table 17 focuses on segmentation models, while Table covers object detection models. All other configurations remain consistent with the settings reported earlier. From the results, the optimal configuration across the three datasets and various concept creation modules is learning rate of 1e-4 combined with a sparsity of 1e-4. Although specific configurations demonstrate improved performance with alternative values, aim to identify baseline configuration that performs robustly across diverse scenarios.
23

---

Data-Efficient Visual Concept Bottleneck Models
Table learning rates and sparsity. Performance across CLIP ResNet-50, VIT-B/16, and VIT-L/14 on ImageNette. Image Woof. and CUB datasets with varying learning rates and sparsity parameters for object detection models.
Coufiguration
CLIP ResNet-5
VIT-MN16
CLIP VIEL/1
Learning Rate
Sparsity
ImageNette
ImageWoof
CUB
Image-Nette
ImageWoof
CUB
ImageNette
ImageWoof
CUB
GDINO Awa
le.4
le-4
98.49
91.27
58.44
99.46
94.94
73.63
99.82
95.67
$1.20
1e.3
le-4
96.29
89.31
46.58
99.64
93.35
68.23
99.71
95.75
76.34
le-2
81.65
99
lo-4
1o-3
99
77.74
1o-4
1o-2
98.44
89.28
48.45
09.44
93.13
63.41
99.77
94.31
71.63
le:3
le-3
98.33
89.31
41.04
99.39
93.08
61.18
99.77
95.50
70,31
Io-2
70.71
4.25
GDINO Awa Low
Ie-4
1e-4
98.55
91.22
57,97
99.47
94,02
73.63
99
95.72
$1.11
Ie-3
lo-2
lo-4
7.13
88.00
17.
1o-4
lo-1
98.60
91.27
55.04
99.44
93.74
69.45
99.92
95.52
77.65
1o-4
1o-2
98.42
88.95
48.55
99.56
92.722
63.339
99.89
95.04
71,59
1e.3
16-3
98.34
96.51
913.
Partimagenet
le-4
1e44
$1.52
1e-3
1o-4
98
77.
1o-2
1o44
97.04
81.75
5.92
99.08
89.25
14.59
99
93.20
20.78
1o-4
16-3
98.47
91.30
56.33
99.45
93.854
70.55
99
95.42
78.39
Ie-4
98.32
le-2
le-2
95.95
70.30
4.49
98.98
$0.99
5.53
13.39
Partimagonet
1e-4
16-4
1o-3
lo44
98
89.92
48.07
09.38
93.61
69. 16
99 80
95.67
77.
16-2
16-4
97:35
81.01
7.23
99.18
891.78
15.59)
99.64
93. 10
17.93
1e-4
98.77
le-3
le:3
98.27
89.92
42.03
99.41
93.33
61.70
77, 29
lo-2
le-2
96.
72.82
4.26
99.21
80.55
7.80
99 72
89.80
12.02
GDINO Pascal
le-4
le-4
98.42
91.09
58,80
99.61
93,26
73.71
99.82
95.75
30.77
le:3
le-4
98.01
89.67
46.70
99.18
93.03
68.47
99.67
95.78
76.39
1e-2
1e-4
le-d
le-2
98.37
48.58
99.31
93:00
99:77
71.33
lo-3
le-3
98: 14
89.03
43:47
99.21
92.85
60 54
99 69
95:44
69 69:42
1o-2
lo-2
96.64
69.41
4.38
98.60
$2,90
7.66
99 39
88.72
10.56
GDINO Pascal Low
le-l
le-4
98.58
90.91
58.25
99.46
9:1.20
73.78
99 85
95.75
$1.89
le-3
le-4
98:40
89 69
46.59
99.46
93.56
67.97
99
95.72
76.73
To-2
97.55
8.84
le-d
le:3
98.57
91.11
55.32
99.52
93.79
77.89
le-4
le-2
98:42
69
48.71
99.34
93.05
63.31
99 85
94
71.19
1o-3
1s-3
98.57
89.20
40.85
99.41
93.05
60.41
99.77
95.20
70.83
1o-2
98.42
GDINO Sun
le-4
le-4
98.09
K2.97
58. 66
99.18
62
73.66
99
93.63
$1.03
1o-3
1o-4
99
1o-2
1644
95.46
77.65
19.28
le-4
le-3
98.39
87.
55.42
99.24
90.58
70.26
99
94.38
77.72
1o-4
lo-2
98.24
57.00
49.31
99.24
90.26
63.20
99
93.74
70.64
1o-3
97.94
95.75
76.43
GDINO
1o4
1p-4
58.63
$1.33
16-3
1644
98
77.55
1e:2
1e-4
97.58
80.71
9.04
99.18
83.10
15.77
99.69
90.91
18.97
1e-4
Ie:3
98.57
91.17
55.85
99.64
93.67
70.42
99.79
95.55
77.84
1o-4
98.42
16-3
98.32
42.58
99.77
71.71
1e2
1e:2
96.33
69.71
4.26
98.96
77.50
8.37
99.67
$7.78
11.91
24

---

Data- Pata-Efficient Visual Concept Bottleneck Models
D.7. Number of images per class
DCBM does not require using all images dataset generate meaningful concepts. In this section, we evaluate the impact of varying the number of images per class (5, 10, 25, and 50) the resulting aceuracy using the standard configurations from the ablation study. The accuracy scores are presented in Table 19. Increasing the number of images per class most often leads to improved accuracy. However, the improvements become marginal beyond 25 images, with the difference between 25 and 50 being minimal. Consequently, we limited our investigation to 50 images per class to balance accuracy with computational efficiency, as a higher number of images requires more segmentation and object detection, significantly increasing computational costs with diminishing returns.
Table 19. Images per class. Performance comparison models using k-means clustering (1024 clusters) with varying numbers of images per class.
CLIP ResNet-50
VIT-B/16
CLIP VIT-L/14
Model
Images/Cls
ImageNette
ImageWoof
CUB
ImageNette
Image Woof
CUB
ImageNette
ImageWanf
CUB
SAM
5
98.47
91.14
58.78
99.52
93.94
80.36
99.80
95.52
80,26
98.73
91.27
57.82
99.54
93.94
74.13
99.80
95.50
80.03
25
98.57
91.07
57.53
99.54
94.077
73.35
99.85
95.50
79.82
50
98.50
90.81
57.71
99.72
93.74
73.14
99.87
95.67
79.73
SAM2
5
98.
99. 46
99.82
63.86
92.21
25
98.68
89 64
63.63
99.54
93.61
76.48
99.85
95.32
82.45
94.45
DETR
5
98.32
88.88
62.75
99.29
93.79
76.29
99.77
95.37
82.38
10
98.70
91.14
63.88
99.59
94.35
76.84
99.80
95.72
82,60
25
98.62
91.80
63.89
99.41
94.58
76.82
99.80
95.67
82,76
50
98.75
92.03
65.20
99.52
94.58
76.91
99.79
95.80
82.64
MastRCNN
65.93
95.78
10
98. 65
91.27
65.60
99.57
94,32
77.70
99.77
95.75
83.26
25
50
98.73
91.86
63.46
99.60
94,60
77.70
99.85
95.65
83.33
GDINO Awa
5
84.78
59.99
90.66
74,42
93.71
$1.69
10
98.34
88.88
59.51
99.28
93.00
74.58
99.75
95.29
81.57
25
98.42
90.56
58.84
99.29
93.82
74.47
99.82
95.55
81,26
50
98.50
91.17
57.97
99.46
94.04
73.63
99.89
95.60
81.12
GDINO Awa Low
5
98
88.90
59.65
99.36
93.10
74.44
99.80
95.39
81.86
25
98.57
90.99
$7.59
99.46
94.02
73.51
99.85
95.67
81.27
57.97
GDINO Parrinagenet
5
97.99
86.33
60.18
99.54
92.16
74.84
98.98
94,94
81.95
10
98.32
89.44
60.17
99.72
93.26
74.75
99.13
95.50
81,81
25
98.32
90.63
59.41
99,80
94.15
73.85
99.31
95.60
81.10
50
98.52
91.14
59.32
99.43
94.17
74.70
99.82
95.60
81,52
Parrinagenet
10
98. 44
90.58
59.82
99.36
93.92
74.44
90.80
95.83
81,46
25
99.37
99.80
50
98.52
90.99
$9.39
99.49
94,22
74,20
99.85
95.78
81,22
GDINO Pascal
5
85.42
$9.95
91.45
74.63
94.66
81.53
10
98.9 08.99
88.3 38.31
59.51
99.08
93.10
74,32
99.69
95.44
81.15
25
98
90.99
58. 39.39
99.39
93.97
74.01
99.82
95.55
80.82
50
98.57
90.91
58.90
99.41
93.17
73.73
99.87
95.67
$1.00
GDINO Pascal Low
5
98.17
89:03
59.23
99.24
92.98
74.18
99.72
95.44
81,27
98.3
90.68
99.36
93.59
74.56
99.80
95.70
81.31
25
98.57
90.99
58.63
99.52
94.10
73.77
99.80
95.75
81.15
50
98.62
58.25
99.52
93.89
73.78
95.75
81.15
GDINO Sun
5
25
50
98.09
58.66
99.18
88.62
73.66
93.63
GDINO Sun Low
5
97.99
83.15
60.03
99.21
89.23
74.72
99.67
93.54
10
98.32
87.48
60.01
99.34
92.42
74.89
99.72
95.22
81.62
25
98.50
90.23
59.48
99.41
93.55
74.40
99.80
95.60
81.36
50
98.60
90.94
39.00
99.47
94.46
74.54
99.80
95.88
81.51
25

---

Data-Efficient Visual Concept Bottleneck Models
E. Implementation details
Our implementation is in Python and Pytorch, and our CBM implementation based on (Rao et al., 2024; Yuksekgonul al., 2023). For GradCAM (Selvaraju et al., 2019) calculations, we use the implementation by (Zakka, 2021), which we adjust to ViT's and concept instead of text matching based on Gildenblat al. implementation (Gildenblat & contributors, 2021).
E.I. Efficiency and runtimes
The creation of concept proposals is influenced by the number of images per class and the foundation model used to generate outputs from these images. Consequently, the computational effort required for concept creation is highly dependent the input characteristics, similar to other approaches that involve training sparse autoencoder for concept generation (Rao et al., 2024), performing non-negative matrix factorization, or utilizing a large language model (Yang et al.. 2023). In the main paper, we provided rough estimations of the computation time required for concept creation within DCBM.
However, once the concepts are generated, we can directly compare the runtime per epoch between our framework and others on the same dataset. To illustrate this, we measured the runtime for a single epoch ImageNet using DCBM and DN-CBM, employing 90/10 train-validation split and adhering the standard hyperparameters specified in (Rao et al., 2024) and work for DCBM. Both methods were evaluated on the same machine using VIT ResNet50 as the backbone. Our measurements reveal that single epoch takes an average of 16 seconds for DN-CBM, while for DCBM, the average runtime per epoch is seconds (with SAM2). This significant reduction in runtime for DCBM is attributed to fewer concepts, which reduces the input size to the linear model compared to DN-CBM.
E.2. Concept size ablation
We remove the smallest and largest segments to ablate their effect on the CBM performance. To this end, we remove all concepts which are 1000 pixel or smaller (GT1000) and 1500 pixel or smaller(GT1500), Additionally, we remove all concepts that contain 200k or more pixels (LT200k).
Table 20, Test accuracy across datasets segmentation methods. Comparison of test accuracy different segmentation methods on CUB, ImageNet, and CVimateTV animals datasets under various ground truth (GT) conditions.
Dataset
Segmentation
Test Ace.
GT 1000
GT 1500
lt200k
CUB
SAM2
82.4
82.7
82.7
82.4
CUB
GDINO JotNOoparttiagenet
81.8
81.8
81.4
81.7
ImageNet
GDINO.partinageem
77.4
77.4
77.4
77.4
ImageNet
SAM2
77.9
77.7
77.9
77.9
SAM2
86.4
84.8
87.1
85.6
ClimateTV_animan
GDINO MattOppartimagenet
83.3
82.6
87.1
85.6
26

---

Cata-Efficient Visual Concept Bottleneck Models
F. Quantitative results
F.I CLIP backbones
We present the accuracies of additional CLIP backbones, ResNet-50 and VIT B/16, in Table 21. We used the same hyperparameters those initially selected for ViT L/14 for consistency. While ViT B/16 performs competitively, ResNet-50 lags behind other CBM techniques. We attribute this disparity ResNet-50's limited capacity capture complex semantic relationships compared transformer-based models, better equipped to handle the nuanced contextual understanding required for this task.
Table 21. Extended CBM benchmark. Performance comparison across different CLIP versions on datasets used for ablation study. Hyperparameters correspond to ones reported for VIT L14.
CLIP ResNet- 500
CLIP VIT-B/16
CLIP VIT-L/14
Model
Cir10
Cir100
IMN
CUB
Ci100
Lineur Prohe
73.3
$3.4
68.9
88.7*
70.3
80.2*
55.1*
81.0
96.2*
83,1
83.9
55.4
85.7
98.0"
87.5
Zero Shot
$9.6*
37.9
46.1
75.6*
41.6*
68.6*
39.5*
55.0
91.6*
68.7
75.3"
40.0
62.2
96.2*
77.9
LF-BM (Oikarinen et 2023)
72.0*
46.8
74.3*
86.4*
65.1*
75.4
48.2
74.0
94.7
77.4*
49.4
80.1
97.2
83.9
LaBo (Yang al., 2023)
87.9*
69.1*
78.9*
95.7"
81,2"
84.0"
97.8
86.0
CDM (Panousis et al,, 2023)
72.2*
52,7*
72.3*
86.5*
67.6*
79.3*
52.69
79.59
95.3*
80.5
83.4"
55.2*
95.9
82.2
DCLIP (Menon Vondrick, 2023)
59.6*
37.9*
49.0
68.0*
40.34
57.8*
75.0"
40.5*
63.5
DN-CBM (Rao al., 2024)
72.9
53.5
87.6*
67.5*
79.5*
96.0"
82.1
83.6
55.6
98.1
86.0
DCBM-SAM2 (Ours)
58.7
48.0
61.4
84.5
61.8
70.4
50.6
75.3
95.2
79.4
77.9
52.1
81.8
97.7
85.4
DCBM- FERM--DIOO (Ours)
58.7
47.8
59.0
83.9
61.2
69.7
50.7
74.
95.1
79.6
77.4
52.2
81.3
97.5
85.3
DCEM-MASKRCNN (Ours)
F.2. Additional datasets
AWA2 performance of DCBM is on par with the linear probe for all embeddings, as shown in Table 22.
ResNct-50
ViT-L/14 VIT-B/16
Zero shot
88.94
94.00
95.94
Linear probe DCBM
93.72
96.51
97.68
93.13
96.43
97.71
Table 22, DCBM performance on AwA2 using GDINO (w/ partimagenet labels) as concept proposal method
CelebA performance of DCBM is superior to other methods reported by (Xu et al., 2024) shown in Table 23, Given the label generation process, we believe that this is not the most suitable labeling scheme. We advocate for future research to devise more distinct, independent, and objective labels.
CBM Model
Acc
CBM
0.246
ProbCBM
0,299
PCBM
0.150
CEM
0.330
BCBM
0.343
DCBM w/ GDINO (ours)
0,354
DCBM w/ MaskRCNN (ours)
0.363
DCBM w/ SAM2 (ours) 0.356
Table 23. DCBM performance on CelebA using GDINO (w/ partimagenet labels) concept proposal method (CLIP ViT-L/14)
BotCL. We conducted experiments the first 200 classes of ImageNet, analogous to BotCL (Wang et al., 2023). DCBM achieved a test accuracy of 84.7% using CLIP ViT-L/14 and GroundingDINO (partimagenet). BotCL achieves 79.5% accuracy for this task. When comparing DCBM to BotCL, one has to bear in mind that the models have different backbones, limiting comparability.
27

---

Data-E Gata-Efficient Visual Concept Bottleneck Models
G. Qualitative results
In the following, we provide additional explanations in Figure 9, similar to those discussed in Section 4.3, focusing on the ablation datasets ImageWoof and ImageNette. These examples further highlight the effectiveness of DCBM on these specific datasets. Since both datasets consist of only classes each, the limited variety of concepts is likely result the reduced range of distinct concept proposals. This occurs because the datasets contain similar images representing closely related classes, leading to overlapping redundant concept representations.
ImageNette
Cordless
Ivtools
Logging
Saw
Saw
Concapts
ImageWoof
Retriever
Axle
Blu
Retriever
Labrador
Inputx
Top 5 Concopts
Class: Golden Retriever
Figure 9. DCBM ablations. Our technique DCBM was applied to Image Woof and ImageNette (SAM2 and VIT L/14),
G.1. DCBM concept comparison
SAM2
Faucet
Toilet
Floors
Closet
Towel
Concepts
GDINO
Inputx
Top & Cancopts
Mask-RCNM
Shower
Jacuzzi
Sink
Baths
Plaster
Imputx
Top & Concepts
Figure 10. Diverse concept proposals. Concept sets differ in terms of granularity depending the segmentation detection method employed, shown on shower (Places365).
The choice of the concept proposal model determines the retrieved concept set. The generic SAM2 model creates large number of concept proposals of image elements, while Mask-RCNN returns object-centric concepts. In contrast,
28

---

Data- Data-Efficient Visual Concept Bottleneck Models
promptable DIOO specifically steered detect common object parts. While the different concept only differ marginally in terms of accuracy, as shown in Table and Table 2, the interpretability is highly influenced. Here, examine on Places365 with CLIP VIT L/14 the top five image concepts in Figure 10, The figure displays interpretations for the same input image, correctly classified as shower by all three models, with each concept image accompanied by the closest matching textual label (g20k name space). The image consistently activates concepts that exhibit semantic alignment with the class shower or the larger context of the bathroom, where towels, faucets, sinks, and bath (tubs) can commonly be found. In particular, GroundingDINO (GDINO) and Mask-RCNN produce top concepts that closely match CLIP's textual embedding for the shower, even though both concepts appear quite different, as the shower, while GDINO's concept corresponds to image of door.
concept resembles
G.2. Concept intervention
DCBM allows for the removal of specific, undesired concepts before its training. This is achieved by leveraging CLIP's multimodal capabilities: given textual prompt, identify and exclude visual concepts that highly similar to the specified concept in the embedding space. For instance, in Figure 11 we remove exemplary concepts by computing the embedding of the word and discarding all visual concepts with high cosine similarity.
CUB
Rocks
Gull
Clicking
Knot
Blu
Inputx
Top 5 Concepts
Class Herring Guil
CUB
Migrant
Avian
Crop
Gull
Gull
Inputxx
Concepts after removel of
Class: Herring
CUB
Crop
Everquest
Long
Banner
Stylus
Ingut
Top Concepts after removal of 'quil"
Class: Herring Guil
Figure 11. Concept intervention in CUB. The first instance shows the top five concepts of a gull without concept intervention. The second one shows the same instance, with trained DCBM with the removal 'stone' concepts. The third one is DCBM with the removal of the concept guull".
The CBM trained with the included stone concept achieves 81.8% classification accuracy. After retraining model without the stone-related the accuracy remains unchanged and even improves by 0.4 with the removal of the gull-related concepts. For both examples, the explanations for the class gull no longer reference the concepts removed, demonstrating that successfully intervened in the model concept space.
29

---

Data-Efficient Visual Concept Bottleneck Models
G.3. Concept visualizations
We compare the concept-based explanations for examples from the Places365 orignal- used in (Rao al., 2024) in Figure 12 and Figure 13.
In Figure 12, two images are shown alongside explanations generated by CDM (Panousis et al., 2023), LF-CBM (Oikarinen et al., 2023), DNCBM (Rao et al., 2024), and our DCBM, which additionally includes visual explanations. For our method, we also present outputs for each of the CLIP backbones used in this study.
In the left example, labeled as "raft," all CBMs correctly classify the image, though the visual explanations from our DCBM vary depending on the backbone, offering different perspectives on the relevant concepts. In contrast, the right image, labeled "swimming hole," is misclassified by our model "creek" or "river." However, the visual concept attributions provide insight into this decision, revealing that the confusion can be attributed to the high semantic similarity between these classes. As stated before, DCBM concepts are visual, thus more attention should be paid to the visual concept than to its textual description. We have discussed this figure in the review phase and have described there the concepts proposed by DCBM with a VIT-B/16 backbone.
Thu Class rott
DNCBM
LF-CBM
CDM
True Class owimning hol
DNCBM
LF-CBM
CDM
4001
12:0
ACT
DCBM RNS0
DCBM RN50
oplic
Preciced ruft
parple
DCBM B/16
DCBM B/16
fand
TEXSS
irodidded Class: creek
DCBM L/14
DCBM L/14
Precidted Chooc mef)
Figure 12. Comparison of concept explanations across CBMs. This figure shows concept-based explanations for two images from the Places365 dataset, generated by four different CBMs: CDM (Panousis al., 2023), LF-CBM (Oikarinen al., 2023), DNCBM (Rao et al., 2024), and our proposed DCBM. The examples for CDM, LF-CBM, and DNCBM are adapted from (Rao et al., 2024).
30

---

Data-E Data-Efficient Visual Concept Bottleneck Models
In Figure 13, we again present examples originally shown in (Rao al., 2024), but this time the comparison is limited to DNCBM and our DCBM. The left image, laheled as "trench" is correctly classified by both models. Interestingly, while DCBM assigns different weights to visual concepts depending the CLIP backbone, the attributions remain meaningful for interpretability.
In contrast, the right image, labeled as "raceway," is misclassified by both DNCBM and most DCBM variants- for the ResNet-50 backbone, which correctly identifies the class. In particular, ResNet-50 exhibits the lowest overall classification accuracy in the Places365 data set, but is successful in this case. The incorrect predictions, such as "auto showroom" and "auto factory", are semantically similar "raceway". highlighting the inherent challenge of distinguishing such closely related categories. Once more, the visual concepts provide valuable insight into the semantic features each model emphasizes in its decision-making process.
The Goss zonch
DNCBM
roodway
DNCBM
Predicted Cheoe beech
Predicted Class: aido shoercom
DCBM RN50
DCBM RN50
Compoe
Predeted Class moceway
Predeted Class barch
DCBM B/16
DCBM B/16
1.t
hacttack
-12
Pradirded Closs trench
Predidte Caace wuto stowrtom
DCBM L/14
maliche
164
Prediceed Closs trenc
Phudided
Figure 1.3. Concept explanations from DCBM and DNCBM. This figure presents a comparison of concept explanations for two images from the Places365 dataset, illustrating outputs from DNCBM (Rao et al., 2024) and our DCBM. For each DCBM example, we show the concept attribution across the three CLIP backbone variants used our study.
31


Reviews:
[{"review_id": "VIs45fdIpz", "reviewer": "Reviewer_LrMk", "summary": "The paper introduces Data-Efficient Visual Concept Bottleneck Models (DCBMs), which generate interpretable visual concepts using segmentation and detection foundation models, enabling Concept Bottleneck Models (CBMs) to work effectively with limited data. By clustering image regions into concepts without relying on text descriptions, DCBMs achieve strong performance on fine-grained and out-of-distribution tasks while maintaining interpretability. The approach is simple, adaptable, and avoids extensive pre-training, offering a practical method for interpretable image classification.\nThe paper is very well written.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive feedback. We appreciate your acknowledgment of DCBM's strengths being simple, adaptable, and avoiding large-scale pre-training. Below, we outline our responses structured by the key points you raised:\n\n---\n\n## Performance on Large-Scale Datasets\n\nIn our opinion, while CLIP is powerful and designed to capture relationships between images and text, projecting concept centroids into this space might not perfectly align with the underlying semantics of the image data. Additionally, large-scale datasets like ImageNet and Places365 contain a high degree of intra-class variability, making it difficult for a concept-based model to generalize effectively. One could improve the discrepancy within large-scale datasets by adapting the centroids gradient-wise during the training phase of the linear layer. Here, the centroids would dynamically adapt to the underlying structure of the dataset. \n\n## Segmentation Challenges in Specialized Domains\nIt is one strength of DCBM that the segmentation model can be exchanged by a domain-specific one, e.g. MedSAM (Ma et al., 2024) for the medical field—offer. We believe that incorporating such domain-adapted segmentation models can help address challenges related to identifying relevant regions in specialized tasks.\n\nWe agree that combining textual domain-specific concepts with segmentation-based visual concepts may be especially beneficial in the medical domain. Some concepts are more effectively expressed visually, while others are better captured through language. \nIn DCBM, it was our primary objective was to develop a CBM that operates independently of textual inputs or LLMs for extracting concepts. We believe, that the combination of DCBM with existing methods like LaBo (Yang et al. 2023) and LF-CBM (Oikarinen et al. 2023), would achieve such a combination.\n\n## Extension to PCBMs and Backbone Flexibility\n\nDCBM cannot be extended as a post-hoc CBM - as neither of the other ante-hoc CBMs. We have chosen this approach as it allows to better understand the model embedding space. This said we believe that DCBM is a valuable framework to better understand any vision embedding space. We utilize text-image aligned backbones (CLIP) for benchmarking against other CBM approaches. This is not inherent to our framework and DCBM can be run using any vision embedding, with a slightly more intricate mapping to the text labels. \n\nWe appreciate the recommendation of Moayeri et al. (ICML 2023) and agree that projecting embeddings from a blackbox into the VLM embedding space is a compelling strategy. We find this approach both intriguing and valuable and are investigating the opportunities of combining post-hoc and ante-hoc CBMs.\n\n## VLM Biases and Spurious Correlations\n\nThe CLIP space is known to contain spurious correlations (Rao et al., 2024 & Oikarinen et al., 2023 & Panousis et al., 2023). By including all segments as concept proposals, DCBMs visualize spurious correlations. As shown in Figure 3, DCBM learns that *rock* is an important concept for predicting the bird class. This further becomes apparent, when we set the weights of *rock* to zero, the model achieves a confidence of 62.49% (-12%). We are currently training a model with interventions.\n\n*Question: We exclude concepts from training, which have been identified as spurious. Alternatively, we were thinking of masking out concept regions in the image and the measure the model's confidence. Is this what you had in mind? We would love to hear more feedback on this from you.*\n\nIn the final version of this paper, we further investigate the behavior of spurious correlations and actively remove such concepts to observe changes in accuracy.\n\n## Systematic Identification of Spurious Concepts\n\nThis is possible by using either slot attention as in BotCL (Wang et al., 2023) or an image tagging model such as RAM (Zhang et al., 2024) with a threshold. We will include this extension in the discussion section.\n\n---\n\nWe appreciate your insightful feedback and hope our responses have effectively addressed your questions. During the remainder of the rebuttal phase, we welcome further discussion on any open issues and invite additional feedback on our proposed experiments.", "claims_and_evidence": "The paper demonstrates that DCBMs maintain classification accuracy within a small margin (roughly 5–6%) of a CLIP linear probe on CIFAR-10, CIFAR-100, ImageNet, Places365, and CUB. The performance of DCBM is subpar with the other methods for large scale datasets like imagenet and places 365. The authors did not discuss the reason for that? why it is so? is it because of the projection of the centroids to the clip space?", "methods_and_evaluation_criteria": "1. Segmenting or detecting specific image regions can be problematic for medical images. For example, for chest x-rays, often the segmentation models segment the right and left lung and heart. So they ignore the anatomical concepts like the lower left lobe or devices like the chesttube. I think DCBM will also have same problem. Is there any way to solve it? Can the best of both worlds (concepts from LLMs or reports or captions and segmentation models together) solve it?\n\n2. This method is for CBM but not for PCBM. For ex, if I want to extract a CBM from any arbitary blackbox (resnet), this method won't work, because of the reliance on aligned text and vision  encoders. Can this method be extended to PCBMs as well? I believe they can project the embedding from the blackbox to the VLM embedding space and still use their method to extract CBM from a blackbox. See this paper for projection:\nText-To-Concept (and Back) via Cross-Model Alignment. Moayeri et al. ICML 2023.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "1. The authors should do a human evaluation to show these discovered concepts truly meaningful to humans. However, the localization results do a decent job as an automated check.", "relation_to_broader_scientific_literature": "1. The paper cites recent developments in segment-anything approaches (SAM, GroundingDINO) and how these can serve as universal “concept proposal” engines.\n2. This strategy extends the prior “visual concept” line (e.g., DN-CBM) but is more data-efficient and requires no large pre-training corpora.", "essential_references_not_discussed": "[1] Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off. Barberio et el. Neurips 2022 for non linear relationship among concepts\n\n[2] Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat. Ghosh et al. ICML 2023 for expert based PCBM and First order logics for concept interactions\n\n[3] Distilling BlackBox to Interpretable models for Efficient Transfer Learning. Ghosh et al. MICCAI 2023. applying CBM to chest-x-rays.", "other_strengths_and_weaknesses": "1. Foundation models might fail or produce random proposals in certain specialized domains (Eg, breast cancer detetction) if no relevant segmentation or detection model is available.\n\n2. The authors use the pretrained frozen VLMs like CLIP. This can be problematic because CLIP inherits many biases/shortcuts/spurious correlations that can influence the decision. For example, Figure 3 shows rock is an important concept for predicting the bird class. Now rock is not a causal feature for bird prediction. I believe this is due to the internal biases of CLIP. Also, this shows the problem of using segmentation regions from models like SAM. As in Figure 3 (top), the image of CUB, look at the concepts identified. One of the concepts is entire bird - Gull. Is this useful in practice? This is what I pointed out in #1 in \"Methods And Evaluation Criteria\". if the explainer says the entire lung is an important concept, this method wont be useful in the real world. Also, the same image in Fig 3 detects \"Clicking\" and this concept is not visual. Ideally, the concepts should be some features of the birds that will be useful for classification.", "other_comments_or_suggestions": "If the authors want further clarity about spurious correlations, they might incorporate a dedicated “concept removal” experiment (filtering out suspicious or intangible concepts) to see if accuracy or interpretability improves.", "questions_for_authors": "1. Sometimes the top concept is semantically related to the class but not visibly present in the image (e.g., “police kit” for an ambulance). Could we systematically identify these spurious concepts?\n2. You rely on Grad-CAM to show that concepts align to image regions. Have you considered a direct concept-intervention test (removing or altering concept crops) to see how predictions change? That might further confirm concept “faithfulness.” This is due to the fact GRAD-CAM has its own problems discussed in this paper: Sanity Checks for Saliency Maps. Neurips 2018.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehl0ydhp1J", "reviewer": "Reviewer_q1H5", "summary": "The paper proposes a novel framework to enhance the practicality of concept bottleneck models (CBMs) by reducing their reliance on extensive labeled concept data. DCBM decouples concept learning from task adaptation through self-supervised pretraining (e.g., using vision-language models like CLIP) to autonomously extract semantic concepts and sparse dynamic masking to selectively activate task-relevant concepts during fine-tuning. This approach achieves competitive accuracy on benchmarks (CUB, ImageNet) with 10× fewer concept labels compared to traditional CBMs while retaining interpretability, enabling human-in-the-loop concept refinement and efficient deployment in low-resource settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate your recognition of our framework’s originality and data efficiency—requiring 10x fewer concept labels than comparable CBM approaches while achieving similar performance. Below, we provide detailed responses organized by the key points raised:\n\n---\n\n## User Studies and Quantitative Metrics for Domain Utility\n\nWe agree that demonstrating practical utility for domain experts is crucial. While our primary focus has been on establishing the theoretical feasibility and data efficiency of our approach, we recognize the value of user studies and quantitative metrics for concept intervention. In the final version, we will include an additional investigation that actively manipulates relevant concepts to observe changes in the model’s predictions and confidence. For instance, as shown in the bird example (concept “rock”), our preliminary analysis indicates that concept removal has a predictable effect on predictions. We plan to extend this analysis by adding concept interventions to further validate interpretability. \nIn addition, we recognize that the risk of \"explanation illusions\" is an important concern in CBM approaches, particularly when using CLIP as a backbone. To mitigate this, we plan to incorporate tests of concept removal and alteration to explicitly address concept grounding. \n\n\n## Analysis of the Sparsity Mechanism and Concept Coverage\n\nAnalog to concurrent work, we ablate the sparsity parameter $\\lambda$ based on model accuracy (Rao et al., 2024 & Oikarinen et al., 2023)\n\n\n## Concept-Level Validation and Baseline Comparisons\n\nWe acknowledge that thorough concept-level validation is essential to reinforce the interpretability of our method. Our study has systematically analyzed the learned concepts across multiple large-scale datasets (ImageNet, Places, and CUB). \n\nWe deliberately compare only methods that offer interpretability, which is a core property of CBMs. By focusing on interpretable approaches, we ensure that our evaluation remains consistent and that our performance improvements are attributable to our architectural innovations rather than differences in method transparency. This is why we do not evaluate against prompt-tuned CLIP. We will include this distinction in the literature review. \n\nHowever, as demonstrated in `[fCuP]`, we incorporate additional datasets and utilize new, randomly selected images for the concept proposal generation phase. Our results consistently show that our technique delivers comparable performance across diverse experimental settings.\n\n## Broader Applicability Beyond Classification\n\nRegarding the current focus on classification tasks, we appreciate the suggestion to explore applications in regression and other settings. However, the investigation of the suitability of CBMs in general is an additional task, which takes more consideration than would be adequate for the purpose of this paper. Therefore, we will include this discussion at the end of our paper to open up new research fields of CBMS for regression and other settings.\n\n---\n\nThank you again for your valuable feedback. We believe these planned additions and clarifications will further strengthen the work, and we look forward to incorporating your suggestions to improve both the interpretability and practical utility of our approach.", "claims_and_evidence": "While DCBM allows concept editing, the paper lacks user studies or quantitative metrics (e.g., concept intervention success rates) to demonstrate practical utility for domain experts. Claims about interpretability remain anecdotal without empirical validation of human-AI collaboration.\n\nThe sparsity mechanism’s effectiveness is asserted via accuracy metrics but lacks analysis of concept coverage (e.g., whether critical concepts are retained or pruned). Without grounding in domain knowledge (e.g., alignment with known semantic attributes), the claim risks conflating sparsity with arbitrary feature selection.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in DCBM largely align with the goals of data-efficient and interpretable concept learning", "theoretical_claims": "The paper does not present formal theoretical claims or proofs. Its claims are empirically validated through experiments, with no explicit theoretical analysis.", "experimental_designs_or_analyses": "While DCBM’s experiments demonstrate label efficiency and task accuracy, the lack of concept-level validation and incomplete baseline comparisons weaken its claims about interpretability and generalizability. The design is sound for initial proof-of-concept but insufficient for asserting real-world applicability.", "relation_to_broader_scientific_literature": "DCBM innovatively synthesizes self-supervised learning, sparsity, and distillation to modernize CBMs, positioning itself as a critical response to the dual challenges of interpretability and data scarcity. Its contributions resonate with broader ML trends but highlight the need for deeper integration with non-CBM efficiency paradigms.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\nDCBM demonstrates originality by creatively integrating self-supervised vision-language models (e.g., CLIP) with concept bottleneck architectures, effectively decoupling concept discovery from task-specific tuning. This reduces reliance on manual concept annotations—a major bottleneck in traditional CBMs—while preserving interpretability. The framework’s significance lies in bridging data efficiency and explainability, making CBMs viable for real-world applications like medical imaging or ecological monitoring where labeled data is scarce. The design is clear, with modular components (pretraining, masking, distillation) that are empirically validated on standard benchmarks.\n\nWeaknesses:\nWhile innovative, DCBM’s concept grounding remains weakly validated; concepts derived from CLIP lack rigorous alignment with domain-specific semantics (e.g., bird parts in CUB), risking \"explanation illusions.\" Additionally, the paper’s focus on classification tasks limits its demonstrated utility for regression or causal reasoning, which are critical for high-stakes domains. Comparisons to non-CBM data-efficient methods (e.g., prompt-tuned CLIP) are missing, leaving open whether the gains stem from architectural novelty or pretraining advantages.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SooG29X3GR", "reviewer": "Reviewer_AELC", "summary": "The paper proposes a Data-efficient CBM (DCBM) that enhances interpretability while reducing the reliance on large datasets. Specifically, DCBM defines concepts as image regions detected through segmentation and object detection foundation models, rather than relying on textual descriptions. This allows DCBM to generate multiple concepts at various levels of granularity depending on different foundation models. The authors validate their approach using attribution analysis with Grad-CAM, demonstrating that DCBM produces interpretable, localized visual concepts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. We appreciate your positive feedback on our concept extraction pipeline that leverages foundation models, and we’re pleased that you recognize the interpretability and localization capabilities of our DCBM.\n\n## Data efficiency in real-world setting\nDCBM is designed for real-world data scarcity, relying on only 50 samples per class during the concept proposal generation phase. Notably, our experiments also show robust performance with even fewer samples in Appendix D.7.\n\nThis can be taken one step further by reducing the number of training samples to the same subset. For ImageNet, we reduce the number of training samples from originally 1,281,167 images to 50,000 images **(-96\\%), and the performance degrades from 77.4% to 75.0%.**\nFor Cifar10, the reduction in training images from \n50000 to 500 **(-99\\%) reduces the performance from 97.5 to 93.1.** The experiments were run using GDINO (partimagenet) and CLIP ViT-L/14. We will include this experiment for all datasets in the paper, thank you for the suggestion.\n\nThe focus of our work is data efficiency with comparable performance levels. We provide the results of another, non-overlapping 50 images subset (RQ2) and report the accuracy when training the CBM with the segments of 100 images per class, combining the two subsets (RQ1).\nFor CUB, the training set consists of less than 50 images per class, thus we already include all images in the concept proposal generation.\n\nTable 1: Performance evaluation of subset selection\n||IMN|Places|Cif10|Cif100|\n|-|-|-|-|-|\n| s1 (main paper) | 77.4 | 52.2 |97.5|85.3 |\n| s2 (new)| 77.5| 52.2|97.6|85.4 |\n| s1+s2 | 77.1| 52.1 |97.7 |85.5|\n||||||\n\nIn Table 1, s1 corresponds to the original set of 50 images per class, s2 represents an additional set of 50 randomly selected images per class, and s1+s2 denotes the combined dataset of 100 images per class.\n\nThe performance is stable between all subsets. We believe that the performance difference to DN-CBM is domain-dependent. Given their vast number of pre-training images (3.3M), general domains are well covered (ImageNet/Places365) whereas specialized domains benefit from the dataset-specific DCBM (CUB). \n\n## Comparison to DN-CBM\nWe agree that the CBM training for DN-CBM and DCBM are quite similar.\nHowever, the required steps prior to the CBM training differ significantly. While DN-CBM trains a SAE to retrieve concepts using 3.3M images in CC3m, we create the segments by applying foundation models to a subset of the training images. We would like to highlight, that for DN-CBM, the download of an additional 3.3M images is needed, whereas, for DCBM, we only require the dataset to be analyzed. This reduces the storage requirements significantly and is highly time efficient. \n\nTable 2: CBM training preparation: DN-CBM vs DCBM\n| | DN-CBM | DCBM - ImageNet |\n|-|-|-|\n|Dataset size| 3,300k image-caption pairs (CC3M) | 50k images (50/class)| \n| Add. memory capcity| 850 GB (assuming 256x256px) | 6 GB |\n| No extra data required |  x |&#x2713; |\n||||\n\n## Generalization capabilities\nWe agree with your criticism of our reporting of DCBM's generalization capabilities. \nWe would like to point you to Table 19 in the supplementary material where we can report a 22-27\\% error rate difference between ImageNet and ImageNet-R for CLIP-ViT/L14. For this embedding model, we achieve error rates of below 50\\% when evaluating on ImageNet-R.\nDue to resource sparsity, only the results for CLIP-RN50 were ready at the time of submission. *We are currently generating the accuracy for DN-CBM and will provide them asap.*\n\n## Misc\nThank you as well for pointing out that some typos and ambiguities exist in the paper - we have fixed them. We believe that your feedback has helped us to improve our paper and would like to thank you for taking the time and sharing your expertise.", "claims_and_evidence": "The primary claims are following: (1) DCBM can handle data-scarce environment and be easily adapted to new datasets, and (2) DCBM bridges the gap between vision and text modalities in concept extraction by generating visual concepts. To validate the claim, the authors demonstrate that they use the subset of training dataset to extract concepts. However, they still utilize the entire training dataset during training CBM. This raises concerns about whether it can truly be considered data-efficient in real-world scenarios. Also, this paper only reduces 7 seconds per epoch compared to DN-CBM in cost of performance. Furthermore, the claim that DCBM achieves better OOD generalization is also difficult to accept. While the performance gap between in-distribution (IN-200) and out-of-distribution datasets (IN-R) is smaller compared to DN-CBM, DCBM performs worse in both settings, making the smaller gap a misleading indicator of generalization.", "methods_and_evaluation_criteria": "The proposed method to use visual concepts to avoid the modality gap in concept construction is novel, however, the improvement is marginal (or even degraded) compared to the baselines. Also, as mentioned in \"Claims And Evidence,\" I belive the evaluation criteria of \"generalization\" is misleading.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "(1) Does increasing the number of training images used for concept extraction lead to performance improvements? If all training images were used for concept extraction, would DCBM outperform DN-CBM?\n\n(2) Since the training images for concept extraction are selected randomly, is there any variance in performance due to this randomness?", "relation_to_broader_scientific_literature": "The key contribution of this paper is an advanced CBM especially concentrated on concept extraction. Specifically, this paper makes a contribution to generate the visual concepts for CBM.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "A major strength of the paper is that the author explores how concepts can be shaped differently depending on the choice of the foundation model (SAM, Grounding DINO, Mask-RCNN), leading to generate concepts at different levels of granularity.", "other_comments_or_suggestions": "There are some typos or ambiguous sentences throughout the paper, so a thorough proofreading and revision would be beneficial. \nFor instance,\n- Citation format (line 100): Language-guided CBM (LaBo) employs GPT-3 for concept creation, but it stands out by using a submodular function to select concepts from candidate sets, building the bottleneck, and training a linear model based on CLIP embeddings **Yang et al. (2023).**\n- Ambiguous sentence (line 243): To this end, we evaluate both visual CBM models, ours and DN-CBM, trained **on ImageNet on ImageNet-R* *(Hendrycks et al., 2021) which contains 200 ImageNet classes in various renditions (e.g.embroidery, painting, comic).", "questions_for_authors": "Please refer to the \"Experimental Designs Or Analyses\" part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2P2DJ9l0t", "reviewer": "Reviewer_fCuP", "summary": "This paper proposed a data-efficient Concept Bottleneck Model (DCBM) that enables concept generation while maintaining interpretability with minimal training samples. DCBM defines concepts as image regions using segmentation and object detection models, eliminating the reliance on textual descriptions or large-scale pretraining datasets. It offers high flexibility and interpretability for fine-grained classification and domain adaptation tasks. The paper evaluates DCBM on various benchmark datasets, demonstrating competitive performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for recognizing the flexible and interpretable design of DCBM. We highly appreciate your helpful comments and hope to provide the missing details in our answers below.\n\n## Efficiency\nWe derive concept proposals based on 50 samples per class and train the CBM on all training samples. However, as we show in response to Review 3 (AELC), DCBM's performance does not deteriorate when we train with only 50 images per class. In contrast to BotCL (Wang et al., 2023), we do not fix the number of concepts, but the number of images that we use for concept extraction. This design choice avoids BotCL's weakness (Wang et al., 2023) of having to tune the number of concepts for each dataset.\n\nWhen comparing DCBM to BotCL, we quantitatively outperform them on CUB (8.4\\%). Our results for ImageNet are not comparable, as they evaluate only 200 out of 1000 classes and report that their method fails for a large number of classes. Given that BotCL's authors do not share their training recipe, we cannot evaluate overall efficiency. \n\n**Question**: Would you like us to report an evaluation of DCBM on the first 200 classes of ImageNet? \n\n## Ablation on the number of concept samples\n\nTable 1: Ablation for DCBM w/ GDINO Partimagenet on CUB (CLIP ViT-L/14) \n\n|Number of Clusters | Accuracy |\n|-|-|\n|128| 75.20|\n|256| 79.03|\n|512| 80.43|\n|1024| 81.51|\n|2048|81.91|\n|||\n\nIn section D.5 we ablate the number of concept samples, which have - in parts - copied here for your convenience. We will add a reference to Appendix D.5 in the paper's section 3.2, where we describe the concept generation process.\n\n## Concept validity\nWe validate our concepts empirically, by calculating the energy pointing game (GridPG) (Bohle et al.,\n2021). Our motivation for this choice is in line with BotCL (Wang et al., 2023), we want to verify whether detected concepts can be traced back to the image region. \nIn DCBM, the concept-image alignment is verified as part of the evaluation. We chose an automated evaluation over a human analysis.\n\n## Theoretical explanation of segmentation\nOur understanding is the following: An object $O$ can be represented as a combination of concepts from a global concept pool $\\mathcal{C}$. Let $C_i \\in \\mathcal{C}$ denote individual concepts, and let each object be characterized by a subset of these concepts with corresponding weights.\n$$\nO = \\sum_{i=1}^{|\\mathcal{C}|} w_i C_i, \\quad w_i \\geq 0\n$$\nwhere:  \n- $C_i \\in \\mathcal{C}$ represents a concept from the global pool,  \n- $w_i$ denotes the contribution of concept $C_i$ to the image,  \n- $C_i$ is located in the image $I$, $R_i \\subseteq I$ is the region of the image where concept $C_i$ is present,  \n\nFurther, we assume visual concepts to be spatially localized in images and therefore conjecture that data efficient concept extraction should build upon image segments or regions rather than entire images. Therefore, we generate the global concept pool $\\mathcal{C}$ by segmentation or detection foundation models, which are then cropped out and used as concept proposals $s_i$. All $s_i \\in \\mathcal{S}$ are then clustered into the global concept set $\\mathcal{C}$.\n\nIn A. Algorithms (appendix) we provide a further theoretical explanation of this process along with pseudocode. We will update this section following your feedback. *Would you like us to include any additional details?* Thank you for carefully reviewing our methods section - we have revised the paper and unified functions and symbols.\n\n\n## Additional datasets\nDCBMs stand out by applying to any domain in a data-efficient manner. We show on 7 diverse datasets, that DCBMs can be applied to animals (CUB, ImageNet), scenes (Places365), social media (ClimateTV), low-resolution images (cifar10 & cifar100) along with ood generalization (ImageNet-R) and state changes (MiT-States). Additionally, we show for 2 novel datasets in the rebuttal, that DCBMs performance is independent of the domain it is trained on. \nThis exceeds the number of datasets BotCL evaluates, i.e. 4. Our main evaluation contains the same datasets as employed for Vlg-CBM (Srivastava et al., 2024) and the same number of datasets as Res-CBM, i.e. 7 datasets. We include these models in our related work and compare against them where possible.\n\nWe thank you for suggesting additional experiments on AwA2 and CelebA. \nWe have run DCBM on AwA using the standard 50:50 train and test split. We created the val set by randomly selecting 10% of train samples. \n\nTable 1: DCBM performance on AwA2 using GDINO (w/ partimagenet labels) as concept proposal method.\n| | ResNet-50 | ViT-B/16 |ViT-L/14 |\n|-|-|-|-|\n| Zero shot | 88.94| 94.00 | 95.94|\n| Linear probe| 93.72| 96.51| 97.68|\n| DCBM | 93.13| 96.43| 97.71|\n|||||\n\n*For CelebA, the experiments are currently running.*\n\n## Misc\nThank you for your interest in seeing more concept visualizations. We have included more examples in the supplemental material.", "claims_and_evidence": "- DCBM is claimed that CBM can be trained using no more than 50 samples per class, making it more data-efficient than existing CBMs.\n- Related research, BotCL [1] also conducted tests using 50 concepts. Therefore, it is necessary to experimentally prove that the proposed DCBM outperforms BotCL and other models in terms of both performance and efficiency.\n- Additionally, an ablation study on the number of concept samples is required.\n\n[1] Wang, Bowen, et al. \"Learning bottleneck concepts in image classification.\" Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023.", "methods_and_evaluation_criteria": "- DCBM is more data-efficient than conventional CBMs and automatically generates concepts using segmentation and detection models. However, it lacks experiments verifying the semantic validity of the generated concepts even though authors presented various experiments in body and suppl.\n- While interpretability analysis using Grad-CAM has been conducted, there is a lack of comparative experiments evaluating the interpretability of concepts against existing CBMs.", "theoretical_claims": "- There is little mathematical proof available for evaluation.\n- A theoretical explanation is needed to better understand how segmentation is utilized in DCBM.", "experimental_designs_or_analyses": "- The experiments did not utilize commonly used datasets in CBM models. Therefore, additional experiments using widely adopted datasets such as AwA2 and CelebA are necessary.\n- In the ablation study, an analysis of performance differences based on the number of concepts is required, along with experiments evaluating the impact of hyperparameter tuning on performance.\n- The analysis of the key characteristics highlighted in Figure 4 is ambiguous, requiring comparative analysis and additional explanation.\n- More experimental results on the visualization of segmented concept parts should be provided.", "relation_to_broader_scientific_literature": "- A key distinction from previous studies is that this paper proposes a CBM that automatically extracts concepts using Segmentation and Detection models, enabling interpretability without relying on text.\n - The paper compares the proposed model with label-free CBMs such as LaBo [2], but it does not include other relevant studies despite their significance.\n - Therefore, additional comparative experiments and discussions on existing research should be incorporated.", "essential_references_not_discussed": "Wang, Bowen, et al. \"Learning bottleneck concepts in image classification.\" Proceedings of the ieee/cvf conference on computer vision and pattern recognition. 2023.\n\nShang, Chenming, et al. \"Incremental residual concept bottleneck models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nSrivastava, Divyansh, Ge Yan, and Lily Weng. \"Vlg-cbm: Training concept bottleneck models with vision-language guidance.\" Advances in Neural Information Processing Systems 37 (2024): 79057-79094.", "other_strengths_and_weaknesses": "<Strengths>\n\n - Data Efficiency: Maintains performance comparable to existing CBMs with only 50 samples per class.\n - Automated Concept Generation: Utilizes segmentation and detection models to automatically extract concepts, improving domain adaptability.\n - Generalization Capability: Demonstrates robust performance compared to existing CBMs in OOD evaluation using ImageNet-R.\n - Enhanced Interpretability: Uses Grad-CAM for visual concept activation analysis, providing insight into the model’s decision-making process.\n\n<Weaknesses>\n\n - Lack of Semantic Validity Verification: No experiments verifying whether automatically generated concepts are truly meaningful.\n - Insufficient Comparison with Recent CBM Models in Terms of Data Efficiency and Performance: Lacks quantitative comparisons to demonstrate how much more efficient DCBM is compared to existing CBMs.\n - Limited OOD Experiments: Needs generalization evaluation across diverse domains (e.g., medical, industrial) beyond ImageNet-R.\n - Lack of Hyperparameter Tuning Analysis: Requires ablation studies on the effects of concept quantity, clustering methods, and model choices.", "other_comments_or_suggestions": "- It is recommended to unify functions and symbols so that Figure 2 aligns with the main text explanation.\n- A detailed explanation is needed on the segmentation methods mentioned in the introduction and how segmentation was used for concept generation.\n- Using the same abbreviation for different terms can cause confusion. For example, out-of-distribution (OOD) and out-of-domain (OOD).\n- The term \"qblation\" on line 1265 should likely be corrected to \"ablation.\"", "questions_for_authors": "- Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces DCBM (Data-Efficient Concept Bottleneck Models), a modular framework that generates visual concepts using segmentation and detection foundation models (e.g., SAM, Grounding DINO), enabling concept bottleneck models to be trained with significantly fewer concept labels. Unlike traditional CBMs that rely on text-annotated concepts or extensive pre-training, DCBM uses image regions as concepts, offering a scalable and interpretable alternative (e.g. for vision tasks with limited annotations). The method is validated across seven datasets including (CUB, ImageNet, Places365, among others), and evaluated along both predictive accuracy and interpretability dimensions.

**Strengths:**
* The paper tackles a clear and practical limitation of CBMs—their high dependency on labeled concept data—and proposes a solution that combines modern vision foundation models with minimal supervision.
* DCBM is simple, modular, and broadly applicable, relying only on pre-trained segmentation/detection backbones and CLIP-like embeddings.
* The approach seems flexible across domains, and concept quality is evaluated through automated metrics (e.g., GradCAM, GridPG) and interventions.
* The authors engage thoroughly in the discussion phase, conducting new experiments on ImageNet-200 and CelebA, visualizing comparative concept explanations, and demonstrating the ability to suppress spurious concepts without loss in accuracy.
* The codebase has been extended to support targeted concept removal, supporting causal reasoning and real-world debugging use cases.

**Reviewer discussions**:
* Reviewer LrMk (Weak Accept) offered thoughtful feedback on practical issues (spurious correlations, domain adaptation, and PCBM integration). They appreciate the paper’s modularity and raised their score after detailed engagement from the authors.
* Reviewer q1H5 (Weak Accept) recognized the originality and modularity of the design but believes that interpretability claims remain partly anecdotal. The reviewer supports the paper overall after the rebuttal.
* Reviewer AELC (Weak Accept) was initially skeptical about generalization claims but was convinced by post-rebuttal results showing better OOD performance on ImageNet-R. Their final rating reflects this.
* Reviewer fCuP (Weak Accept) requested additional evaluation on recent CBMs (e.g., BotCL, ECBM) and more intuitive visualizations. The authors provided both and added new results for AwA2 and CelebA, leading the reviewer to raise their score.

**Limitations:**
* Interpretability claims are largely supported via automated analysis (e.g., via GradCAM and GridPG), but lack human evaluation or concept-level alignment with domain knowledge.
* While the paper argues DCBM is “data efficient,” it still uses the full training set for CBM training. The efficiency pertains primarily to concept extraction, which could/should be better clarified.
* Comparisons are focused on interpretable CBMs; comparisons to non-CBM data-efficient alternatives (e.g., prompt-tuned CLIP) are not included, though reasonably justified by the authors.
* DCBM is not directly compatible with post-hoc CBMs (e.g., PCBM), though the authors openly discuss this and explore integration possibilities.

**Conclusion**
This paper offers a well-rounded contribution to interpretable machine learning. It presents a simple, reproducible framework that bridges modern segmentation/detection tools with CBMs, demonstrating improved data efficiency and broader applicability. The authors were highly responsive to feedback, and the final set of experiments and clarifications resolve most of the initial concerns. Assuming the revisions outlined in the rebuttal are incorporated, I am supportive of accepting this paper.
Decision: Accept (poster)