VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Authors: Divyansh Srivastava
Ge Yan
Tsui-Wei Weng
Venue: NeurIPS 2024 poster (2024)
Keywords: Interpretable machine learning
concept bottleneck models
Primary area: interpretability_and_explainability

Abstract: Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes human-understandable concepts to explain models' decision. Recent works proposed to utilize Large Language Models and pre-trained Vision-Language Models to automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, it has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to state-of-the-art CBMs. To address these critical limitations, in this work, we propose a novel framework called Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelf open-domain grounded object detectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) to control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM, outperforms existing methods by at least 4.27\% and up to 51.09\% on *Accuracy at NEC=5* (denoted as ANEC-5), and by at least 0.45\% and up to 29.78\% on *average accuracy* (denoted as ANEC-avg), while preserving both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiments.

VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Divyansh Srivastava, Ge Yan', Tsui-Wei Weng [ddivyansh, geyan, Iweng) @ ucsddeddu UC San Diego
Abstract
Concept Bottleneck Models (CBMs) provide interpretable prediction by intro- ducing an intermediate Concept Bottleneck Layer (CBL), which encodes human- understandable concepts to explain models' decision. Recent works proposed utilize Large Language Models and pre-trained Vision-Language Models automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to hate-of-theearr CBMs. address these critical limitations, this work, we propose novel framework called Vision Lantoon-angggee---C Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelff open-domain grounded object de- tectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM, outperforms existing methods by least 4.27% and up to 51.09% at NEC=5 (denoted as ANEC-5), and by at least 0.45% % and up to 29.78% on average accuracy (denoted as ANEC-avg), while preserving both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiment
Accuracy
Introduction
As deep neural networks become popular real-world applications, is crucial to understand the decision of these black-box models. One approach provide interpretable decisions is the Concept Bottleneck Model (CBM) (6], which introduced an intermediate concept layer to encode human- understandable concepts. The model makes final predictions based these concepts. Unfortunately, one major limitation of this approach is that it requires concept annotations from human experts, making it expensive and less applicable in practice concept labels may always be available.
Recently, a line works utilized the powerful Vision-I Language Models (VLMs) to replace manual annotation [15] 27 (25). They used Large Language Models (LLMs) to generate set of concepts, and then trained the models in a post-hoc manner under the guidance of VLMs or neuron-level interpretability tool [14. By eliminating the expensive manual annotations, some of these CBMs [3] could be scaled to large datasets such as ImageNet [8]. However, these CBMs 15 25 still face two critical challenges:
Equual contribution
"Our code is available at http:///tth.cccm///wwwtttMttaaat
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1. Challenge #1: Inaccurate concept prediction. The concept predictions in these CBMs often contain factual errors i.e. the predicted concepts do not match the image. Moreover, as concepts are generated by L.Ms,, there are some non-visual concepts, for example "loud music" or "location" used in L.-CBM 15 which further hurt the faithfulness of concept prediction.
2. Challenge #2: Information leakage. Recently, [3 [2]
observed the information leakage in CBMs through empirical experiments they found that the concept prediction encodes unintended information for downstream tasks, even if the concepts are irrelevant to the task.
In this paper, we propose new framework called Vision-Lanna Larton-LangaggeGGii Guided Concept Bottleneck Model (VLG-CBM) to address these two major challenges. Our contributions summarized below:
1. To address Challenge #1, we propose to use the open-domain grounded object detection model to generate localized, visually recognizable concept annotations in Section 3 This approach automatically filters the non-visual concepts. Furthermore, the location information is utilized to augment the data. As far as we know, our VLG-CBM is the first end-to-end pipeline to build CBM with vision guidance from open-vocabulary object detectors.
2. To address Challenge #2, we provide the first rigorous theoretical analysis which proves that CBMs have serious issues on information leakage in Section 4. whereas previous study on information leakage 22 25] only provides empirical explanations. Building on our theory, we further propose
metric called the Number of Effective Concepts (NEC) in Section 4.2) which facilitates fair comparison between different CBMs. We also show that using NEC can help to effectively control information leakage and enhance interpretability in our VLG-CBM.
3.
We conduct series of experiments in Section 5 and demonstrate that our VLG-CBM outper- forms existing methods across 5 standard benchmarks by at least 4.27% and up to 51.09% on Accuracy at NEC=5 (denoted as ANEC-5), and by least 0.45% and up to 29.78% on average accuracy across different NECs (denoted as ANEC-avg). Our Iearned CBM achieves a high sparsity of 0.2% in the final layer even on large datasets including Places365, preserving interpretability even with large number of concepts. Additionally, we qualita- tively demonstrate that our method provides more accurate concept attributions compared to existing methods.
VLG-CBM provide accurate explanations while prior work provide inacourate/wronngless useful explanations!
VLG-CBM (ours)
LF-CBM
LM4CV
short inld book (050)
NOT a brown and whith color achame (1.77)
Color- - Blue haadd, ol back, yollow underpeert (101 .00
2.
blue head (1)
NOT white and black co oration (1.98)
2. grastihh head, back, wings and tail with blue highrghts (94.03)
3.
grewn bacck (0.00)
indescent teathers (1 (.37)
3. bright tlue and orange plumage (91 (44)
4.
short slout bill (0.01)
NOT a black bib with white stripes (1 200
large red bill with si lighly hooked sp (80.00)
6,
small songbird 10 01)
NOT a black and white oolor schomo (1.01)
5. d stinstive while throat (:78,87 77888)
Sumn other concapte 00.00) de
Sum of other concepts (4.22
St. of other concepaa (344.60 (.)))
blue gray wings (9.40)
yellow haed 98)
long: straight bill with a bdack (100.07)
[141 40)
2. bright den ye yellorpaa plu mage (1 31)
NOT a red creit the head 10 99)
3. yallow head and breast (1.07)
orange legs (0.99)
3. pointeed orange bill (95 (257222
large contccll bill 1o 531
yellow onange plumage (0 (0)
yellow and black plumage (84.85)
black mask
(0 2..2
is hiight oningee bewset (0.76)
5 brighnl tile and oninge plumage SIT)
Burn other concapts 0.000
Sumn of other concepts (5.53)
Bum of other cor noppps (-257 02)
small forked tal 22)
NOT dark winglips (2.00
redi black and while feathers 12)
2. black and white winge (3 57)
white brcast with brown spots 11 .93
2. bright red head and broast (84
3. small delizatte body (2.75)
NOT a cans brown or black oolor 11 (25)
3.
medirmneent black ben with bronce or broowiihh gnay high igit (-54.8))
4. long slendar beak (0. 12)
yollow leet 12)
dark gray wings with pale gray edgingg (-18.50)
poined beak (0.50)
NOT lona. blue-pray wings (0.81)
bright red and napa (75.39)
Stmm other concepts 000
Sum of other concepts (3.
9um other ponoepts (118.99)
Figure I: We compare the decision explanation of VLG-CBM with existing methods by listing top-5 contributions for their decisions. Our observations include: (I) VLG-CBM provides concise and accurate concept attribution for the decision; (2) LF-CBM [IS] frequently uses negative concepts for explanation, which is less informative; (3) LM4CV (25 attributes the decision to concepts that do not match the images, a reason for this that LM4CV uses a limited number of concepts, which hurts CBM's ability to explain diverse images; (4) Both LF-CBM and LM4CV have significant portion of contribution from non-top concepts, making decisions less transparent. Full figure is in Appendix
Fig. D.I
2

---

Evaluation
Flexibility
Interprelability
Method
Control on information leakage
Unlimited concepi numbers
Flexible backbone
Accurate concept prediction
Vision-guided concept filtering
Interpretahle decision
Raselines:
LF-CBM
A X
4
4
LaBo[27
X
A
4
LM4CV 25
X
4
4
This work: VLG-CBM
Table 1: Comparative analysis of methods based on evaluation, flexibility, and interpretability. Here, v denotes the method satisfies the requirement, A denotes the method partially satisfies the requirement, and denotes the method does not satisfy the requirement. We compare with SOTA methods including LF-CBM [5], Labo [27] and LM4CV [25].
2
Related work
Concept Bottleneck Model (CBM), The seminal paper [ first proposed self-explaining models by leveraging the idea of autoencoder to learn interpretable basis concepts in an unsupervised manner without pre-specified concepts. Later, [6] proposed to learn interpretable concepts with human specifications (labels) in the concept bottleneck layer (CBL), and coin the term Concept Bottleneck Models (CBM). CBL is followed by a linear prediction layer, which maps concepts to classes, enabling interpretable final decisions. Formally, let feature representation generated by a frozen backbone represented by 2 = ((x). CBL concept prediction as g(2) = Waz, and the final prediction layer as h(.) Wrg(2) + br. The final class prediction of the CBM is given by ÿ = h(]=)) = h o a ((x).
Under this setting, the key in training CBM is obtaining an annotated ((image, concept)) paired dataset for training concept bottleneck layer g. (6), the authors used human-specified labels to train the CBL in a supervised way. However, obtaining labels with human annotators could be very tedious and costly. Recently, [[5). [25). and 227 proposed to utilize Large Language Models (LLM) to generate set of concepts S, then train CBL by aligning image and concepts with the guidance of vision language models (c.g. CLIP). For example, Oikarinen et al. [33] proposed LF-CBM to train CBM by directly learning mapping from the embedding space of backbone concept values in the CLIP space using cosine cubed loss function with the neuron interpretability tool and then mapping concepts to classes using sparse linear layer. 22 proposed LM4CV, task-guided concept searching method that learns text embeddings in the CLIP space, and then maps the learned embeddings concepts obtained from LLM using nearest neighbor. Yang al. [27 proposed
LaBo, using submodular optimization to reduce the concept set, followed by using CLIP backbone for obtaining concept values. However, as we show in Sections and [..]. these methods suffer from multiple issues: (i) The concept prediction is often incorrect and does not capture the visual attributes required for downstream class prediction (e.g. see Fig. b (ii) VLMs like CLIP suffer from modality gap between image and text embeddings [9] resulting in encoding unintended information, and even random concepts can achieve high accuracy 25 To address these issues, we explicitly ground the concepts on the training dataset using open-domain object detection model and then using the obtained concepts for learning CBL this can ensure a more faithful representation of fine-grained concepts and avoids the modality gap issues introduced by VLMs. Table I demonstrates the superiority of VLG-CBM over existing methods [15 25 27 on properties including controlling information leakage, flexibility to use any backbone, and accurate concept prediction.
There are some recent works aim addressing the challenges of CBMs. Similar us. Pham al. [16] uses an open-vocabulary object detection model to provide an explainable decision. However, their model is directly adapted from an OWL-ViT model, while our VLG-CBM uses an open-vocabulary object detection model to train a CBL over any base model, providing more flexibility. Additionally, their model requires pretraining to get best performance, while our VLG-CBM could be applied post-hoc to any pretrained model. Kim et al. 4 proposed to filter non-visual concepts by adding a vision activation term to the concept selection step, whereas VLG-CBM uses an open-vocabus vocabulary object detectors in multiple stage of CBM pipeline: for filtering non-visual concepts and the guiding training of concept bottleneck layer. Sun al. 20 aims eliminating the information leakage and the authors evaluate the information leakage by measuring the performance drop speed after
3

---

Obtain Candidate Concepts
Grounded Bounding Box Generation
Auto-Labelled Dataset
Whrtet
footuros cluss bind?
Ogjad Delicto
Beak Taal
(mage, Concepts, Class Label) OK
(Box, Concestts Class Label)
CBL Training
Learning Predictor
Bacidoon
CEL
Feathers:
Sparse Layer
Monkky
Concepts
Multilabel BCE Loss
Multiclass CE Loss
Class Label
Figure 2: VLG-CBM pipeline: We design automated Vision+Language Guided approach train Concept Bottleneck Models.
removing top-contributing concepts. This metric can be controlled by our proposed NEC metric, because the performance reach minimum after removing contributing concepts. Roth et [17 demonstrate that random words and characters achieve comparable CLIP zero-shot performance on visual classification tasks. However, their work does address information leakage problem and is very different setting from our work. To date, most of the CBMs focused on vision domains, including this work. There are some recent work applying CBM approach to different domains and different tasks, e.g. interpretable language models for text classifications
[2]] 22 11 and for
continual learning [26]. We refer the interested readers to their papers for more details.
Open Domain Language Grounded Object Detection. Recent works, including GLIP 88, GLIPv2 29 and GroundingDINO [00] detect objects in images an open-vocabulary manner conditioned on natural language queries. In this work we propose to utilize open- vocabulcyyy object detectors for automatically generating grounded concept dataset for training CBMs. This removes the need for human labelers, which is costly, tedious, and does not scale to large datasets. Further, the detected objects provide necessary vision-guidance for CBMs training as demonstrated in our experiments.
3 Method
In this section, we describe our novel automated approach train CBM with both Vision and Language Guidance to ensure faithfulness, which is currently lacking in the field. Our approach, abbreviated as VLG-CBM in the paper, generates an auxiliary dataset grounded on fine- mne-graieed concepts present in images for training sequential CBM. Section 3.1 describes our approach to generating an auxiliary dataset used in training CBM, Section (3. (3.2 describes our approach training concept bottleneck layer, and Section 3.3 describes the training of sparse layer to obtain class labels from concepts in an
manner. The overall pipeline shown Fig.
3.1 Automated generation of auxiliary dataset
Here we describe our novel automated approach for generating labeled datasets for training CBMs. Let f X' - be the neural network mapping images corresponding class labels, where X' = = #xx x3 denotes the input image space and y = {1, 2, ........r C*} denotes the label space, C is number of classes. Denote D ((r, ).))22 x, X, y/4 E y the dataset used for training f. where at; is the i-th image and is the corresponding label. Let S be set of natural-language concepts describing the fine-level visual details from which classes are composed. We propose to generate modified and auxiliary dataset D' from D such that each image contains finer-grained concepts that

---

are useful in predicting the classes, along with the target class. The overall process of obtaining the modified dataset D' can be divided into two steps:
Language supervision from LLMs to generate a set of candidate concepts: We follow the steps proposed in LF-CBM [15] for generating candidate concepts S. for each class e by prompting LLM to obtain visual features describing the class.
Vision supervision from Open-domain Object Detectors to ground candidate concepts to spatial information: We propose using Soauudinn-DINOOAA Swin-B, current state-of- the-art grounded object detector, for obtaining bounding boxes of candidate concepts in the dataset. For each image I; with class label and candidate concepts Se, we prompt Grounding DINO model with S. and obtain K; bounding boxes:
B; = {(bj. f, 8 K.
(1)
where by E IR4x2 is the j-th bounding box coordinates, i E R is the corresponding confidence given by the model and 8j S. is the concept of this bounding box. We define a confidence threshold T and remove bounding boxes with confidence less than T to get filtered bounding boxes for each image:
B₁ (b,t, E B, 1>>
(2)
After collecting bounding boxes for every image, we filter out the concepts that do not appear in any bounding box, and get our final concept set S:
$ = {s E S 3(;,,s) B.).
(3)
The one-hot encoded concept label vector o; E {0, 13 for image 2; is thus defined as:
if sy appears in B,,
(o.); = (o. otherwise.
Our final concept-labeled dataset D' for training CBM can be written as:
D' = [11, x))
(5)
3.2 Training Concept Bottleneck Layer
After constructing the concept-labeled dataset D', we now define our approach to train the concept bottleneck layer for predicting the fine-grained concepts in the input image in multi-label classifi- cation setting. Let Φ X + Ad be backbone that generates d-dimensional embeddings for input image x. Note that (xx) can be pre-trained backbone or trained from scratch. Define to be the Concept Bottleneck Layer (CBL) which maps embeddings to concept logits. We train sequential CBM 66 [3] g( (()))) to predict concepts in an image using Binary Cross Entropy (BCE)
6(x)
loss for multi- prediction. Additionally, to improve the diversity of the concept- dataset D', we augment the training dataset by cropping images to randomly selected bounding box and modifying the target one-hot vector to predict the concept corresponding to the bounding box. Our optimization objective in terms of BCE loss can be written as:
D'I
1
min LOBL, LCBL g
BCET o ((I))
(6)
D
3.3 Mapping Concept to Classes
In this section, we define our approach training sparse linear layer to obtain class labels from concepts in an incepretabiiitypreeevvvlll preservable manner. Let h IRd - RC be the sparse linear layer with weight matrix Wr and bias br. which maps concept logits to class logits. We train the sparse layer using the original dataset D by first obtaining concept logits from the trained (fooeeeze,) normalizing the concept logits with the mean and variance on training set, and then using them to predict class logits. Our optimization objective in terms of Cross Entropy (CE) loss can be written as:
min Lst, LSL =
1 D
Chlhagoooa v] + XBar (2DD)DD
Σ
5

---

where Ra = ( a 11 is the elastic-net net regularization [3 on weight matrix Wp. is a hyperparameetrr controlling regularization strength. We use GLM-SAGA 24 solver solve this optimization problem.
4 Unifying CBM evaluation with Number of Effective Concepts (NEC)
Besides training, another important challenge for CBM is: how to evaluate the semantic information learned in the CBL? Conventionally, the classification accuracy for final class labels is an important metric for evaluating CBMs, with the intuition that a good classification accuracy indicates that useful semantic information is learned in the CBL. However, purely using accuracy as the evaluation metric could be problematic, as it has been shown that information leakage exists in jointly or sequentially That is say, the CBL could contain unintended information that could be used for downstream classification hence achieving high classification accuracy, even if the concept is irrelevant to the task. In fact, recently [25] showed that, when increasing the number of concepts, a randomly selected concept set could even approach the accuracy of the concept set chosen with sophistication, supporting the existence of information leakage.
trained CBM [3 [2].
To better understand this phenomenon, in section 4.1 we conduct a first theoretical analysis to investigate random CBI. and its capability. To the best of our knowledge, this is the first formal analysis of random CBL. Next, inspired by our theoretical result, we propose a new evaluation metric for CBM, named NEC in section 4.2 NEC provides way to control information leakage and enhance the interpretability of model decisions.
4.1 Theoretical analysis of the Random CBL
We start by defining the notations. Denote & the number of concepts in CBL. We assume that the CBL g consists of single linear layer: (2)) Wez, where Wa Ø and 2 E 12, and the final layer h is also linear: h o g(z) Wrg(2> by, where WF RCck by E RC_ This is the common setting for CBMs. The following theorem suggests a surprising conclusion: a linear classifier upon random (i.e. untrained) CBL could accurately approximate any linear classifier trained directly on the representation, as the number of concepts in the CBL goes up.
Theorem 4.1. Suppose Σ Bddd is the variance matrix of the representation 2 which is positive definite, Xmaz is the largest eigenvalue of Σ, and the weight matrix W. E Bkxd is sampled i.i.d from a standard Gaussian distribution. For any linear classifier f which is built directly on the representation i.e. I a + R, [(z) w + b, it could be upproximated by another linear classifier f on concept logits g(z) Wez, iee. f(2) R0 f(z) w g(z) + b, with the expected square error Ekk upper-bounded, by
Amea (1 1) k < d: k d.
E(k)= 1 0,
(8)
Here E(k) = Ew: min (1),)) E. 1()=))(()}PP) denotes the average square error, w E Rdd
Φ E IR*, 1 is the number of concepts in CBL.
Remark 4.1. In Theorem 4.1 we consider a I-D regression problem where we use linear combination of concept bottleneck neurons to approximate any linear function. The multi-class classification result could be derived by applying Theorem 4. to each class logit (see Corollary A.T From Eq. (8) we could see that the expected error goes down linearly when concept number le increases, and achieves 0 when 2 d, where is the dimension of backbone representation 2. This suggests that, even with random CBL (i.c. W. is simply drawn from standard Gaussian distribution without any training), the classifier could still approximate the original classifier well and achieve good accuracy, when concept number l is large enough. We defer the formal proof of Theorem 4.1 to Appendix
4.2 A New Evaluation Metric for CBM: Number of Effective Concepts (NEC)
Theorem 4.1 provides a formal theoretical explanation on 25 ss observation. Moreover, it raises a concern on the evaluation of CBMs: model classification accuracy may not be a good metric for evaluating the semantical information learned in CBL. because a random CBI. could also achieve high accuracy. To address this concern, need to control the concept number k so that the

---

semantically meaningful CBLs can be distinguished with random CBLs w.n.t the final classification performance.
We notice that previous works mainly use two approaches to control k:
1. Control the total number of concepts: 25 used a more concise concept layer, i.e. reduce the total number of concepts. However, this approach may miss some important concepts due to limitations in total concept numbers. Additionally, they used a dense final layer which is less interpretable for humans, as each decision is related the whole concept set.
2. [15 28 suggested using a sparse linear layer for final prediction to enhance interpretability. Though sparsity is initially introduced to enhance interpretability, we note that this also reduces the number of concepts used in the decision, thus controlling the information leakage. However, the problem is, these works lack the quantification for sparsity, which is necessary for fair comparison between methods.
To provide a unified metric for both approaches, we propose to measure the Number of Effective Concepts (NEC) for final prediction as sparsity metric. It is defined
NEC(W,) ΣΣ ((WF), #
Intuitively, NEC measures the average number of concepts the model uses predict class. Using NEC to evaluate CBM provides the following benefits:
1. A smaller NEC reduces the information leakage. As shown Fig. B with large NEC, even random CBL could achieve near-optimal accuracy, suggesting potential leakage in information. However, by reducing NEC, the accuracy of random concepts drops quickly. This implies enforcing small NEC could help control information leakage.
2. A model with smaller NEC provides more interpretable decision explanations. Humans can recognize an object with several important visual features. However, models can utilize tens or hundreds of concepts for the final prediction. By using smaller NEC, the model's decision could be attributed mainly to several concepts, making more interpretable to human users.
3. NEC enables fair comparison between CBMs. Comparing the performance of CBMs has long been challenging problem, as different models use different numbers of concepts and different styles of final layers (sparse/dense). NEC considers both, thus providing fair metric to compare different models.
Given these benefits, we suggest to control the NEC when comparing the performance of CBMs. In Section we provide experiments with controlled NEC, where we observed our VLG-CBM outper- forms other baselines.
3.55
2.00
2.75
Oun
Ours
Ours
Durs
1.55
LFCBM
CBM
LF CBM
CBM
140
Ronoom
64
154
16
Random
10a
0.3
16
Bandom
124
10
Bandom
104
(a) CIFARIO
(b) CIFAR100
(c) CUB
(d) Places365
Figure 3: Accuracy comparison between our VLG-CBM, LF-CBM and randomly initialized concept bottleneck layer under different NEC. The experiment is conducted on the CIFARIO dataset. From the results, we could see that (I) for NEC large enough, even a random CBL could achieve near-optimal accuracy, supporting the existence of information leakage; (2) when NEC decreases, the accuracy of LF-CBM and random weights begin to drop, while our VLG-CBM does not have significant decrease.
7

---

Experiments
In this section, we conduct series of experiments to evaluate our method, including illustrating the faithfulness of concept prediction, interpretability of model decisions, and performance with controlled NEC.
5.1 Performance comparison
Setup. Following prior work [5] we conduct experiments on five image recognition datasets: CIFAR1O, CIFAR ()) CUB 23
Places365 (30] and ImageNer[ [8]. For the choice of backbone,
we categorize the experiments into two categories:
1. CLIP backbone: For CLIP backbone, we choose CLIP-RN50 for all datasets.
2. non-CLIP backbobe: We follow the choice of [15] to use ResNet- for CUB and ResNet-50 (trained on ImageNet) for Places365 and ImageNet as the backbone.
The reason of this categorization some previous works (e.g. LaBo 27 only supports CLIP backbone.
Baselines. We compare our method with three major baselines when applicable: LF-CBM LaBo[27], and LM4CV (5). These are SOTA methods for constructing scalable CBMs, with [15] most flexible and 27 25 limited by specific architecture and not available for certain dataset. Additionally, we present the results from a randomly initialized CBL for comparison.
Metrics. As discussed in Section 4 in order to evaluate the final classification power of CBM, we should acculate the Accuracy under specified NEC(ANEC). Therefore, we measure the following two metrics:
1. Accuracy at NEC-S(ANEC-5): This metric is designed to show the performance of CBM which could provide an interpretable prediction. We choose the number 5 so that human users could easily inspect all concepts related the decision without much effort.
2. Average performance, we also calculate the average accuracy under different NECs. In general, higher NEC indicates more complex model, which may achieve better performance but also hurt interpretabilityy We choose six different levels: 5, 10, 15, 20, 25, 30 and measure the average accuracy.
To evaluate the trade-off between interpretability and
Controlling NEC. As we discussed in Section there are two approaches to control NEC: (1) using a dense final layer and directly controlling the number of concepts and (2) training sparse final layer with appropriate sparsity. LM4CV used the first approach. where the number of concepts could be directly set target NEC. For LF-CBM and our VLG-CBM, the second approach is utilized: To achieve target sparsity, we control the regularization strength À in GLM-SAGA. GLM-SAGA provides a regularization path, which allows us to gradually reduce regularization strength and get series of weight matrices with different sparsity. Specifically, we start with A = Amaz which gives the sparsest weight. Then, gradually reduce the À by At+1 aA₁ until achieve the desired NEC. We choose the weight matrix with the closest NEC to our target and prune the weights from smallest magnitude to largest to enforce accurate NEC. LaBo 27 did not provide a NEC control method. Hence, we apply sparse final layer training of LaBo's 's concept prediction to control NEC.
Results. We summarize the test results for with backbone in Table 2 and results with non-CLIP backbones in Table From the results, we could see that:
1. The accuracy at NEC = 5 provides a good metric for evaluating the semantic information in CBL: As shown in the table, the accuracy of random CBL much lower with NEC which implies the information leakage is controlled and the accuracy could better reflect the useful semantic information learned in the CBL.
2. The performance of LM4CV is even worse than random CBL. An explanation to this is LM4CV utilizes a dense final layer, which is intrinsically inefficient to interpret as each class is connected to all the concepts, including the irrelevant ones. When limiting the NEC to small value to provide concise explanation, the model has to largely reduce the concept number which sacrifices the prediction power.
8

---

3. Our method significantly outperforms all the baselines at least 4.27% and up to 51.09% on accuracy NEC=5, and by least 0.45% and up 29.78% on average accuracy across different NECs, illustrating both high performance and good interpretability.
Dalaset
CIFARIO
CIFARI00
ImageNet
CUB
Metries
ANFC-5
ANRC-avv
ANRC-5
ANEC-avg
ANEC-5
ANRC-avg
ANEC-5
ANEC-avg
Random
67.55%
77.45%
29.52%
47.21%
18.04%
39.63%
25.37%
40.13%
LF-CBM
84.05%
85.43%
56.52%
62.24%
52.88%
62.24%
31.35%
52.70%
LM4CV
53.72%
69.02%
14.64%
36.70%
3.77%
26.65%
3.63%
15.25%
LaBo
78.69%
82.05%
44.82%
55.18%
24.27%
45.53%
41.97%
59.27%
VLG-CBM(Ours)
88.55%
88.63 %
65.73%
66.48 %
59.74%
62.70%
60.38 %
66.03% %
Table 2: Performance comparison with CLIP-RN50 backbone. We compare our method with
random baseline, LF-CBM((S), LM4CV (25) and LaBo[[7], The random baseline has 1024 neurons for CIFARIO and CIFAR 100, 512 for CUB and 4096 for ImageNet.
Dataset
CUB
Places365
ImageNet
Metrics
ANEC-5
ANEC-avg
ANEC-5
ANEC-avg
ANEC-5
ANEC-avg
Random
68.91%
73.44%
17.57%
28.62%
41.49%
61.97%
LF-CBM
53.51 %
69. 11%
37.65%
42. 42.10%
60.30%
67. 07.92%
VLG-CBM (Ours)
75.79% %
75.82 %
41.92%
42.55%
73.15%
73.98%
Table 3:
Performance comparison with non con-CLIP backbone. We compare against LF-CBM and a random baseline, as LM4CV(25) LaBo (27 does not support non-CLIP backbone. The random baseline has 512 neurons for CUB, 2048 for Places365, and 4096 for ImageNet.
5.2 Visualization of CBL neurons
In order to examine whether our CBL learns concepts aligned with human perception, we list the top-5 activated images for example concept neurons the model trained the CUB dataset in Fig. 4 As shown in the figure, our CBL faithfully captures the corresponding concept. We provide
more visualization results Appendix K
(a) Concept 19: black face.
(b) Concept 29: black plumage
(d) Concept 182: small black head Figure 4: Top-5 activated images example concepts neurons VLG-CBM on CUB dataset.
(c) Concept 117: brown or gray body
5.3
Case study
study to compare the concept prediction between VLG-CBM, LF-CBM [5] and LM4CV 25 as shown in Fig We provide extended results and comparison with
In this section, we conduct
LaBo 27 in Appendix G.2 For method, use the final layer with NEC 5. We show that method provides more accurate concept prediction and more interpretable decisions for users.
Decision interpretability. We examine the explanation of each CBM model example images by showing the top-5 concept contributions. The contribution each concept is calculated as the product of the concept prediction value and corresponding weight. Formally, the contribution of

---

i-th concept to j-th class is defined as: Contribution((..) gz(2) contributing concepts for the final predicted class and visualize in Fig. T
(WF) js- We pick the top-5 We could see that:
1. For other CBMs, large portion of the final decision is attributed to the "Sum of other features". This part hurts the interpretability of CBM, as it's difficult for users to manually inspect all these concepts in practice. We conduct further study on this in Section 5.4 Our model, however, provides concise explanation from a few concepts because we apply the constraint NEC=5. This ensures users can understand model decisions without difficulties.
2. Our VLG-CBM provides explanation more aligned with human perception. From the example, we also see that our model explains the decision with clear visual concepts. Other CBMs attribute the decision to non-visual concepts (c.g. LaBo), concepts that do not match the image (e.g. LM4CV), or negative concepts (LF-CBM).
5.4 Do Top-5 concepts fully explain the decision?
Besides training a final layer with a small NEC, another common approach to provide a concise explanation is showing only the top contribution concepts. However, we argue that this approach may not faithfully explain the model's behavior, as the non-top concepts also make a significant contribution the decision. To verify this, conduct the following experiment: On the CUB dataset, we prune the final weight matrix WF to leave only the top-5 concepts for each class, whose weight has the largest magnitude. Then, we use the pruned model to make predictions and compare them with the prediction results from the original model. Table shows results for our VLG-CBM which uses NEC= 5 to control sparsity, and other three baselines, [2B LF-CBM and LM4CV 25), without any constraint on NEC. As shown in the table, for all three baselines, a large portion of predictions changes after pruning. This suggests that without explicitly controlling NEC, only showing top-5 contributing concepts does not faithfully explain all of the model decisions. Hence, we recommend training the final layer with NEC controlled obtain a concise and faithful explanation as we proposed in Section
Method
VLG-CEM (Ours)
LF-CBM
LM4CV
LaBo
% changed decisions
0.12%
49.21%
98.34%
81.40%
Table 4: Portion of model predictions that changes after pruning. The results suggest that for existing methods (LF-CBM, LM4CV, LaBo) without NEC control, large portion of predictions changes with top-5 concepts, implying potential risk when using top-5 contributions to explain model decisions.
Conclusion, Potential Limitations and Future work
In this work, we study how to improve the interpretability and performance of concept bottleneck models. We introduce novel approach VLG-CBM based on both vision and language guidance, which successfully improves both the interpretability and utility of existing CBMs in prior work. Additionally, our theoretical analysis show that information leakage may exist even in the un- trained CBLs, serving the foundations for our proposed new NEC-controlled metrics (ANEC-5 and ANEC-avg). We show that NEC not only allow fair evaluation of CBMs but also can be used to effectively control information Icakage of CBM and ensure interpretability. Extensive experiments on image classification benchmarks demonstrated our VLG-CBM largely outperform previous baselines especially for small NEC, providing more interpretable decisions for users.
Despite the superior performance of VLG-CBM over prior work as demonstrated in extensive experiments, one potential limitation the dependence on large pretrained models (e.g. the success of open-domain grounded object detection model that we use to enforce vision guidance). However, prior work (e.g. LaBo, LM4CV, LF-CBM) also shared similar limitation on the reliance of large pre-trained models (e.g. CLIP). Nevertheless, it also means that our techniques have the potential to be further improved with the advancement of large pre-trained models. the future, we plan to explore training CBL with even more vision guidance, such using segmentation maps of concepts.
10

---

Acknowledgement
The authors thank the anonymous reviewers for valuable feedback on the manuseript. The authors are partially supported by National Science Foundation awards CCF-2107189, IIS-2313105, IIS-2430539 Hellman Fellowship, and Intel Rising Star Faculty Award. The authors also thank ACCESS for support in this work.
References
[1] David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018.
[2] Morris L Eaton. Multivariate statistics; vector space approach, volume 512. Wiley New York, 1983.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[4] Injae Kim, Jongha Kim, Joonmyung Choi, and Hyunwoo J. Kim. Concept bottleneck with visual concept filtering for explainable medical image classification, 2023. URL https
//arxiv. abwpa/////////////////oe/ 2308. 11920
[5] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6980, 2014.
[6] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020.
[7] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[8] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training, 2022.
[9] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.
[10] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- pre-training for open-set object detection. arXiv preprint arXiv:2303.. 05499, 2023.
[11] Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris Callison- Burch. Interpretalee-y-deesiin text classification with iteratively generated concept bottleneck. CoRR, 2023.
[12] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 10..2/016333.3 2021.
[13] Andrei Margeloiu. Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn intended? arXiv preprint arXiv:2105.04220 2021.
[14] Tuomas Qikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron repre- sentations in deep vision networks. In ICLR, 2023.
[15] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. ICLR, 2023.
[16] Thang M Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, and Anh Nguyen. Peeb: Part-based image classifiers with an explainable and editable language bottleneck. arXiv preprint 2502007. 2024.
11

---

[17] Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts, 2023. URL
wwt.aaawwwww...ra.raaaaaaaaaaa 07282
[18] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-255 2015.
[19] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprini arXiv: 1.08731, 2019.
[20] Ao Sun, Yuanyuan Yuan, Pingchuan Ma, and Shuai Wang. Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning. arXiv preprint arXiv:2402.05945, 2024.
[21] Chung-En Sun, Tuomas Oikarinen, and Tsui- Wei Weng- Crafting large language models for enhanced interpretability. ICML Mechanistic Interpretability workshop, 2024.
[22] Zhen Tan, Lu Cheng, Song Wang, Yuan Bo, Jundong Li, and Huan Liu. Interpreting pretrained language models via concept bottlenecks. CoRR, 2023.
[23] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[24] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In ICML, 2021.
[25] An Yan, Yu Wang. Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, 2023.
[26] Sin-Han Yang, Tuomas Oikarinen, and Tsui-Wei Weng. Concept-driven continual learning. TMLR, 2024.
[27] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023.
[28] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. ICLR, 2023.
[29] Haotian * Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint 220::2200008366, 2022.
[30] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 145-1144,, 2017.
[31] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
12

---

Appendix
Table of Contents
A Proof of Theorem 4.1
14
B Implementation details
17
Ablation Studies
Ablation study confidence threshold
D Evaluating annotations from Grounding DINO
18
Distribution of nonzero weights among class
20
Constructing model with specified NEC
20
Additional case study examples
20
G.1
Negative concepts reasoning
20
G.2 Impact of NEC
20
H Further discussion on decision explanations
23
H.I
Negative contributions
23
Additional experiment results
23
I.1
Generalizability to OOD dalasels
23
.2 Ablation study
23
Human study
23
Visualizing VLGG-BM explanations
24
13

---

A Proof of Theorem 4.1
In this section, we present formal definition of the expected square error in Theorem and show the proof. First, we define the square approximation error as
E2 [I(()) - ]
(A.1)
which is the average square distance between f(z) and f(z). Given a specific CBL Wa, we seek a final layer i to minimize the square error:
min E, [15(s) - f(=)) (w,5)
(A.2)
For randomly Gaussian initialized We, we care about the minimal error we could achieve on average. Thus, for each Wes we choose and achieve minimum approximation error, then take the expectation over We to define the expected square error as
((k) = 1Ew, min E. ( {(=))²] (w,bb
(A.3)
Setting Suppose the representation has variance E E. Rddd which is positive definite. The weight matrix W. E Rixd is sampled i.i.d from a standard Gaussian distribution, Here, we show that any linear classifier which is built directly representation i.e. f(z) = u + b, could be approximated by linear classifier concept logits g(z) Wez, i.e. f(z) f(z) T TT)()))()))))))))))))())) + b.
Theorem 4.1. Suppose Rdxd is the variance matrix of the representation z which is positive definite, Xmaz is the largest eigenvalue of Σ, and the weight matrix W. E R is sampled ii.d from a standard Gaussian distribution. For any linear classifier f which is built directly on the representation i.e. f R + R, fz) w + b. it could be approximated by another linear classifier f on concept logits g(z) Wez, ie. f(2) = f(z) w" g(2) b, with the expected square error E(k) upper-bounded by
E(k) 1 Amar (1 1
k < d: k d.
(8)
0,
Here E(k) Ew. min (w.,)) B.
denotes the average square error, w E Rd.
W E B k is the number of concepts in CBL.
Proof. Based on the value of ki we can consider two cases: (I) k: < d, and (II) k 2 d, and derive the E(k) respectively.
Case (I): < First, we consider under a fixed Wo, what the minimum error we could achieve. The expected approximation error is:
E. [1f(z) - f(=)p] E, [w + b (w We + 6)]2]
F. [ww" u W.)= b b12]
(A.4)
Vsl(w W.T w) T T2] + [.l - 2 w.) + b- 6
(**)
The last equality is from E(X2)= = VX + (EX)² The second term (**) takes minimum when
i - [=[(w" T W.)= = + The remaining question is to choose a proper v to minimize (+).
Notice that
V, [w W. w) [2] (w W w) F(w W i)
=1}E> W. 1
(A.5)
= =11 W to > 2 1
14

---

where Σ is the covariance matrix of 2, Σa is the principal square root of Σ, Σ E Raxd, Σ E Rdxd Now the problem in Eq. A.2 can be reduced to a linear least square problem:
min Es [if(s) [ f()))} min | - (w
(A.6)
Since E is positive definite, is Σt. Thus, the eigen decomposition of Σ satisfies the following: E = Q AQ. where E [ddd x is an orthogonal matrix and A IR is diagonal matrix with positive entries. With Gram-Schmin process, we could derive QR factorization of W W. QR. where Q E Rdxd is orthogonal and Rt E gdxk is upper triangular. Plugging above decomposition of Σ and W now we have
= min IIÂ AQQRO QT AQ min E W i
"
min |AQQR - AQwI1
( is orthogonal, thus preserves 2-norm)
min |A(QQRO Qw)113
"
min X QQR Qwi?3
(Since all entries of A are positive.)
min
RRù
Q l
(Multiply by Q Q preserves the norm)
maz
(A.7)
where Amaz is the largest eigenvalue of Σt. In short, we have derived the minimum square error for given We, which is upper bounded by
min
E2 (((=))-(s))0) < X X min R (
Q
2
(A.8)
Secondly, we consider when Wo is sampled i.i.d. from standard normal distribution, and calculate the expected error. From above derivation,
Ew. min E. (17,8)
[[r(e)-((or]]
< Xmaa ((n min RR @Twl3
(A.9)
Note that since W. QR, the randomness in W. is reflected in Q and R. The matrices Q and R satisfies the following properties:
1. Q is random rotation following uniform distribution. This is intuitive because standard Gaussian distribution is Ektation-invarant For formal statement and proof, we refer to Proposition 7.2 of Eaton (2).
2. range(R) -span(e1, C2, .x) with probability 1. Since rank R) ran and W.T is full rank with probability 1, rank(It) min(k, d) = & with probability 1. From upper-triangularity of R, know that
range(R) S span(es, e2. ,ex)
(A.10)
With probability 1, rank(It = k, thus we conclude
range(R) = span 62, ,e))
(A.11)
In the following derivation, since we only cares about the expectation, we omit "with probability 1" for brevity.
From the above properties of Q and R, the expectation term in the RHS of Eq. (A.9) can be derived as:
E.R.Q) min Rw QTw w0 = EQ ((Q@T))xaall))
(A.12)
This is because range((1) span(e), .., ek), and < d. Thus, ming | Q "wll3 is the squared distance from Q w to subspace span(e1, ... ,ex), which equals to the squared sum of last d - l coordinates of Q? w.
15

---

Because Q is random rotation, Q w is uniformly distributed on a sphere with radius ||w]2. Denote v w. From symmetricity, we have
ot = E
vz.
(A.13)
Furthermore, ||o| = | gives Ei= us Σ Ey of 1|w112 thus E of E uz
-
Take expectation of both sides gives
E u = || tw012// d. The target quantity becomes
Boll(Q' w) w)x+1dd12 En Σ
of
1= 1
(A.14)
Ev
k=++1
d
2wl12
d
In conclusion, we derive an upper bound of approximation error for any linear function f:
Ew,
min Exg (((x)) (v0.6)
< Xmaan (1
d 12
(A.15)
Look at the bound in Eq. regarding the norm of weight vector we want to approximate; (1 ()) is a linear term shows that the expected square error goes down linearly when we increase the number of concepts k. and achieves zero when d.
A.15
A is constant regarding the scale of data; || is a constant
Case (II): d. For the case that k 2 it could be derived from our main results that E(k) 0. Additionally, with probability could find f(x) f(x) as will be derived below. As discussed, with probability 1, W. has full rank. Given that, we have
W W
where W is the Moore- inverse of W.. For any linear classifier f(z) w z + b. Let T = (W.+) w, b = b. we have
f(z) - T. g()))))))))))))))))))(()) - ww Woz + b ===bb=ff f(2)
and thus E(k) O.
Corollary A.1. For I and J with output classes, i.e. I IRd RC, f: Rd - R.C w E Rdd i E IR. the expected error upper-bound is
E(k) & CAman
(A.16)
Here E(k) Ew, mim (e, llf(2)) [(=))[]]
denotes the average square error.
Remark A.2. The statement could be verified by applying Theorem 4.1 to each ft and f, output, then summing up the error.
16

---

B Implementation details
Computational resources and codes. Our experiments run on server with CPU cores, 64 GB RAM, and Nvidia 2080Ti GPU. Our implementation builds on the open-source implementation of the LF-CBM [5] available: https: //t:://iiiit com -tt-sstooottt--LLa---LL---- wareaabeel--e--eeeeeeeee CBM For training the final predictive layer, we use publicly available code for GLM-SAGA [24].
pepraarraeeeer tuning. We tune the hyperparameters for our method using 10% of the training data as validation for the CI,AR1O, CIFAR1O0 100, CUB and ImageNet datasets. For Places365, we use 5% of the training data as validation. We use CLIP(RN50) image encoder as the backbone for CIFAR1O and CIFAR 100, Resnet- 1imeet-8[[3 [8 trained on CUB for CUB dataset, and Resnet-50 pretrained for Places365 following setup similar to LF-CBM. We tune the CBL with Adam optimizer with learning rate x 10-4 and weight decay xx 10-5 The concept dataset obtained from GroundingDINO inherently unbalanced since there a much lower proportion positive datapoints for concept. Consequently,
scale the CBL loss by multiplying it with positive value to balance the tradeoff between precision and recall and improve the imbalance of positive data points. We set T 0.15 in Eq. (2 in all our experiments. We seed the random number generator with a fixed seed to ensure the results can be reproduced.
C Ablation Studies
Confidence threshold
CUB200
Places365
Metrics
ANEC-5
ANEC-avg
ANEC-5
ANEC-avg
0.10
75.75%
75.75%
41.84%
42.50%
0.15
75.75%
75.73%
41.84%
42.51 %
0.20
75.73%
75.73%
41.25%
42.15%
Table C.1: ANEC-5 and ANEC-avg for different confidence threshold 7.
C.1 Ablation study for confidence threshold
Confidence threshold T in Eq filters concepts with bounding boxes' confidence less than T. In this experiment, we study the affect of on the VLG-CBM's accuracy. The results are shown in Table We observe that ANEC-5 and ANEC-avg first increases (or stays constant) and then decreases. attribute this effect to to the fact that increases, the number of false-positive decreases leading to better learning of concepts, however, the number of annotations available for learning a concept decreases.
C.1
17

---

D Evaluating annotations from Grounding DINO
This section quantitatively evaluates concept annotations obtained from Grounding DINO. We use CUB dataset for comparison which contains ground-trus for fine- fine-grained concepts present in each image. We use the label set from Koh al. 6 which has 1:1 mapping with the ground-truth concepts in the CUB dataset. We use precision and recall metric to measure the quality of annotations from Grouding DINO for each concepts. Table [.I] present mean precision and mean recall value at different confidence threshold. We observe that the obtained annotations have very high recall i.e if the concept is present the image, grounding DINO is able retrieve the object. The precision also sufficiently high though it suffers from relatively higher false- pells--ossiive detection rate compared to false-negative detection rate. However, as demonstrated in our qualitative and quantitative studies (Table 3 Fig K.2 K.I the effect of false-pos false-positive is minimal and VLG-CBMis able to faithfully represent concepts in the Concept Bottleneck Layer
Confidence threshold
Mean Precision
Mean Recall
0.10
0.7150 + 0.07
0.9930 + 0.08
0.15
0.7156 0.07
0.9693 + 0.11
0.20
0.7121 + 0.10
0.8713 + 0.21
Table D.1: Quantitative evaluation of concepts obtained from Grounding DINO model with Mean Precision and Recall for concepts at different confidence thresholds.
18

---

3. medium-sizes black bird with bronze or brownish-grai highlligtts (-84.05)
VLG-CBM provide accurate explanations while prior work provide hnacuurate/wronglless useful explanations!
1. Color Blue head, olive back, yellow underparts (1011 1.08) grayish head, back. wings and tail with blue highlights (94.03)
dark gray wings with pale gray gray edging (-78.56)
large red bill with a sligh fly hooked (89 (9)
bright blue and orange plumage (91 1.44)
2. large, orange bill with black (100.01
5. bright blue and orange plumage (76.97)
1. red, black and white feathers (93.1 12)
1. long, straight orange bill (141 1.48)
4. yellow and black plumage (84. 85)
2. bright red head and breast (84, (84.55)
bright red head and nape (75.39)
distinctive white throat (-76. 91)
Sum other concepts (-34.8) (4.89)
Sum other concepts (-257 (0)
Sum other concepts (11 3.93))
3. pointed orange bill (95. 72)
LM4CV
1. NOT a brown and white color schome (1.77)
5. NOT a black and white color scheme (1 (1.01)
4. NOT a black bib with white stripes (1 (129)
3. NOT a dark brown or black color (1 (1.25)
2. NOT white and black coloration (1. (.66)
2. NOT a red crest the head (.99)
2. white breast with brown spots (1 1.93)
5. NOT long. blue-gray wings (0.81)
4. yellow orange plumage (.80)
5. a bright orange breast (0. (.76)
3. Inieescent feathers (1 (1.37)
Sum of other concepts (4.22) 4.22)
Sum of other concepts (5.53) 53)
NOT dark wingtips (2.07)
Sum of other concepts (5. 12)
LF-CBM
1. yellow head 1.96)
3. orange legs (0 (9)
4. yellow feet (1.15 (.12)
2. bright golden yellow plumage (1.31)
Sum of other concepts (0.00)
3. yellow head and breast (1 1.07)
5. black mask on the face (0 26)
Sum of other concepts (0 m
2. black and white wings (3.57)
Sum of other concepts m
VLG-CBM (ours)
short pointed beak (0.65)
3. small delicate body (2. (2.75)
4. long slender beak (0.12)
5. small songbird (0.00 (0.01)
1. blue gray wings (9 (040)
4. large conical bill (0 (,53)
small forked (7 (2)
4. short stout bill (0.01)
5. pointed beak (0 (0)
2. blue head (..11
3. green back 0.09)
Figure D.I: Full version of Fig I comparing explanation of LF-CBM and LM4CV with VLG- CBM(surs)
19

---

E Distribution of nonzero weights among class
The NEC metric controls the average number of non-zero weights among classes. Further, study the distribution non-zero weight numbers between different classes. We choose our VLG-CBM model trained on CUB and places365 datasets, which have 200 and 365 classes, respectively, and plot the distribution of non-zero weights. Both models are trained to have NEC=5 The results are shown in Fig. [I.]] The figure suggests most classes have non-zero weight numbers around 5, while small number of classes utilize more concepts to make decisions.
80
70
50
00
40
30
30
20-
5
15
20
25
30
2.3
5.0
10.0
12.3
15.0
17.
(a) CUB
(b) Places365
Figure E. distribution of non-zero weight numbers from CUB and Places365 dataset. The models trained to have NEC=5.
Constructing model with specified NEC
In this section, discuss how construct models with specified NEC. When using methods with dense final layers (e.g. (251), controlling NEC is simply controlling total number of concepts in the concept set. Hence, below we mainly focus models with sparse final layers.
When training the final linear layer, larger lambbatreguularizztion strength) pushes the model to be sparser. Hence, utilize GLM-SAGA (24) which allows to obtain regularization path consists of different lambdas. To more specific, we choose Amaz and train models with À in [Amin Amaz/5000 /500Amarr), and take 50 X evenly from the interval in log space. Then, we choose the weight matrix with the closest NEC and pruning the weights from smallest magnitude to largest to enforce strict NEC. Hence, the actual NEC is enforced to be exactly as prespecified ones.
G Additional case study examples
G.1 Negative concepts in reasoning
In LF-CBM [5] and our VLG-CBM, normalization is applied on concept logits before the final decision layer. Hence, a negative value of concept logits indicates corresponding concept does not appear in the image. Following LF-CBM, we mark these concepts as "NOT [concept)" in explaining the decision. To study the frequency of this negative reasoning. concepts appear in top-5 contributing concepts on CUB dataset. The results show that, for VL.G-BM, 162 out of 28950(0.56%) reasonings through negative concepts. For comparison, LF-CBM utilizes 6687 out of 28950(23. (2000%%) negative reasoning.
count the times these negative
G.2 Impact NEC
The study in Section 5.3 shows that our VLG-CBM provides more interpretable decisions than baseline methods. To better understanding where these advantages comes from, we conduct further study to set the baselines with NEC=5 and compare the decision interpretation, see Figs. G.2 to G.4 The results suggest setting NEC=5 alleviate the problem from non-top-5 concepts. However, wrong/inacccrate/less useful explanations still exist.
20

---

Ground truth:
Bobolink.
VLG-CBM prediction: Bobolink
black
back(4.30)
black and white striped h1.9()
3. black head and back(0.27)
4. small round body(0.07)
5. NOT white stripes above the eyes(0.01)
Sum of other concepts: (0.03)
Figure G.I: Image 307: An example of negative reasoning of VLG-CBM
LF-CBM
LM4CV
LaBo
1. NOT a brown and white color scheme (1.77)
1. Color - Blue head, olive back, yellow underparts (101 (88
1. beautiful bird with brightly colored body
(0.022 (0.02)
2. NOT white and black coloration
2. graylsh head, back, wings bright
with
2. small, songbird with short and pointed bill 02)
Indescent feathers (1.37)
highlights (34.03)
black with stripes
orange plumage (91.4))
brightly
(1.28 (.229)
4. large red with slightly hooked lip
plumage (0 (0.02)
5. NOT a black and white color schome
(BS 09)
4. one of the most beautifuf north american
(1.01)
5. white rump patch at the base of the tail
songbirds (0.02)
Sum other concepts (4
(59.69)
5. colors vibrant and beautiku (9.02)
Sum concepts
Sum concepts (51.25)
LF-CBM (NEC=5)
LM4CV (NEC=5)
LaBo(NEC=5)
NOT a brown and white color
bright reddish brown head, crown and
beauttful little with very colerful plumage(3.
scheme(2.33) 39)
back of (382.66)) bright
black with stripes(1.
2. very colorful bird, with blue and green(0.4) .33)
black white scheme(0 61)
plumage(95.6
bright yellow throat, breast, flanks
3. very pretty and very colorful(0 441
iridescant feathers(0 27)
with black bars (51.22
4. NOT white stripes on white stripes
5.
NOT yellowish brown wings(0.22) 25)
Broad tail that is shorter than other pelican species (-36.48) nape, mantle, and
brown(0:
Sum other concepts: 00)
"rainbow jey" bright
Mottled brown
plumage(0.
scapularst
Sum of other concepts: (0. (0.00)
other concepts. (0.00)
Figure G.2: Comparing baselines with different NECs
21

---

Gills knyjllir Brypltton
LF-CBM
LM4CV
LaBo
1. NOT dark winglipo 07)
1. red, black and white feathhrss(93. 11)
1. male goldfinch the more brightly colored of the
2. white breast with brown spota(1 93)
2. bright red head and breast(844 05)
saxes, with n2)
3. NOT dark brown black color(1 25)
bright red head and nape(75 39)
flocks of other goldlinchess0 62)
yallow feet(1: 12)
bright red creacent below its beak (63.
often forming flacks with other goldinches(0
NOT long. gray wings 81)
White neck
black collar
bird tables feeders(0. 02)
Sum of other concepts: (5. (2)
chestnut red head and breastç60 52)
young goldfinches are drabber then adults, with brownish plumage; 02)
Sum of other concapts: (-172 32)
Sum of other concopts: (41.69)
LF-CBM (NEC=5)
LM4CV (NEC=5)
LaBo(
1. NOT dark winglips(1 80)
1. bright reddish brown head, crown and back of neck (344 38)
dlosely related to the goldfinch 83)
2. NOT long. blue-gray wings(0 69)
2. often forming flocks with other goldlichees( 16) 3.NOT from alaska and canada the
3. yellow feet(0 66)
bright yellow, green and blue plumage/07 41)
red faca(0.00
southwestem united states{1
Scarlet-red body(0.00)
bright yellow
forming flocks
goldfinches and
Sum of other concepts: (0.00 00.00)
with black bars (39 26)
similar small birds(0 17)
Brood tall that is shortor than other pelican species (:3,77)
5. NOT found in eastern and central united states
year round(0. 0.3
5. Mottled brown the nape, mantle, and scapularsi ::27 34)
Sum of other concepts: (2)
concepts: (0.00)
Figure G.3: Comparing baselines with different NECs
LF-CBM
LM4CV
LaBo
1. a yellow head[1190 1.96)
long. straight orange
.48)
1. is the only warbler with entirely(0 02)
2. NOT a red crest on the heed[o 99)
2. large, orange with black fip (100.00 07)
2. largast warbier in north americag
3. orange lega(0 99)
3. polntee orange bill (95 72)
3. plumage bright yellow(0. 02) largesi heaviest member the
4 yellow orange
5. bright orange breast(0 other concepts: (5.53)
80)
5. bright blue and orange plumage (76 97)
4. yellow and black plumage(84 85)
family(0..0 0.002)
Sum other concepts: (:257.00 .2)
yellow bird(0 02)
Sum of othar concapts: (42 29)
LF-CBM (NEC=5)
LM4CV (NEC=5)
LaBo(NEC=5)
1. yellow head(2 36)
bright throat, breast, fankk with black hars (320 88)
1. NOT sometimes called the "sea sparrow" due to its black and while plumage(t.10)
2. yellow
plumage(0.60) 60)
orange legs(0: 17)
2. bright yellow, green and blue
black ring around bill/o
plumago/205. 24)
head, chest, belly(0.
5. Glossy black winge/0 00)
3. bright reddish brown head, crown and back of neck (-89 49)
3. yellow maturity and wisdom 37)
thought
sign
Sum of other concapts: (0.00 00)
Broad that shortor pelican specles (-126 68)
4. orange in color(( 26)
5. series of high, thin "peaps" (0.15)
5. Mottled brown on the nape, mantle, and
Sum other concepts. (0.37)
scapulars 1551 665)
Sum of other concopts: (0.00)
Figure G.4: Comparing baselines with different NECs
22

---

H Further discussion on decision explanations
In this section, we further discuss some interesting phenomena observed in the decision explanations generated by different models.
H.1 Negative contributions
In Fig. [ we could that LM4CV generates negative contribution values, while LF- L5 and our VI.G-BMM do not. We hypothesize the reason different training methods: LM4CV trains dense final layer, hence the concepts irrelevant the class may provide negative contribution. LF-CBM and VLG-CBM, however, train a sparse final layer. To enforce sparsity, the model only captures relevant concepts for decision. Hence, it's natural to expect most contributions should be positive.
I Additional experiment results
L.1 Generalizability OOD datasets
In this section, we study question: will our VLGG-BM hurts the generalization ability of original model to MiitOO-DDitributionOOONN
dataset? To study this problem, we conduct experiment on Waterbirds dataset [19]. Waterbirds is an OOD dataset adapted from the CUB dataset, which combines bird photos from CUB with image backgrounds from Places365. We use the same ResNet model as we used in Table CUB dataset. For VLG-CBM, we choose NEC=5 and compare the results with the standard, non-interpretable models. On this dataset, the results are shown below: can be seen
Method
CUB Accuracy
Waterbirds Aceuracy
Standard model (black-box)
76. 70%
69.83%
VLG-CBM
75.79%
69.83%
Table I.1: Accuracy of VLG-CBM and standard blackbox model on CUB and Waterbirds datasets.
that our VLG-CBM generalizes well as the standard model does, which shows that VLG-CBM is competitive and has very small accuracy trade-off with the interpretability compared with the standard black-box model.
1.2
Ablation study
1.2.1
Ablation on augmentation probability
In this section, we conduct an ablation study on the probability of applying our crop-to cop-tto- data augmentation introduced in Section 3.1 The dataset used is the CUB dataset and the backbone is ResNet as we used in the main experiment. The results are listed below. From the table, we could see
Crop-to-Concept-Prb
ANEC-5
ANEC-avg
0.0
75.73
75.76
0.2
75.83
75.88
0.4
75.71
75.72
0.6
75.57
75.62
0.8
75.52
75.57
1.0
72,29
73.15
that the performance is best with augmentation probability 0.2.
Human study
In this section, we present human study following the practice of Oikarinen et al. 15 on Amazon MTurk platform. To briefly summarize, we show the annotator top-5 contributing concepts of our method (VLG-CBM) and bascline (LF-CBM LM4CV) and asking them which one is better.
23

---

Task
Expleratioe:
Sessuo imuge fillwiing mpop featume in eltoettrrdd:
dhooody 22d
2 Faly fur purii su Phi
ess)
teclage on comtaluue
6 Ofhire -3.0%
made or inookaano, ou nunder an tu auchu the cond sitbbtttt bocken))
image "sanding
Exp Poploration:
hollowinn
Bul for (440,55)
Fulrant eannt wwwinngtt-o 771
Sighty Moceco Eqaccyy
Moder Cenrry
Why? Select that apply
The masomablee ixplanation uar fuatree are relevenn the image.
rassnable roplaration informative.
MA son epp leatto equally ressonable
Figure J.1: An example of human study interface
The scores for each method are assigned as 1-5 according to the response of annotators: 5 for the explanations from VLG-CBM is strongly more reasonable, 4 for VLG-CBM is slightly more reasonable, 3 for both models are equally reasonable, 2 for the baseline is slightly more reasonable, and for the baseline is strongly more reasonable. Thus, if our model provides better explanations than the baselines, then we should see score higher than 3. We show an example screenshot of our study in Fig. J.]
We report the average score in Table J.1 for two baselines: LF-CBM and LM4CV. For each baseline, randomly sample 200 images and collect results from 3 different annotators. It can be seen that VLG-CBM has scores higher than 3 for both baselines, indicating our VLG-CBM provides better explanations than both baselines. LaBo is excluded in our experiment due its dense layer and large number of concepts: the top-5 concepts usually account for less than 0.01 % of final prediction.
Experiment
Score ( (VLG-CBM)
Score (Baseline)
ILG-CBM vs. LF-CBM
3.33 (1,54)
2.67 (1.54)
VLG-CBM LM4CV
3.38 (1.54)
2.62 (1.54)
Table J.1: Average Mturk score for our VLG-CBM and two baselines.
K
Visualizing VLG-CBM explanations
This section presents an extended version of Table 4 visualizing top-5 images for randomly picked
concepts for CUB and Places365 dataset. The results are shown Fig Fig K.I K.3 K.4 and K.5 for
Places365 and K.2
Fig K.6
Fig K.7 and Fig K.8 for CUB.
24

---

(a) Concept: Armrest
(b) Concept: Balance beam
(c) Concept: Barren Landscape
(d) Concept: Baseball
(e) Concept: Bookcase
(f) Concept:
counter
(g) Concept: Statue
(h) Concept: Spoon
(i) Concept: Used for grazing animals
(j) Concept: Pool Table
Golatii
(k) Concept: Shelves Sewing supplies
(I) Concept: Plastic and chairs
(m) Concept: Steering Wheel
(n) Concept: Vacation
F
(o) Concept: Barbershop
(p) Concept: Bank
Figure K.1: Top-5 activating images for randomly selected P6aces365 concepts
25

---

(a) Concept: Black and White head
(b) Concept: Black feathers
(c) Concept: Black legs
(d) Concept: Blue body
(c) Concept: Blue wings
(f) Concept: Brown cap
Concept: Brown tail
(h) Concept: Yellow bill with red spot
ii) Concept: Thin beak
(i) Concept: White body
(k) Concept: White stripes above the eycs
(I) Concept: Streaked breast
(m) Concept:
White underwings
(n) Concept: Small slender body
(0) Concept: Black mask on the face
(p) Concept: Small dark body
Figure K.2: Top-5 activating images for randomly selected CUB concepts
26

---

library, outdoor logit: bias:
front stops(poc)
14.24
stone brick walls(pos)
+2.77
lendscaping(pos)
work/neg)
place wash before prayer(pos)
+0
place to ta k(neg)
0
place to swim or cool offineg)
21/9 other features +0
Concept contributions
Pred class: cockpit
logit: 9.717 / bias: -0.879
computer screens(pos)
controls(pos)
+1.95
controls for the airplane(pos)
+1.69
Instruments for navigation(pos)
+1.5
flight engineer(posi
where the pilot sits(pos)
+0.77
(pptbsits(pos)
+0.4
Sum 2179 other features
0.0
0.5
1.0
1.5
2.0
Concept contr buttiibbtiins
Pred class: marsh
logit: 7.780 bias: 0.247
often murky or stagmant water(pos)
+2.53
reeds(pas)
wetland(pos)
+1.22
habitat(pas)
moving
+0.52
Lanoe(ppss)
+0.14
place k(neg)
Sum of 2179 other features
+0
0.0
0.5
1.0
1.5
2.0
2.5
Concapt contributions
logit: bias:
slides or other water features(pos)
+3.27
recreation arca(pos)
+3.19
pools water(pos)
sunshinellpos)
+1.33 Io
var ous attractions and rides(pos)
+0.17
space(pos)
+0.06
pool cover(pos) +0
Surr 2179 other features
3.5
Concept contributions
Figure K.3: Randomly selected explanations for Places365 (Part 1)
27

---

doorway,
logit:
large rectan opening(pos)
+3.35
exitposs)
+2.98
door handle(pos)
people gatherineg)
place to work(pes)
10
place to wash before prayer(pes)
place talk(pes)
Sum 2179 other features
+0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Concept
Pred class: veterinarians office
logit: 8.467 bias: -0.286
cat(pos)
+3.04
gerbillpas)
2.888
receptionist cesk(pos)
kennel area(pos)
+0.94
offices for the vets and sta ff(ps))))
++0.48
place where people gather(pos)
prayer(neg)
-0
Sum of 2179 other features
+0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Conceot contributions
Pred class: hockey
11.303
-0.857
nets either end(pos)
+5.21
lines marking out boundaries(pos)
puck(pos)
+2.38
rink(pos)
+0.66
dlace where people gather(neg)
before prayarineg)
place to talk(neg)
-0
Sum 2179 other features
Concept contributions
Pred cless: church, indoor
logit: 7.147 bias: -0.195
rows of pews(poc)
13.67
stained gless window(pos)
pews(pos)
+0.63 ceneeneee an
lectern(pos)
0..14
oulpit(pos)
prayer(pos)
place to k(neg)
-0
Sum of 2179 other features
0.0
1.0
1.5
2.0
2.5
3.0
3.5
Concept contributions
Figure K.4: Randomly selected explanations for Places365 (Part 2)
28

---

Pred class: home office logit: 9.178 bias: -0.053
bookshelies or cabinets(pos)
+7.32
phones(pos)
dud player(pos)
10.44
scanners(pas)
10.4
place to work(pos)
prayer(neg)
-0
olace tal ((pos)
+0
Sum of 2179 other tectures
6
8
Concept contributions
logit:
made of stone adobe(pos)
+2.37
found in north africa(pos)
+2.16
thick walls(pos)
+1.02
square(pos)
+0.96
minaret(pos)
+0.06
place to wash before prayer(pos)
+0
place k(neg)
Sum 2179 other features
+0
0.0
0.5
1.5 Concept contributions
1.0
2.0
2.5
Figure K.5; Randomly selected explanations for Places365 (Part 3)
29

---

eyed Vireo
logit:
grayish wings tall(pos)
+4.35
olive upperparts(pos)
+3.76
yellowish sides and flanks(pos)
+2.07
greenish upperparts(pos)
+0.47
while eye ring(pus)
+0.44
dark streaks breast(pos)
10
hard crystalline structure(pos)
10
Sum of 662 other festures
+0
1
2
3
4
Concept contributions
Indigo Bunting logit:
beak(pos)
+3.67
white streaks on the ing(lpos)
+3.44
short beak(pos)
+1.64
small bird(pas)
plurnage(pas)
+0.97
blue body(pps)
10.42
hard crystalline structure(pos)
+0
Sum of 662 other features
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Concept contributions
class: Leder Waxwing
logit: 8.305 bias: -0.027
pointy head(pos)
+2.61
small brown bodylpos;
+2.14
yellow tipped tail(pos)
+2.13
yellow tips on the (pogs(pos)
grey sidesipas]
++0
dark streaks on the breast[pos]
+0
Iridescent blue green back(pos)
+0
other features
0.0
0.5
1.3
1.5
2.5
Concept contributions
Pred clace: Gray crewned Racy Finch
logit: 6. 6.393 blas: -0.017
brown wings(pos)
+1.81
pinkish
+1.75
crown(pos)
1.66
rosy breast(pos)
dark streaks the breast(pos)
+0
hard crystalline structure(pos)
+0
grayish head(pos)
Sum of 662 other foatures
+0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Concept
bbttions
Figure K.6: Randomly selected explanations for CUB (Part 1)
30

---

Flycatcher logit: bias:
eye stripe(pos)
+2.2
olive (pperparts(pos)
+1.71
yellowish underparts(pas)
+1.55
pointed bill(pos)
+0.89
white wing bars(pos)
gray apperparb(puo))
+0.04
hard crystalline structure(neg)
0
Sum of 662 other features
10
0.0
0.5
1.0
1.5
2.0
2.5
Grng LesieyCCLO
Concept
Pred clare: Painted Bunting
logit: 13.865 bias 0.022
shart painted beak(pos)
head(pos)
green hack(pos)
+1.76
short stout bilsp
+1.15
small songbirci(uss)
+00663
grey sides(pos)
dark streaks on the breast(pos)
+0
Concept contributions
Pred class: Northern Flicker logit: 8.439 bias: -0.009
red breast males onlylpos)
+2.94
brownish red breast(pos)
+22.58
stripes/pos)
black white bars(pos)
+0.89
white rump(pas)
10.42
grey sides(pos)
+0
Iridescent blue green back(pos)
Sum of other teatures
-0
1.0
1.5
Concept co nttibbtiions
Pred class: Cape Glossy Starling
logit: 10.22 10.223 blas: -0.039
glossy black plumagelpos)
glossy sheen to feathers(pos)
+2. 64
small black birdipos)
+1.6
white spots on wingstpos)
pointy beak(pos)
+1.41
lang pointed tail(pos)
+0.18
crystalline structure(pos)
+0
Sum of 662 other features
+0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Concept contributions
Figure K.7: Randomly selected explanations for CUB (Part 2)
31

---

Pred ny Cowbird
log 5.787
conicca bill(pos)
+2.07
shiny ny black plumage(pos)
+2.46
small bill(pos)
Iridescent green back(neg)
hard crystalline structure(neg)
0
gr greyish brown head((eg))
black sltipe down neck(nep)
features
+0
0.5
1.5
2.0
2.5
3.5
Concept contributions
Predi rlacs: Homed Grehn
logit: 7.186 bias: 00.011
long sharp bpol(pos)
+3.75
small compact body(pos)
small pointed (pos)
winus(pos)
+0.7
black Lai (pos)
+0.12
long thin neck(pos)
+0.08
dark streaks on the breast(pos)
+0
Sum of 662 other features
0.0
1.0
2.0
2.5
3.0
3.5
Concept contributions
Figure K.8: Randomly selected explanations for CUB (Part 3)
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our introduction in Section summarizes the contribution and scope of this paper accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

We provide full set of assumptions in our Theorem 4. and we provide complete and correct proof in Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
we present the settings of our experiments. We present more implementation details on Appendix
Justification: Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The link to the project webpage is provided in the abstract and the code will be released by poster deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Soaaad/ss//oeeSSummssssiooooee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include experimental details in Section 5 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments are very computationally expensive to repeat and measure the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
35

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational resources in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research is conducted following NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focus on technical development of making neural network models more interpretable.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
36

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite the dataset and models we used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
37

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "uPxzY591nZ", "reviewer": "Reviewer_hsJW", "summary": "This paper proposed a framework that leverages existing grounding models to refine the accurate concept annotations, and therefore enables better CBM training. Also, a new metric is proposed to avoid leakage and improve interoperability. Experiments and visualizations are conducted to demonstrate the superior performance of the proposed method.", "questions": "In summary, my concerns mainly focus on the lack of comprehensive comparison, ablation, and analysis.", "limitations": "Yes, it is discussed in Section 6.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. For methodology, incorporating a grounding model to get faithful concepts, and proposing a metric - Number of Effective Concepts to control the information leak and supervise sparse layer optimization are interesting and novel to a certain degree in this field.\n\n2. The proposed method outperforms previous approaches in multiple widely used benchmarks.", "weaknesses": "1. Missing comparison with recent works in CMB, for example,  [1] and [2].\n\n2. How good is LLM at generating candidate concepts, and how the different LLMs' quality would impact the generated candidate quality as well as final performance? An ablation/analysis is needed.\n\n3. How good is the grounding model in terms of object class matching and box location prediction? The ablation of different grounding models and even a random baseline should be added. I am also curious about the training augmentation based on bounding box, how much gain it can bring?\n\n4. It's natural that different classes have different numbers of concepts to distinguish the class because some are complex and some are simple. If forcing the NEC to certain number, would the model automatically learn to assign different numbers of concepts to different classes? An analysis is needed.\n\nRefs:\\\n[1]. Pham, Thang M., et al. \"PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck.\" arXiv preprint arXiv:2403.05297 (2024). \\\n[2]. Sun, Ao, et al. \"Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning.\" arXiv preprint arXiv:2402.05945 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hsJW,\n\nThank you for the positive feedback! Please see our responses below to address your comments. \n\n**Q1**. Missing comparison with recent works in CMB, for example, [1] and [2]\n\n**A1**: Thank you for providing the reference [1, 2]. Below we discuss the differences between our method and [1] and [2]\n- [1]: Similar to us, [1] also uses an open-vocabulary object detection model to provide an explainable decision. However, their model is directly adapted from an OWL-ViT model, while our VLG-CBM uses an open-vocabulary object detection model to train a CBL over any base model, providing more flexibility. Additionally, their model requires pretraining to get best performance, while our VLG-CBM could be applied post-hoc to any pretrained model.\n- [2]: This paper also aims at eliminating the information leakage. They evaluate it by measuring the performance drop speed after removing top-contributing concepts, which can be controlled by our proposed NEC metric. This is because the performance should reach minimum after removing all contributing concepts. \n\nWe will cite the papers [1,2] and discuss the differences in the revised manuscript.\n\n[1]. Pham, Thang M., et al. \"PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck.\" arXiv preprint arXiv:2403.05297 (2024).\n\n[2]. Sun, Ao, et al. \"Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning.\" arXiv preprint arXiv:2402.05945 (2024)\n\n---\n\n**Q2**: How good is LLM at generating candidate concepts, and how the different LLMs' quality would impact the generated candidate quality as well as final performance?\n\n**A2**: We would like to clarify that the focus of this work is on proposing a novel CBM training pipeline by grounding concepts with open-domain object detectors and not on obtaining the concept set. As mentioned in Ln 110-112, we use the concept set from LF-CBM for training VLG-CBM and this could be replaced with concept sets obtained from other methods. \t\n\n----\n\n**Q3**: How good is the grounding model in terms of object class matching and box location prediction? \n\n**A3**: In the Appendix D, we have provided a study evaluating concept annotations obtained from Grounding-DINO. We use CUB dataset for comparison which contains ground-truth for fine-grained concepts present in each image and report precision and recall metric to measure the quality of annotations from Grounding-DINO. As demonstrated in the appendix along with Table 2, Fig 4, Fig G.1 and G.2, the effect of false-positives is minimal and VLG-CBM is able to faithfully represent concepts in the CBL. \n\n---\n\n**Q4**: The ablation of different grounding models and even a random baseline should be added. \n\n**A4**: Following your suggestion, we have conducted additional experiments and provided an ablation study with two different versions of GroundingDINO: Swin-T and Swin-B, where Swin-B is a larger model and has better open-vocabulary detection performance. We observe an increase in accuracy when using Swin-B for both NEC=5 and avg accuracy metric. This shows the potential of our method to scale as the performance of open-domain object detectors continues to improve. \n\nFor the random baseline, we have provided results in Table 2 in the original draft by starting with a randomly initialized CBL layer and training a linear layer with our sparsity constraints. This would be equivalent to randomly assigning concepts from the concept set to each image and training the final layer.\n\n| DINO-Model | Acc@NEC=5 | Avg. Acc |\n|------------|-----------|----------|\n| Swin-T     | 75.30     | 75.54    |\n| Swin-B     | 75.79     | 75.82    |\n\n\n---\n\n**Q5**: Gains with training augmentation based on bounding box\n\n**A5**: Following your request, we reported our results in the Global Response #2.1.\n\n---\n\n**Q6**: If forcing the NEC to certain number, would the model automatically learn to assign different numbers of concepts to different classes? An analysis is needed.\n\n**A6**: In Appendix E, we have shown the distribution of non-zero weight on CUB and Places365 models. As the Fig E.1 shows, our method automatically learns to use a different number of concepts for classifying different classes.\n\n---\n\n**Summary**\n\nIn summary, \n- In **Q1**, we discussed and compared with recent works [1] and [2]\n- In **Q2**, we answered the question on concept set generation. \n- In **Q3**, we discussed the effect of grounding models on concept annotations \n- In **Q4**, we provided an ablation study on the grounding model used in VLG-CBM. \n- In **Q5**, we added an ablation study to study the impact of our augmentation probability.\n- In **Q6**, we answered the question on the distribution of non-zero weights between different classes. \n\nIn response to the weakness part, we have addressed the reviewer’s concerns \n- in weakness #1 by providing a comparison in **Q1**.\n- in weakness #2 in **Q2**\n- in weakness #3 by adding ablation study on object detection models and augmentation based on bounding box in **Q3** and **Q4** \n- in weakness #4 in **Q6**\n\nWe believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RccrtOG6f8", "reviewer": "Reviewer_UnrN", "summary": "The paper introduces Vision-Language-Guided Concept Bottleneck Model (VLG-CBM), an innovative approach to training Concept Bottleneck Models (CBMs) using vision-language models (VLMs). This method aims to improve the faithfulness and performance of CBMs by addressing the limitations of existing models, specifically inaccurate concept predictions and information leakage. VLG-CBM leverages grounded object detectors for visually recognizable concept annotations and introduces a new metric, the Number of Effective Concepts (NEC), to control information leakage and enhance interpretability. The method demonstrates significant performance improvements across multiple benchmarks.", "questions": "see weaknesses", "limitations": "see weaknesses", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well written with a clear logical flow.\n\n2) The approach automates the creation of concept datasets using grounded object detectors, which means there's no need for manual annotations. This saves time and effort, making the method easier to scale up for larger datasets. Plus, it reduces the chances of human error in the data, leading to more reliable training results.\n\n3) The paper provides theoretical explanations for information leakage in Concept Bottleneck Models (CBMs). By introducing the Number of Effective Concepts (NEC) metric, the authors offer a way to control and measure information leakage, enhancing the overall integrity and interpretability of the model.\n\n3) The VLG-CBM model shows significant improvements in accuracy across five standard benchmarks, with gains ranging from 2.4% to 7.6% over current methods.", "weaknesses": "1）The proposed pipeline integrates several independent models, ie, LLM for concept candidate generation, and GroundingDINO for bounding box detection. I'm wondering how well the grounding detector can do in discovering the concept candidate.  And if the detector fails to detect a key concept, will the final recognition result be affected? \n\n2) Dependence on Pre-trained Models: The reliance on pre-trained VLMs and object detectors may limit the approach's applicability to domains where such models are not readily available or effective. \n\n3) My major concern is in generalization. While the method shows impressive results on standard image recognition benchmarks, its effectiveness in more diverse or less-structured real-world scenarios needs further validation. For example, how would the VLG-CBM perform for OOD image recognition like NICO or Water-Bird, or how could it handle more complex tasks like VQA?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UnrN,\n\nThank you for the positive feedback! Please see our responses below to address your comments. \n\n---\n**Q1:** How well the grounding detector can do in discovering the concept candidate. And if the detector fails to detect a key concept, will the final recognition result be affected?\n\n**A1:** In Appendix D, we provided a study evaluating concept annotations obtained from Grounding-DINO. We use CUB dataset for comparison which contains ground-truth for fine-grained concepts present in each image and report precision and recall metric to measure the quality of annotations from Grounding-DINO.\n\nThe confidence threshold controls when a concept will be used in training the model. It is possible that a concept is present with a confidence score below threshold in an image and will  not be used during training. Please see the ablation study for this quantity in Appendix C.1. \n\n---\n\n**Q2:** The reliance on pre-trained VLMs and object detectors may limit the approach's applicability to domains where such models are not readily available or effective.\n\n\n**A2:** We would like to clarify that this is a general drawback with the use of foundational models and most existing CBMs [1-3] use foundation models to remove the need for human annotation in various phases of the CBM pipeline. For example, LF-CBM[1] uses CLIP model for assigning score to concepts, and existing methods like LaBO[2], LM4CV[3], and LF-CBM use LLMs for generating concept sets.\n\n---\n\n**Q3:** How would the VLG-CBM perform for OOD image recognition like NICO or Water-Bird, or how could it handle more complex tasks like VQA?\n\n**A3:** Following your suggestion, we have conducted additional experiments to evaluate how VLG-CBM performs in OOD datasets. We trained our VLG-CBM on the CUB dataset and tested it on the Water-Bird dataset to evaluate the OOD performance. The Water-bird dataset is constructed by cropping out birds from the CUB dataset and transferring them onto backgrounds from the Places dataset. We compare the performance of VLG-CBM with the standard black-box model trained on the CUB dataset. It can be seen that our VLG-CBM generalizes well as the standard model does, which shows that our VLG-CBM is competitive and has very small accuracy trade-off with the interpretability compared with the standard black-box model. \n\n| Method                     | CUB-Acc | waterbird-Acc |\n|----------------------------|---------|---------------|\n| Standard model (black-box) |  76.70% |        69.83% |\n| VLG-CBM                    |  75.79% |        69.83% |\n\nFor the question on more complex tasks like VQA, it is a very different task than the image classification task that current CBMs [1-3] are designed for, as a typical model for VQA task would involve the image embedding and question embeddings. Nevertheless, we think that our VLG-CBM can be potentially useful to convert the non-interpretable image embedding to an interpretable embeddings through a concept bottleneck layer. We will include this interesting question to future work and add a discussion in the revised draft. \n\n---\n\n**Reference**\n\n[1] Oikarinen etal. Label-free concept bottleneck models. ICLR 23\n\n[2] Yang etal. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. CVPR 23. \n\n[3] Yan etal. Learning concise and descriptive attributes for visual recognition. ICCV 23.\n\n---\n\n**Summary**\n\nIn summary, \n\n* In **Q1**, we added discussion on the performance of the grounding detector. \n\n* In **Q2**, we respond to the concern on reliance on pretrained VLMs.\n\n* In **Q3**, we conducted an extensive study on the waterbird dataset to show generalizability of our model to OOD datasets. \n\nIn response to the weakness part, we have addressed the reviewer’s concern\n\n* in weakness #1 in **Q1**.\n\n* in weakness #2 by comparing with other baselines in **Q2**.\n\n* in weakness #3 by adding an extensive study on the waterbird dataset in **Q3**.\n\nWe believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAGmdogKvF", "reviewer": "Reviewer_QueL", "summary": "This work proposed a new way to implement concept bottleneck models (CBMs), which use pretrained object detectors (Grounding-DINO) to filter and annotate the concepts. This step can make the concepts more visual and groundable, which improves the reliability of the concept predictor. This paper proposed a new metric (NEC) to control the number of concepts that the model uses when making the predictions. In the experiments of 5 datasets, the proposed model shows better performance using the NEC metric.", "questions": "How is the threshold of detection confidence (T=0.15) chosen? Can this generalize to different datasets?", "limitations": "The authors discussed the limitations in Section 6.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of using the grounding model can help improve concepts' groundability. The paper is easy to follow in general.", "weaknesses": "1. **Limited contribution.** This paper is an incremental work to previous LLM-guided CBMs [1, 2] by adding a step to filter concepts with an object detection model. The findings that \"CBM with random concepts can achieve good performance\" have also been shown in previous works [3, 4]. From the technical part, the only difference between the proposed model and prior CBMs is the object detection model for concept filtering. However, the detected bounding boxes are not used in the model training and inference. Therefore, the technical contribution is also limited. Overall, this paper makes limited contributions to the community and does not provide enough insights for future work.\n\n2. **Generalizability of the method.** The proposed method relies on an open-vocab detection model. However, open-vocab detection is still a very challenging task, and from my experience, the off-the-shelf models do not work well in many cases. For example, part-based detection is very difficult. In some datasets like DTD or action detection, the concepts are hard to capture in bounding boxes. Even in the evaluated datasets in this paper, like CIFAR, which is in low resolution, the performance of object detection is also quite skeptical. Although the authors have mentioned this limitation in the last section, I don't agree with their arguments that \"prior work (e.g., LaBo, LM4CV, LF-CBM) also shared similar limitations on the reliance of CLIP\". CLIP definitely has better generalization than the open-vocab detection model because the latter is a more challenging task. This concern is also reflected by the small set of datasets in their experiments, only 5 datasets, compared to what LaBo evaluated (11 datasets).\n\n3. **Baseline Comparison.** The linear probing baseline is missing, which is necessary to justify the effectiveness of CBMs. For CUB, Place365, and ImageNet, the numbers of LM4CV and LaBo are missing. The reason shown in the caption that \"they could not be applied on non-CLIP backbones\" cannot convince me. I didn't see any difficulties in letting your model support CLIP. You can simply fine-tune CLIP using BCE loss in Eq (6) or just use CLIP as a feature extractor and learn a linear layer over it. In general, the numbers shown in Table 2 are quite low, far behind linear probing based on my experience. It seems the proposed method sacrifices a lot of performance for interpretability, which is not validated properly (see weakness 4). I feel that the performance gain of the proposed method comes from the priors of the object detection model.\n\n4. **Human evaluation to justify faithfulness.** This paper claims that the proposed VLG-CBM is more interpretable and faithful. However, without concrete evidence like large-scale human evaluation, it is hard to justify VLG-CBM as more interpretable by just showing a few qualitative examples.\n\n[1] Yang et al. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023.\n[2] Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR 2023.\n[3] Roth et al. Waffling around for Performance: Visual Classification with Random Words and Broad Concepts. ICCV 2023.\n[4] Yan et al. Learning Concise and Descriptive Attributes for Visual Recognition. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QueL,\n\nThank you for the feedback, we believe there are some misunderstandings based on the comments. Please allow us to clarify below to address your comments. \n\n**Q1:** This paper is an incremental work to previous LLM-guided CBMs [1, 2] by adding a step to filter concepts with an object detection model.\n\n**A1:** We would like to clarify our contributions below and demonstrate that our contribution is much more than just filtering the concepts with object detection models. As described in the end of Sec 1, we have 3 main contributions spanning from a new methodology in sec 3, rigorous theoretical analysis in sec 4, to extensive evaluation in sec 5:\n1. We are the first to propose an end-to-end pipeline for training CBM with annotations from grounding DINO and provide a method to utilize bounding boxes information obtained from the Grounding DINO, as shown in Sec 3. \n2. As demonstrated and discussed in Sec 4, our work is the first to provide rigorous theoretical analysis for information leakage in CBMs and demonstrate that CBM can achieve optimal accuracy even on random concepts when the number of concepts is large enough\n3. As discussed in Sec 4, we propose a new metric called the Number of Effective Concepts (NEC), which facilitates fair comparison between different CBMs by controling information leakage and helps improve interpretability of the CBMs.\n---\n**Q2:** The findings that \"CBM with random concepts can achieve good performance\" have also been shown in previous works [3, 4]...\n\n**A2:** CBM with random concepts can achieve good performance as has been observed before, as you point out. However, we are the first to provide a theoretical analysis for it. As we prove in Thm. 4.1, The random concept CBL can approximate any linear function with approximation error that goes down linearly with the number of concepts, and reaches zero error when the number of concepts gets to the dimension of embedding. \n\n---\n**Q3:** However, the detected bounding boxes are not used in the model training and inference. Therefore, the technical contribution is also limited…\n\n**A3:** We would like to clarify that the detected bounding boxes are indeed used during model training. As mentioned in Ln 134-136, we augment the training dataset by cropping images to a randomly selected bounding box and modifying the target one-hot vector to predict the concept corresponding to the bounding box.\n\n---\n\n**Q4:** The proposed method relies on an open-vocab detection model. However, open-vocab detection is still a very challenging task, and from my experience, the off-the-shelf models do not work well in many cases. For example, …\n\n**A4:** We would like to highlight that the open-vocabulary object detectors primarily suffer from false-positive detections, and we address this issue in the following way:\n\n* Rather than using the entire concept set for detecting concepts in an image, we limit the concepts relevant to the ground truth class of the image. This ensures that the concepts have a high likelihood of being present in the image.\n\n* We used confidence threshold to filter bounding boxes with low confidence. In the Appendix D, we have evaluated concept annotations at different thresholds. We use CUB dataset for comparison which contains ground-truth for fine-grained concepts present in each image and report precision and recall metric to measure the quality of annotations from Grounding-DINO. As demonstrated in the appendix along with Table 2, Fig 4, Fig G.1 and G.2, the effect of false-positives is minimal and VLG-CBM is able to faithfully represent concepts in the CBL. \n\n---\n\n**Q5:** This concern is also reflected by the small set of datasets in their experiments, only 5 datasets, compared to what LaBo evaluated (11 datasets).\n\n**A5:** Results on Flower102 and Food101: The Food-101 dataset achieved an ACC@NEC=5 of 81.68% and an Avg. Acc of 80.31%, while the Flower-101 dataset achieved higher scores with an ACC@NEC=5 of 90.58% and an Avg. Acc of 92.94% with CLIP-Resnet50 backbone.\n\n---\n\n**Q6:**  The linear probing baseline is missing, which is necessary to justify the effectiveness of CBMs.\n\n**A6:** The linear probing results for Table 2 models are: \n88.80% for CIFAR10, 70.10% for CIFAR100, 76.70% for CUB, 48.56% for Places and 76.13% for ImageNet. The gaps between our models (avg. acc) to the linear probing is 0.17% for CIFAR10, 3.62% for CIFAR100, 0.88% for CUB, 5.99% for Places and 2.15% for ImageNets. \n\n---\n\n\n**Q7:** For CUB, Place365, and ImageNet, the numbers of LM4CV and LaBo are missing. \n\n**A7:** Following your request, we reported our results in the **Global Response #1 Additional models and datasets.** It can be seen that our VLG-CBM still outperforms the baselines under this setting.\n\n---\n\n**Q8:** In general, the numbers shown in Table 2 are quite low, far behind linear probing based on my experience. It seems the proposed method sacrifices a lot of performance for interpretability, which is not validated properly (see weakness 4)...\n\n**A8:** First, we want to clarify that our accuracy is not far behind linear probing baseline, only 0-6% gap (refer to the numbers in A6) while other baselines have much larger gaps . In Table 2, we control NEC to be small (NEC=5) mainly for fairly compare different models: as shown in Figure 3, under dense setting (linear probing) even random CBL could achieve comparable performance to SOTA CBMs, which largely weaken its usefulness in comparing different CBMs.  In practice, users have the flexibility to choose the NEC to trade-off between performance and interpretability. \n\n---\n\n**Q9:** …without concrete evidence like large-scale human evaluation, it is hard to justify VLG-CBM as more interpretable by just showing a few qualitative examples.\n\n**A9:** Following your suggestion, we have conducted additional large-scale human evaluations and demonstrated that our method outperformed the baselines. Please see the results in **Global Response #3 Human study.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7M7ixSdzIo", "reviewer": "Reviewer_MvHs", "summary": "The paper uses foundational models to automate the generation of concept annotations which are then used to train a concept bottleneck model (CBM). To generate the concept set, the paper extends prior scalable CBM approaches[1] by using open-domain object detectors to weed out concepts that are not visually grounded. Concepts that are not present in any bounding-box for any training image are removed. Following existing work,  a CBM is trained sequentially – first, the concept layer is trained and frozen, then a sparse linear layer is trained. The paper introduces a theoretically justified metric, Number of effective concepts (NEC), which is the average number of concepts the sparse final layer uses to make the class prediction. NEC can be used to compare between various CBMs. Results are shown on 5 standard datasets, and they compare against 3 scalable CBM baselines.\n\n[1] Oikarinen, Tuomas P. et al. “Label-Free Concept Bottleneck Models.” ArXiv abs/2304.06129 (2023): n. Pag.", "questions": "1. A study of the impact of proposed visually-grounded concept filtering would be interesting. For ex. What fraction of quantitative gains are because of the removed concept set? How much does augmentation contribute to overall performance gains?  \n2. As mentioned in the weaknesses section, a section on time complexity of concept set generation would be useful.\n3. The theoretical analysis, though appreciated, seems out of place. The result presented in Thm 4.1 is not specific to concept models, but can be seen as a general statement on fidelity of performing dimensionality reduction with a linear random projection. It is not clear to me how NEC follows from here, as NEC is a measure of the sparsity of the final layer, which can be low even for large concept sets.", "limitations": "Yes, limitations have been addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "\\+ The paper is written well and easy to follow.  \n\\+ The bounding boxes generated from open-domain object detectors are used to i) filter out concepts which are not visually grounded ii) augment training of the concept layer by cropping images to bounding-boxes\nBoth these design decisions seem crucial for learning a better, more interpretable concept layer.  I would find it interesting if this was studied in greater depth.  \n\\+ The paper quantifies existing thought that having a dense prediction layer limits the applicability of CBMs. NEC can be used to fairly compare across CBMs.  \n\\+ Empirical results are convincing. Their CBM outperforms existing baselines on classification accuracy across various levels of final layer sparsity.", "weaknesses": "\\- It is not clear if NEC controls information leakage. As defined in [2], information leakage happens because of a `soft’ concept layer (that predicts probability of a concept instead of presence and absence). Though a small NEC reduces the possibility of information leakage and is a useful metric to check the goodness of the concept set.  \n\\- A study of time and compute complexity is needed, as every image has to be passed through a large object-detector. The effort to generate concepts is much higher (compared to [1], who generate concepts class wise).  \n\n[1] Oikarinen, Tuomas P. et al. “Label-Free Concept Bottleneck Models.” ArXiv abs/2304.06129 (2023): n. Pag.\n[2] Mahinpei, Anita et al. “Promises and Pitfalls of Black-Box Concept Learning Models.” ArXiv abs/2106.13314 (2021): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer MvHs,\n\nThank you for the positive feedback. Please see below our responses to address your comments. \n\n**Q1:** It is not clear if NEC controls information leakage. As defined in [2], information leakage happens because of a `soft’ concept layer (that predicts probability of a concept instead of presence and absence). Though a small NEC reduces the possibility of information leakage and is a useful metric to check the goodness of the concept set.\n\n**A1:** Information leakage is a problem that haunts CBM models. As discussed in Sec 4 of [2], not only soft CBL has this problem, using hard concepts still shows information leakage, suggesting it’s hard to fully eliminate it in CBM. Though the reason for this has not been fully understood, we present Thm. 4.1 to provide a theoretical understanding of this problem. Additionally, in order to understand the real semantic information learned by CBL, we compare model performance with a random baseline as a “control group”, where no semantic information is learned. Our results in Figure 3 show that this random baseline drops significantly when NEC is low. This suggests that reducing NEC is a way to control information leakage. \n\n---\n\n**Q2:** A study of time and compute complexity is needed, as every image has to be passed through a large object-detector. The effort to generate concepts is much higher (compared to [1], who generate concepts class wise).\n\n**A2:** Following your suggestion, we provide a comparative analysis of GPU-hours required to obtain annotations for three datasets: CUB, ImageNet, and Places365 in the table below. We would also like to clarify that we use concepts relevant to the class of an image when using Grounding-DINO. This helps reduce the false-positive rate of the open-vocabulary detection model and significantly decrease computation time. Please refer to Appendix D for more details.\n \n| Dataset           | GPU-hours |\n|-------------------|-----------|\n| CUB               | 0.5       |\n| ImageNet          | 110       |\n| Places365         | 138       |\n\n---\n\n**Q3:** A study of the impact of proposed visually-grounded concept filtering would be interesting. For ex. What fraction of quantitative gains are because of the removed concept set? How much does augmentation contribute to overall performance gains?\n\n**A3:** We would like to clarify that concept filtering is a natural step in our training pipeline. An ablation here may not change the performance as it would be equivalent to training in a multi-label setting with certain labels never appearing in the training dataset.\n\nFurther, following your request, we reported our provided an ablation study on data augmentation in the **Global Response #2.1.**\n\n---\n\n**Q4:** The theoretical analysis, though appreciated, seems out of place. The result presented in Thm 4.1 is not specific to concept models, but can be seen as a general statement on fidelity of performing dimensionality reduction with a linear random projection. It is not clear to me how NEC follows from here, as NEC is a measure of the sparsity of the final layer, which can be low even for large concept sets.\n\n**A4:** Thank you for the comments, please allow us to clarify. Thm. 4.1 shows that the approximation error goes down linearly with the number of concepts, when CBL weights are randomly selected. This supports previous observations that a random CBL layer could also achieve good performance, suggesting the existence of information leakage. By applying NEC constraint, we actually constraint the number of concepts in Thm 4.1 (as other concepts do not contribute to the decision and could be ignored). Thus, the information leakage could be controlled as Thm. 4.1 suggested.\n\n ---\n\n**Reference**\n\n[1] Oikarinen et al. “Label-Free Concept Bottleneck Models.” ICLR 2023\n\n[2] Mahinpei et al. “Promises and Pitfalls of Black-Box Concept Learning Models.” 2021\n\n---\n\n**Summary**\n\nIn summary,  \n\n* In **Q1**, we discuss why controlling NEC could help control information leakage\n\n* In **Q2**, we provide a study on computational time and complexity. \n\n* In **Q3**, we provide an ablation study on the contribution of our data augmentation.\n\n* In **Q4**, we discuss how our theoretical result is connected to the NEC. \n\nIn response to the weakness part, we have addressed the reviewer’s concern\n\n* in weakness #1 in **Q1**.\n\n* in weakness #2 by providing an analysis on computational time and complexity in **Q2**\n\nWe believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfwJxU5EPv", "reviewer": "Reviewer_i5YZ", "summary": "In this paper, the authors propose a learning pipeline named VLG-CBM, which enhances CBM with the aid of open-domain object detectors to create more accurate concept labels, while also excluding image-irrelevant concepts (e.g,. \"loud music\").\nAlso, in order to prevent the 'information leak' in CBMs, which stands for a phenomenon where the exploitation of irrelevant concepts may help downstream accuracy, authors propose a metric named Number of Effective Concepts (NEC) which enables the fair comparison of different CBMs within a similar amount of information being provided.\nExperimental results show that with the proposed VLM-CBM, high-quality concept labels are obtained therefore achieving better performances within similar NEC.", "questions": "1. Questions regarding Table 3. 1) How is the performance of each method? Only the percentage of changed decisions are reported, therefore providing accuracies may help a better understanding of the effectiveness of each method. 2) Isn't it obvious that the percentage of changed decisions is lower in VLG-CBM since the sparsity constraint is not applied to other methods? Can you also provide the same analysis with sparsity constraint applied to other methods?\n\n2. Providing qualitative examples of labeling results using Grounding-DINO would help in understanding the quality of the auto-labeled dataset, which seems to be absent in the current version of the paper.\n\n3. Also, please address the concerns elaborated in the weakness section.", "limitations": "Authors have described limitations and possible societal impacts in the main paper and the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of applying a grounding model to generate an accurate concept label is sound and seems effective as shown in the experimental results.\n\n2. The theoretical analysis of random CBM well explains the problem of information leaks in CBMs, and the proposed NEC is a proper method to measure the information being provided to a CBM classifier, enabling fair comparison of CBMs within a similar amount of information being provided.\n\n3. The paper is generally well-written, making it easy to follow the main ideas and claims.", "weaknesses": "1. Only experimental results under controlled NEC are reported, making it difficult to compare the performance of VLG-CBMs with previous works in conventional settings. For instance, the average accuracy of LaBo in the CIFAR-100 dataset under an NEC-controlled setting is reported as 55.18%, while the full performance in the original paper is around 86% (86.82 in the dev set, 86.04 in the test set). Therefore, also comparing the performance of VLG-CBM under conventional settings in previous work would make the comparison easier.\n\n2. The idea of applying a detection or a grounding module to generate an image annotation is already widely explored (e.g., KOSMOS-2 [1]). The proposed data generation process does not significantly differ from previous approaches, limiting the originality of the proposed method.\n\n\n[1] Peng et al., KOSMOS-2: Grounding Multimodal Large Language Models to the World.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer i5YZ,\n\nThank you for your feedback. Below are responses to your comments. \n\n**Q1:** Therefore, also comparing the performance of VLG-CBM under conventional settings in previous work would make the comparison easier.\n\n**A1:** In this paper we do not compare the results under a dense final layer setting mainly due to this setting is less informative: as shown in Fig. 3 of our draft, even random baseline achieve similar performance to SOTA CBM models, when NEC becomes large enough.\n\n---\n\n**Q2:** The idea of applying a detection or a grounding module to generate an image annotation is already widely explored (e.g., KOSMOS-2 [1]). The proposed data generation process does not significantly differ from previous approaches, limiting the originality of the proposed method.\n\n**A2:** We would like to clarify our contribution in regards to grounding concepts for training CBMs. CBMs rely on LLMs for a concept set which usually provide generic concepts and might not be possible to visualize in images. We are the first to use open-vocabulary grounding models to improve the effectiveness of CBMs. \n\n---\n\n**Q3:** How is the performance of each method? Only the percentage of changed decisions are reported, therefore providing accuracies may help a better understanding of the effectiveness of each method\n\n**A3:** We calculate the performance of each method:\n\n\n|        | Full   | Truncated |\n|--------|--------|-----------|\n| LF-CBM | 73.92% |    44.80% |\n| LM4CV  | 43.36% |     0.01% |\n| LaBo   | 81.70% |    17.53% |\n\nWe could see here most methods suffer significant performance drops. LF-CBM has a smaller gap, partially due to its sparse final layer. \n\n---\n\n**Q4:** Isn't it obvious that the percentage of changed decisions is lower in VLG-CBM since the sparsity constraint is not applied to other methods? Can you also provide the same analysis with sparsity constraint applied to other methods?\n\n**A4:** Yes, what we want to illustrate here is to emphasize the importance of NEC control. A quite popular method to explain model decisions for CBMs is to show the users the top-5 contributing concepts. However, we found this approach has potential risk without NEC control: as shown in Tab. 3, if limited to only top-5 concepts, the decision of the model will change significantly. This suggests that showing top-5 concepts does not faithfully explain the model behavior. \n\nTo better illustrate this, we choose a baseline, LF-CBM and apply our NEC control on it. The below table shows that controlling NEC effectively reduces the decision change. \n\n| NEC                         | Change percentage |\n|-----------------------------|-------------------|\n|                           5 |            15.77% |\n|                          10 |            35.39% |\n|                          15 |            37.70% |\n| original LF-CBM (NEC=31.52) |            40.79% |\n\n---\n\n**Q5:** Providing qualitative examples of labeling results using Grounding-DINO would help in understanding the quality of the auto-labeled dataset, which seems to be absent in the current version of the paper.\n\n**A5:** We have provided few examples of annotation obtained from GroundingDINO in **global response supplementary pdf**.\n\n---\n\n**Summary**\n\nIn summary, we have\n\n* In **Q1**, we discuss why conventional benchmark is not that informative.\n\n* In **Q2**, we clarify our difference with previous work. \n\n* In **Q3**, we provide the performance change of each method in Table. 3.\n\n* In **Q4**, we discuss how NEC constraint affect the decision changes and add LF-CBM experiments to support it. \n\n* In **Q5**, we provide a few examples from GroundingDINO.\n\nIn response to the weakness part, we have\n\n* addressed the reviewer’s concern in weakness #1 in **Q1**.\n\n* addressed the reviewer’s concern in weakness #2 by clarifying our difference with previous work in **Q2**.\n\n\nWe believe that we have addressed all your concerns. Please let us know if you still have any reservations and we would be happy to address them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General response: New Experiments**

Thank you for all the thoughtful reviews. In response, we have performed many experiments during the rebuttal period as requested by the reviewers, mainly to evaluate our method on more model architectures and datasets, as well as several ablation studies to understand the importance of each part in our pipeline.

1. **Additional models and datasets**: 

- **We add CLIP-RN50 backbone results to compare with LaBo and LM4CV in CUB and ImageNet datasets**. 
The results show that our method consistently outperformed both LaBo and LM4CV.  
Table R1: Performance comparison of CLIP-RN50 backbone on all baselines  
  | Method       | ImageNet  | ImageNet  | CUB    | CUB        |
  |--------------|----------------|-------------------|-----------|--------------|
  |                     | Acc@5      | Avg. Acc   | Acc@5 | Avg. Acc |
  | VLG-CBM (ours) | **59.74%**         | **62.70%**            | **60.38%**    | **66.03%**       |
  | LF-CBM       | 52.88%         | 62.24%            | 31.35%    | 52.70%       |
  | LaBo         | 24.27%         | 45.53%            | 41.97%    | 59.27%       |
  | LM4CV        | 3.77%          | 26.65%            | 3.63%     | 15.25%       |

- **We add experiments on Waterbird dataset to show generalizability to OOD datasets**:  In order to see how VLG-CBM performs in OOD datasets, we train our VLG-CBM on the CUB dataset and test our VLG-CBM on Water-Bird dataset to evaluate the OOD performance. The Water-bird dataset is constructed by cropping out birds from the CUB dataset and transferring them onto backgrounds from the Places dataset. We compare the performance of VLG-CBM with the standard black-box model trained on the CUB dataset. It can be seen that our VLG-CBM generalizes well as the standard model does, which shows that our VLG-CBM is competitive and has very small accuracy trade-off with the interpretability compared with the standard black-box model.   
Table R2: OOD Performance of VLG-CBM
  | Method                        | CUB-Acc | Waterbird-Acc |
  |-------------------------------|---------|---------------|
  | Standard model (black-box)    | 76.70%  | 69.83%        |
  | VLG-CBM                       | 75.79%  | 69.83%        |

2. **Ablation Studies**

- **Data Augmentation**: To understand the effect of augmentation, we provide an ablation study on the CUB dataset in the table below. We observe that a crop-to-concept-prob of 0.2 works best for the CUB dataset, and augmentation improve the accuracy. We will provide an ablation study on remaining datasets in the revised manuscript.
  | Crop-to-Concept-Prob | Acc@NEC=5 | Avg. Acc |
  |----------------------|-----------|----------|
  | 0.0                  | 75.73     | 75.76    |
  | 0.2                  | 75.83     | 75.88    |
  | 0.4                  | 75.71     | 75.72    |
  | 0.6                  | 75.57     | 75.62    |
  | 0.8                  | 75.52     | 75.57    |
  | 1.0                  | 72.29     | 73.15    |

3. **Human study**  
  We conduct a human study following the approach of LF-CBM on Amazon MTurk, showing the annotator top-5 contributing concepts of our method (VLG-CBM) and baseline (LF-CBM or LM4CV) and asking them which one is better. The scores for each method are assigned as 1-5 according to the response of annotators: 5 for the explanations from VLG-CBM is strongly more reasonable, 4 for VLG-CBM is slightly more reasonable, 3 for both models are equally reasonable, 2 for the baseline is slightly more reasonable, and 1 for the baseline is strongly more reasonable. Thus, if our model provides better explanations than the baselines, then we should see a score higher than 3.   
We report the average score in below table for two baselines: LF-CBM and LM4CV. It can be seen that VLG-CBM has scores higher than 3 for both baselines, indicating our VLG-CBM provides better explanations than both baselines.   
  LaBo is excluded in our experiment due to its dense layer and large number of concepts: the top-5 concepts usually account for less than 0.01% of final prediction. We provide an example in the supplementary pdf to show this.   
  Table R3: Human study result (standard deviation in parentheses)
  | Experiment                | Score (VLG-CBM) | Score (Baseline) |
  |---------------------------|-----------------|------------------|
  | VLG-CBM vs. LF-CBM        | 3.33 (1.54)     | 2.67 (1.54)      |
  | VLG-CBM vs. LM4CV         | 3.38 (1.54)     | 2.62 (1.54)      |

Meta-review: After the rebuttal, the submission received relatively positive reviews (1 Accept, 3 Borderline Accept, and 1 Borderline Reject). The main concerns about the papers are: 1) The novelty of the proposed idea is limited. 2) More comprehensive studies are expected. During the rebuttal, the authors provided extensive new results to show the generalization ability of the proposed method. Overall, I think the submission is worth to be published and recommend **Accept** for this submission. As suggested by reviewers, in the camera-ready version, a more detailed comparison with previous works should be added.
Decision: Accept (poster)