Title: VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance
Venue: NeurIPS 2024
Accepted: yes
Keywords: Interpretable machine learning

concept bottleneck models
Abstract: Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes human-understandable concepts to explain models' decision. Recent works proposed to utilize Large Language Models and pre-trained Vision-Language Models to automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, it has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to state-of-the-art CBMs. To address these critical limitations, in this work, we propose a novel framework called Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelf open-domain grounded object detectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) to control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM, outperforms existing methods by at least 4.27\% and up to 51.09\% on *Accuracy at NEC=5* (denoted as ANEC-5), and by at least 0.45\% and up to 29.78\% on *average accuracy* (denoted as ANEC-avg), while preserving both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiments.

VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance

Abstract

Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes humanunderstandable concepts to explain models' decision. Recent works proposed to utilize Large Language Models and pre-trained Vision-Language Models to automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, it has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to state-of-the-art CBMs. To address these critical limitations, in this work, we propose a novel framework called Vision-Language-Guided Concept Bottleneck Model ( VLG-CBM ) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelf open-domain grounded object detectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) to control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM , outperforms existing methods by at least 4.27% and up to 51.09% on Accuracy at NEC=5 (denoted as ANEC-5), and by at least 0.45% and up to 29.78% on average accuracy (denoted as ANEC-avg), while preserving both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiments 2 .

1 Introduction

As deep neural networks become popular in real-world applications, it is crucial to understand the decision of these black-box models. One approach to provide interpretable decisions is the Concept Bottleneck Model (CBM) [6], which introduced an intermediate concept layer to encode humanunderstandable concepts. The model makes final predictions based on these concepts. Unfortunately, one major limitation of this approach is that it requires concept annotations from human experts, making it expensive and less applicable in practice as concept labels may not always be available.

Recently, a line of works utilized the powerful Vision-Language Models (VLMs) to replace manual annotation [15, 27, 25]. They used Large Language Models (LLMs) to generate set of concepts, and then trained the models in a post-hoc manner under the guidance of VLMs or neuron-level interpretability tool [14]. By eliminating the expensive manual annotations, some of these CBMs [15] could be scaled to large datasets such as ImageNet [18]. However, these CBMs [15, 27, 25] still face two critical challenges:

Challenge #1: Inaccurate concept prediction. The concept predictions in these CBMs often contain factual errors i.e. the predicted concepts do not match the image. Moreover, as concepts are generated by LLMs, there are some non-visual concepts, for example "loud music" or "location" used in LF-CBM [15], which further hurt the faithfulness of concept prediction.

Challenge #2: Information leakage. Recently, [13, 12] observed the information leakage in CBMs through empirical experiments - they found that the concept prediction encodes unintended information for downstream tasks, even if the concepts are irrelevant to the task.

In this paper, we propose a new framework called V isionL anguageG uided C oncept B ottleneck M odel ( VLG-CBM ) to address these two major challenges. Our contributions are summarized below:

To address Challenge #1 , we propose to use the open-domain grounded object detection model to generate localized, visually recognizable concept annotations in Section 3. This approach automatically filters the non-visual concepts. Furthermore, the location information is utilized to augment the data. As far as we know, our VLG-CBM is the first end-to-end pipeline to build CBM with vision guidance from open-vocabulary object detectors.

To address Challenge #2 , we provide the first rigorous theoretical analysis which proves that CBMs have serious issues on information leakage in Section 4.1, whereas previous study on information leakage [12, 25] only provides empirical explanations. Building on our theory, we further propose a new metric called the Number of Effective Concepts (NEC) in Section 4.2, which facilitates fair comparison between different CBMs. We also show that using NEC can help to effectively control information leakage and enhance interpretability in our VLG-CBM .

We conduct a series of experiments in Section 5 and demonstrate that our VLG-CBM outperforms existing methods across 5 standard benchmarks by at least 4.27% and up to 51.09% on Accuracy at NEC=5 (denoted as ANEC-5), and by at least 0.45% and up to 29.78% on average accuracy across different NECs (denoted as ANEC-avg). Our learned CBM achieves a high sparsity of 0.2% in the final layer even on large datasets including Places365, preserving interpretability even with a large number of concepts. Additionally, we qualitatively demonstrate that our method provides more accurate concept attributions compared to existing methods.

VLG-cBM provide accurate explanations while prior work provide inaccurate/wrong/less useful explanations!

2 Related work

Concept Bottleneck Model (CBM). The seminal paper [1] first proposed self-explaining models by leveraging the idea of autoencoder to learn interpretable basis concepts in an unsupervised manner without pre-specified concepts. Later, [6] proposed to learn interpretable concepts with human specifications (labels) in the concept bottleneck layer (CBL), and coin the term Concept Bottleneck Models (CBM). CBL is followed by a linear prediction layer, which maps concepts to classes, enabling interpretable final decisions. Formally, let feature representation generated by a frozen backbone represented by z = ϕ ( x ) , CBL concept prediction as g ( z ) = W c z , and the final prediction layer as h ( · ) = W F g ( z ) + b F . The final class prediction of the CBM is given by ˆ y = h ( g ( z )) = h ◦ g ◦ ϕ ( x ) .

Under this setting, the key in training a CBM is obtaining an annotated {(image, concept)} paired dataset for training concept bottleneck layer g . In [6], the authors used human-specified labels to train the CBL in a supervised way. However, obtaining labels with human annotators could be very tedious and costly. Recently, [15], [25], and [27] proposed to utilize Large Language Models (LLM) to generate a set of concepts S , then train CBL by aligning image and concepts with the guidance of vision language models (e.g. CLIP). For example, Oikarinen et al. [15] proposed LF-CBM to train CBM by directly learning a mapping from the embedding space of backbone to concept values in the CLIP space using cosine cubed loss function with the neuron interpretability tool[14], and then mapping concepts to classes using sparse linear layer. [25] proposed LM4CV, a task-guided concept searching method that learns text embeddings in the CLIP space, and then maps the learned embeddings to concepts obtained from LLM using nearest neighbor. Yang et al. [27] proposed LaBo, using submodular optimization to reduce the concept set, followed by using CLIP backbone for obtaining concept values. However, as we show in Sections 5.1 and 5.3 , these methods suffer from multiple issues: (i) The concept prediction is often incorrect and does not capture the visual attributes required for downstream class prediction (e.g. see Fig. 1 b )(ii) VLMs like CLIP suffer from modality gap between image and text embeddings [9] resulting in encoding unintended information, and even random concepts can achieve high accuracy [25]. To address these issues, we explicitly ground the concepts on the training dataset using an open-domain object detection model and then using the obtained concepts for learning CBL - this can ensure a more faithful representation of fine-grained concepts and avoids the modality gap issues introduced by VLMs. Table 1 demonstrates the superiority of VLG-CBM over existing methods [15, 25, 27] on properties including controlling information leakage, flexibility to use any backbone, and accurate concept prediction.

There are some recent works aim at addressing the challenges of CBMs. Similar to us, Pham et al. [16] uses an open-vocabulary object detection model to provide an explainable decision. However, their model is directly adapted from an OWL-ViT model, while our VLG-CBM uses an open-vocabulary object detection model to train a CBL over any base model, providing more flexibility. Additionally, their model requires pretraining to get best performance, while our VLG-CBM could be applied post-hoc to any pretrained model. Kim et al. [4] proposed to filter non-visual concepts by adding a vision activation term to the concept selection step, whereas VLG-CBM uses an open-vocabulary object detectors in multiple stage of CBM pipeline: for filtering non-visual concepts and the guiding training of concept bottleneck layer. Sun et al. [20] aims at eliminating the information leakage, and the authors evaluate the information leakage by measuring the performance drop speed after

ObtainCandidateConcepts

GroundedBoundingBoxGeneration

Auto-LabelledDataset

important

features

LLM

Grounded

Object

+

ofclass

Beak

Detector

Feathers

Tail

( Image, Concepts, Class Label)

Prompt

OR

( Box, Concepts, Class Label)

LearningPredictor

Beak

Bird

Wings

6o0

Backbone

CBL

9

Feathers!

Sparse Linear

Monkey

Layer h

Concepts

Multilabel BCELoss

MulticlassCELoss

Class Label

removing top-contributing concepts. This metric can be controlled by our proposed NEC metric, because the performance reach minimum after removing all contributing concepts. Roth et al. [17] demonstrate that random words and characters achieve comparable CLIP zero-shot performance on visual classification tasks. However, their work does not address information leakage problem and is a very different setting from our work. To date, most of the CBMs focused on vision domains, including this work. There are some recent work applying CBM approach to different domains and different tasks, e.g. interpretable language models for text classifications [21, 22, 11] and for continual learning [26]. We refer the interested readers to their papers for more details.

Open Domain Language Grounded Object Detection. Recent works, including GLIP [8], GLIPv2 [29], and GroundingDINO [10] detect objects in images in an open-vocabulary manner conditioned on natural language queries. In this work we propose to utilize open-vocabulary object detectors for automatically generating grounded concept dataset for training CBMs. This removes the need for human labelers, which is costly, tedious, and does not scale to large datasets. Further, the detected objects provide necessary vision-guidance for CBMs training as demonstrated in our experiments.

3 Method

In this section, we describe our novel automated approach to train a CBM with both Vision and Language Guidance to ensure faithfulness, which is currently lacking in the field. Our approach, abbreviated as VLG-CBM in the paper, generates an auxiliary dataset grounded on fine-grained concepts present in images for training a sequential CBM. Section 3.1 describes our approach to generating an auxiliary dataset used in training CBM, Section 3.2 describes our approach to training concept bottleneck layer, and Section 3.3 describes the training of sparse layer to obtain class labels from concepts in an interpretability-preservable manner. The overall pipeline is shown in Fig. 2.

3.1 Automated generation of auxiliary dataset

Here we describe our novel automated approach for generating labeled datasets for training CBMs. Let f : X → Y be the neural network mapping images to corresponding class labels, where X = R H × W × 3 denotes the input image space and Y = { 1 , 2 , . . . , C } denotes the label space, C is the number of classes. Denote D = { ( x i , y i ) } , x i ∈ X , y i ∈ Y the dataset used for training f , where x i is the i -th image and y i is the corresponding label. Let S be a set of natural-language concepts describing the fine-level visual details from which classes are composed. We propose to generate a modified and auxiliary dataset D ′ from D such that each image contains finer-grained concepts that

are useful in predicting the classes, along with the target class. The overall process of obtaining the modified dataset D ′ can be divided into two steps:

Language supervision from LLMs to generate a set of candidate concepts : We follow the steps proposed in LF-CBM [15] for generating candidate concepts S c for each class c by prompting LLM to obtain visual features describing the class.

Vision supervision from Open-domain Object Detectors to ground candidate concepts to spatial information : We propose using Grounding-DINO[10] Swin-B, current state-ofthe-art grounded object detector, for obtaining bounding boxes of candidate concepts in the dataset. For each image x i with class label c and candidate concepts S c , we prompt Grounding DINO model with S c and obtain K i bounding boxes:

where b j ∈ R 4 × 2 is the j -th bounding box coordinates, t j ∈ R is the corresponding confidence given by the model and s j ∈ S c is the concept of this bounding box. We define a confidence threshold T and remove bounding boxes with confidence less than T to get filtered bounding boxes for each image:

After collecting bounding boxes for every image, we filter out the concepts that do not appear in any bounding box, and get our final concept set ˜ S :

The one-hot encoded concept label vector o i ∈ { 0 , 1 } | ˜ S | for image x i is thus defined as:

Our final concept-labeled dataset D ′ for training CBM can be written as:

3.2 Training Concept Bottleneck Layer

After constructing the concept-labeled dataset D ′ , we now define our approach to train the concept bottleneck layer for predicting the fine-grained concepts in the input image in a multi-label classification setting. Let ϕ : X → R d be a backbone that generates d -dimensional embeddings z = ϕ ( x ) for input image x . Note that ϕ ( x ) can be a pre-trained backbone or trained from scratch. Define g to be the Concept Bottleneck Layer (CBL) which maps embeddings to concept logits. We train a sequential CBM [6, 13] g ( ϕ ( x )) to predict concepts in an image using Binary Cross Entropy (BCE) loss for multi-label prediction. Additionally, to improve the diversity of the concept-labeled dataset D ′ , we augment the training dataset by cropping images to a randomly selected bounding box and modifying the target one-hot vector to predict the concept corresponding to the bounding box. Our optimization objective in terms of BCE loss can be written as:

3.3 Mapping Concept to Classes

In this section, we define our approach to training a sparse linear layer to obtain class labels from concepts in an interpretability-preservable manner. Let h : R d → R C be the sparse linear layer with weight matrix W F and bias b F , which maps concept logits to class logits. We train the sparse layer using the original dataset D by first obtaining concept logits from the trained CBL(frozen), normalizing the concept logits with the mean and variance on training set, and then using them to predict class logits. Our optimization objective in terms of Cross Entropy (CE) loss can be written as:

where R α = (1 -α ) 1 2 ∥ W F ∥ 2 2 + α ∥ W F ∥ 1 is the elastic-net regularization [31] on weight matrix W F , λ is a hyperparameter controlling regularization strength. We use GLM-SAGA[24] solver to solve this optimization problem.

4 Unifying CBM evaluation with Number of Effective Concepts (NEC)

Besides training, another important challenge for CBM is: how to evaluate the semantic information learned in the CBL? Conventionally, the classification accuracy for final class labels is an important metric for evaluating CBMs, with the intuition that a good classification accuracy indicates that useful semantic information is learned in the CBL. However, purely using accuracy as the evaluation metric could be problematic, as it has been shown that information leakage exists in jointly or sequentially trained CBM [13, 12]. That is to say, the CBL could contain unintended information that could be used for downstream classification hence achieving high classification accuracy, even if the concept is irrelevant to the task. In fact, recently [25] showed that, when increasing the number of concepts, a randomly selected concept set could even approach the accuracy of the concept set chosen with sophistication, supporting the existence of information leakage.

To better understand this phenomenon, in section 4.1, we conduct a first theoretical analysis to investigate random CBL and its capability. To the best of our knowledge, this is the first formal analysis of random CBL. Next, inspired by our theoretical result, we propose a new evaluation metric for CBM, named NEC in section 4.2. NEC provides a way to control information leakage and enhance the interpretability of model decisions.

4.1 Theoretical analysis of the Random CBL

We start by defining the notations. Denote k the number of concepts in CBL. We assume that the CBL g consists of a single linear layer: g ( z ) = W c z , where W c ∈ R k × d and z ∈ R d , and the final layer h is also linear: h ◦ g ( z ) = W F g ( z ) + b F , where W F ∈ R C × k , b F ∈ R C . This is the common setting for CBMs. The following theorem suggests a surprising conclusion: a linear classifier upon random (i.e. untrained) CBL could accurately approximate any linear classifier trained directly on the representation, as the number of concepts in the CBL goes up .

Theorem 4.1. Suppose Σ ∈ R d × d is the variance matrix of the representation z which is positive definite, λ max is the largest eigenvalue of Σ , and the weight matrix W c ∈ R k × d is sampled i.i.d from a standard Gaussian distribution. For any linear classifier f which is built directly on the representation z , i.e. f : R d → R , f ( z ) = w ⊤ z + b , it could be approximated by another linear classifier ˜ f on concept logits g ( z ) = W c z , i.e. f ( z ) ≈ ˜ f ( z ) = ˜ w ⊤ g ( z ) + ˜ b , with the expected square error E ( k ) upper-bounded by

Here E ( k ) = E W c [ min ( ˜ w, ˜ b ) E z [ | f ( z ) -˜ f ( z ) | 2 ]] denotes the average square error, w ∈ R d , ˜ w ∈ R k , k is the number of concepts in CBL.

Remark 4.1 . In Theorem 4.1, we consider a 1-D regression problem where we use a linear combination of concept bottleneck neurons to approximate any linear function. The multi-class classification result could be derived by applying Theorem 4.1 to each class logit (see Corollary A.1). From Eq. (8), we could see that the expected error goes down linearly when concept number k increases, and achieves 0 when k ≥ d , where d is the dimension of backbone representation z . This suggests that, even with a random CBL (i.e. W c is simply drawn from a standard Gaussian distribution without any training), the classifier could still approximate the original classifier well and achieve good accuracy, when concept number k is large enough. We defer the formal proof of Theorem 4.1 to Appendix A.

4.2 A New Evaluation Metric for CBM: Number of Effective Concepts (NEC)

Theorem 4.1 provides a formal theoretical explanation on [25]'s observation. Moreover, it raises a concern on the evaluation of CBMs: model classification accuracy may not be a good metric for evaluating the semantical information learned in CBL, because a random CBL could also achieve high accuracy. To address this concern, we need to control the concept number k so that the

semantically meaningful CBLs can be distinguished with random CBLs w.r.t. the final classification performance.

We notice that previous works mainly use two approaches to control k :

Control the total number of concepts: [25] used a more concise concept layer, i.e. reduce the total number of concepts. However, this approach may miss some important concepts due to limitations in total concept numbers. Additionally, they used a dense final layer which is less interpretable for humans, as each decision is related to the whole concept set.

[15, 28] suggested using a sparse linear layer for final prediction to enhance interpretability. Though sparsity is initially introduced to enhance interpretability, we note that this also reduces the number of concepts used in the decision, thus controlling the information leakage. However, the problem is, these works lack the quantification for sparsity, which is necessary for fair comparison between methods.

To provide a unified metric for both approaches, we propose to measure the Number of Effective Concepts (NEC) for final prediction as a sparsity metric. It is defined as

̸

Intuitively, NEC measures the average number of concepts the model uses to predict a class. Using NEC to evaluate CBM provides the following benefits:

A smaller NEC reduces the information leakage. As shown in Fig. 3, with large NEC, even random CBL could achieve near-optimal accuracy, suggesting potential leakage in information. However, by reducing NEC, the accuracy of random concepts drops quickly. This implies enforcing a small NEC could help to control information leakage.

A model with a smaller NEC provides more interpretable decision explanations. Humans can recognize an object with several important visual features. However, models can utilize tens or hundreds of concepts for the final prediction. By using a smaller NEC, the model's decision could be attributed mainly to several concepts, making it more interpretable to human users.

NEC enables fair comparison between CBMs. Comparing the performance of CBMs has long been a challenging problem, as different models use different numbers of concepts and different styles of final layers (sparse/dense). NEC considers both, thus providing a fair metric to compare different models.

Given these benefits, we suggest to control the NEC when comparing the performance of CBMs. In Section 5, we provide experiments with controlled NEC, where we observed our VLG-CBM outperforms other baselines.

(a) CIFAR10

(b) CIFAR100

(c) CUB

(d) Places365

0.90

0.8

0.5

0.85

0.6

0.7

0.4

accurac

2

0.80

accuracy

0.4

accuracy

0.6

accuracy

0.3

0.75

0.70

Ours

司

0.2

Ours

Test

0.5

Ours

0.2

sno

0.65

LF-CBM

LF-CBM

0.4

LF-CBM

0.1

LF-CBM

0.60

4

8

16

32

Random

64

128

0.0

4

8

16

32

Random

64

128

0.3

4

8

16

32

Random

64

128

0.0

4

8

16

32

Random

64

128

NEC

NEC

NEC

NEC

5 Experiments

In this section, we conduct a series of experiments to evaluate our method, including illustrating the faithfulness of concept prediction, interpretability of model decisions, and performance with controlled NEC.

5.1 Performance comparison

Setup. Following prior work [15], we conduct experiments on five image recognition datasets: CIFAR10, CIFAR100[7], CUB[23], Places365[30] and ImageNet[18]. For the choice of backbone, we categorize the experiments into two categories:

CLIP backbone: For CLIP backbone, we choose CLIP-RN50 for all datasets.

non-CLIP backbobe: We follow the choice of [15] to use ResNet-18[3] for CUB and ResNet-50 (trained on ImageNet) for Places365 and ImageNet as the backbone.

The reason of this categorization is some previous works (e.g. LaBo[27] only supports CLIP backbone.

Baselines. We compare our method with three major baselines when applicable: LF-CBM[15], LaBo[27], and LM4CV[25]. These are SOTA methods for constructing scalable CBMs, with [15] most flexible and [27, 25] limited by specific architecture and not available for certain dataset. Additionally, we present the results from a randomly initialized CBL for comparison.

Metrics. As discussed in Section 4, in order to evaluate the final classification power of CBM, we should acculate the Accuracy under specified NEC(ANEC). Therefore, we measure the following two metrics:

Accuracy at NEC=5(ANEC-5): This metric is designed to show the performance of CBM which could provide an interpretable prediction. We choose the number 5 so that human users could easily inspect all concepts related to the decision without much effort.

Average accuracy(ANEC-avg): To evaluate the trade-off between interpretability and performance, we also calculate the average accuracy under different NECs. In general, higher NEC indicates a more complex model, which may achieve better performance but also hurt interpretability. We choose six different levels: 5 , 10 , 15 , 20 , 25 , 30 and measure the average accuracy.

Controlling NEC. As we discussed in Section 4, there are two approaches to control NEC: (1) using a dense final layer and directly controlling the number of concepts and (2) training a sparse final layer with appropriate sparsity. LM4CV[25] used the first approach, where the number of concepts could be directly set as target NEC. For LF-CBM[15] and our VLG-CBM , the second approach is utilized: To achieve target sparsity, we control the regularization strength λ in GLM-SAGA. GLM-SAGA provides a regularization path, which allows us to gradually reduce regularization strength and get a series of weight matrices with different sparsity. Specifically, we start with λ 0 = λ max which gives the sparsest weight. Then, we gradually reduce the λ by λ t +1 = αλ t until we achieve the desired NEC. We choose the weight matrix with the closest NEC to our target and prune the weights from smallest magnitude to largest to enforce accurate NEC. LaBo [27] did not provide a NEC control method. Hence, we apply sparse final layer training of LaBo's concept prediction to control NEC.

Results. We summarize the test results for with backbone in Table 2 and results with non-CLIP backbones in Table 3. From the results, we could see that:

The accuracy at NEC = 5 provides a good metric for evaluating the semantic information in CBL: As shown in the table, the accuracy of random CBL is much lower with NEC = 5 , which implies the information leakage is controlled and the accuracy could better reflect the useful semantic information learned in the CBL.

The performance of LM4CV is even worse than random CBL. An explanation to this is LM4CV utilizes a dense final layer, which is intrinsically inefficient to interpret as each class is connected to all the concepts, including the irrelevant ones. When limiting the NEC to a small value to provide a concise explanation, the model has to largely reduce the concept number which sacrifices the prediction power.

Our method significantly outperforms all the baselines at least 4.27% and up to 51.09% on accuracy at NEC=5, and by at least 0.45% and up to 29.78% on average accuracy across different NECs, illustrating both high performance and good interpretability.

5.2 Visualization of CBL neurons

In order to examine whether our CBL learns concepts aligned with human perception, we list the top-5 activated images for example concept neurons on the model trained on the CUB dataset in Fig. 4. As shown in the figure, our CBL faithfully captures the corresponding concept. We provide more visualization results in Appendix K.

5.3 Case study

In this section, we conduct a case study to compare the concept prediction between our VLG-CBM , LF-CBM [15] and LM4CV[25] as shown in Fig 1. We provide extended results and comparison with LaBo [27] in Appendix G.2. For our method, we use the final layer with NEC = 5 . We show that our method provides more accurate concept prediction and more interpretable decisions for users.

Decision interpretability. We examine the explanation of each CBM model on example images by showing the top-5 concept contributions. The contribution of each concept is calculated as the product of the concept prediction value and corresponding weight. Formally, the contribution of

i -th concept to j -th class is defined as: Contribution ( i, j ) = g i ( z ) · ( W F ) ji . We pick the top-5 contributing concepts for the final predicted class and visualize it in Fig. 1. We could see that:

For other CBMs, a large portion of the final decision is attributed to the "Sum of other features". This part hurts the interpretability of CBM, as it's difficult for users to manually inspect all these concepts in practice. We conduct further study on this in Section 5.4. Our model, however, provides a concise explanation from a few concepts because we apply the constraint NEC=5. This ensures users can understand model decisions without difficulties.

Our VLG-CBM provides explanation more aligned with human perception. From the example, we can also see that our model explains the decision with clear visual concepts. Other CBMs attribute the decision to non-visual concepts (e.g. LaBo), concepts that do not match the image (e.g. LM4CV), or negative concepts (LF-CBM).

5.4 Do Top-5 concepts fully explain the decision?

Besides training a final layer with a small NEC, another common approach to provide a concise explanation is showing only the top contribution concepts. However, we argue that this approach may not faithfully explain the model's behavior, as the non-top concepts also make a significant contribution to the decision. To verify this, we conduct the following experiment: On the CUB dataset, we prune the final weight matrix W F to leave only the top-5 concepts for each class, whose weight has the largest magnitude. Then, we use the pruned model to make predictions and compare them with the prediction results from the original model. Table 4 shows results for our VLG-CBM which uses NEC = 5 to control sparsity, and other three baselines, LaBo[27], LF-CBM[15] and LM4CV[25], without any constraint on NEC. As shown in the table, for all three baselines, a large portion of predictions changes after pruning. This suggests that without explicitly controlling NEC, only showing top-5 contributing concepts does not faithfully explain all of the model decisions. Hence, we recommend training the final layer with NEC controlled to obtain a concise and faithful explanation as we proposed in Section 4.

6 Conclusion, Potential Limitations and Future work

In this work, we study how to improve the interpretability and performance of concept bottleneck models. We introduce a novel approach VLG-CBM based on both vision and language guidance, which successfully improves both the interpretability and utility of existing CBMs in prior work. Additionally, our theoretical analysis show that information leakage may exist on even in the untrained CBLs, serving the foundations for our proposed new NEC-controlled metrics (ANEC-5 and ANEC-avg). We show that NEC not only allow fair evaluation of CBMs but also can be used to effectively control information leakage of CBM and ensure interpretability. Extensive experiments on image classification benchmarks demonstrated our VLG-CBM largely outperform previous baselines especially for small NEC, providing more interpretable decisions for users.

Despite the superior performance of VLG-CBM over prior work as demonstrated in extensive experiments, one potential limitation is the dependence on large pretrained models (e.g. the success of open-domain grounded object detection model that we use to enforce vision guidance). However, prior work (e.g. LaBo, LM4CV, LF-CBM) also shared similar limitation on the reliance of large pre-trained models (e.g. CLIP). Nevertheless, it also means that our techniques have the potential to be further improved with the advancement of large pre-trained models. In the future, we plan to explore training CBL with even more vision guidance, such as using segmentation maps of concepts.

References

David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems , 31, 2018.

Morris L Eaton. Multivariate statistics: a vector space approach , volume 512. Wiley New York, 1983.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Injae Kim, Jongha Kim, Joonmyung Choi, and Hyunwoo J. Kim. Concept bottleneck with visual concept filtering for explainable medical image classification, 2023. URL https: //arxiv.org/abs/2308.11920 .

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML , 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training, 2022.

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning, 2022.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris CallisonBurch. Interpretable-by-design text classification with iteratively generated concept bottleneck. CoRR , 2023.

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314 , 2021.

Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 , 2021.

Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In ICLR , 2023.

Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. ICLR , 2023.

Thang M Pham, Peijie Chen, Tin Nguyen, Seunghyun Yoon, Trung Bui, and Anh Nguyen. Peeb: Part-based image classifiers with an explainable and editable language bottleneck. arXiv preprint arXiv:2403.05297 , 2024.

Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts, 2023. URL https://arxiv.org/abs/2306.07282 .

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Ao Sun, Yuanyuan Yuan, Pingchuan Ma, and Shuai Wang. Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning. arXiv preprint arXiv:2402.05945 , 2024.

Chung-En Sun, Tuomas Oikarinen, and Tsui-Wei Weng. Crafting large language models for enhanced interpretability. ICML Mechanistic Interpretability workshop , 2024.

Zhen Tan, Lu Cheng, Song Wang, Yuan Bo, Jundong Li, and Huan Liu. Interpreting pretrained language models via concept bottlenecks. CoRR , 2023.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In ICML , 2021.

An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV , 2023.

Sin-Han Yang, Tuomas Oikarinen, and Tsui-Wei Weng. Concept-driven continual learning. TMLR , 2024.

Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR , 2023.

Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. ICLR , 2023.

Haotian* Zhang, Pengchuan* Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836 , 2022.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452-1464, 2017.

Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67(2):301-320, 2005.

Appendix

Table of Contents

A Proof of Theorem 4.1

In this section, we present a formal definition of the expected square error in Theorem 4.1 and show the proof. First, we define the square approximation error as

which is the average square distance between f ( z ) and ˜ f ( z ) . Given a specific CBL W c , we seek a final layer ˜ w to minimize the square error:

For randomly Gaussian initialized W c , we care about the minimal error we could achieve on average. Thus, for each W c , we choose ˜ w and ˜ b to achieve minimum approximation error, then take the expectation over W c to define the expected square error as

Setting Suppose the representation z has variance Σ ∈ R d × d which is positive definite. The weight matrix W c ∈ R k × d is sampled i.i.d from a standard Gaussian distribution. Here, we show that any linear classifier which is built directly on representation z , i.e. f ( z ) = w ⊤ z + b , could be approximated by a linear classifier on concept logits g ( z ) = W c z , i.e. f ( z ) ≈ ˜ f ( z ) = ˜ w ⊤ g ( z ) + ˜ b .

Theorem 4.1. Suppose Σ ∈ R d × d is the variance matrix of the representation z which is positive definite, λ max is the largest eigenvalue of Σ , and the weight matrix W c ∈ R k × d is sampled i.i.d from a standard Gaussian distribution. For any linear classifier f which is built directly on the representation z , i.e. f : R d → R , f ( z ) = w ⊤ z + b , it could be approximated by another linear classifier ˜ f on concept logits g ( z ) = W c z , i.e. f ( z ) ≈ ˜ f ( z ) = ˜ w ⊤ g ( z ) + ˜ b , with the expected square error E ( k ) upper-bounded by

Here E ( k ) = E W c [ min ( ˜ w, ˜ b ) E z [ | f ( z ) -˜ f ( z ) | 2 ]] denotes the average square error, w ∈ R d , ˜ w ∈ R k , k is the number of concepts in CBL.

Proof. Based on the value of k , we can consider two cases: (I) k < d , and (II) k ≥ d , and derive the E ( k ) respectively.

Case (I): k < d . First, we consider under a fixed W c , what is the minimum error we could achieve. The expected approximation error is:

The last equality is from E ( X 2 ) = V X +( E X ) 2 . The second term ( ∗∗ ) takes minimum 0 when ˜ b = E z [( w ⊤ -˜ w ⊤ W c ) z ] + b . The remaining question is to choose a proper ˜ w to minimize ( ∗ ) . Notice that

where Σ is the covariance matrix of z , Σ 1 2 is the principal square root of Σ , Σ ∈ R d × d , Σ 1 2 ∈ R d × d . Now the problem in Eq. (A.2) can be reduced to a linear least square problem:

Since Σ is positive definite, so is Σ 1 2 . Thus, the eigen decomposition of Σ 1 2 satisfies the following: Σ 1 2 = ˜ Q ⊤ Λ ˜ Q , where ˜ Q ∈ R d × d is an orthogonal matrix and Λ ∈ R d × d is a diagonal matrix with positive entries. With Gram-Schmidt process, we could derive QR factorization of W ⊤ c : W ⊤ c = QR , where Q ∈ R d × d is orthogonal and R ∈ R d × k is upper triangular. Plugging above decomposition of Σ 1 2 and W ⊤ c , now we have

where λ max is the largest eigenvalue of Σ 1 2 . In short, we have derived the minimum square error for a given W c , which is upper bounded by

Secondly, we consider when W c is sampled i.i.d. from standard normal distribution, and calculate the expected error. From above derivation,

Note that since W ⊤ c = QR , the randomness in W c is reflected in Q and R . The matrices Q and R satisfies the following properties:

Q is a random rotation following uniform distribution. This is intuitive because standard Gaussian distribution is rotation-invarant. For a formal statement and proof, we refer to Proposition 7.2 of Eaton [2].

range ( R ) = span ( e 1 , e 2 , · · · , e k ) with probability 1. Since rank ( R ) = rank ( W ⊤ c ) and W ⊤ c is full rank with probability 1, rank ( R ) = min( k, d ) = k with probability 1. From upper-triangularity of R , we know that

With probability 1, rank ( R ) = k , thus we conclude

In the following derivation, since we only cares about the expectation, we omit "with probability 1" for brevity.

From the above properties of Q and R , the expectation term in the RHS of Eq. (A.9) can be derived as:

This is because range ( R ) = span ( e 1 , e 2 , · · · , e k ) , and k < d . Thus, min ˜ w ∥ R ˜ w -Q ⊤ w ∥ 2 2 is the squared distance from Q ⊤ w to subspace span ( e 1 , e 2 , · · · , e k ) , which equals to the squared sum of last d -k coordinates of Q ⊤ w .

Because Q is a random rotation, Q ⊤ w is uniformly distributed on a sphere with radius ∥ w ∥ 2 . Denote v = Q ⊤ w . From symmetricity, we have

Furthermore, ∥ v ∥ 2 2 = ∥ w ∥ 2 2 gives ∑ d i =1 v 2 i = ∥ w ∥ 2 2 . Take expectation of both sides gives ∑ d i =1 E v v 2 i = ∥ w ∥ 2 2 , thus E v 2 1 = E v 2 2 = · · · = E v 2 d = ∥ w ∥ 2 2 /d . The target quantity becomes

In conclusion, we derive an upper bound of approximation error for any linear function f :

Look at the bound in Eq. (A.15): λ 2 max is a constant regarding the scale of data; ∥ w ∥ 2 2 is a constant regarding the norm of weight vector we want to approximate; (1 -k d ) is a linear term shows that the expected square error goes down linearly when we increase the number of concepts k , and achieves zero when k = d .

Case (II): k ≥ d . For the case that k ≥ d , it could be derived from our main results that E ( k ) = 0 . Additionally, with probability 1 we could find ˜ f ( x ) = f ( x ) as will be derived below. As we discussed, with probability 1, W c has full rank. Given that, we have

where W + c is the Moore-Penrose inverse of W c . For any linear classifier f ( z ) = w ⊤ z + b . Let ˜ w = ( W + c ) ⊤ w , ˜ b = b , we have

and thus E ( k ) = 0 .

Corollary A.1. For f and ˜ f with C output classes, i.e. f : R d → R C , ˜ f : R d → R C , w ∈ R d , ˜ w ∈ R k , the expected error upper-bound is

Here E ( k ) = E W c [ min ( ˜ w, ˜ b ) E z ∥ f ( z ) -˜ f ( z ) ∥ 2 ] denotes the average square error.

Remark A.2 . The statement could be verified by applying Theorem 4.1 to each f i and ˜ f i output, then summing up the error.

B Implementation details

Computational resources and codes. Our experiments run on a server with 10 CPU cores, 64 GB RAM, and 1 Nvidia 2080Ti GPU. Our implementation builds on the open-source implementation of the LF-CBM [15] available: https://github.com/Trustworthy-ML-Lab/Label-free-CBM . For training the final predictive layer, we use publicly available code for GLM-SAGA [24].

Hyperparameter tuning. We tune the hyperparameters for our method using 10% of the training data as validation for the CIFAR10, CIFAR100, CUB and ImageNet datasets. For Places365, we use 5% of the training data as validation. We use CLIP(RN50) image encoder as the backbone for CIFAR10 and CIFAR100, Resnet-18[3] trained on CUB for CUB dataset, and Resnet-50 pretrained for Places365 following setup similar to LF-CBM. We tune the CBL with Adam[5] optimizer with learning rate 1 × 10 -4 and weight decay 1 × 10 -5 . The concept dataset obtained from GroundingDINO is inherently unbalanced since there is a much lower proportion of positive datapoints for a concept. Consequently, we scale the CBL loss by multiplying it with a positive value to balance the tradeoff between precision and recall and improve the imbalance of positive data points. We set T = 0 . 15 in Eq. (2) in all our experiments. We seed the random number generator with a fixed seed to ensure the results can be reproduced.

C Ablation Studies

C.1 Ablation study for confidence threshold

Confidence threshold T in Eq 2 filters concepts with bounding boxes' confidence less than T . In this experiment, we study the affect of T on the VLG-CBM 's accuracy. The results are shown in Table C.1. We observe that ANEC-5 and ANEC-avg first increases (or stays constant) and then decreases. We attribute this effect to to the fact that as T increases, the number of false-positive decreases leading to better learning of concepts, however, as the number of annotations available for learning a concept decreases.

D Evaluating annotations from Grounding DINO

This section quantitatively evaluates concept annotations obtained from Grounding DINO. We use CUB dataset for comparison which contains ground-truth for fine-grained concepts present in each image. We use the label set from Koh et al. [6] which has 1:1 mapping with the ground-truth concepts in the CUB dataset. We use precision and recall metric to measure the quality of annotations from Grouding DINO for each concepts. Table D.1 present mean precision and mean recall value at different confidence threshold. We observe that the obtained annotations have a very high recall i.e if the concept is present in the image, grounding DINO is able to retrieve the object. The precision is also sufficiently high though it suffers from a relatively higher false-positive detection rate compared to false-negative detection rate. However, as demonstrated in our qualitative and quantitative studies (Table 3, Fig 4, K.2, K.1) the effect of false-positive is minimal and VLG-CBM is able to faithfully represent concepts in the Concept Bottleneck Layer.

VLG-CBM provide accurate explanations while prior work provide inaccurate/wrong/less useful explanations!

large, orange billwith a black tip (100.07) (6'9) ahenid abuo pue anig syg long, straight orange bill (141.48) (s8'te) abewnid yoeiq pue moien pointed orange bill (95.72)

Sum of other concepts (-257.02)

NOT a red crest on the head (0.99) yellow or orange plumage (0.80) a bright orange breast (0.76) Sum of other concepts (5.53) a yellow head (1.96) (66'0) s6al abueo 12 3 5

(e') abeund moi uapon yug yellow head and breast (1.07) black mask on the face (0.26) (0t6) sbum Xenb anjg large conical bil (0.53) 123 4 5

dark gray wings with pale grayedging (-78.55) red, black and white feathers (93.12) (so'ta) iseaia pue peau pau gya (6992) Sum of other concepts (1i3.93) bright red head and nape ( 123４5

NOT a dark brown or black color (1.25) , blue-gray wings (0.81) Sum of other concepts (5.12) NOT dark wingtips (2.07) yellow feet (1.12) NOT long, 1234 5

(s'e) shum alm pue yog Sum of other concepts (0.00) small delicate body (2.75) long slender beak (0.12) small forked tail (7.22) pointed beak (0.00) 1234 5

large red bill with a slightly hooked tip (89.09) (+tl6) abewnid abueno pue ang nyug distinctive white throat (-76.91)

Sum of other concepts (-34.89)

Sum of other concepts (4.22)

LM4CV

LF-CBM

VLG-CBM (ours)

Sum of other concepts (0.00) short pointed beak (0.65) short stout bill(0.01) (Lo'o) pqbuos lews blue head (0.21) green back (0.09) 123 5

E Distribution of nonzero weights among class

The NEC metric controls the average number of non-zero weights among classes. Further, we study the distribution of non-zero weight numbers between different classes. We choose our VLG-CBM model trained on CUB and places365 datasets, which have 200 and 365 classes, respectively, and plot the distribution of non-zero weights. Both models are trained to have NEC=5 The results are shown in Fig. E.1. The figure suggests most classes have non-zero weight numbers around 5, while a small number of classes utilize more concepts to make decisions.

(a) CUB

80

70

60

50

40

30

20

10

0

5

10

15

20

25

30

(b) Places365

80

70

60

50

40

30

20

10

0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

F Constructing model with specified NEC

In this section, we discuss how to construct models with specified NEC. When using methods with dense final layers (e.g. [25]), controlling NEC is simply controlling total number of concepts in the concept set. Hence, below we mainly focus on models with sparse final layers.

When training the final linear layer, larger lambda(regularization strength) pushes the model to be sparser. Hence, we utilize GLM-SAGA[24], which allows us to obtain a regularization path consists of different lambdas. To be more specific, we choose a λ max and train models with λ in [ λ min = λ max / 500 λ max ] , and take 50 λ evenly from the interval in log space. Then, we choose the weight matrix with the closest NEC and pruning the weights from smallest magnitude to largest to enforce strict NEC. Hence, the actual NEC is enforced to be exactly as prespecified ones.

G Additional case study examples

G.1 Negative concepts in reasoning

In LF-CBM [15] and our VLG-CBM, normalization is applied on concept logits before the final decision layer. Hence, a negative value of concept logits indicates corresponding concept does not appear in the image. Following LF-CBM, we mark these concepts as "NOT {concept}" in explaining the decision. To study the frequency of this negative reasoning, we count the times these negative concepts appear in top-5 contributing concepts on CUB dataset. The results show that, for VLG-CBM, 162 out of 28950(0.56%) reasonings are through negative concepts. For comparison, LF-CBM utilizes 6687 out of 28950(23.10%) negative reasoning.

G.2 Impact of NEC

The study in Section 5.3 shows that our VLG-CBM provides more interpretable decisions than baseline methods. To better understanding where these advantages comes from, we conduct a further study to set the baselines with NEC=5 and compare the decision interpretation, see Figs. G.2 to G.4. The results suggest setting NEC=5 alleviate the problem from non-top-5 concepts. However, wrong/inaccurate/less useful explanations still exist.

Groundtruth:Bobolink. VLG-CBM prediction:Bobolink

black v on the back(4.30)

black and white striped head(1.94)

black head and back(0.27)

4.small round body(0.07)

NOT white stripes above the eyes(0.01)

Sum of other concepts: (0.03)

LF-CBM

LM4CV

LaBo

1. NOT a brown and white color scheme

1. Color-Blue head, olive back, yellow

1. beautiful bird with a brightly colored body

(1.77)

underparts (101.08)

(0.02)

2. NOT white and black coloration (1.66)

2.

2.

small, plump songbird with a short tail and

3. iridescent feathers (1.37)

blue highlights (94.03)

a pointed bill (0.02)

4. NOT a black bib with white stripes

bright blue and orange plumage (91.44)

3.

beautiful bird with a brightly colored

(1.29)

4.large red bill with a slightly hooked tip

plumage (0.02)

5. NOT a black and white color scheme

(89.09)

4.

one of the most beautiful north american

(1.01)

5. white rump patch at the base of the tail

songbirds (0.02)

Sum of other concepts (4.22)

(59.69)

5. colors are very vibrant and beautiful (0.02)

Sum of other concepts (-171.50)

Sum of other concepts (41.25)

LF-CBM (NEC=5)

LM4CV (NEC=5)

LaBo(NEC=5)

1.

NOT a brown and white color

1.

bright reddish brown head,crown and

1. beautiful little bird with a very colorful

scheme(2.39)

back of neck.(382.61)

plumage(3.98)

2.

NOT a black bib with white stripes(1.08)

2.

bright yellow, green and blue

2. very colorful bird, with a lot of blue and

3.

NOT a black and white color

plumage(95.61)

green(0.43)

scheme(0.61)

3.

brightyellowthroatbreast,andflanks

3. very pretty and very colorful(0.41)

4.

iridescent feathers(0.27)

with black bars (51.36)

4. NOT white stripes on white stripes on

5.

NOT yellowish-brown wings(0.25)

4.

Broad tail thatisshorterthanother

brown(0.22)

Sum of other concepts:(0.00)

pelican species (-36.48)

5. known as the "rainbow jay" due to its bright

5.

Mottledbrownnthenapemantle,and

plumage(0.11)

scapulars(-243.90)

Sum of other concepts:(0.00)

Sum of other concepts: (0.00)

@2006BrynjurBjifsson

LF-CBM

LM4CV

LaBo

1. NOT dark wingtips(2.07)

1. red, black and white feathers(93.11)

1.male goldfinch is the more brightlycolored of the

2. white breast with brown spots(1.93)

2. bright red head and breast(84.05)

sexes,with(0.02)

3. NOT a dark brown or black color(1.25)

3. bright red head and nape(75.39)

2.seen in flocks of other goldfinches(0.02)

4. yellow feet(1.12)

4. bright red crescent below its beak (63.14)

3.often forming flocks with other goldfinches(0.02)

5. NOT long, blue-gray wings(0.81)

5.White neckwithablackcollar and

4.visit bird tables and feeders(0.02)

Sum of other concepts: (5.12)

chestnutredhead andbreast(60.52)

5. young goldfinches are drabber than adults, with

Sum of other concepts:(-172.32)

brownish plumage(0.02)

Sum of other concepts: (41.69)

LF-CBM (NEC=5)

LM4CV (NEC=5)

LaBo(NEC=5)

1. NOT dark wingtips(1.80)

1. bright reddish brown head, crown and

1. closely related to the goldfinch(3.83)

2. NOT long, blue-gray wings(0.69)

back of neck.(344.38)

2.often forming flocks with other goldfinches(1.16)

3. yellow feet(0.66)

2.bright yellow, green and blue

3. NOT from alaska and canada to the

4. a red face(0.08)

plumage(87.41)

southwestern united states(1.10)

5. a Scarlet-red body(0.00)

3.bright yellow throat,breast, and flanks

4.oftenforming flocks with other goldfinches and

Sum of other concepts: (0.00)

with black bars (39.26)

similar small birds(0.17)

4.Broad tail that is shorter than other

5. NOT found in eastern and central united states

pelican species (-34.77)

year-round(0.03)

5. Motled brown on the nape, mantle, and

Sum of other concepts: (0.02)

scapulars(-227.34)

Sum of other concepts:(0.00)

LF-CBM

LM4CV

LaBo

1. a yellow head(1.96)

1. long, straight orange bill(141.48)

1. is the only warbler with entirely(0.02)

2. NOT a red crest on the head(0.99)

2.large, orange bil with a black tip (100.07)

2. largest warbler in north america(0.02)

3. orange legs(0.99)

3.pointed orange bill (95.72)

3. plumage is bright yellow(0.02)

4. yellow or orange plumage(0.80)

4. yellow and black plumage(84.85)

4.largest and heaviest member of the

5. a bright orange breast(0.76)

5. bright blue and orange plumage(76.97)

wood-warbler family(0.02)

Sum of other concepts:(5.53)

Sum of other concepts: (-257.02)

5. yellow bird(0.02)

Sum of other concepts:(42.29)

LF-CBM (NEC=5)

LM4CV (NEC=5)

LaBo(NEC=5)

1. a yellow head(2.36)

1. bright yellow throat, breast, and flanks

1. NOT sometimes called the "sea

2. yellow or orange plumage(0.60)

with black bars (320.88)

sparrow" due to its black and white

3. orange legs(0.17)

2. bright yellow, green and blue

plumage(1.10)

4.a black ring around the bill(0.08)

plumage(285.24)

2. yellow head, chest, and belly(0.95)

5. Glossy black wings(0.00)

3.bright reddishbrown head,crown and

3. yellow head is thought to be a sign of

Sum of other concepts:(0.00)

back of neck.(-89.49)

maturity and wisdom(0.37)

4.Broad tail that is shorter than other

4. orange in color(0.28)

pelican species (-126.68)

5. series of high, thin "peeps".(0.15)

5. Motled brown on the nape, mantle, and

Sum of other concepts:(0.37)

scapulars(-151.65)

Sum of other concepts: (0.00)

H Further discussion on decision explanations

In this section, we further discuss some interesting phenomena observed in the decision explanations generated by different models.

H.1 Negative contributions

In Fig. 1, we could see that LM4CV[25] generates negative contribution values, while LF-CBM[15] and our VLG-CBM do not. We hypothesize the reason is different training methods: LM4CV trains a dense final layer, hence the concepts irrelevant to the class may provide a negative contribution. LF-CBM and VLG-CBM, however, train a sparse final layer. To enforce sparsity, the model only captures relevant concepts for decision. Hence, it's natural to expect most contributions should be positive.

I Additional experiment results

I.1 Generalizability to OOD datasets

In this section, we study a question: will our VLG-CBM hurts the generalization ability of original model to Out-Of-Distribution(OOD) dataset? To study this problem, we conduct experiment on Waterbirds dataset [19]. Waterbirds is an OOD dataset adapted from the CUB dataset, which combines bird photos from CUB with image backgrounds from Places365. We use the same ResNet model as we used in Table 3 for CUB dataset. For VLG-CBM, we choose NEC=5 and compare the results with the standard, non-interpretable models. On this dataset, the results are shown below: It can be seen

that our VLG-CBM generalizes well as the standard model does, which shows that our VLG-CBM is competitive and has very small accuracy trade-off with the interpretability compared with the standard black-box model.

I.2 Ablation study

I.2.1 Ablation on augmentation probability

In this section, we conduct an ablation study on the probability of applying our crop-to-concept data augmentation introduced in Section 3.1. The dataset we used is the CUB dataset and the backbone is ResNet as we used in the main experiment. The results are listed below. From the table, we could see

that the performance is best with augmentation probability 0.2.

J Human study

In this section, we present a human study following the practice of Oikarinen et al. [15] on Amazon MTurk platform. To briefly summarize, we show the annotator top-5 contributing concepts of our method (VLG-CBM) and baseline (LF-CBM or LM4CV) and asking them which one is better.

Task

Model 1 predicts this image is a *parking meter*

Image:

Explanation:

Because the image has the following features(in order of importance):

1. sometimes has a green or red light to indicate if it is open or closed(78.2%)

2.may have a purple hue(29.7%)

3.other components (e.g,handlebars,seat)may be a variety of colors(29.4%)

4.taxi sign on the roof(28.6%)

5.a washing machine is typically large and box-shaped(28.0%)

6. Others: -93.9%

(in order of importance,number in thebracket() is thecontribution of eachfeature.)

Model 2 predicts this image is a “parking meter”

Explanation:

Because the image has the following features:

1.meter maid(44.8%)

2.button for adding time(40.0%)

3.meter(13.6%)

4.button to start the timer(1.7%)

5.Others:0.0%

(in orderof importance,number in thebracket() is thecontributionof eachfeature.)

WhichExplanationismorereasonable?

OModel 1 Clearly More Reasonable O Model 1 Slightly More Reasonable O Both Models Equally Reasonable O Model 2 Slightly More Reasonable O Model 2 Clearly More Reasonable

Why?Select allthat apply

Themorereasonable explanation usesfeaturesthat aremorerelevant to theimage.

The more reasonable explanation usesfeatures are more relevant to the prediction.

The more reasonable explanation is more informative.

N/A, both explanations equally reasonable.

The scores for each method are assigned as 1-5 according to the response of annotators: 5 for the explanations from VLG-CBM is strongly more reasonable, 4 for VLG-CBM is slightly more reasonable, 3 for both models are equally reasonable, 2 for the baseline is slightly more reasonable, and 1 for the baseline is strongly more reasonable. Thus, if our model provides better explanations than the baselines, then we should see a score higher than 3. We show an example screenshot of our study in Fig. J.1.

We report the average score in Table J.1 for two baselines: LF-CBM and LM4CV. For each baseline, we randomly sample 200 images and collect 3 results from 3 different annotators. It can be seen that VLG-CBM has scores higher than 3 for both baselines, indicating our VLG-CBM provides better explanations than both baselines. LaBo is excluded in our experiment due to its dense layer and large number of concepts: the top-5 concepts usually account for less than 0.01% of final prediction.

K Visualizing VLG-CBM explanations

This section presents an extended version of Table 4 visualizing top-5 images for randomly picked concepts for CUB and Places365 dataset. The results are shown in Fig K.1, K.3, K.4, and K.5 for Places365 and K.2, Fig K.6, Fig K.7, and Fig K.8 for CUB.

39

17

ilGelataio

eshtogo..

Pred class: library, outdoor|logit:7.272|bias: 0.257

frontentrance withsteps(pos)

+4.24

stone or brick walls(pos)

+2.77

landscaping(pos)

+0

place towork(neg)

-0

place towashbefore prayer(pos)

+0

place to talk(neg)

-0

-0

Sumof2179otherfeatures

+0

0

1

2

3

4

Concept contributions

Pred class: cockpit|logit:9.717|bias:-0.879

computerscreens(pos)

+3

controls(pos)

+1.95

controlsfor the airplane(pos)

+1.69

instrumentsfor navigation(pos)

+1.5

flight engineer(pos)

+1.3

theareawherethepilotsits(pos)

+0.77

seatbelts(pos)

+0.4

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Conceptcontributions

Pred class:marsh|logit: 7.780|bias: 0.247

oftenmurkyorstagnantwater(pos)

+2.53

reeds(pos)

+2.46

wetland(pos)

+1.22

habitat(pos)

+0.66

slow moving water(pos)

+0.52

canoe(pos)

+0.14

place to talk(neg)

-0

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

Conceptcontributions

Pred class:water park|logit:9.286|bias:-0.284

slidesorotherwaterfeatures(pos)

+3.27

recreationarea(pos)

+3.19

pool or pools of water(pos)

+1.55

sunshine!(pos)

+1.33

various attractions and rides(pos)

+0.17

large outdoorspace(pos)

+0.06

pool cover(pos)

+0

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.52.02.5

3.0

3.5

Conceptcontributions

Pred class: doorway,outdoor|logit:8.379|bias:-0.077

largerectangular opening(pos)

+3.36

exit(pos)

+2.98

doorhandle(pos)

+2.12

place wherepeople gather(neg)

-0

place to work(pos)

+0

place towashbeforeprayer(pos)

+0

place to talk(pos)

0+

Sum of 2179otherfeatures

+0

0.0

0.5

1.01.52.02.5

3.0

3.5

Concept contributions

Pred class:veterinarians office|logit:8.467|bias:-0.286

cat(pos)

+3.04

gerbil(pos)

+2.88

receptionist desk(pos)

+1.41

kennel area(pos)

+0.94

officesforthevets and staff(pos)

+0.48

place wherepeople gather(pos)

+0

placeto washbeforeprayer(neg)

-0

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Concept contributions

Pred class: arena,hockey|logit: 11.303|bias:-0.857

nets at eitherend(pos)

+5.21

linesmarking outboundaries(pos)

+3.91

puck(pos)

+2.38

FROSTY

rink(pos)

+0.66

placewherepeoplegather(neg)

-0

place towashbeforeprayer(neg)

-0

place to talk(neg)

-0

Sumof2179otherfeatures

+0

0

1

2

3

4

5

6

Concept contributions

Pred class: church, indoor|logit: 7.147|bias:-0.195

rows of pews(pos)

+3.67

stained glasswindow(pos)

+2.79

pews(pos)

+0.63

lectern(pos)

+0.14

(sod)a!dind

+0.11

placetowashbeforeprayer(pos)

+0

place to talk(neg)

-0

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.52.02.53.0

3.5

4.0

Conceptcontributions

Pred class: home office|logit: 9.178|bias:-0.053

bookshelves or cabinets(pos)

+7.32

phones(pos)

+1.07

+0.44

scanners(pos)

+0.4

place towork(pos)

+0

placetowashbeforeprayer(neg)

-0

place to talk(pos)

+0

Sumof2179otherfeatures

+0

0

1

2

3

4

5

6

7

8

Conceptcontributions

Pred class:kasbah|logit: 6.381|bias:-0.184

made ofstone oradobe(pos)

+2.37

found innorthafrica(pos)

+2.16

+1.02

square(pos)

+0.96

minaret(pos)

+0.06

placetowashbeforeprayer(pos)

+0

place totalk(neg)

-0

Sumof2179otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

Concept contributions

Pred class:White eyed Vireo|logit:11.082|bias:-0.008

grayish wings and tail(pos)

+4.35

olive upperparts(pos)

+3.76

yellowish sides and flanks(pos)

+2.07

greenish upperparts(pos)

+0.47

white eye ring(pos)

+0.44

darkstreaks on thebreast(pos)

+0

hard crystalline structure(pos)

+0

Sumof662otherfeatures

+0

0

1

2

3

4

5

Conceptcontributions

Pred class:Indigo Bunting|logit:11.418|bias:-0.027

thin pointed beak(pos)

+3.67

whitestreaksonthewings(pos)

+3.44

shortbeak(pos)

+1.64

small bird(pos)

+1.3

blue plumage(pos)

+0.97

blue body(pos)

+0.42

hard crystalline structure(pos)

+0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.52.02.53.0

3.5

4.0

Conceptcontributions

Pred class: Cedar Waxwing|logit:8.505|bias:-0.027

pointy crest on the head(pos)

+2.61

small brown body(pos)

+2.14

yellowtipped tail(pos)

+2.13

yellow tips on the wings(pos)

+1.65

grey sides(pos)

+0

darkstreaksonthebreast(pos)

+0

iridescentbluegreenback(pos)

+0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Concept contributions

Pred class:Gray crowned Rosy Finch|logit:6.393|bias:-0.017

pinkish breast(pos)

+1.75

gray crown(pos)

+1.66

rosybreast(pos)

+1.2

darkstreaksonthebreast(pos)

+0

hard crystalline structure(pos)

+0

grayish brown head(pos)

+0

Sumof662otherfeatures

+0

0.00

0.25 0.50 0.75 1.001.25 1.50 1.75 2.00

Conceptcontributions

blackeye stripe(pos)

+2.2

olive gray upperparts(pos)

+1.71

yellowish underparts(pos)

+1.55

pointed bill(pos)

+0.89

white wing bars(pos)

+0.12

hard crystallinestructure(neg)

-0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

GregW.Lasley/CLO

Conceptcontributions

short pointed beak(pos)

+5.72

blue head(pos)

+4.59

green back(pos)

+1.76

shortstoutbill(pos)

+1.15

small songbird(pos)

+0.63

grey sides(pos)

+0

darkstreaksonthebreast(pos)

+0

Sumof662otherfeatures

+0

0

1

2

3

4

5

6

Conceptcontributions

Pred class:Northern Flicker|logit:8.439|bias:-0.009

red breastmalesonly(pos)

+2.94

brownish red breast(pos)

+2.38

blackbibwithwhite stripes(pos)

black tailwith whitebars(pos)

+0.89

white rump(pos)

+0.42

grey sides(pos)

+0

iridescentblue greenback(pos)

+0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Conceptcontributions

glossy black plumage(pos)

+3.02

glossy sheen to its feathers(pos)

+2.64

small black bird(pos)

+1.6

white spots on its wings(pos)

+1.41

long pointy beak(pos)

+1.41

long pointed tail(pos)

+0.18

hard crystalline structure(pos)

+0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Conceptcontributions

Pred class: Shiny Cowbird|logit: 5.787|bias:-0.025

conical bill(pos)

+3.07

shinyblackplumage(pos)

+2.46

small bill(pos)

+0.29

iridescentbluegreenback(neg)

-0

hard crystalline structure(neg)

-0

-0

blackstripedownitsneck(neg)

-0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.52.0

2.5

3.0

3.5

Conceptcontributions

Pred class:Horned Grebe|logit: 7.186|bias:-0.011

long sharp bil(pos)

+3.75

small compactbody(pos)

+1.46

small pointed bill(pos)

+1.1

blackandwhitewings(pos)

+0.7

black tail(pos)

+0.12

long thin neck(pos)

+0.08

darkstreaksonthebreast(pos)

+0

Sumof662otherfeatures

+0

0.0

0.5

1.0

1.52.02.53.03.5

4.0

Conceptcontributions

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our introduction in Section 1 summarizes the contribution and scope of this paper accurately.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a full set of assumptions in our Theorem 4.1 and we provide complete and correct proof in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 5, we present the settings of our experiments. We present more implementation details on Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The link to the project webpage is provided in the abstract and the code will be released by poster deadline.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include experimental details in Section 5 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiments are very computationally expensive to repeat and measure the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide computational resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research is conducted following NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:This work focus on technical development of making neural network models more interpretable.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite the dataset and models we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a learning pipeline named VLG-CBM, which enhances CBM with the aid of open-domain object detectors to create more accurate concept labels, while also excluding image-irrelevant concepts (e.g,. \"loud music\").\nAlso, in order to prevent the 'information leak' in CBMs, which stands for a phenomenon where the exploitation of irrelevant concepts may help downstream accuracy, authors propose a metric named Number of Effective Concepts (NEC) which enables the fair comparison of different CBMs within a similar amount of information being provided.\nExperimental results show that with the proposed VLM-CBM, high-quality concept labels are obtained therefore achieving better performances within similar NEC.\n\nSTRENGTHS:\n1. The idea of applying a grounding model to generate an accurate concept label is sound and seems effective as shown in the experimental results.\n\n2. The theoretical analysis of random CBM well explains the problem of information leaks in CBMs, and the proposed NEC is a proper method to measure the information being provided to a CBM classifier, enabling fair comparison of CBMs within a similar amount of information being provided.\n\n3. The paper is generally well-written, making it easy to follow the main ideas and claims.\n\nWEAKNESSES:\n1. Only experimental results under controlled NEC are reported, making it difficult to compare the performance of VLG-CBMs with previous works in conventional settings. For instance, the average accuracy of LaBo in the CIFAR-100 dataset under an NEC-controlled setting is reported as 55.18%, while the full performance in the original paper is around 86% (86.82 in the dev set, 86.04 in the test set). Therefore, also comparing the performance of VLG-CBM under conventional settings in previous work would make the comparison easier.\n\n2. The idea of applying a detection or a grounding module to generate an image annotation is already widely explored (e.g., KOSMOS-2 [1]). The proposed data generation process does not significantly differ from previous approaches, limiting the originality of the proposed method.\n\n\n[1] Peng et al., KOSMOS-2: Grounding Multimodal Large Language Models to the World.\n\nQUESTIONS:\n1. Questions regarding Table 3. 1) How is the performance of each method? Only the percentage of changed decisions are reported, therefore providing accuracies may help a better understanding of the effectiveness of each method. 2) Isn't it obvious that the percentage of changed decisions is lower in VLG-CBM since the sparsity constraint is not applied to other methods? Can you also provide the same analysis with sparsity constraint applied to other methods?\n\n2. Providing qualitative examples of labeling results using Grounding-DINO would help in understanding the quality of the auto-labeled dataset, which seems to be absent in the current version of the paper.\n\n3. Also, please address the concerns elaborated in the weakness section.\n\nLIMITATIONS:\nAuthors have described limitations and possible societal impacts in the main paper and the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses foundational models to automate the generation of concept annotations which are then used to train a concept bottleneck model (CBM). To generate the concept set, the paper extends prior scalable CBM approaches[1] by using open-domain object detectors to weed out concepts that are not visually grounded. Concepts that are not present in any bounding-box for any training image are removed. Following existing work,  a CBM is trained sequentially – first, the concept layer is trained and frozen, then a sparse linear layer is trained. The paper introduces a theoretically justified metric, Number of effective concepts (NEC), which is the average number of concepts the sparse final layer uses to make the class prediction. NEC can be used to compare between various CBMs. Results are shown on 5 standard datasets, and they compare against 3 scalable CBM baselines.\n\n[1] Oikarinen, Tuomas P. et al. “Label-Free Concept Bottleneck Models.” ArXiv abs/2304.06129 (2023): n. Pag.\n\nSTRENGTHS:\n\\+ The paper is written well and easy to follow.  \n\\+ The bounding boxes generated from open-domain object detectors are used to i) filter out concepts which are not visually grounded ii) augment training of the concept layer by cropping images to bounding-boxes\nBoth these design decisions seem crucial for learning a better, more interpretable concept layer.  I would find it interesting if this was studied in greater depth.  \n\\+ The paper quantifies existing thought that having a dense prediction layer limits the applicability of CBMs. NEC can be used to fairly compare across CBMs.  \n\\+ Empirical results are convincing. Their CBM outperforms existing baselines on classification accuracy across various levels of final layer sparsity.\n\nWEAKNESSES:\n\\- It is not clear if NEC controls information leakage. As defined in [2], information leakage happens because of a `soft’ concept layer (that predicts probability of a concept instead of presence and absence). Though a small NEC reduces the possibility of information leakage and is a useful metric to check the goodness of the concept set.  \n\\- A study of time and compute complexity is needed, as every image has to be passed through a large object-detector. The effort to generate concepts is much higher (compared to [1], who generate concepts class wise).  \n\n[1] Oikarinen, Tuomas P. et al. “Label-Free Concept Bottleneck Models.” ArXiv abs/2304.06129 (2023): n. Pag.\n[2] Mahinpei, Anita et al. “Promises and Pitfalls of Black-Box Concept Learning Models.” ArXiv abs/2106.13314 (2021): n. pag.\n\nQUESTIONS:\n1. A study of the impact of proposed visually-grounded concept filtering would be interesting. For ex. What fraction of quantitative gains are because of the removed concept set? How much does augmentation contribute to overall performance gains?  \n2. As mentioned in the weaknesses section, a section on time complexity of concept set generation would be useful.\n3. The theoretical analysis, though appreciated, seems out of place. The result presented in Thm 4.1 is not specific to concept models, but can be seen as a general statement on fidelity of performing dimensionality reduction with a linear random projection. It is not clear to me how NEC follows from here, as NEC is a measure of the sparsity of the final layer, which can be low even for large concept sets.\n\nLIMITATIONS:\nYes, limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a new way to implement concept bottleneck models (CBMs), which use pretrained object detectors (Grounding-DINO) to filter and annotate the concepts. This step can make the concepts more visual and groundable, which improves the reliability of the concept predictor. This paper proposed a new metric (NEC) to control the number of concepts that the model uses when making the predictions. In the experiments of 5 datasets, the proposed model shows better performance using the NEC metric.\n\nSTRENGTHS:\nThe idea of using the grounding model can help improve concepts' groundability. The paper is easy to follow in general.\n\nWEAKNESSES:\n1. **Limited contribution.** This paper is an incremental work to previous LLM-guided CBMs [1, 2] by adding a step to filter concepts with an object detection model. The findings that \"CBM with random concepts can achieve good performance\" have also been shown in previous works [3, 4]. From the technical part, the only difference between the proposed model and prior CBMs is the object detection model for concept filtering. However, the detected bounding boxes are not used in the model training and inference. Therefore, the technical contribution is also limited. Overall, this paper makes limited contributions to the community and does not provide enough insights for future work.\n\n2. **Generalizability of the method.** The proposed method relies on an open-vocab detection model. However, open-vocab detection is still a very challenging task, and from my experience, the off-the-shelf models do not work well in many cases. For example, part-based detection is very difficult. In some datasets like DTD or action detection, the concepts are hard to capture in bounding boxes. Even in the evaluated datasets in this paper, like CIFAR, which is in low resolution, the performance of object detection is also quite skeptical. Although the authors have mentioned this limitation in the last section, I don't agree with their arguments that \"prior work (e.g., LaBo, LM4CV, LF-CBM) also shared similar limitations on the reliance of CLIP\". CLIP definitely has better generalization than the open-vocab detection model because the latter is a more challenging task. This concern is also reflected by the small set of datasets in their experiments, only 5 datasets, compared to what LaBo evaluated (11 datasets).\n\n3. **Baseline Comparison.** The linear probing baseline is missing, which is necessary to justify the effectiveness of CBMs. For CUB, Place365, and ImageNet, the numbers of LM4CV and LaBo are missing. The reason shown in the caption that \"they could not be applied on non-CLIP backbones\" cannot convince me. I didn't see any difficulties in letting your model support CLIP. You can simply fine-tune CLIP using BCE loss in Eq (6) or just use CLIP as a feature extractor and learn a linear layer over it. In general, the numbers shown in Table 2 are quite low, far behind linear probing based on my experience. It seems the proposed method sacrifices a lot of performance for interpretability, which is not validated properly (see weakness 4). I feel that the performance gain of the proposed method comes from the priors of the object detection model.\n\n4. **Human evaluation to justify faithfulness.** This paper claims that the proposed VLG-CBM is more interpretable and faithful. However, without concrete evidence like large-scale human evaluation, it is hard to justify VLG-CBM as more interpretable by just showing a few qualitative examples.\n\n[1] Yang et al. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023.\n[2] Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR 2023.\n[3] Roth et al. Waffling around for Performance: Visual Classification with Random Words and Broad Concepts. ICCV 2023.\n[4] Yan et al. Learning Concise and Descriptive Attributes for Visual Recognition. ICCV 2023.\n\nQUESTIONS:\nHow is the threshold of detection confidence (T=0.15) chosen? Can this generalize to different datasets?\n\nLIMITATIONS:\nThe authors discussed the limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Vision-Language-Guided Concept Bottleneck Model (VLG-CBM), an innovative approach to training Concept Bottleneck Models (CBMs) using vision-language models (VLMs). This method aims to improve the faithfulness and performance of CBMs by addressing the limitations of existing models, specifically inaccurate concept predictions and information leakage. VLG-CBM leverages grounded object detectors for visually recognizable concept annotations and introduces a new metric, the Number of Effective Concepts (NEC), to control information leakage and enhance interpretability. The method demonstrates significant performance improvements across multiple benchmarks.\n\nSTRENGTHS:\n1) The paper is well written with a clear logical flow.\n\n2) The approach automates the creation of concept datasets using grounded object detectors, which means there's no need for manual annotations. This saves time and effort, making the method easier to scale up for larger datasets. Plus, it reduces the chances of human error in the data, leading to more reliable training results.\n\n3) The paper provides theoretical explanations for information leakage in Concept Bottleneck Models (CBMs). By introducing the Number of Effective Concepts (NEC) metric, the authors offer a way to control and measure information leakage, enhancing the overall integrity and interpretability of the model.\n\n3) The VLG-CBM model shows significant improvements in accuracy across five standard benchmarks, with gains ranging from 2.4% to 7.6% over current methods.\n\nWEAKNESSES:\n1）The proposed pipeline integrates several independent models, ie, LLM for concept candidate generation, and GroundingDINO for bounding box detection. I'm wondering how well the grounding detector can do in discovering the concept candidate.  And if the detector fails to detect a key concept, will the final recognition result be affected? \n\n2) Dependence on Pre-trained Models: The reliance on pre-trained VLMs and object detectors may limit the approach's applicability to domains where such models are not readily available or effective. \n\n3) My major concern is in generalization. While the method shows impressive results on standard image recognition benchmarks, its effectiveness in more diverse or less-structured real-world scenarios needs further validation. For example, how would the VLG-CBM perform for OOD image recognition like NICO or Water-Bird, or how could it handle more complex tasks like VQA?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nsee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a framework that leverages existing grounding models to refine the accurate concept annotations, and therefore enables better CBM training. Also, a new metric is proposed to avoid leakage and improve interoperability. Experiments and visualizations are conducted to demonstrate the superior performance of the proposed method.\n\nSTRENGTHS:\n1. For methodology, incorporating a grounding model to get faithful concepts, and proposing a metric - Number of Effective Concepts to control the information leak and supervise sparse layer optimization are interesting and novel to a certain degree in this field.\n\n2. The proposed method outperforms previous approaches in multiple widely used benchmarks.\n\nWEAKNESSES:\n1. Missing comparison with recent works in CMB, for example,  [1] and [2].\n\n2. How good is LLM at generating candidate concepts, and how the different LLMs' quality would impact the generated candidate quality as well as final performance? An ablation/analysis is needed.\n\n3. How good is the grounding model in terms of object class matching and box location prediction? The ablation of different grounding models and even a random baseline should be added. I am also curious about the training augmentation based on bounding box, how much gain it can bring?\n\n4. It's natural that different classes have different numbers of concepts to distinguish the class because some are complex and some are simple. If forcing the NEC to certain number, would the model automatically learn to assign different numbers of concepts to different classes? An analysis is needed.\n\nRefs:\\\n[1]. Pham, Thang M., et al. \"PEEB: Part-based Image Classifiers with an Explainable and Editable Language Bottleneck.\" arXiv preprint arXiv:2403.05297 (2024). \\\n[2]. Sun, Ao, et al. \"Eliminating information leakage in hard concept bottleneck models with supervised, hierarchical concept learning.\" arXiv preprint arXiv:2402.05945 (2024).\n\nQUESTIONS:\nIn summary, my concerns mainly focus on the lack of comprehensive comparison, ablation, and analysis.\n\nLIMITATIONS:\nYes, it is discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}