Label-free Concept Bottleneck Models
Authors: Tuomas Oikarinen
Subhro Das
Lam M. Nguyen
Tsui-Wei Weng
Venue: ICLR 2023 poster (2023)
Keywords: Interpretability
Explainability
Concept Bottleneck Models

Abstract: Concept bottleneck models (CBM) are a popular way of creating more interpretable neural networks by having hidden layer neurons correspond to human-understandable concepts. However, existing CBMs and their variants have two crucial limitations:  first, they need to collect labeled data for each of the predefined concepts, which is time consuming and labor intensive; second, the accuracy of a CBM is often significantly lower than that of a standard neural network, especially on more complex datasets. This poor performance creates a barrier for adopting CBMs in practical real world applications. Motivated by these challenges, we propose Label-free CBM which is a novel framework to transform any neural network into an interpretable CBM without labeled concept data, while retaining a high accuracy. Our Label-free CBM has many advantages, it is: scalable - we present the first CBM scaled to ImageNet, efficient - creating a CBM takes only a few hours even for very large datasets, and automated - training it for a new dataset requires minimal human effort. Our code is available at https://github.com/Trustworthy-ML-Lab/Label-free-CBM.

Published as a conference paper at ICLR 2023
LABEL-FREE CONCEPT BOTTLENECK MODELS
Tuomas Oikarinen
Subhro Das
UCSD CSE
MIT-IBM Watson Al Lab, IBM Research subhro. das@ ibm. com
toikarinen@ucsd.eer
Lam M. Nguyen
Tsui-Wa Weng
IBM Research
UCSD HDSI
lamnguyen. ml td hmm.bo...
Iweng@ucsd. edu
ABSTRACT
Concept bottleneck models (CBM) are a popular way of creating more inter- pretable neural networks by having hidden layer neurons correspond to human- understandable concepts. However, existing CBMs and their variants have two crucial limitations: first, they need collect labeled data for each of the predefined concepts, which is time consuming and labor intensive; second, the accuracy of a CBM often significantly lower than that of standard neural network, especially on more complex datasets. This poor performance creates a barrier for adopting CBMs in practical real world applications. Motivated by these challenges, we pro- pose Label-free CBM which is novel framework to transform any neural network into interpretable CBM without labeled concept data, while retaining a high ac- curacy. Our Label-free CBM has many advantages. it is: scalable we present the first CBM scaled ImageNet, efficient creating a CBM takes only few hours even for very large datasets, and automated training it for new dataset requires minimal human effort. Our code available https://github.com wwwtt/orttstttty...o......o Hil-Laab/Labl--fre-CCC
INTRODUCTION
Deep neural networks (DNNs) have demonstrated unprecedented success in a wide range of ma- chine learning tasks such computer vision, natural language processing, and speech recognition. However, due to their complex and deep structures, they are often regarded as black-box models that difficult understand and interpret. Interpretable models are important for many reasons such as creating calibrated trust in models, which means understanding when we should trust the models. Making deep learning models more interpretable is active yet challenging research topic.
Input
concepts
Output y
O
mast and sails
O
ship
Pirate ship
Any
:
Backbone
O
carnival
f(x)
Label-free CBM:
automated,scalale efficient, concept labels required
Figure 1: Our proposed Label-free CBM has many desired features which existing CBMs lack, and can transform any neural network backbone into an interpretable Concept Bottleneck Model.

---

Published as a conference paper at ICLR 2023
One approach to make deep learning more interpretable is through Concept Bottleneck Models (CBMs) (Koh et al., 2020). CBMs typically have a Concept Bottleneck Layer before the (last) fully connected layer of the neural network. The concept bottleneck layer is trained to have each neuron correspond to a single human understandable concept. This makes the final decision a linear function of interpretable concepts, greatly increasing our understanding of the decision making. Importantly, CBMs have been shown to be useful in a variety of applications, including model debugging and human intervention on decisions. However, there are two crucial limitations of existing CBMs and their variants (Koh et al., 2020; Yuksekgonul et al., 2022; Zhou et al., 2018): (i) labeled data is required for each of the predefined concepts, which time consuming and expensive to collect; (ii) the accuracy of a CBM is often significantly lower than the accuracy of a standard neural network,
especially on more complex datasets.
To address the above two challenges, we propose new framework named Label-free CBM, which capable of transforming any neural network into an interpretable CBM without labeled concept data while preserving accuracy comparable to the original neural network by leveraging foundation models (Bommasani et al. 2021). Our Label-free CBM has many advantages:
it is scalable to our best knowledge, it is the first CBM that scales to ImageNet it is efficient creating CBM takes only few hours even for very large datasets it is automated training it for new dataset requires minimal human effort
2 RELATED WORK
Post-hoc explanations Samek et al. 2021): The approach of post-hoc explanations includes some classic methods such as LIME (Ribeiro et al. 2016) and SHAP (Lundberg & Lee 2017). which try explain individual model decisions by identifying which parts of the input data (e.g- pixels)
the most important for a given decision. However, these methods are based on local approximations of the DNN model and as such are not always accurate. Further, the explanations the granularity of input pixels may not always be helpful and could require substantial subjective analysis from human. In contrast, our explanations in Section 4 are not approximated and explain predictions in terms of human- mnmrann-ndesttadbbllee concepts.
(Wong et al., 2021) proposes making the FC layer sparse, and develop an efficient algorithm for doing so. They show that sparse models are more interpretable in many ways, but still suffers from the fact the previous layer features are not interpretable. NBDT (Wan et al. 2020) propose replacing the final layer with neural backed decision tree for another form of more interpretable decisions. Other approaches to make NNs more interpretable include Concept Whitening (Chen et al. 2020 and Concept Embedding Models (Zarlenga et al. 2022).
More interpretable final layer:
CBM: Most related to our approach are Concept Bottleneck Models (Koh et al. 2020 Losch et al., 2019) which create layer before the last fully connected layer where each neuron corresponds to human interpretable concept. CBMs have been shown to be beneficial by allowing for human test- time intervention for improved accuracy, well being easier to debug. To reduce the training cost of a CBM, a recent work (Yuksekgonul et al. 2022) proposed Post-Hoc CBM that only needs to train the last FC layer along with an optional residual fitting layer, avoiding the need to train the backbone from scratch. This is done by leveraging Concept Activation Vectors (CAV) (Kim) al. 2018) or the multi-modal CLIP model (Radford al., 202T). However, the post-hoc CBM does not fully address the problems of the original CBM as using TCAV still requires collecting annotated concept data and their use of CLIP model can only be applied if the NN backbone the CL.IP image encoder. Additionally, the performance of post-hoc CBMs without uninterpretable residual fitting layers often significantly lower than the standard DNNs. Similarly, an earlier work Interpretable Basis Decomposition (Zhou et al. 2018) proposes learning concept bottleneck layer based on labeled concept data for explanable decisions, even though they do not call themselves a CBM. Comparison between the features our method and existing approaches is shown in Table D
Model editing//ebugging: Our approach is related to a range of works proposing ways to edit networks, such (Bau et al. 2020. Wang et al. 2022) for generative vision models, (Bau et al.) 2020) for classifiers, or (Meng et al. 2022; Mitchell et al. 202I) for language models. In addition Abid et al. 2021) propose way debug model mistakes using TCAV activation vectors.
2

---

Published as a conference paper at ICLR 2023
() Flexibility
(lI Interpretability
(III) Performance
Without labeled
Any network architecture
Sparse
All features interprelable
Preserves accuracy
Extends to
Method:
concepl data
final layer
ImageNet scale
CBM
No
Yes
No
Yes
No.
No
IBD
No
Yes
No
No
Yes
No
P-CBM
No
Yes
Yes
Yes
No
No
P-CBM (CLIP)
No
Yes
Maybe
P-CBM-h
No
Yes
Yes
No
Yes
No
P-CBM-h (CLIP)
Yes
No
Yes
No
Yes
Maybe
Label-free CBM
Yes
Yes Yes
Yes
Yes
Yes
(This work)
Table 1: Comparison of our method against existing methods for creating Concept Bottleneck mod- els, CBM (Koh et al. (020), IBD) (Zhou et al.] (2018) and versions of P-CBM (Yuksekgonul et al.) 2022), where '-h" indicates the hybrid model that uses uninterpretable residual term, "(CLIP) means models using CLIP concepts. We used maybe to indicate models that could in theory extend to Im- ageNet but have not been tested.
CLIP-Dissect (Oikarinen & Weng p022): CLIP-Dissect recent method for understanding the roles of hidden layer neurons by leveraging the CLIP multimodal model (Radford al., [22TT) can provide a score of how close any neuron to representing any given concept without the need of concept annotation data, which makes it useful optimization target for learning interpretable projection Step 3.
LABEL-FREE CBM: A NEW FRAMEWORK TO BUILD CBM
In this section, we propose Label-free CBM, novel framework that builds a concept bottleneck model (CBM) in an automated, scalable and efficient fashion and addresses the core limitations of existing CBMs. Given neural network backbone, Label-free CBM transforms the backbone into an interpretable CBM without the need concept labels with the following steps which are illustrated in Fig 2 Step 1: Create the initial concept set and filter undesired concepts; Step 2: Compute embeddings from the backbone and the concept matrix on the training dataset; Step 3: Learn projection weights W. create Concept Bottleneck Layer (CBL); Step 4: Learn the weights Wr of the sparse final layer to make predictions.
Note that the backbone model trained on a different task. The details of Label-free CBM for Step are provided in Sec 3.1, Step and Step 3 in Sec 3.2, and finally Step 4 in Sec 3.3.
either be model trained the target task, general model
Step Compute embedding fix) & concept
Step 3: Compute coefficient W;
Step 4: Train a sparse FC layer W, on f((x)
by max similarity between
Input imgs Xy Xx
Embeddings
Concept bottleneck layer (CBL)
Predicted class
argmax ata) X)
Label-free CBM
concept
Bridge, can, cloud
Backbone
transform
91
9:
9x
f(x)
f.00 (6)
WeleOo)
(Gs) ((m)
[=Om))
Image Enc
Concept Matrix Git) E4t)))
(Notations) x, E Rd Rd ganaa
and filter concept
Step
Target concepts
Inner product
Antertica cashier butter dirt markings
Elky) Edit))
Text Enc E
F2gy) E,0g)
lele nM
gd,
Filtering
Figure Overview our pipeline for creating label-free CBM.
3

---

Published as a conference paper at ICLR 2023
3.1 STEP 1: CONCEPT SET CREATION AND FILTERING
In this step, we will describe how to create a concept set to serve as the basis of human-interpretable concepts in the Concept Bottleneck Layer. This step consists of two sub-steps: A. Initial concept set creation and B. Concept set filtering.
A. Initial concept set creation: A concept set refers to the set of concepts represented in the Con- cept Bottleneck Layer. In the original CBM paper (Koh et al. 2020), this is decided by domain experts as the set of concepts that are important for the given task. However, since our objective is to automate the entire process of generating CBMs, we don't 't want to rely on human experts. Instead, we propose generating the concept set via GPT-3 (Brown et al. 2020) using the OpenAl API. Some- what surprisingly, GPT-3 has good amount of domain knowledge of which concepts are important for detecting each class when prompted in the right way. Specifically, we ask GPT-3 the following:
List the most important features for recognizing something as a {class}:
List the things most commonly seen around a {class):
Give superclasses for the word {class):
Note that the {class} in here refers to the class name in the machine learning task. For GPT-3 to perform well on the above prompt, we provide two examples of the desired outputs for few-shot adaptation. Note that those two examples can be shared across all datasets, input is needed for generating concept set for a new dataset. Full prompts and example outputs are illustrated in the Appendix Figures 10 and [T]] To reduce variance combine the results. Combining the concepts received from different classes and prompts gives us large, somewhat noisy set of initial concepts, which we further improve by filtering. We found using GPT-3 to generate initial concepts to perform better than using the knowledge-graph ConceptNet (Speer & Havasi 2013) which was used in Post Hoc-CBM (Yuksekgonul et al., 2022). Reasons for this and a comparison between the two are presented in Appendix|A.0, AA.
additional user
each prompt twice and
B. Concept set filtering: Next we employ several filters improve the quality and reduce the size of our concept set, as stated below:
1. Concept length: We delete any concept longer than 30 characters in length, to keep concepts simple and avoid unnecessary complication.
2. Remove concepts too similar classes: We don't want our CBM to contain the output classes themselves as that would defeat the purpose of an explanation, To avoid this we remove all concepts that are too similar the names of target classes. We measure this with cosine similarity in text embedding space. In particular we use an ensemble of similarities in the CLIP VřT-B/16 text encoder as well the all-mpnet-buse- encoder space, so our measure can be seen as a combination of visual and textual similarity. For all datasets, we deleted concepts with similarity > 0.85 to any target class.
sentence
3. Remove concepts too similar each other: We also don't want duplicate or synonymous concepts in the bottleneck layer. We use the same embedding space as above, and remove any concept that has another concept with > 0.9 cosine similarity already in concept set.
4. Remove concepts not present in training data: To make sure our concept layer accurately presents its target concepts, we remove any concepts that don't 't activate CLIP highly. This cut-off is dataset specific, and we delete all concepts with average top-5 activation below the cut-off.
5. Remove concepts we can't ' project accurately: Remove neurons that are not interpretable from the CBL. This step is actually performed after step 3 and is described in section 3.2
We discuss the reasoning behind the filters in detail in Appendix A.5 and perform abblation study on their effects in Appendix A.4
3.2 STEP 2 AND 3: LEARNING THE CONCEPT BOTTLENECK LAYER (CBL)
Once the concept set is obtained, the next step is to learn a projection from the backbone model's feature space into space where axis directions correspond to interpretable concepts. Here,
4

---

Published as a conference paper at ICLR 2023
present way of learning the projection weights We without any labeled concept data by utilizing CLIP-Dissect (Oikarinen & Weng 2022). To start with, we need a set of target concepts that the bottleneck layer is expected to represent C = (ti,..,tal
well as training dataset (e.g. images) D { ) of the original task. Next we calculate and save the CLIP concept activation matrix P where P.,, = (x())) (t,) and Fy and Er are the CLIP image and text encoders respectively. W. is initialized as a random M x do matrix where do is the dimensionality of backbone features f(z). The initial set C' is created in Step 1 and the training set D is provided by the downstream task. We define fc(x) = W (z), where fc(z;) E RM, We use k to denote a neuron of interest in the projection layer, and its activation pattern is denoted as qu where 9k (Se, (a.),...,, Sa. (xx))] with Ou R. and f=,(()) = [fe(z)]a.
To make the neurons in the CBL interpretable, we need to enforce the projected neurons to activate in correlation with the target concept, which we do by optimizing W. to maximize the CLIP-Dissedd similarity between the neuron's s activation pattern and the target concept. To optimize this similarity, we have designed new fully differentiable similarity function sim(t,, 9:) that can be applied to CLIP-Dissect, called cos cubed, that still achieves very good performance in explaining the neuron functionality as shown in Appendix A.3 Our optimization goal is to minimize the objective L over We as defined in Equation (
M
3 P.a
L(W.) = Σ sim(l,, 9:)
=
11 1a2l
Here indicates vector q normalized to have mean 0 and standard deviation 1, and the cubed similarity sim(t,, q)) is simply the cosine similarity between two activation vectors after both have been normalized and raised to third power wise The third power is necessary make the similarity more sensitive to highly activating inputs. As this is still cosine similarity, it takes values between [- We optimize I((W)) using the Adam optimizer on training data D, with early stopping when similarity on validation data starts to decrease. Finally to make sure our concepts are truthful, we drop all concepts with sim(t,, 9) < 0.45 on validation data after training W.. This is the 5th concept set filter from Sec 3.1. This cutoff was selected manually as good indicator of a neuron being interpretable. During this filtering, the number of concept reduced: M +- M A, where A is non-negative integer representing the number of concepts being removed in this step. Note that matrix Wa has also be updated accordingly by removing the rows that corresponds to the removed concepts, and with our notation, W. IR al xd To simplify Figure we omit plotting this concept removal step.
3.3 STEP 4: LEARNING THE SPARSE FINAL LAYER
Now that the Concept Bottleneck Layer learned, the next task is to learn the final predictor with the fully connected layer Wp E Rd XM where d, is the number of output classes. The goal is to keep WF sparse, since sparse layers have been demonstrated to more interpretable (Wong et al. (021]) I). Given that both the backbone (xx and learned concept projection W. are fixed, this is problem of learning a sparse linear model, which can be solved efficiently with the clastic net objective:
min Σ Lee (Wefo(x.) bp. M1 + (WFF) We.be
allWr|l1.1. || ||r denotes the Frobenius norm, II.1 denotes element wise matrix norm, by denotes the bias of the FC layer, L. is the standard cross- entropy loss and y/4 is the ground-truth label of data z, We optimize Equation (2) using the GLM- SAGA solver created by (Wong et al. 2021). For the sparse models, we used chosen such that each model has 25 to 35 nonzero weights per output class. This level was found to still result in interpretable decisions while retaining good accuracy. Depending on the number of features/concepts in the previous layer this corresponds to 0.7-15% of the weights of the model being nonzero.
where (Wr) = (1)
0.99 and À was
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENT RESULTS
We present three main results on evaluating the accuracy and interpretability of the Label-free CBM in this section. Due to page limit, additional experiments and discussions are in Appendix A.2 A.12 An overview of additional experiments is provided in Appendix A.
Datasets. To evaluate our approach, we train Label-free CBMs on 5 datasets. These CIFAR-IO.
CIFAR-100 (Krizhevsky et al.] 2009). CUB (Wah et al. ( Places365 (Zhou et al. 2017) and ImageNet (Deng et al. |ll (009). This is a diverse set of tasks, where CIFAR-10/100 and ImageNet are general image classification datasets, CUB is a line-graine
bird-species classification dataset and Places365 is focused on scene recognition. Their sizes vary greatly, with CUB having 5900 training samples, CIFAR datasets 50,000 each and ImageNet and Places365 have 1-2 million training images. CUB contains annotations with 312 concepts for each datapoint, such as has wing color:blue or has head ppttern:spotted. We used neither the concept names or the concept annotations to train our LF- CBM models to showease our ability to perform without labels, yet our method discovered similar concepts and is competitive with methods that utilize the available concept information.
Setup. For CIFAR and CUB we use the same backbone models as YYuksckgonul et al. 2022) for fair comparison, we use CLIP(RN50) image encoder as the backbone for CIFAR, and ResNet- trained on CUB from imgelsmob| for CUB. For both ImageNet and Places365 we use ResNet-50 trained on ImageNet as the backbone network. The number of concepts each model uses is roughly proportional to the number of output classes for that task, as each class adds more initial concepts. The number of concepts for our models are as follows: 128 for CIFAR-IO, 824 for CIFAR-100, 211 for CUB, 2202 for Places-365 and 4505 for ImageNet. CUB has a smaller number of concepts because we only used the important features prompt. All models are trained on a single Nvidia Tesla P100 GPU, and the full training run takes anywhere from few minutes to 20 hours depending on the dataset size. The majority of runtime is taken by step 2, where we save activations for both the backbone and CLIP over the entire training dataset. Fortunately, these results only need to be calculated once and can be reused for training new CBMs on the same dataset. Once the activations have been saved, learning the model takes less than 4 hours on all datasets.
Result (I): Accuracy, Table shows the performance of Label-free CBM on all 5 datasets. We can that our method can create CBM with sparse final layer on with little loss in accuracy all datasets, including a model with 72% top-1 accuracy on the ImageNet. Our Label-free CBM has significantly higher accuracy than Post-hoc CBM the datasets we evaluated, but some rows for P-CBM are N/A as they do not provide results those results and it is unclear how to scale P-CBM to larger datasets. For Table we excluded methods with non-interpretable components (i.e. P-CBM-h or IBD) as we want to focus on fully interpretable CBM models. Note that P-CBM uses the expert provided concept set for CUB200, yet we can outperform using our fully GPT-3 derived set of features. The standard sparse models were finetuned by us by learning a sparse final layer directly after feature layer f() as described in (Wong et al. (021) and also have 25-35 nonzero weights per class. The accuracies of sparse standard models are comparable to our CBM, indicating the CBM does not reduce accuracy.
Dataset
Model
Sparse final layer
CIFARIO
CIFARI0O
CUB200
Places365
ImageNet
Standard
No
88.80%
70.10%
76.70%
48 15.56%
76.13%
Standard (sparse)
Yes
82.96%
58.34%
75.96%
38.46% N/A
74.35% N/A
P-CBM
Yes
70.50%
43.20%
59.66 59.60%
P-CBM (CLIP)*
Yes
84.50%
56.00%
N/A
N/A
N/A
Label-free CBM (Ours)
86.40 %
65.13%
74.31%
43.68 %
71.95%
Yes
+ 0.06%
+: 0.12%
0.29%
+ 0.10%
+ 0.05%
Table 2: Accuracy comparison, best performing sparse model bolded. We can see our method outperforms Posthoc-CBM and performs similarly sparse standard model. The results for our method are mean and standard deviation over three training runs. *Indicates reported accuracy.
Result (II: Explainable decision rules. Perhaps the biggest benefit of Concept Bottleneck Models is that their decisions can be explained as a simple linear combination of understandable features. To showcase this, in Figure we visualize the final layer weights for the classes "Orange" and "Lemon"

---

Published as a conference paper at ICLR 2023
ImageNet CBM Orange Lemon
Places365 CBM
Mountain Mountain Snowy
Concept
Prediction
Concept
Prediction
rodiy and
large: sheer roochice
a CXER0
layare of fuu
chockout nourtor
peel
Imange
garge
piece
telingg
shapas
ciiua Fruit
eleeation
lerg rockyy pane.
Inton
la gu. Ful exparse of shote lare: place p280)
S0OW
mandei
Figure 3: Visualization of the final layer weights of Label-free CBM. Showcasing how models differentiate between two similar classes.
on ImageNet, and classes "Mountain" and "Mountain Snowy" on Places365. This visualization is Sankey diagram of the final layer weights coming into two output classes, with the weight between concept and output class displayed as the width of the line connecting them. We have only included weights with absolute value greater than 0.05 (for comparison the largest weights are usually 0.5- 1). Negative weights are denoted as NOT concept. The learned decision rules largely align with our intuitions. For ImageNet, the concept "citrus" fruit is highly connected both "Orange" and "Lemon", while orange colors activate the "Orange" class and yellow colors and limes activate the "Lemon" class. On Places, concepts like "a high elevation" activate both classes, while "Mountain Snowy" is activated by snow and ice related concepts and "Mountain" is activated by concepts related to volcanoes like "lava" and "crater". Visualizations like this allows us to gain a global understanding of how our model behaves. Additional visualizations are shown Appendix
Result (III:: Explainable individual decisions. In addition to global understanding. CBMs allow understand the reasoning behind individual decisions. Since decisions linear functions of interpretable features, they can be explained in a simple and accurate way. After learning the Concept Bottleneck Layer, we normalize the activations of each concept to have mean 0 and standard deviation 1 on the training data. Given normalized features, the contribution of feature j to output i on input F& can be naturally calculated as Contrib( j. = WKk) fe(x)) Since our Wr is sparse, most contributions will be O and the important contributions can be easily visualized with a bar plot. Example visualizations for explaining the models predicted class are shown Figure and more visualizations are available in Appendix A. These visualizations show the concepts with highest absolute value contribution. Concepts with negative activation can still be important contributors, and they shown "NOT concept" in visualizations. We see for example the CUB model correctly identifies "a red head" as the most important feature in recognizing this image "Red headed Woodpecker", while the fact that there is no "a rosy breast" present also increases models confidence.
CASE STUDY: MANUALLY IMPROVING AN IMAGENET MODEL
In this section we take a deep dive into our ImageNet Label-free CBM - we analyze the errors it makes and design ways to debug/improve show how we can inspect a handful of individual incorrect decisions and manually change our model weights not only fix those predictions but also multiple other predictions, improving the networks
network based these findings. In particular,

---

Published as a conference paper at ICLR 2023
Prod-Aod hosded Woooceectrr Cont 0.706 Logi:7.80 7.80 Bloc- 0.32
rod houd
+1.06
head and
+2.18
CUB
NOT yellwish wingss
rasy
Sum 201 other teatures
0.60
0.25
0.30
0.75
3,00
1.25
150
1.75
Cancept
Predzunkyard Coné.
Logit:0.00
0.30
Jnnk
old cons
+2.05
dabrib
-1:1
+0.73
Places
365
windhtrald
Riled with tresh or dubris
+0.4
Sum 2200 other featunes
+2.56
contributtons
Figure 4: Visualization of two correct decisions made by our Label-free CBM.
overall accuracy. To our knowledge, this is the first example of manually editing a large well-trained neural network in way that improves test accuracy the same distribution.
5.1 TYPES MODEL ERRORS
During the course of experiment, were able to identify 4 different types of mistakes model makes, with different strategies for fixing them:
Type 1: Incorrect/ambiguous label: Despite looking like errors in terms of accuracy, these simply a consequence having noisy labels and are unavoidable. We will not attempt to fix this type of mistakes
No sufficient concept in CBL: Despite having a large set of concepts, our CBL does not include all concepts important for detecting certain classes. For example, detecting the difference between two species snakes may require recognizing very fine-grained patterns that can't be easily explained in words. While these could often fixed by adding new concepts to the concept set, do not focus on it as it requires retraining the projection We and the final layer.
Type 2:
Type 3: Incorrect concept actívations: Some of the time, the activations in our CBL are incorrect and do not match the image, causing the prediction to become incorrect. Since the network up to the Concept Bottleneck Layer is mostly black box, we can't easily improve the predictions, but predictions can be improved with test time interventions as shown in Figure 5
Type 4: Incorrect final layer weight WF: Sometimes even if all the concept activations are correct, the models final layer weights still cause it to make an erroneous prediction. As the final layer is fully interpretable, fixing these errors is the main focus our study described in Sec
Examples of Type and Type errors are shown in Figure 13 in the Appendix.
5.2 EDITING FINAL LAYER WEIGHTS
In this section we describe our procedure for manually editing final layer weights of our model. This done with the following procedure:
1. Find input where the model makes a Type 4 error: This is done by visualizing incorrect model predictions and their explanations, and identifying Type errors, i.e. errors where the highly activating concepts look correct and the label correct and unambiguous.
2. Identify concept to edit.: Perhaps the most tricky step, to change prediction need to select a concept that is highly activated in this image, important for the ground truth class and not important for the incorrect (using domain knowledge). and will have small impact other
8

---

Published as a conference paper at ICLR 2023
Predictionpoke bonnet Confi: 9.176 Logit Blas:
headgear
milkmadd
+1.09
hat
+0.93
baby's
strap or band around the hat Sum of 4516 other features
+0. 95
0.00
0.29
0.50
0.73
1.00
1.25
1.30
1.75
Concapt
Intervene: activation of "a hat" 2.70 -> O. New prediction: strainer
Figure 5: Here we fix an incorrect model prediction on ImageNet by simply zeroing out the incorrect activation for concept "a hat". Model originally predicted "poke bonnet" which is a type of hat.
Gt: Grasshopper
Hamper
Orig prect: Cricket Insect
Org pred: Shopping Basket
Grasshopper
New red: Hamper
Other inputs: predictions correctee
Other Inputs: 46 preclitiion corrected
-11 turned incorrect
turned Incorrect
Edit W,;
W,:
"a green color" "Grasshapper" :0 -+ 0.197 "Cricket Insest" 0.1197
"made of rope string" "Hamper" -. 0.205 "made rope string" "Shooping
Figure 6: Examples of two edits performed model predictions. Both edits fix the error found, and fix more other predictions than they cause mistakes on other unseen inputs.
predictions. For example to flip prediction from "cricket insect" to "grasshopper" we chose the concept "a green color".
Change weights by sufficient magnitude: Next we choose magnitude of weight change Aw, and edit the weights in the following way: W cogeepp + Wiigt,conoept + Aw and WFlpred,concepp/ <- W Fllpre,.concept
Aw. Typically we want Aw to be slightly more than required to flip the prediction our example, but understanding its effect other predictions is hard and should be checked with a validation dataset.
Fig shows two examples of model edits we performed, and their effects on model predictions on the validation set. In total we identified 5 such beneficial edits during our quick exploration, the other are shown in Fig 12 After applying all of them, the total validation accuracy of our model goes from 71.98% to 72.02%, correcting 38 predictions while turning 17 incorrect. While this might seem like small difference, it is worth noting we only change 10/4.5 million total weights in the final layer, of which 34,550 are non-zero. Our edits only affect 10 classes which have a total of 500 validation examples, so effect we increase accuracy on this subset by 4.2% which is significant boost. We believe this editing approach is promising direction for future use, where practitioners could for example notice an incorrect prediction in a production system, quickly devise fix for that prediction, and check how it affects other predictions using validation data.
CONCLUSION
We have presented Label-free CBM, fully automated and scalable method for generating concept bottleneck models and used it to create the first high performing CBM on ImageNet. In addition, have demonstrated how our models are easier understand in terms of both global decision rules and reasoning for individual predictions. Finally, we show how to use this understanding to manually edit weights to improve accuracy in trained ImageNet model.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The authors would like to thank anonymous reviewers for valuable feedback to improve the manuscript. The authors also thank MIT-IBM Watson AI lab for support in this work. This work was done during T. Oikarinen's ' internship at MIT-IBM Watson Lab. T.-W. Weng is supported by National Science Foundation under Grant No. 2107189.
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully explaining model mistakes using conceptual counterfactuals. arXiv preprint arXiv:2106. 12723, 2021.
David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba. Rewriting a deep generative model. In European conference on computer vision, pp. 351-369. Springer, 2020.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolf- sson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing. Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Ku- ditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma. Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben New- man, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matci Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kait- lyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2021. URL https: wwwpvv.vvvvvv..///ii//aa./a...v. org/ htrg/abs//ee abs//10/2 2108 07258
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal. Ariel Herbert-Voos Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL httpp::/pproeeed.nnnt neurips. httpp//ww.....ppppp///////////ee
ppdf
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition.
Nature Machine Intelligence, 2(12):772-782, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-24422555 2009. doi: 10. LOW.CC/V..20099220666666
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-55 5348. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classificattin output: Semantic bottleneck networks, 2019, htt.:/ttsss//axx...rr///b/////// 100.9002.882
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint arXiv:2202.05262, 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale, 2021. URL /ttplttts///arsii.orr///e/e /ttp:///ww..oo//////1.o.o
Tuomas Oikarinen and Tsui-Wes Weng. Clip-dissect: Automatic description of neuron representa- tions in deep vision networks. arXiv preprint arXiv:2204. 10965, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https://arxis: wwwzxiiv.rr//aaa.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 2016.
Wojeiech Samek, Grégoire Montavon, Sehastian Lapuschkin, Christopher Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: review methods and applica- tions. Proceedings of the IEEE, 109(3):247-278, 2021.
Robyn Speer and Catherine Havasi. Conceptnet 5: A large semantic network for relational knowl- edge. The people's 's web meets NLP, theory and applications of natural language processing, pp- 161-176, 02 2013. doi: 10. 010007/978-3-42-35500
C. Wah, S. Branson. P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-200 dataset. Technical Report CNS-TR-200 1-001, California Institute of Technology. 2011.
Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E. Gonzalez. Nbdt: Neural-backed decision trees, 2020.
Sheng- Yu Wang, David Bau, and Jun- Yan Zhu. Rewriting geometric rules of a gan. ACM Transac- tions Graphics (TOG), 41(4):1-16, 2022.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for de- buggable deep networks. In International Conference on Machine Learning, pp. 11205-11216. PMLR, 2021.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint wwxXi::2005.54440 2022.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv: 220.002000000066 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 119-134,20 2018
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A. APPENDIX OVERVIEW
In this section we provide brief overview of the Appendix contents. The Appendix is mostly focused on Ablation studies identifying the importance and effectiveness of different components of our pipeline, as well additional Figures to showcase more examples of our results.
First in section A.2 we discuss the limitations of our method. then in section A.3 we show that our proposed cos cubed similarity function performs well describing neuron functionality. In section A.4 we perform an ablation study on the effect of different concept set filters we have proposed, and in See |A.5 we discuss our reasoning behind each filter in detail through an example. section [A.6] show that our GPT-3 generated concept set helps us reach higher accuracy than previous methods for creating initial concept set. In Appendix A.7 we show the effects of removing the sparsity models, leading to higher accuracy but reduced interpretability. In Appendix [A.8 we show our results are consistent despite inherent randomness in our pipeline. Next, Appendix A.9 we provide a detailed explanation of the procedure we used for model editing described in Sec 5.2 Finally in Sections [A. 10 [A. and A2 we provide additional figures for model editing, individual decision explanations and model weight visualizations respectively.
constraint
A.2 LIMITATIONS
While our model performs well in terms of both interpretability and accuracy, it still has some limitations. For example, using a model like GPT-3 to produce concepts is stochastic, and sometimes may fail generate important concepts for detecting certain class. However robust to small changes in the concept set as shown in sections A.4 and A.8 In general, automatic ways of generating concept set may sometimes lack the required domain knowledge include some important concepts, and would perhaps be best used collaboration with a human expert.
model is quite
Second, our model is aimed to extend Concept Bottleneck Models large datasets where labeled concept data is not available. However it works best on domains where CLIP performs well and as such may perform as well on small datasets that require specific domain knowledge and labeled concept data is available. such as medical datasets. For such datasets we believe it is still better to use method that can effectively leverage the labels available such original Concept Bottleneck Models Koh et al. (2020) or M-CCBMYukseeenn et al. (2022).
A.3 DISSECTION PERFORMANCE OF COS CUBED SIMILARITY FUNCTION
In this section we discuss how good the cos cubed similarity function we defined in section 3.2 is at describing neuron functionality, i.c. how well it performs similarity function for CLIP- Dissect. Specifically we measured its performance on describing the roles of final layer neurons on ResNet- 50 trained ImageNet (where we know the ground truth role of the neurons) in terms of both accuracy and average distance from the correct description in a text embedding space. We followed the quantitative evaluation methodology of (Oikarinen & Weng 2022), which describes this process in more detail. We compared the results against using CLIP-Dissed with two different similarity functions. softWPMI which was the best function identified by (Oikarinen & Weng 2022) and cos which is simple differentiable baseline. We can see cos cubed performs es well SofTWMI in terms of similarity in text embedding space, while being slightly worse terms of accuracy, and is much better than simple cos similarity. This is impressive given cos cubed is fully differentiable while SoftWPMI is not differentiable.
12

---

Published as a conference paper at ICLR 2023
probe
Similarity function
CIFARI00
Broden
ImageNet val
ImageNet val Broden
Average
Differentiable
Metric
train
mpnet
COs
0.2761
0.215
0,2823
0,2584
0.2580
Yes
cos similarity
SoftWPMI
0.3664
0.3945
0.5257
0,5233
0.4525
cos cubed
0.3579
0.4101
0.5187
0.5223
0.4523
Yes
Topl accuracy
cos
8.50%
5.70%
15.9%
11.4%
10.38 %
Yes
SoftWPMI 46.20%
70.50 %
95.00%
95.40%
76.78%
No
cubed 31.00%
49.40 %
87.40%
85.50%
63.33%
Yes
Table 3: Comparison of the performance between similarity functions. We look at the final layer of ResNet-50 trained on ImageNet. We use 20,000 most common English words as the concept set for mpnet cos similarity and ImageNet classes the concept set for top1 accuracy. We cubed performs much better than simple cos, and almost as well as SoftWPMI.
A.4 ABLATION: EFFECT OF CONCEPT FILTERS
In this section we study how each step in our proposed concept filtering effects the results of our method. In general our use of filters has two main aims:
To improve the interpretability of our models
Improve computational efficiency and complexity by reducing the number of concepts
The filters are not designed to improve performance of the models in terms accuracy, in fact we would expect them slightly reduce accuracy as model with more concepts is generally larger and more powerful.
To evaluate the effect each individual filter has trained models both CIFAR1O and ImageNet while deactivating one filter at a time, as well as one without using any filters at all. The results are displayed in Table We of filters, with ImageNet accuracy remaining constant and CIFAR 10 slightly increasing its accuracy with less filters. We were unable train an ImageNet model without any filters as training with the large number of concepts required more memory than we had available in our system. Additionally the ImageNet model without similarity to other concepts filter had worse accuracy and less sparsity than other tested models, which we think may be caused by the GLM-SAGA optimizer not finding as good solution when the number concepts increased too much.
that the accuracy our models is not at all sensitive to the choice
CIFAR10
CIFAR10 #concepts
ImageNet Accuracy
ImageNet #concepts
Filters
Accuracy
All filters
86.26%
142
71.89%
4380
No length filter
86.55%
146
71.88%
5347
No similarity to classes filter
86. 66%
148
71.93%
4671
No similarity to other concepts filter
86.42%
1.11
70.90%
6515
No CLIP activation filter
86.41%
147
71.92%
4478
No projection accuracy filter
86.33%
147
71.88%
4462
No filters at all
86.56%
177
N/A
9087
Table 4: Effect of individual concept filters the final accuracy and number of concepts used by our models. #We were unable to train the model to be sparse enough, results from a less sparse model.
13

---

Published as a conference paper at ICLR 2023
A.5 FURTHER DISCUSSION AND EXAMPLE OF CONCEPT FILTERING
To further show the effects of each filter, we will showcase the full filtering procedure for our CI- FARIO model with all filters. We start with freshly generated initial concept set from GPT-3, which has a total of 177 concepts. We then take the following steps to filter down concepts:
1.
Delete concepts that are too long. This step is important make sure concepts are easy to visualize and simple non-convoluted concepts. For CIFAR-I0 this leads to deletion of concepts:
white spots on the fur (in some cases), feline features (e.g,, whiskers, ears) and legs with two toes pointing forward and two toes pointing backwar.
174 concepts remain.
2. Delete concepts too similar output classes. This step required for the explanations to be informative, it helps avoid trivial explanations such as: "This is image is cat because it is cat". This step removes the following CIFAR-10 concepts (with the class name is too close to shown in brackets):
a cat (cat), a deer (deer), a horse (horse), a plane (airplane), a car (automobile), car (automobile), cars (automobile), vehicle (automobile), animal (dog), truck driver (truck) 164 concepts remain
3. Delete concepts too similar to other concepts. In this step we aim delete duplicate concepts from the concept set, i.e. two concepts that have the same semantic meaning should not both be part of the concept set. Duplicate concepts will cause unnecessary computational cost and confusing explanations. For CIFAR-10 we delete the following concepts (with the concept it is too similar to in brackets):
4 wheels (four wheels), food bowl (a bowl), furry, four-legged animal (a large, four- legged mammal), a gasoline station (a gas station), a large boxy body (a large body), a large, muscular body (a large body), street (a road), strong engine (an engine), large, bulging eyes (large eyes), the ocean (the sea) 154 concepts remain
4. Remove concepts not present in training data. In this step we remove all concepts that CLIP thinks are not really present in training data. The purpose is avoid learning neurons that don't I correctly represent their target concepts, as concepts missing from the dataset are unlikely be learned correctly in the Concept Bottleneck Layer. For CIFAR-10 this step deletes the following concepts:
bit, a mechanic, legs, long legs, passengers, several seats inside, windows all around 147 concepts remain
5. Remove concepts we can't project accurately. The purpose of this step is similar to the previous step, want to remove all concepts that are not faithfully represented by the CBL. To do this we evaluate the similarity score between the target concept and the acti- vations of new neuron using CLIP-Dissect Oikarinen & Weng (2022) the validation DikeaiiseetOikkaiieen
data, and delete concepts with similarity less than 0.45 which was determined to be good indicator of faithful concept representation. For CIFAR-10 we delete the following con- cepts:
bed, a coffee mug, crew, a house, wheel We are left with our 142 final concepts.
In total our method has different cutoff parameters that can be tuned. Since the main purpose of these filters is to make the network more interpretable, their values were mainly chosen through trial and error find ones that produce the most useful and explainable models. Most of these cutoffs are independent of the dataset chosen, and the only one we changed from one dataset to another was the value of cutoff for images being present in the dataset (filter 4). We used a cutoff of 0.25 for CIFAR- 10 and CIFAR- 100, 0.26 for CUB-200 and 0.28 for Places and ImageNet. We found we had to change this cutoff as CLIP doesn't 't activate very highly on low resolution images of CIFAR, and in general we get higher top5 activations larger datasets. Since the other 4 cutoffs are fixed, using our method for a new dataset won't require a large hyperparameter search, and as seen in Table the overall performance is not very sensitive specific filters.
To visualize the effect our filters, in Figure 7 we visualize the final layer weights for CIFAR-10 class "automobile" trained with and without filters. We can the model with filters has found
14

---

Published as a conference paper at ICLR 2023
reasonable decision rule, with largest weights corresponding to four wheels and steering wheel. On the other hand, the weights for the model without filters are quite problematic. First, the largest weights
car and car. Not only do they have the same meaning the final class itself (filter 2), thus lacking any explanatory power, they are duplicates of each other (filter 3), making the explanation unnecessarily compilicated. Finally see the unrelated concept bed which is not present in CIFAR-10 having a small positive weight, which should be removed by either filter 4 or
CIFAR-10 final layer weight visualization
LF-CBM with all concept filters
LF-CBM without concept filters
beete
cab for the driver
cars
deshboard
dashboard
seats
@treet
NOT feeder
steering wheel
passenger
NOT rider
steering wheel
sutombble
NOT
NOT trailer
cab for the dfiver
traleer
NOT bow and stem
four wheels
NOT large anders (on meles)
tallights
NOT bal
object
NOT large entirs (on metes)
tailights
NOT a crew
Figure 7: Comparison of the final layer weights between CIFAR-10 model trained with and without filters.
A.6 ABLATION: EFFECT OF INITIAL CONCEPT SET
In this section we evaluate the effect our initial concept set generator has on our results. In our paper proposed
method of generating descriptions using GPT-3 Brown et al. (2020), and in this section we compare those results to the case where we generate the initial concept set using Con- ceptNet Speer & Havasi (2013 as proposed by Yukkekkonuul et al. (2022). In general ConceptNet only works well when prompted with relatively common single word phrases, but many of class names for CUB200, Places365 and ImageNet consist of multiple words, such as the ImageNet class great white shark. To overcome this issue we split multi-worg class names into single word compo- nents and add concepts related to each word to the concept set, this case great, white and shark. After creating the initial concept set we use the same filtering procedure described in section 3.1 for both models. Table 5 displays the accuracy comparison between models trained on different initial concept setst. In general using GPT3 seems provide a small (0.1-1.5%) accuracy boost on all datasets, with the exception of CUB200 where the ConceptNet model fails completely. This is because ConceptNet is unable generate concepts for the highly specific class names in CUB200, such as Groove billed Ani, while GPT-3 is not troubled by this.
Model Dataset
CIFARIO
CIFAR100
CUB200
Places365
ImageNet
LF-CBM (GPT-3) (original) LF-CBM (ConceptNet)
86.40%
65.13%
74.31% %
43.68 %
71.95%
86.30%
63.62%
2.19%
42.49%
71.52%
Table 5: Accuracy comparison between using different methods for generating initial concept set.
15

---

Published as a conference paper at ICLR 2023
A.7 ABLATION: LF-CBM WITHOUT SPARSITY CONSTRAINT
To estimate the effect sparsity has on our models, we show the accuracy of LF-CBM models trained without sparsity constraints (LF-CBM (dense)) in Table 6 We see that using sparsity constraint greatly improves accuracy (except for CUB200 which starts to overfit), but it comes at very large cost for interpretability can be seen in Figure where almost none of the decision is explained by the 10 most highly contributing concepts.
Model Dataset
CIFARIO
CIFAR100
CUB200
Places363
ImageNet
LF-CBM (sparse (original]) 86.40%
65.13%
74.31%
43.68%
71.95 71.95%
LF-CBM (dense)
87.50%
67.93%
74.25%
48.25%
74.09%
Table 6: Results of training our CBM without sparisity constraints the final layer.
LF-CBM (dense)
Image: 13260, Ground truth: Toy Poodle
Class Miniature Poodle Cont. 0.963 Logit 25.29 29 Blas-O
+0.54
-0.47
long. curly hair
+0.4
pepper
marshmallow
poodle
synthetic
top
+0.3
features
+24.2
15
25
Concept contributions
Figure 8: Dense CBM explanation for the same image as the fourth image in Figure the dense model is practically uninterpretable.
We can see
16

---

Published as a conference paper at ICLR 2023
A.8 CONSISTENCY CONCEPT SET GENERATION
Since the initial concept set is generated using GPT-3, it is random process and in this section explore how much this noise effects final results. When regenerating initial concept set and running our concept filtering pipeline for our ImageNet model, we got 4380 concepts, compared to the original 4523 concepts. In terms of accuracy, the model learned with regenerated concept set reached an accuracy of 71.89%, compared to the average of 71.95% with the original concept set. exact concepts
This indicates that the difference very small terms of accuracy. However, used and model weights can be pretty different as shown in Figure despite all being relevent (e.g. there are many concepts that appear on both concept sets, e.g. Image 49962 has important concepts both, though they have different corresponding contributions). Figure
of "wipes", "toiletries" 9 shows the explanations for decisions on two random images for the original model and model trained with regenerated concept set.
17

---

Published as a conference paper at ICLR 2023
Original concept set
paper -Logit:9992
wipes
paper
Image: 49962, GT: toilet paper
+0.88
stack paper
+0.86
toilet cleaner
+0.64
attached to toilet bow!
+0.63
napkin
+0.61
Sum of 4516 other features
+2.35
0.0
0.5
1.0
1.5
2.0
2.5
Concept contributions
Regenerated concept set
Pred: toiletilet Conf: 0.458 Logit:9 2..6.
Bias:0.00
tollet paper holder
+3.98
+1.56
tissue
+0.79
stack of paper
+0.47
strong, unpleasant odor
+0.41
absorbent
+0.37
toiletries
+0.33
Sum 4373 other features
Concept contributions
Original concept set
Pred
Cont.
Logit:11.73 Blas:0.12
knife sharpener
+3.02
Image: 35528 GT: pencil sharpener
stamps
pencil
+1.21
stapler
+0.89
hole punch
+0.03
pencils
Sum 4516 features
2.5
3.0
Concept contributions
Regenerated concept set
UR SHIWEE
Precipencil sharpener
Logit:11.40
+2.02
pencil cup
+1.02
spoot tape
chalk
+1.32
hole punch
stamp
+0.51
Sum 4373 other
Figure 9: Difference concepts used for decision and their weights for models trained with original vs regenerated concept set, shown two random images.
18

---

Published as a conference paper at ICLR 2023
A.9 MORE DETAILS ON MODEL EDITING PROCEDURE
Here we provide an improved explanation of Sec 5.2 on how we can edit the final layer weights of our proposed Label-Free CBM correct incorrect predictions. Our procedure includes 3 steps:
Step 1: Find an input image I; where the model makes a Type 4 Error (incorrect final layer weight):
This is done by visualizing incorrect model predictions and their explanations similar to Figure To identify if it is Type 4 error, first we check that the ground truth label of the input image correct and unambiguous (therefore not type I error described in sec 5.1). Next, we check that the highly activating concepts look correct (therefore not type or type 3 error described in see 5.1).
Step 2: Identify concept to edit:
We start by listing the highest activating concepts for the chosen input, i.e. the highest elements of fe(x;) (after normalization). This allows us to identify concepts that are the most important for this specific image while not too important for other images. From these concepts we use our domain knowledge (and/or internet search) to understand which concepts are relevant to making this decision.
For example, the image in the right panel of Figure ili originally predicted as a "Shopping basket", while the ground truth is a "Hamper". The most highly activating concepts for this image are: "a basket", "made of rope string". "a rope". "a laundry basket" and "a fishing net". After short investigation we find that although the classes "Hamper" and "Shopping basket" are very similar, hampers more often constructed in woven like manner. Therefore, identify the concept of next step.
"made of rope or string" as relevant for capturing this difference, and choose it for editing in the
Step 3: Change weights by sufficient magnitude:
Once we have selected the concept be corrected, will compute how much the magnitude of the associated final layer weight should be changed (denoted as Aw, Aw E R) and then edit the weights in the following way:
WF cloteco <- [oggecoo + Aw, W Flpred,comospp) +- WF concepd
Aw
Note that WF(.,j denotes the (i,j) element in the matrix WF. The goal of editing is to correct the inaccurate prediction on this specific instance while minimizing effect on other predictions. To calculate Aw needed flip prediction, first calculate the difference in logits An (before softmax) before the edit, Aa = W WFipred.]Jeee - WFlot.see(t:)
Since would like the prediction be corrected gt class, goal is to design Aw such that
(WFlgt.) + Au e))(( - (W (Wrlpred.) Aw e))r(;;))) > 0
(4)
where is one-hot vector with the entry ((cooeed) 1. Let b (W [WF[t..] + Aw fxs)
(W [Wrpredd: Aw e))(( where b is nonnegative constant deciding how large of margin we want the correct prediction to have. Since Aa W (c(r)) WF(ot.:)eaazs)) we have:
2Aw fe((x)) Aa = + Aw (Aa + (2f((((( ( (coneep))
(5)
Typically we use values for b between 0.2 and 2 to calculate required Av.
It is worth noting that when the model weights are edited, might affect other image's predictions too due to the change of weight parameters. Thus, with the goal to correct the wrong predictions while not affecting other already correct predictions, we suggest each edit should be checked with validation dataset before applying a target model. As described in the end of Sec 5 in the manuscript, with the above proposed model weight editing, we are able to improve the overall model accuracy from 71.98% to 72.02% by performing the model editing for only 5 different images, which is a non-negligible improvement. Since the edits only affect weights for 10/100 (0/1000 classes, this corresponds to 4% accuracy boost on the affected classes.
19

---

Published as a conference paper at ICLR 2023
A.10 ADDITIONAL FIGURES
Figures 10 provide examples of the full prompts we used for GPT-3, as well as GPT outputs. For all experiments we used the text-davinci-002 model available through OpenAl API.
Figure 12 shows the additional model edits performed in our ImageNet CBM experiment, and Figure 13 showcases Type and Type 2 errors made by our ImageNet CBM.
things most commonly seen around "tench".
Give superclacses for the word "tench":
pond
-fish
-vertebrate -animal
-a net
rod
Give superclasses for the word "beer glass":
-a hook
-bait
gglas
-container
List the things most commonly seen around "beer glass":
-object
beer
Give superclasses for the word "grasshopper":
bar
-a coaster
+insect
napkin
straw
-arthropod
-a lime
-animal
List the things most commonly seen around "mountain":
peak
ra valley
forest
-dirt
trail
Figure 10: Full prompts used for GPT-3 concept set creation. Text in green generated by GPT, rest is our prompt.
20

---

Published as a conference paper at ICLR 2023
List the most important features for recognizing something as "goldfish":
-bright orange color
-a small, round body
long, flowing tail
-a small mouth
-orange fins
List the most important features for recognizing something "beerglass":
-a tall, cylindrical shape
-clear translucent color
opening at the top
-a sturdy base
-a handle
List the most important features for recognizing something as "scuba diver":
person wearing scuba mask and breathing apparatus
-a wet suit
-flippers
-a diving tank
Figure 11: Full prompts used for GPT-3 concept set creation. Text in green generated by GPT, rest prompt.
21

---

Published as a conference paper at ICLR 2023
Gt: Scuba Diver Orig pred: Mask Respirator New pred:
Greenhouse Orig pred: Sliding Door
New pred: Greenhouse
inputs: +9 predictions corrected turned incorrect
Other Inputs: predictions
-1 turned incorrect
Edit "steam" "Scuba Diver"
W,:
:0 0.197
"plants" "Greenhouse"" 0.724 0.800
"steam" "Gas Mask Respirator" :0 -0.197
"plants" "Siding Door": 0.000 -0.076
Gt: Stinkhorn Mushroom Orig pred: Slug
Stinkhorn
Other inputs: +3 predictions corrected turned incorrect
Edit W,:
"a long, thin, reat" "Stinkhorn Mushraom": 0.012 0.141 long, thin, orange root" "Slug"
0.039 -0.091
Figure 12: The other 3 model edits performed in experiment
Type
Incorrect label
Ambiguous label
Gt: patas monkey, pred: lion o 2009 G. Kiter
Gt: stethoscope, pred: maltese (dog)
Type2: sufficient concept in CBL
Gt: barn spider, pred: European garden spider
PredictinnEErqpenn garden spider Conf. 0.430 Logit Blas:-0.18
spider
arachnid
+1.79
web Siee structure
+1.78
web page
+1.18
yellow spots on body
Sum 4516 other features
+1.74
Slage Dash
0.0
Figure 13; Examples Type and type model errors.
22

---

Published as a conference paper at ICLR 2023
A.11 ADDITIONAL WEIGHT VISUALIZATIONS FOR RANDOM CLASSES
In figures a44 15 we have visualized the final layer weights leading to 4 randomly chosen output classes for both Places363 and ImageNet. These were created using the same procedure Figure 3] with minor differences such as focus on single class at time and no manual editing of concept colors according semantic similarity. We can see that the concepts that have large weights are indeed relevant to the class, and overall the weights look reasonable.
Places365 Final layer weight visualizations for random classes
projection
slage
room
stage bright light
NOT vister
shower curten
sink
or porcelan surface
orchestra
toieet
sudiofum
bathroom
legisiators
deaner
tollet paper
state
Theater
Il checkoot line
menu with inerpersive terre
pharmadis
mursing
serving counter
variety ice cream Ravors
drugstore
drinking estabilohment
partor
sigh with the stone's name
NOT
Ice cream
supples
games
restourant feature
costs facing the counter
ictons
"bakery"
toppings
suppless
Figure 14: Weight visualizations for 4 randomly chosen output classes for our CBM trained on Places365.
23

---

Published as a conference paper at ICLR 2023
ImageNet Final layer weight visualizations for random classes
Climbing
body
herd, red exteroo
dinossu
SPF information
poologist
NOT green apple
long thin orange
praiie
rocky craggy
bright orange color
outoropping
guard
agama
a vitrant blue color
nevts
Izard
NOT black and while shipes
NOT
celomander
iquana
equiress
NOT rediating ribs or spokes
NOT botte of champagne
bars
clog dance
buuttan-dosen shitt
conductor
-OT a conterts list
drum
NOT long, Barked longue
machine
drum-ike body
piano
NOT messurement
NOT handaol
NOT headiggtt
nubber
NOT arygen lank
NOT
NOT of boots
plano bench
NOT
pingipong table
marimba
trace
Rashing
tured wooden bars
fostball pads
hand
High speed
tasbch
robilit
Prafumentalata
protectve gear
NOT messurement
NOT scert
performers
Figure 15: Weight visualizations for 4 randomly chosen output classes for our CBM trained on ImageNet, as well as an example image of each class to clarify their meaning.
24

---

Published as a conference paper at ICLR 2023
A.12 ADDITIONAL EXPLANATIONS FOR RANDOM IMAGES
In figures 16 17 18 we display additional samples of decision explanations for randomly chosen input images for each of Places365, ImageNet and CUB-200 respectively. Overall, it can be seen that the concepts and explanations are of good quality, even with the CUB-200 dataset which may require very specific knowledge and concepts to create CBMs. This result demonstrates the effectiveness of our LF-CBM, which does not use the knowl-ek concept sets unlike existing CBMs.
25

---

Published as a conference paper at ICLR 2023
Places365
Image:1958/ Ground truth: Jacuzzi, indoor
Prediacuzzi, Indoor Cont: 0.889 Loggt:11.02 Bias:-0.68
hot tub
+2.84
spa
+2.07
a large, round tub
+1.83
bathtub
.1:12
+0.92
room designated for bathing
+0.86
pool
+0.82
Sum of 2200 other features
+1.23
0.0
0.5
1.0
2.0
2.5
3.0
Concept contributions
Image:5776, Ground truth: boardwalk
Pred:pier- Loggit:3..5
dock
+10.1
dock levelers
+1.17
salt water
crossings
people
inside
+0.15
long, conveyor
+0.12
walking
+0.06
Sum 2200 features
+0.14
Concept contributions
Image: 17442, Ground truth: heliport
Pred:heliport Logit:7.15 B.as:-0.43
helicopter
+4.07
aircraft
helipad
planes
landing
fighters
+0.15
mushrooms
+0.13
+0.31
Concept contributions
Image: 33310, Ground truth: Toplary Garden
Preditopiary garden Conf: 0.1 Logit:8.23 Bias: 0.16
well-trimmed hedges
+1.44
extensive grounds and gardens
+1.13
gardening gloves
+0.76
sculptures
outdoor sculptures
statues
+0.41
Sum of 2200 other features
+1.16
0.0
0.5
1.0
1.5
2.0
2.5
Concept contributions
Figure 16: Explanations for randomly chosen input images for our CBM trained on Places365.
26

---

Published as a conference paper at ICLR 2023
ImageNet
Image:17233 Ground truth: hippopotamus
Predhhippopotamus
0.986 Logit:15.55 Blas:-0.45
hippos
hippos
long,
+0.63
+0.46
horselike face
+0.16
crocodilian
+0.05
large, red crustacean
+0.04
Sum of 4516 other features +0.18
10
Concept contributions
13age:33159, Ground truth: monastery
Pred:monastery Conf: 0.634 Logit:10.74 Bias:-0.43
a community of monks or nuns
+2.46
courtyard
cathedral
+1.48
court
+0.66
town square
+0.62
cassock
village
+0.4
Sum of 4516 other features
+3.44
0.0
0.5
1.0
1.5
2.0
2.5
3.5
Concept contributions
Image: 23074, Ground truth: breastplate
Pred.breastplate
Logit:13.09 Bias:0.11
soldiers and
+1.23
corset
+1.01
side shields
+0.99
knight
+0.75
foll
+0.65
Sum of 4516 other features
+4.14
Concept contributions
Image: 13260, Ground truth: Toy Poodle
Logit:14.62
+1.6
long,
+1.03
shaggy mane
+0.59
cotton ball
+0.49
a dog park
+0.49
wheaten color
Concept contributions
Figure 17: Explanations for randomly chosen input images for our CBM trained ImageNet.
27

---

Published as a conference paper at ICLR 2023
CUB200
Image:1169, Ground truth: Yellow bellied Flycatcher
Yellow bellied Plycatcher- - Cont: 0.242 Logit: 14 Blas:-0.18
NOT large webbed feet
+0.9
+0.71
olive colored sides
+0.67
wings
NOT small, dark body
+0.39
wings
plumage
+0.27
Sum of 201 other features
+1.59
0.00
0.75
1.00
1.75
Image:4121, Ground truth: Black Tem
Pred:Black Tern Conf: 0.341 Logit:6.77 6.77 Blas:0.19
swift, acrobatic flight
+2.28
white
white wings
+1.28
+0.72
white
+0.68
seabird -0.57
groy
+0.47
black white plumage
+0.3
features
-0.
0.0
0.5
1.0
2.5
Concept contributions
Image: 5159, Ground truth: Tennessee Warbler
Pred:Warbling Vireo Conf: 0.346 Logit:6.45 Blas:-0.04
live-colored sides
+1.22
white throat and belly
+0.71
olive green
greenish-gray gray
+0.6
small, white bird
+0.54
NOT black and orange plumage
green wings
Sum of 201 other features
+1.86
0.00
0.25
0.50
0.75
1.5
1.75
2.00
Concept contributions
Image: 4823, Ground Kentucky Warbler
Pretuce Werbler Conf: 0,923 Logit:9.85 Blas--..30
yellow throat and breast
+1.46
NOT a long, thin neck
+0.74
wings
NOT black and white head
+0.47
NOT black bib with white stripes
+0.46
Sum of 201 other features
33.86
0.5
1.0
15
2.0
3.5
4.0
Concept contributions
Figure 18: Explanations for 4 randomly chosen input images for our CBM trained CUB-200.
28


Reviews:
[{"review_id": "yi3CnJzEQP", "reviewer": "Reviewer_CRuQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a way to create a concept bottleneck paper without having to provide the concept datasets (which is usually user-specified) (this can be seen as a bug or a feature in the community). The way they achieve this is to first use GPT-3 to query related concepts given an output class. Using these GPT-created concepts they then use CLIP embeddings to match the concerted to images in the training data and hence create their concept dataset without labels. Once this dataset is created they can then train sparse layers i.e. sparse matrices using elastic net regularization to obtain their final predictions.\n\nThey then show that their method is able to perform well on datasets like imagenet while still being interpretable.", "summary_of_the_review": "I have mentioned all my concerns as well as thoughts on the paper in the review above.\n\nI am more than happy to increase my score if the authors are able to answer the concerns above.", "strength_and_weaknesses": "Strengths:\n- I have not seen people utilize GPT and CLIP in this manner to construct concept datasets (not expert in this field though)\n- The idea to create your label-free concept bottleneck paper is very interesting and a step forward to interpretable ML models\n- the paper is mostly well and clearly written.\n- Given that they use spare matrices they are also able to investigate the model on a sample level as well as edit the model accordingly. i.e. only changing a few weights in the sparse layers.\n- This opens up a new avenue for interpretable Deep models.\n\nWeakness:\n- I might be wrong but there are some typos in the paper that make it hard for me to understand what steps 2 and 3 are doing. \n - in 3.2 you have $W_c \\in d \\times N$ shouldn't that we M given figure 2? if f(x) \\in R^d what is the dimension for f_c(x)? and therefore what is q_k? I am very confused with the notation.\n - I am pretty sure eq(1) cannot be right. the index of t only goes from 1-M and the index for q (I don't understand). Should this be M?\n - EXPERIMENTS: Please add the 2 standard deviations in the table of results and how many times the algorithm has been run (I might have missed this detail. please point me to the part in the paper)\n- EXPERIMENTS: Could the authors please clarify what \"Standard (sparse)\" means?\n- EXPERIMENTS: I would like to investigate the code and see if the authors have cherry-picked any of the examples in the paper. Could the authors please confirm that the examples were not cherry-picked?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i9gptIlXNzT", "reviewer": "Reviewer_8fC7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a methodology to train concept bottleneck models, which automates the process of concept definition and it is empirically shown to outperform existing baselines making use of user-defined concepts. The methodology consists of four stages: \nIn stage 1, concepts are defined by generating lists of words conditioned on information related to each class in the downstream task using GPT-3. Additionally, a set of heuristics is applied to filter out generated concepts and to produce a curated concept list which is used in the subsequent stages. In stage 2, CLIP embeddings for the image and the concepts are used to produce a similarity matrix. In stage 3, the concept bottleneck layer is trained to maximise a similarity score between the weights of the layer and the similarity matrix. Finally, In stage 4, a linear layer is learnt to solve the downstream task.\n\nExperiments are conducted on 5 common vision benchmarks, including CIFAR10, CIFAR100, CUB200, Places365 and ImageNet, by comparing the proposed strategy against post-hoc concept bottleneck models [1] and a black-bock baseline. Additionally, the work proposes a new strategy intervening on the weights of the final prediction layer as demonstrated with a use case on ImageNet.\n\n[1] Post-hoc Concept Bottleneck Models. ICLR 2022 PAIR2Struct Workshop", "summary_of_the_review": "I recommend for an initial score of 5, as the weaknesses are currently outweighing the strengths.\n\n----- POST REBUTTAL ------\n\nI went through the rebuttal. The authors have addressed many of my concerns, (i) by including an ablation analysis on concept filtering, (ii) discussing the limitations of the proposed approach and (iii) providing substantial clarifications to all my questions.\nThe quality of the paper has consequently increased thanks to the ablation analysis. The scope of the proposed solution has been made clearer thanks to the discussion on the limitations.\nOverall, I'm quite positive about the work. The idea is novel and original, pushing forward research on explainability for concept bottleneck models. The only drawback is related to the reproducibility. Certainly, building solutions around proprietary software (like GPT-3) is not a good practice towards open science. However, the authors plan to release their code, all generated concepts and also make the necessary steps to ensure that all experiments are reproducible.\n\nBased on these considerations, I increase my score.", "strength_and_weaknesses": "**Strenghts**\n- The paper is clear and easy to follow.\n- The idea is simple, intuitive and original. I appreciated the idea of reusing existing technology and to apply it to concept bottleneck models.\n- The visualisations in the experiments are compelling and informative.\n\n**Weaknesses**\n- The proposed strategy includes several heuristics and an ablation study is missing.\n- It’s unclear what is the scope of application of the proposed strategy, as it seems to be limited to traditional vision benchmarks. For instance, what happens if classes are imbalanced or if data come from specific domains?\n- The experimental evaluation (Section 4.2) is non-standard and the authors should consider the same experiments performed by post-hoc networks. Additionally, the authors should report the set of whole results in the original paper.\n- From a computational point of view, the proposed strategy is quite complex.\n- Code is not available. However, the authors have stated that they will release it prior to publication.\n\n**Detailed comments and constructive feedbacks**\n1. Several heuristics are used throughout the different stages of training. For instance, in stage 1 several rules are used to filter out the generated concepts.  How are these rules and hyperparameters affecting the final performance? Are these configurations data/task dependent?\n2. At which stage is the main backbone trained?\n3. The experimental analysis focuses on 5 traditional vision datasets, namely CIFAR10, CIFAR100, CUB200, Places365 and ImageNet. I can imagine that the concept generated by GPT-3 can transfer to these benchmarks and therefore help in the automation process. However, what does it happen when the downstream tasks include rare classes (e.g. iNaturalists) or it requires concepts which requires domain specific knowledge (e.g. medical data)?\n4. Why aren’t the experiments on accuracy following the ones in the paper of post-hoc concept bottleneck models (P-CBM)? The author should start from the results in Table 1 of the paper of P-CBM, rather than running different experiments.\n5. In Section 5.2, only 5 interventions are found to be useful. Are there any examples of interventions in Section 5 which hamper the predictive performance? And how hard is to find such beneficial interventions?\n6. What are the limitations of the proposed methodology?\n\n**Additional missing references**\n\n[2] Concept Whitening for Interpretable Image Recognition. Nature Machine Intelligence 2020\n[3] Concept Embedding Models. NeurIPS 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dLe65Aja3YT", "reviewer": "Reviewer_FHRn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a framework to transform a neural network model into an interpretable CBM model that does not require a domain expert to provide concepts and does not require labeled concept data. The authors leveraged GPT3 model by prompting it in a certain way to get concepts related to each output class. They later perform a set of heuristics to clean the concept list. In order to train a concept bottleneck layer and to learn a projected layer, authors exploited the CLIP-dissect. Finally, they introduced a sparse final layer in order to keep it more interpretable by using the ElasticNet regularization in the loss function. The experiments are conducted using 5 different datasets. The results showed that their method is able to maintain decent performance. Later in the paper, the authors perform preliminary experiments on changing the model's prediction by manipulating the neurons.", "summary_of_the_review": "- The idea of using GPT3 is smart. I am wondering how the effectiveness of the proposed approach changes, with the change in the model, used to extract concepts.\n- The concept filtering is a bit ad-hoc. e.g remove concepts we can't project accurately, remove concepts too similar to each other, concept length longer than 30 characters. I am wondering how much noisy were the initial concepts and how much careful cleaning is required.\n- Authors mentioned that the number of concepts is related to the number of classes. Does it mean that it is good to have a one-to-one mapping between concepts and classes? Restricting the neurons to learn a single concept extracted using GPT3, would it not affect the generalization and robustness of the model? \n- The number of concepts mentioned in section 4.1. Are these the remaining concepts after cleaning or some further restriction is applied on the number of concept to consider based on the number of classes?\n- The evaluation can be improved. The paper focuses a lot on qualitative evaluation. The main quantitative result is on maintaining the overall performance of the model. Authors may add a correlation between the selected concepts and the predicted class. They may also consider ablating concepts and present their effect on the output class.\n- Figure 4, I am wondering if authors found false negatives and false positive cases. If a correct concept is ranked the lowest for a certain prediction, this raises the question of the faithfulness of the interpretation.\n\nAfter rebuttal:\nThank you for the detailed reply and for making substantial improvements to the paper. \n- It would be great to add a point on how did you select the number of concepts for a dataset e.g. how did you come up with the number 128 for CIFAR10?\n", "strength_and_weaknesses": "Strength: the method provides a framework to build concept bottleneck models that do not require domain experts and labeled concept data.\nWeakness: The evaluation is rather weak. The downsides or the limitations of using a model to create concepts are not discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4YUO4X20JBL", "reviewer": "Reviewer_mHS9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Citing two limitations of Concept bottleneck model (CBM),  (1) the time consuming and labor intensive need to collect labeled data for each of the predefined concepts and (2) their accuracy s often significantly lower than that of a standard neural network, the authors propose Label-free CBM; a framework to transform any neural network into an interpretable CBM without labeled concept data, while retaining nearly the same accuracy as the original model.  They prompt CLIP based on templates filled in with task class names to generate an initial concept set (ie, label free) and then filter that down based on many heuristics.  They then learn a concept matrix ( the dotproduct of task image encodings with each concept text embedding ) and subsequent concept bottleneck layer of concept weights for a given image and then finally learn a sparse final layer using elastic net penalty and solved via GLM-SAGA ( from Wong et al 21 ).  They compare there results with CBMs and PostHoc CBMs (a similar-ish prior method).  They additionally show how to explain individual instances via concepts ( like CBMS ), how to generate global explainability rules from the concept weights and final layer weights and discuss model editing, namely the final sparse layer weights.", "summary_of_the_review": "The use of CLIP to generate task specific concepts sets seems like a useful method, though the filtering methods felt like heuristics that should have included ablation studies to justify certain decisions ( concepts not being near classes for instances, and all the  thresholds that are included in section 3.1 B.   The use of CLIP-Dissect (CD) and the GLM-SAGA solver also seem novel and well motivated though I’m not sure of what the exact difference between CD and what they propose in 3.2 is.   The paper makes many unsubstantiated claims, under explains important elements of the paper, in particular with regards to having human validation of the filtered concepts and I don’t find section 5.2  terribly convincing.  The paper is a useful contribution, but could be made much stronger by addressing some of the issues and performing certain ablations to justify decisions/claims made..  \n", "strength_and_weaknesses": "**Strengths:**   \nThe use of CLIP to generate task specific concepts sets seems like a useful method and the use of CLIP-Dissect (CD) and the GLM-SAGA solver also seem novel and well motivated. \n\n**Weaknesses**  \nThe filtering methods felt like heuristics that should have included ablation studies to justify certain decisions ( concepts not being near classes for instances, and all the  thresholds that are included in section 3.1 B ).\n\nThe paper makes a few unsupported claims such as   \n1) in general we found GPT-3 concepts to be of higher quality (than ConceptNet), but you never actually do a qualitative comparison or study in the paper to back that ( and a concept quality analysis is one of the major things missing from the paper).   Using a human curated ontology would seem to have some advantages over task specific concept sets ( namely using the same concepts over different data and tasks) though the paper shows accuracy wise they get better result, but this would have been made stronger had they included an experiment showing the results they would have obtained by using Concept Net to generate their concept set ( ie, how much is it the concepts vs the architecture which gives this paper a performance bump.  \n2) criticizing the use of CLIP in Posthoc CBMs when its a substantial part of the architecture here  ( I honestly could be missing something, but it doesn’t seem like Post CBMs tie them to CLIP anymore than the Label Free model is? )  \n3) In Table 2 you show Label-Free CBM  outperforms Post-hoc CBM in 3 of 5 and 2 of 5 tasks for the CLIP variant . You can show values for other datasets if you have them, but without that you can’t say your work clearly outperforms PCBMs “on all datasets”.  \n4) Section 5 starts, “this is the first example of manually editing a large neural network with no obvious flaws”.  This is a vague assertation. What methods of manually editing are you referring to that have obvious flaws? PCBMs propose model editing in their paper so if you are commenting on their proposal or another papers proposal you should specify why they have obvious flaws.\n\nThe paper under explains important elements of the paper, in particular with regards to having human validation of the filtered concepts.\nAblations and some under explained areas include:  \n1) How important are each of the filtering techniques really?  An ablation on these, in particular points 1 + 2, would be useful.  The number of cutoffs and thresholds makes this feel not very “automated”.   \n2) What would have happened without the sparseness  constraint on the last layer in terms of accuracy?    \n3)  How are the decision rules  in Fig 3 learned? Is this showing the average Wf layer for training examples of each class and then showing relative weight.  This vizs are nice, but you should explain this better ( maybe swap 4.4 and 4.3 in order and then add additional explanation )  \n4) Its possible I’m missing something, but the algorithm in section 5.2 is quite vague and not terribly convincing because of it.  It seems like its really flipping model weights and hoping more examples are corrected than made incorrect by the new model and is not a systemic, principled way to do things.  \n\n**Nitpicks:**\n1) Why do you spend time introducing IBDs as interpretable CBMs only to not compare against them because you say they have non-interpretable components?    \n2) I personally think its ok to have a residual layer to improve CBM performance ( PCBM-h ) and it’d be better to include those results with that caveat of the model not being fully interpretable   \n3) For Fig 2: you should show  \n- there are M concepts in concept set ( which is implied by numbers in Concept Matrix ) AND  \n- f_c(x) = Wc * f(x) AND  \n- the output of step 4 are classes and not concepts ( so add that each x has an associated y in R^p for instance ) \n   so the architecture its not misunderstood to be completely unsupervised.    \n   you then need to write that W_f is M by p   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster