Post-hoc Concept Bottleneck Models
Authors: Mert Yuksekgonul
Maggie Wang
James Zou
Venue: ICLR 2023 notable top 25% (2023)
Keywords: concepts
interpretability
concept bottleneck models
model editing

Abstract: Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts (``the bottleneck'') and use the concepts to make predictions. A concept bottleneck enhances interpretability since it can be investigated to understand what concepts the model "sees" in an input and which of these concepts are deemed important. However, CBMs are restrictive in practice as they require dense concept annotations in the training data to learn the bottleneck. Moreover, CBMs often do not match the accuracy of an unrestricted neural network, reducing the incentive to deploy them in practice. In this work, we address these limitations of CBMs by introducing Post-hoc Concept Bottleneck models (PCBMs). We show that we can turn any neural network into a PCBM without sacrificing model performance while still retaining the interpretability benefits. When concept annotations are not available on the training data, we show that PCBM can transfer concepts from other datasets or from natural language descriptions of concepts via multimodal models. A key benefit of PCBM is that it enables users to quickly debug and update the model to reduce spurious correlations and improve generalization to new distributions. PCBM allows for global model edits, which can be more efficient than previous works on local interventions that fix a specific prediction. Through a model-editing user study, we show that editing PCBMs via concept-level feedback can provide significant performance gains without using data from the target domain or model retraining.

Published as a conference paper at ICLR 2023
POST-HOC CONCEPT BOTTLENECK MODELS
Mert Yuksekgonul, Maggie Wang, James Zou Stanford University {merty, waggiewang. Jamesz]@stanforodl
ABSTRACT
Concept Bottleneck Models (CBMs) map the inputs onto a set of interpretable concepts ("the bottleneck") and use the concepts make predictions. A concept bottleneck enhances interpretability since it can be investigated to understand what concepts the model "sees" in an input and which of these concepts are deemed important. However, CBMs are restrictive in practice as they require dense concept annotations in the training data to learn the bottleneck. Moreover, CBMs often do not match the accuracy of an unrestricted neural network, reducing the incentive deploy them in practice. In this work, we address these limitations of CBMs by introducing Post-hoc Concept Bottleneck models (PCBMs). We show that we can turn any neural network into PCBM without sacrificing model performance while still retaining the interpretability benefits. When concept annotations not available on the training data, we show that PCBM can transfer concepts from other datasets from natural language descriptions of concepts via multimodal models. A key benefit of PCBM that it enables users to quickly debug and update the model to reduce spurious correlations and improve generalization new distributions. PCBM allows for global model edits, which can be more efficient than previous works local interventions that fix a specific prediction. Through a model-editing user study, we show that editing PCBMs via concept- level feedback can provide significant performance gains without using data from the target domain or model retraining. The code for our paper can be found in hoc-com
https:/ /ttp://////tuuuuuooe/ www.c//w...eeeceeee...cce.eeee.. htt///ertteee www.tttttypsttt.eeeeeeem hotp://c/oo//tttt//ooootooo----
INTRODUCTION
There is growing interest in developing deep learning models that are interpretable and yet still flexible. One approach concept analysis (Kim et al. (018), where the goal is to understand if and how high-level human-understandalee instance, we may like probe skin lesion classifier understand if the Irregular Streaks concept is encoded in the embedding space of the classifier and used later to make the prediction.
features are "engineered" and used by neural networks. For
Our work builds on the earlier idea of concept bottlenecks, specifically Concept Bottleneck Models (CBMs) (Koh et al. 2020). Concept bottlenecks are inspired by the idea that we can solve the task of interest by applying a function an underlying set of hhwaan-inteppa interpeeateee concepts. For instance. when trying to classify whether a skin tumor is malignant, dermatologists look for different visual patterns, e-g. existence of Blue- Whitish Veils can be useful indicator of melanoma (Menzies et al., 1996] Lucieri et al. 2020). CBMs train entire model in end-to-end fashion by first predicting concepts (e.g. the presence Blue- Whitish Veils), then using these concepts to predict the label.
By constraining the model to only rely set of concepts and an interpretable predictor, we can: explain what information the model is using when classifying an input by looking at the weights/rules in the interpretable predictor and understand when the model made particular mistake due to incorrect concept predictions.
While CBMs provide several of the benefits mentioned above, they have several key limitations:
1. Data: CBMs require access to concept labels during model training, i.e. training data should be annotated with which concepts are present. Even though there number of densely annotated datasets such as CUB (Wah et al., (01T)) this particularly restrictive for real-worlds use cases, where training datasets rarely have concept annotations.

---

Published as a conference paper at ICLR 2023
2. Performance: CBMs often do not match the accuracy of an unrestricted model, potentially reducing the incentive to use them in practice. When the concepts are not enough to solve the desired task, it is not clear how to improve the CBM and match the original model performance, while retaining the interpretability benefits.
3. Model editing: Koh et al. (2020) discuss intervening on the model to fix the prediction for single imput, yet it not shown changes the model behavior for single sample, but global editing changes the model behavior completely. When the model pieks up an unintended cue, or learns spurious associations, using the latter approach and editing the concept bottleneck can improve the model performance more generally than an intervention tailored toward one specific input. Prior work on CBMss does not discuss how to globally edit a model's behavior. Idcally, we would like to edit models with the help of human input order to lower computational costs and remove assumptions about data access.
to holistically edit and improve the model itself. Intervening only
Our contributions. In this work, we propose the Post-hoc Concept Bottleneck Model (PCBM) to address these important challenges. PCBMs convert any pre-trained model into concept bottleneck model in data-efficient manner, and enhance the model with the desired interpretability benefits. When the training data does not have concept annotations, which is often the case, PCBM can flexibly leverage concepts annotated in other datasets and natural language descriptions of concepts. When applicable, PCBMs can remove the laborious concept annotation process by leveraging multimodal models to obtain concept representations; this results in richer and more expressive bottlenecks using natural language descriptions of concept, making PCBMs more accessible in various settings. Furthermore, when the available concepts are not sufficiently rich, we introduce a residual modeling step to the PCBM recover the original blackbox model's performance. In experiments across several tasks, we show that PCBMs can be used with comparable performance compared to black-box models. While prior work (Koh et al. 2020) demonstrated the possibility of performing local model interventions change individual predictions, here propose interventions for changing global model behavior. Through user studies, we show that PCBM enables efficient global model edits without retraining access to data from the target domain and that users can improve PCBM performance by using concept-level feedback to drive editing decisions.
POST-HOC CONCEPT BOTTLENECK MODELS
Post-hoc (PCBM)
Concopt
Multimodal
(CAVA)
Backbone
Embeddings
Projection ento the
Predictions R
Concent Subsoiice
f(
"stripes" "norse"
f:X + Rd
f(x) E Rd
proje (z) E R
stripes
Stripes:
M(
Handsz
Horse:12
Interpretable Predictor (egg. Sparse Linear Leyer)
PCRM predicttoo
Baacko07
π(
Text Encoder
Mane: og
pizebraj 0.0
PCBM-h prediction
Residual Fitting
Stripes Ghands
Crann
Cplack
pizebra): 0:55
R + y
C M.
Hybrid Post-hoc CBM (PCBM-h)
Learning the Concept Bank
Fitting the Concept Bottleneck
Figure 1: Post-hoc Concept Bottleneck Models. First, we learn the vectors in our concept bank. With the CAV approach, for each concept, e.g. stripes, we train a linear SVM to distinguish the embeddings of examples that contain the concept and use the vector normal to the boundary (CAV). When annotations are hard to obtain, we can leverage multimodal models and use the text encoder to map each concept to vector. Next, we project the embeddings produced by the backbone onto the concept subspace defined by the set of vectors. We then train interpretable predictor to classify the examples from their projections. When the concept library is incomplete, we can construct a PCBM-h by sequentially introducing residual predictor that maps the embeddings to the target space.
2

---

Published as a conference paper at ICLR 2023
There are two main steps to building PCBM. We let f X - Rd be any pretrained backbone model, where d is the size of the corresponding embedding space and X' is the imput space. For instance, f can be the image encoder of CLIP (Radford et al., (02T) or the model layers up to the penultimate layer of a ResNet (He et al. 2016). An overview of PCBMs can be found in and we describe the steps in detail below.
Learning the Concept Subspace (C); To learn concept representations, we make use of CAVs (Concept Activation Vectors) (Kim al., 2018). In particular, we first define concept library I= {ū,, d2,,.., ...; J. where N₂ denotes the number of concepts. The concepts in the library can be
selected by a domain expert or learned automatically from the data (Ghorbani al. 2019: Yeh et al.)
2020). For each concept i, we collect embeddings for the positive examples, denoted by the set P. (T(p)))..... (FFP,, }}, that exhibit the concept, and negative examples N; = (f )..... (Fpx..) }
that do not contain the concept. In practice, densely annotated datasets (Caesar et al.. 2018 Fong
Vedaldi 2018) are used to collect points that are positive/negative for the concept. Importantly. unlike
CBMs, these samples can be different from the data used to train the backbone model. Following (Kim et al. 2018), we train linear SVM using P; and N; to learn the corresponding CAV, that is, the vector normal to the linear classification boundary. We denote the CAV for concept as c, Let xd
C' e IRN, xdd denote the matrix of concept vectors, where each row c; represents the corresponding concept i. Given an input, we project the embedding of the input onto the subspace spanned by concept vectors (the concept subspace). Particularly, we let fc(x)) proje fx:) RRNe where the
ith entry is e (3:) = 1 E R. It is important to observe that we do not need to annotate the training data with concepts. Namely, the dataset used to learn concepts can be different from the original training data. In several of our experiments, we use held-out datasets to learn the concept
(f())_c.)
subspace.
Leveraging multimodal models to learn concepts: Annotating images with concepts is a laborious process. In practice, this be an important roadblock before using CBMs. To address this, here we show that we can also leverage natural language concept descriptions and multimodal models implement concept bottlenecks. This approach alleviates the need of collecting labeled data to construct the concept subspace. Multimodal models such as CLIP (Radford et al. 202T) have a text encoder fuxt along with an image encoder, which maps description to the shared embedding space. We leverage the text encoder to augment the process of learning concept vectors. For instance, we can have clext Stripes = fiex "stripes") where the concept vector for stripes is obtained by mapping the prompt using the text encoder. For each concept description, we can collect the text embeddings and construct our multimodal concept bank Crext as our subspace.
For given classification task, we can use ConceptNet (Speer et al., 2017) to obtain concepts that are relevant to these classes. ConceptNet is an open knowledge graph, where we can find concepts that have particular relations to query concept. For instance, we can find relations of the form "A Cat has [whiskers, four legs, sharp claws, ..."'. Similarly, we can find "parts" of given class (e.g. "'bumper', "roof" for "truck"), or the superclass of given class (c.g. "animall' "canine" for "dog"). We restrict ourselves to five sets of relations for each class: the hasA, isA, partOf, HasProperty MadeOf relations in ConceptNet. We collect all concepts that have these relations with classes in each classification task to build the concept subspace.
Next, we define an interpretable predictor that maps the concept subspace to the model prediction. Concretely, let g : IR + ) be interpretable predictor, such as a sparse linear model or a decision tree, where ) = (1, 2..., K) denotes the label space. An interpretable predictor is desirable because it provides insight into which concepts the model is relying on when making decision. If a domain expert observes a counter-intuitive phenomenon in the predictor, they can edit the predictor improve the model. To learn the PCBM, we solve the following problem:
Learning the Interpretable Predictor:
À N.K
min E (C) (E.
2(g)
where fc = f() is the projection onto the concept subspace, L(j, y) is a loss function such as cross-entropy loss, 22(g) is a complexity measure to regularize the model, and is the regularization strength. Note that the concept subspace is fixed during PCBM training. In this work, we use sparse linear models to learn the interpretable predictor, where ((fc(z)) = w" fe(x) + b. We apply the softmax function to g if the problem is a classification problem. Similarly, define
3

---

Published as a conference paper at ICLR 2023
((g)) a =a||||11 + [
[ to be the elastic-net penalty parameterized by a, then normalize by the number of classes and concepts.
The expressiveness of the concept subspace is crucial to PCBM performance. However, even when we have a rich concept subspace, concepts may not be enough to solve task of interest. When the performance of the PCBM does not match the performance of the original model, users have less incentive to use interpretable models like concept bottlenecks. Next, we aim to address this limitation.
Recovering the original model performance with residual modeling: What happens when the concept bank is not sufficiently expressive, and PCBM performs worse than the original model? For instance, there may be skin lesion descriptors that are not available in the concept library. Ideally, we would like to preserve the original model's Is accuracy while retaining the interpretability benefits. Drawing inspiration from the semiparametric model literature on fitting residuals (Härdle al. 2004), we introduce Hybrid Post-hoc CBMs (PCBM-h). After fixing the concept bottleneck and the interpretable predictor, we re-introduce the embeddings 'fit the residuals'. In particular, we solve the following:
min ((ss))-((((((((s)) + + (f(())) v)
(2)
where : Rd + y is the residual predictor. Note that in Equation 2 while training the residual predictor, the trained concept bottleneck (the concept subspace (fc) and the interpretable predictor(g)) is kept fixed, i.c. fitting PCBM-hs is a sequential procedure. We hypothesize that the residual predictor will compensate for what is missing from the concept bank and recover the original model's 's accuracy. We implement the residual predictor as a linear model, i.e. r(f(z)) = = ff fxx ) + b,. Given a trained PCBM-h, if we would like to observe model performance in the absence of the residual predictor, we can simply drop r (in other words, concept contributions do not depend on the residual step).
3 EXPERIMENTS
We evaluate the PCBM and PCBM-h in challenging image classification and medical settings, demonstrating several use cases for PCBMs. We further address practical concerns and show that PCBMs can be used without loss in the original model performance. We used the following datasets to systematically evaluate the PCBM and PCBM-h:
CIFARIO, CIFARIOO (Krizhevsky et al., 2009) We use CLIP-ResNet50 (Radford al. 202 as the backbone model. For the concept bottleneck, we use 170 concepts introduced in (Abid et al. 2022) which are extracted from the BRODEN visual concepts dataset (Fong Vedaldi 2018). These include objects (e.g. dog), settings, (e.g. snow) textures (e.g. stripes), and image qualities (e.g. blurriness). The full list of concepts can found (Abid et al. 2022).
COCO-Stuff (Caesar et al. 2018) is dataset derived from MS-COCO (Lin et al. 2014), consisting of scenes with various object annotations. Previous work (Singh et al.] (020) has shown that there are severe co-occurrence biases in this dataset. For instance, images from the wine glass often also have a dining table in the image. (ingh et al. (2020) identifies these 20 classes with heavy co-occurrence biases and we train PCBMs to recognize these 20 objects, where we minimize the binary cross-entron loss individually for each class. In this scenario, we again use CLIP ResNet50 (Radford et al. 2021 as the backbone and BRODEN visual concepts as the set of concepts.
CUB (Wah et al., 2011 In the 200-way bird identification task, we use a ResNet18 (He et al. 2016) trained on the CUB [ We use the same waaning/validation splits and 112 concepts in (Khh et al. 2020). These concepts include wing shape, back pattern, and eye color.
HAM10000 (Tschandi et al., 2018) is a dataset of dermoscopic images, which contain skin lesions from a representative set of diagnostic categories. The task is detecting whether a skin lesion is benign or malignant. We the Inception (Szegedy al., 2015) model trained this dataset, which is available from (Daneshjou et al. 2021). Following the setting in (Lucieri et al. 2020). we collect concepts from the Derm7pt (Kawahara al. 2018) dataset. The 8 concepts obtained from this dataset include Blue Whitish Veil, Pigmented Networks, Regression Structures, which are reportedly associated with the malignancy of a lesion.
'The CUB pretrained model is obtained from https::/githhb..ccctttm wwm http://b..ooo//o////o/o/ mggo 1mmbb
4

---

Published as a conference paper at ICLR 2023
SIIM-ISIC (Rotemberg et al., 2021) To test weal-world transfer learning use case, evaluate the model trained on HAMI0000 on subset of the SIIM-ISIC Melanoma Classification dataset. We use the same concepts described in the HAM10000 dataset.
We use linear probing on top of the backbones for CIFAR, COCO, and ISIC to obtain the baseline model performance. For CUB and HAM10000, use the out-of-the-bon models trained on the respective datasets without any additional training. We emphasize that in all of our experiments except CUB, we learn concepts using held-out dataset. Namely, while the original CBMs need to have concept annotations for training images, we remove this limitation by post-hoc learning of concepts with any dataset. To learn the concept subspace, we use 50 pairs of images for each concept to train a Linear SVM. We refer the reader to the Appendix for further details on datasets and training.
PCBMs achieve comparable performance to the original model: In Table ! we report results over these five datasets. PCBM achieves comparable performance to the original model in all datasets except CIFAR 100, and PCBM-h matches the performance in all scenarios. Strikingly, PCBMs match the performance of the original model in HAM10000 and ISIC, with as few as 8 human-interpretable concepts. In CIFAR I00, we hypothesize that the concept bank available is not sufficient to classify finer-grained classes, and hence there is performance gap between the PCBM and the original model. When the concept bank is not sufficient to solve the task, PCBM-h can be introduced to recover the original model performance while retaining the benefits of PCBM (see next sections). Comparing to CBM: A comparison to CBM was not possible in most datasets, as CBMs cannot be trained due the lack of dense annotations. To compare the data efficiency of both methods, report a comparison of CBM to PCBM when trained on CUB in Appendix [C] finding that CBM can only reach a similar performance PCBM with 20x more annotations. Analyzing the residual predictor: To understand whether PCBM-h overrides PCBM predictions, in Appendix B we look at the consistency between PCBM and PCBM-h predictions. We show that the residual component in PCBM-h intervenes only when the prediction is wrong, and fixes mistakes. When PCBM is confident, PCBM-h does not modify the prediction or significantly increase the confidence. general, the residual component may dominate when the bottleneck is insufficient and future work can aim to explicitly limit the residual component, c.g. PIE (Wang et al. 202I) regularizer.
Table I: PCBMs achieve comparable performance to the original model. We report performance over different scenarios for the original model and PCBMs with concept datasets. In CIFAR 100, PCBM performs poorly since the concept bank is not expressive enough to solve a finerggrined task; however, PCBM-h recovers the original model's accuracy. Strikingly, PCBMs match the performance of the original model in HAM10000 and ISIC, with as few as 8 human-interpretable concepts. Original CBMs cannot be trained CIFAR/HAMI WTTT1/00///////CC/COOC00O concept labels in the training dataset. The mean and standard errors are reported over 10 random seeds. We report AUROC for HAM10000 and ISIC, mAP for COCO-Stuff, and accuracy for CIFAR and CUB.
as they do not have
CIFARIO
CIFARI00
COCO-Stuff
CUB
HAM10000
ISIC
Original Model
0.888
0.701
0.770
0.612
0.963
0.821
PCBM
0.777 0.003
0.520 + 0.005
0.741 + 0.002
0.588 + 0.008
0.947 = 0.001
0.736 0.012
PCBM-h
0.871 1 0.001
0.680 1 0.001
0.768 0.01
0.610 1 0.010
0.962 1 0.002
0.801 1 0.056
PCBM using CLIP concepts: Here, we show that when la- beled examples to learn concepts are not available, we can use multimodal representations such as CLIP to generate concepts without the laborious annotation cost. Using ConceptNet, we ob-
Table 2: Concept Bottlenecks with CLIP concepts. When a
concept bank is not available or is insufficient, we can use natural language descriptions of concepts with CLIP to implement CBMs.
CIFARIO
CIFARI00
COCO-Stuff
Original Model (CLIP)
0.888
0.701
0.770
PCEM & labeled coneepts
0.777 = 0.003
0.520 + 0.000
0.741
= 0.002
PCBM CLIP concepts
0.833
0.003
0.600 + 0.003
0.755
0.001
tain concept descriptions for CI- FARI0, CIFAR 100, and COCO-
PCBM-h CLIP concepts
0.874 = 0.001
0.691 1 0.006
0.769
0.001
Stuff tasks (206. 527, and 822 concepts, respectively), using the relations described in Section 2 by using text embeddings for the concept descriptions generated by the ResNet50 variant of CLIP. In Table 2
The concept subspace is obtained
we give overview of the results. We observe that with more expressive multimodal
5

---

Published as a conference paper at ICLR 2023
bottleneck, we almost recover the original model accuracy in CIFARIO, and reduce the performance gap in CIFARI00. CLIP contains extensive knowledge of natural images, and thus captures natural image concepts (e.g. red, thorns). However, CLIP is less reliable at representing more granular or domain-specific concepts (e.g. blue whitish veils) and hence this methodology is less applicable more domain-speciffe applications. While there is recent trend in training multimodal models for biomedical applications (Zhang et al. [020; EleutherAI). these models are trained with smaller datasets and are less competitive. We hope that as these models improve, PCBMs will make it easier to deploy concept bottlenecks in more specialized domains.
In Figure 2. we provide sample concept weights in the corresponding PCBMs. For instance, in HAM10000, PCBMs use Blue Whitish Veils, Atypical Pigment Networks, and Irregular Streaks to identify malignant lesions, whereas Typical Pigment Networks, Regular Streaks, and Regular Dots and Globules are used to identify benign lesions. These associations are consistent with medical knowledge (Menzies et al. 1996, Lucieri et al., 2020). We observe similar cases in CIFAR-I0 and CIFAR-100, where the class Rose is associated with the concepts Flower, Thoms and Red. In Appendix Section F, we further analyze the global explanations of COCO-Stuff and show that PCBM reveals co-occurrence biases in the training data. Note that the concept weights for PCBM-h are exactly the same as for PCBM since the residual model is fit after the concept weights are fixed. However, given the addition of the residual component, the effect of concepts may be different. We also note that the concept weights discussed may not reflect the causal effect. Estimating the causal effect of a concept requires careful treatment (Goyal et al. 2019 e.g. accounting for the interactions between concepts, and we leave this to future work.
Explanations in PCBMs:
HAM10000
CIFAR10Q
w proje f(x) +
Class: Malignant
Class: Wolf
Whitish Volls
04459
Conina
4972
Interpretable Predictor (Sparse Linear Layer)
Atypical Pigment
Fur
3860
Imegular Streakss
Long Ears
364
W
Class: Benign
Regular Streaks
Class: Rose
Flower
Regular Dots and Globulos
Analyzing Concept importances
0.151
Thoms
Typical Pigment Nets
0.091
Red
2109
PCBM
PCBM with CLIP Concepts
Figure 2: Explaining Post-hoc CBMs. We report the top 3 largest weights in the linear layer for the shown classes. For instance, Blue Whitish Veils, Atypical Pigment Networks, and Irregular Streaks have large weights for classifying whether skin lesion is malignant. These are consistent with dermatologists' findings (Menzies al. 1996).
4 MODEL EDITING WITH POST-HOC CONCEPT BOTTLENECKS
When we observe a counter-intuitive phenomenon or a spurious correlation in the concept bottleneck, can we make the model perform better by simple edit operations? In this section, we show that PCBMs come with the benefit of easy and global concept-level model feedback. KKhh et al. (2020) demonstrate the ability of CBMs to incorporate local interventions. Namely, a user identifies misprediction in the concept bottleneck, they can intervene on the concept prediction, and change the prediction for the particular instance, we call this local intervention or edit. Here, we are interested in editing models to perform 'global' edits. Namely, with simple concept-level feedback, we would like to edit a PCBM and change the entire model behavior. For instance, in our experiments, we investigate the use case of adapting PCBM to new distribution. Unlike most existing model editing approaches, we do not need any data from either the training or target domains to perform the model edit, which can be significant advantage in practice when data is inaccessible. Given trained PCBM, we edit our concept bottleneck by manipulating the concept weights. For simplicity, in the experiments below only focus on positive weights in the linear model.

---

Published as a conference paper at ICLR 2023
4.1 CONTROLLED EXPERIMENT: EDITING PCBM WHEN THE SPURIOUS CONCEPT KNOWN
For our editing experiments, we use the Metashift (Liang & Zou 2022) dataset to simulate distribution shifts. We use 10 different scenarios where there is a distribution shift for a particular class between the training and test datasets. For instance, during training, we only use table images that also contain a dog in the image and test the model with table images where there is not a dog in the image. We denote the training domain as table(dog). We give more details and the results of all domains in the Appendix. We use an ImageNet pretrained ResNet50 variant as a backbone and the visual concept bank described in the Experiments section. Given a PCBM, we evaluate three editing strategies:
1. Prune: We set the weight of a particular concept a particular class prediction to 0, i.c. for concept indexed by i, we let i0 = O.
2. Prune+Normalize: After applying pruning, we rescale the concept weights. Let P denote the indices of positive weights that pruned, p denote the indices of weights that remain, and wp, wp be corresponding weight vectors for the particular class. We rescale each weight to match the original norm of the vector by letting Vi P, ú; w; (1(1+ || leading ||عl|| = ||w]| 1
The normalization step alleviates the imbalance hetween class weights upon pruning concepts with large weights for a particular class.
3. Fine-tune (Oracle): We compare our simple editing strategies to fine-tuning on the test domain, which can be considered an oracle. Particularly, we fine-tune the PCBM using samples from the test domain and then test the fine-tuned model with set of held-out samples.
In the context of Metashift experiments, we simply edit the concept spuriously correlated with a particular class. Concretely, for the domain table(dog). we prune the weight of the dog concept for the class table. In Table 3 we report the result of our editing experiments over 10 scenarios. We observe that for PCBMs, the Prune+Normalize strategy can recover almost half of the accuracy gains achieved by fine-tuning on the original test domain. Further, we observe that PCBM-h obtains lower accuracy gains with model editing. In PCBM, we can remove the concept from the model, but in PCBM-h, there may still be leftover information about the spurious concept in the residual part of the model. This is in line with our expectations, i.e. PCBM-h is a 'less' interpretable but more powerful variant of PCBM. While it still offers partial editing improvements that would not be possible with the vanilla model, it does not bring as much improvement as the PCBM.
Table 3: Model edits with Post-Hoc CBMs. We report results over 10 distribution shift experiments generated using Metashift. Accuracy and edit gains are averaged over 10 scenarios and are reported as mean + standard error. We observe that very simple editing strategies the concept subspace provide almost 50% of the gains made by fine-tuning on the test domain.
Unedited
Prune
Prune + Normalize
Fine-tune (Oracle)
PCBM Accuracy PCBM Edit Gain
0.656 + 0.025
0.686 + 0.026
0.750 + 0.019
0.859 + 0.028
0.029
0.093
0.202
PCBM-h Accuracy
0.657 + 0.039
0.672 + 0.033
0.713 + 0.027
0.861 + 0.032
PCBM-h Edit Gain
0.017
0.058
0.190
Even though our edit strategy extremely simple, we can recover almost half of the gains made by fine-tuning the model. It is particularly easy to use since it can be applied without fine-tuning or using any knowledge or data from the target domain. However, this methodology requires knowledge of the spurious concept in the training domain. This may not be realistic in practice, and in the next section, we turn to human users for guidance.
4.2 USER STUDY: EDITING PCBM WITH HUMAN GUIDANCE
One of the advantages of pruning is that it is naturally amenable to bringing humans into the loop. Rather than attempting to automate the edits, we can rely on a human to pick concepts that make logical sense to prune. We show that users make fast and insightful pruning decisions that improve model performance on test data when there is a distribution shift. Notably, these decisions can be made even when the user knows little about the model architecture and has no knowledge of the true
7

---

Published as a conference paper at ICLR 2023
underlying spurious correlation in the training data. We conduct user study where explore the benefits of coman-guidee pruning on PCBM and PCBM-h performance.
Study Design:
We construct 9 MetaShift scenarios with a distribution shift for one of the classes between the training and testing datasets, which refer to as the "shifted class". For instance, all of the training images for the class Keyboard also contain Cat in the image, whereas test images do not have a cat in the image. The list of the 9 scenarios can be found in the Appendix. For each scenario, we train a PCBM and a PCBM-h with CLIP concepts on the training dataset and task the users with making edits to the trained models by selecting concepts to prune. We use CLIP's ResNet50 variant as our backbone, and leverage ConceptNet to construct the concept subspace.
Rempoe concepts from the morkl that you think Kayboard
perierelization class yng
The mockel uow los to daling Beach, Bux. Airplane: Keyboard Bird
Largest 10 Concept Weights for Keyboard
spacebar
tab key
6.52
feline
4.27
sharp claws
whisker
claws
keys
1.26
fur 1.05
Figure 3 shows the concept-selection interface. We refer to the Appendix for the full set of instructions given to the user. The display consists of the classification task the model was trained on and the concepts with the ten most positive weights for the shifted class. The user selects subset of the concepts to prune. Neither the original model accuracy nor the accuracy after pruning is revealed completed. The target audience
device 0.96
giblet 0.60
Weight
Choose option
the user until all scenarios for the study was machine learning practitioners and researchers; 30 volunteers participated for total of 30 x 270 experiments. The IRB has determined that this does not require review.
Figure 3: User Study In- terface. We train PCBMs on MetaShift scenarios, each with distribution shift be- tween the training and test datasets. The user selects con- cepts to prune from the model.
Human-guided editing is fast and improves model accuracy: On average, users prune 3.65 + 0.39 concepts in 34.3 = 6.4s per scenario. We evaluate the effectiveness of user pruning by comparing the model accuracy after user pruning to the accuracy of the unedited model on a held-out dataset from the test domain. All 30/30 users improve the model accuracy when averaged over scenarios. 8/30 users improve model accuracy in all scenarios and 26/30 users improve model accuracy in at least 6 scenarios. We compare ouman-guided editing to three baselines:
1. Random Pruning: We uniformly select without replacement subset of the top ten concepts to prune, matching the number of concepts pruned by the user for fair comparison.
2. Greedy Pruning (Oracle): We greedily select one of the top 10 concepts that, when pruned, improves the model accuracy the most. We match the number of concepts pruned by the user.
3. Fine-tune (Oracle): The model is fine-tuned samples from the test domain.
Table 4: Human-guided editing improves model accuracy for PCBMs with CLIP concepts (N=30), We show the test accuracy on the shifted class averaged across 9 MetaShift scenarios, before and after the model is edited. Accuracies and edit gains are reported as mean standard error. (For the pruning strategies, we first compute the mean test accuracy across users within each scenario, then calculate the overall mean and standard error from the within-scenarri means). We see that user pruning surpasses random pruning and can achieve nearly 50% of the accuracy gains made by fine-tuning and over 80% of the accuracy gains made by greedy pruning.
Unedited
Random Prune
User Prune
Greedy Prune (Oracle)
Fine-tune (Oracle)
PCBM Accuracy
0.620 + 0.035
0.604 0.039
0.719 + 0.042
0.740 0.041
0.824 + 0.049
PCBM Edit Gain
-0.016
0.099
0.120
0.204
PCBM-h Accuracy PCBM-h Edit Gain
0.642 1 0.034
0.622 0.037
0.736 1 0.034
0.766 0.034
0.856 0.018
-0.020
0.094
0.124
0.224
We consider greedy pruning and retraining to be oracles since they require "leaked" knowledge of the test domain. reblotsr the model performance improvements for PCBM and PCBM-h, averaged over all 270 experiments in the user study. Improvements within each separate scenario can be found in the Appendix. We note that even with the residual component in PCBM-h -h, we can still improve model performance by editing the concept bottleneck. We observe that pruning achieves marked improvement in model performance compared to the unedited model, attaining around 50% of the accuracy gains from fine-tuning and 80% of the gains from greedy pruning.
8

---

Published as a conference paper at ICLR 2023
5 RELATED WORKS
Concepts Using human concepts to interpret model behavior has been drawing increasing inter-
est (Kim et al. 2018 Bau et al. 2017, 2020). Related work focuses on understanding if neural
networks encode and use concepts (Lucieri al. 2020; Kim et al. 2018 McGrath et al. (022), or
generate counterfactual explanations to understand model behavior (Ghandeharioun et al. 2021 Abid
ce al.
2022 Akula et al. 2020). Recent works evaluated the causal validity of explanations ( Feee
et al.
2021 Elazar et al. 2021 Goyal et al., 2019), e.g. to eliminate potential confounding effects.
There is further increasing interest in automatically discovering the concepts that are used by a
model (Yeh et al. 2020 Ghorbani et al.
2019:
Lang et (021). Concept-based Models: Concept
bottleneck models (CBMs) (Koh et al.
2020) extend the earlier idea (I (Lampert et al. 2009] Kumar
ct al. 2009) of first predicting the concepts, then using concepts to predict the target. CBMs bring about interpretability benefits but require training the model using concept labels for the entire training dataset, which is key limitation. CBMs have not been analyzed in terms of model edits. Recent work
reveals that end-to-end learned CBMs encode information beyond the desired concept (Mahinpei) al. [02T] Margeloiu et al. (02T). Concept Whitening (Chen et al., (020) aims to align each concept with an individual dimension in layer. While a subset of dimensions is aligned, this approach lacks a bottleneck since there are dimensions that are potentially not aligned with a concept (comparable to PCBM-h). [Brnntt et al. (202l) learns prototypes and uses them as concepts, where the distance to prototypes determines the model prediction. PCBM-h is inspired by semiparametric models on fitting residuals (Härdle et al., 2004). PIE (Wang et al., 1021 has similar approach, where they combine individual features with more complicated interaction terms for tabular data.
Model Editing Model editing aims to achieve the removal modification of information in given neural network. Several models investigated editing factual knowledge in language models: iZh et al. (220)) use variants of fine-tuning to achieve this objective while retaining performance on unmodified factual knowledge and Mitchell et al. (202T); De Cao et al. (202T): Hase et al. (202T) update the model by training a separate network modify model parameters to achieve the desired edit. Similarly, Sotoudeh & Thakur (022)) proposes "repairing" models by finding minimal parameter updates that satisfy a given specification. One thread of work focuses on intervening on the latent space of neural networks to alter the generated output towards the desired state, e.g. removal of artifacts or manipulation of object positions (Sinitsin et al.] 2020: Bau et al., 2020, Santurkar et al. 2021). For instance, |Bau et al. (2020) edits generative models. Santurkar et al. (2021) edits classifiers by modifying 'rules'. such as making a model perceive the concept of a "snowy road' as the concept of 'road', and they achieve this by modifying minimal updates to specific feature extractors. FIND (Lertvitayykkumjo
et al., 2020) prunes individual neurons chosen by users and later fine-tunes the model. Right for the Right Concepts (Stammer et al. 2021] similarly fine-tune a neuro-symbolic model with user sser-provided feedback maps. [oottempelli et al.] (2021) give an overview of concept-based models and a discussion on debugging strategies while concurrent work ProtoPDebug (Bontempelli et al.) 2022) proposes an efficient debugging strategy.
LIMITATIONS AND CONCLUSION
In this work, presented Post-hoc CBMs way of converting any model into a CBM, retaining the original model performance without losing the interpretability benefits. We leveraged multimodal models as an interface use concepts, without the laborious concept annotation steps. Further, in addition to the local intervention benefits of CBM, we demonstrated that PCBMs can be leveraged to perform global model interventions. Many benefits of CBMs depend on the quality of the concept library. The concept set should be expressive enough to solve the task of interest. Users should be careful about the concept dataset used to learn concepts, which can reflect various biases. While there are several such real-world tasks, it is an open question if conntruccost concept bottlenecks can solve larger-scale tasks(e.g. ImageNet level). Hence, finding concept subspaces for models in an unsupervised fashion is active area of research that will help with the usability and expressivity of concept bottlenecks. Multimodal models provide an effective interface for concept-level reasoning, yet it is unclear what is the optimal way to have humans in the loop. Here, we presented a simple way of taking human input via concept pruning; how to incorporate richer feedback is an interesting direction for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank Duygu Yilmaz, Adarsh Jeewajee, Carlos Guestrin, Edward Chen, Kyle Swanson, Omer Faruk Akgun, and Roxana Daneshjou for their support and comments on the manuscript, and all members of the Zou Lab and Guestrin Lab for helpful discussions. We thank the anonymous reviewers for their suggestions to improve this manuscript. J.Z. supported by NSF CAREER 1942926 and the Chan-Zuckerberg Biohub.
ETHICAL STATEMENT
our user study, we did not collect any user information. We applied the IRB within our institution and "IRB has determined that this research does not involve human subjects as defined in 45 CFR 46. 46. 102(f) and therefore does not require review by the IRB".
REPRODUCIBILITY STATEMENT
The code to reproduce all experiments will be released at past-hoc-che All of the backbones we used to construct concept bottlenecks are released public checkpoints, and we stated all of these in the manuscript.
com me yg/
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In International Conference Machine Learning, pp. 66-88. PMLR, 2022.
Arjun Akula, Shuai Wang, and Song-Chun Zhu. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In Proceedings of the AAAI Conference on Arlificial Intelligence, 2020.
Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Y Lo, and Cynthia Rudin. case-based interpretable deep learning model for classification mass lesions in digital mammography. Nature Machine Intelligence,
2021.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541 -6549, 2017.
David Bau, Steven Liu, Tongzhou Wang, Yan Zhu, and Antonio Torralba. Rewriting deep generative model. In European Conference on Computer Vision, pp. 351-369. Springer, 2020.
Andrea Bontempelli, Fausto Giunchiglia, Andrea Passerini, and Stefano Teso. Toward unified framework for debugging gray-box models. arXiv preprint 2rXiv:2109.1110 2021.
Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. Concept-level debug-
ging of part-prototypp networks. arXiv preprint arXiv:2205. 2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff' classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218, 2018.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Roxana Daneshjou, Kailas Vodrahalli, Weixin Liang. Roberto Novoa, Melissa Jenkins, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, al. Disparities in dermatology ai: Assessments using diverse clinical images. arXiv preprint 1..0002 2021.
10

---

Published as a conference paper at ICLR 2023
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models, arXiv preprint arXiv:2104 1404:22104.08164,, 2021.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021.
EleutherAI. CLASP: Contrastive Language Aminoacid Sequence Pretraining, 2021. URL https
//github. wwt.c///////e/caagpppppac
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through
counterfactual language models. Computational Linguistics, 47(2):333-386 2021.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730-8738, 2018.
Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Eoff, and Rosalind W Pi- card. Dissect: Disentangled simultaneous explanations via concept traversals. arXiv preprint arXiv:2105.15164, 2021.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, 32, 2019.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv: 190.007650..0 2019.
Wolfgang Härdle, Marlene Müller, Stefan Sperlich, and Axel Werwatz. Nonparametric and semi- parametric models, volume 1. Springer, 2004.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2 13644, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 770-778, 2016.
Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, and Ghassan Hamarnh.. Seven -pient checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538-544 2018.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp.
PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machirane Learning, pp. 9338-5348.. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Autribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pp. 365-372. JEEE, 2009.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference computer vision and pattern recognition, pp. 951-958. IEEF, 2009.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William Freeman, Phillip Isola, Amir Globerson, Michal Irani, al. Explaining in style: Training a gan to explain a classifier in stylespace. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 693-702, 2021.
11

---

Published as a conference paper at ICLR 2023
Piyawat Lertvitayakumjoo. Lucia Specia, and Francesca Toni. Find: Human-in Numan-in-the-looe the-loop debugging deep text classifiers. arXiv preprint arXiv:2010.04987, 2020.
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. arXiv preprint arXiv:2202.06652, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. 2020 International Joint Conference on Neural Networks (LJCNN), pp. 1-10. IEEE, 2020.
Anita Mahinpei. Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314, 2021.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXi::2105.04289 2021.
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, UIrich Paquet, and Vladimir Kramnik. Acquisition chess knowledge in alphazero. arXiv preprint arXiv:2111.09259, 2021.
SW Menzies, C Ingvar, and WH McCarthy. A sensitivity and specificity analysis of the surface microscopy features of invasive melanoma. Melanoma research, 6(1):55-62, 1996.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D) Manning. Fast model editing scale. arXiv preprint arXiv:2110.11309, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 8748-8763. PMIR, 2021.
Veronica Rotemberg, Nicholas Kurtansky, Brigid Betz-Stablein, Liam Caffery, Emmanouil Chousakos, Noel Codella, Mare Combalia, Stephen Dusza, Pascale Guitera, David Gutman, et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific data, 8(1):188, 2021.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Alek- sander Madry. Editing classifier by rewriting its prediction rules. Advances in Neural Information Processing Systems, 34, 2021.
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: learning overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11070-11078, 2020.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2020. URL https:
//openreview net. net. forum 2ot.ee/eEEoomm222ceeeee..EE Pid.teeeeXBtts
Matthew Sotoudeh and Aditya V Thakur. Provable repair of deep neural networks. In Proceedings of the 42nd ACM SIGPLAN International Conference Programming Language Design and Implementation, pp. 588-603 2021.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference artificial intelligence, 2017.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolie concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3619-3629, 2021.
12

---

Published as a conference paper at ICLR 2023
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 1000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 1(1):1-9, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Tong Wang, Jingyi Yang. Yunyi Li, and Boxiang Wang. Partially interpretable estimators (pic): black-box-refined interpretable machine learning. arXiv preprint arXiv:2105.02410, 2021.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-LLiang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations deep neural networks. Advances in Neural Information Processing Systems, 33:20554-20566 2020.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint 1'Xiv:2010.007477 2020.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.03363, 2020.
A TRAINING DETAILS
Hyperparameters: In all of our experiments FlasticNet sparsity ratio parameter
0.99.
We trained all our models on single NVIDIA-Titan Xp gpu. All of the models were trained for total number of 10 epochs. We tune the regularization strength on subset of the training set, that is kept as validation set. PCBMs are fitted using scikit-learn's 's SGDClassifier class, with 5000 maximum steps. Hybrid parts are trained with PyTorch, where we used Adam as the optimizer with 0.01 learning rate, with 0.01 L2 regularization on the residual classifier weights, and trained for 10 epochs.
Dataset Details:
1. Metashift: For all metashift experiments, we have class classification problems. We have 50 images for each class as the training set, and 50 images for each class in the test dataset, which is different from the training images. The regularization strength for Metashift experiments 0.002.
2. CIFAR: We the original training and test splits of CIFAR datasets. The regularization strength for CIFARIO and CIFAR100 is 2.0 We use linear probing to evaluate the original model
KN.
3. COCO-Stuff: We use the original training and test splits of COCO dataset. We sample 500 training and 250 test images from the dataset for each class, and we upsample the images whenever there are not enough images. Recognizing each of the 20 biased classes itepsingh2020don is treated entropy loss separately for 20 classes, and compute the mean average precision metric. The regularization strength for COCO-Stuff is 0.001. We evaluate the original model performance using linear probes with CLIP.
binary classification task where minimize binary cross
4. H0ml0k: We split 80% of the HAM10K dataset and use it as the training set, and use the remaining 20% as the test data. The regularization strength for Ham10k is 2.0
KN.-
5. ISIC: We use 2000 images for training (400 malignant, 1600 benign) and evaluate the model on a held-out set of 500 images (100 malignant, 400 benign). The regularization strength for ISIC 0.001 We evaluate the original model performance using linear probe.
KN.
6. CUB: We use the training and test splits provided in Koh et al. (Koh et al. 2020). The regularization strength for CIFAR 10 and CUB is 0.01 KN.
13

---

Published as a conference paper at ICLR 2023
Visual Concepts: We used Broden Visual concept bank for CIFAR and controlled editing experiments, CUB's training data for CUB concepts, and derm7pt dataset for dermatology concepts. For each of these, use 50 pairs of positive and negative images, and learn linear SVM. We use the vector normal to the decision boundary as the concept vector.
For concept learning, leveraged the ConceptNet hierarchy. For each classification task, we searched concept net for the class name and obtained concepts that have the following relation with the query concept: hasA, isA, partOf, HasProperty, MadeOf. We share the code for obtaining natural language concepts using ConceptNet. For each of these concepts, we obtain the text embedding using CLIP and use those as the concept vector.
Multimodal concepts:
B RESIDUAL COMPONENT INTERVENES ONLY WHEN NECESSARY
Does PCBM-h alter predictions made by the PCBM? We hypothesized that the residual component would intervene only when the concept bottleneck is not sufficient. To better understand the effect of residual predictor, we analyzed the prediction consistency between the two models for CIFARI 10 and CIFARI00, where we looked at the models with labeled concepts (i.e. 170 concepts). In Figure 4 and x-axis denotes the confidence of PCBM, the orange line gives the accuracy for samples with the given confidence, and the blue line gives the consistency between PCBM and PCBM-h for the same samples, i.c. whether they make the same prediction. In CIFARIO 10 and CIFARIO0, PCBM and PCBM-h consistency is high when the model confidence is high, and the model accuracy high. Consequently, PCBM-h changes the model prediction mostly when the PCBM prediction is likely a mistake, and the confidence low. In Figure 5 we show that all of the predictions changed by PCBM-h are to fix model mistakes. Further, Figure we show the PCBM confidence and the mean absolute deviation between the PCBM confidence and the PCBM-h confidence. We observe that PCBM-h has less effect on model confidence as the model gets more confident.
5
show that
CIFAR10
CIFAR100 00
1.0
1.0
0.8
0.8
Metric
Metric
0.6
0.6
0.4
PCBM-h Consistency
0.4
PCBM-h Consistency
Accuracy
Accuracy
0.2
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-1.0
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-1.0
PCBM Confidence
PCBM Confidence
Figure 4: Residual component intervenes mostly when the confidence is low. Here, we look at the consistency between PCBM and PCBM-h predictions (i.e. whether both models make the same prediction). Namely, at each confidence level for the PCBM, we report the accuracy and consistency with the PCBM-h predictions. Overall, we see that PCBM-h most likely to change the model prediction when the model is making mistake, and otherwise, predictions are consistent.
C COMPARISON TO CBM
In most of our experiments, there is clear way to CBM as benchmark, of the datasets have dense concept annotations except CUB. CUB, we run CBM as a baseline. Particularly, try the joint training strategy from Koh al. (Koh al. 2020), where it reported to
14

---

Published as a conference paper at ICLR 2023
CIFAR1O
CIFAR100
3000
# of PCBM mistakes
800
# of Mistakes fixed PCBM-h
2500
# of Predictions Changed
Metric
Metric
2000
1500
400
1000
200
500
0.0-0.2
0.2-0.4
0.4-0.6 6
0.6.0.8
0.8-1.0
0.0-0.2
0.2-0.4 4
0.4-0.6
0.6.0.a
0.8-1.0
PCBM Confidence
PCBM Confidence
Figure 5: Residual component intervenes only to fix mistakes. Here we show the number of mistakes, the number of predictions changed by PCBM-h, and the number of mistakes fixed by PCBM-h. We see that PCBM-h only changes the model predictions to fix model mistakes.
CIFAR10
CIFAR100
Mean PCBM Confidence
0.8
Mean Abs. Dev. in Confidence
Metric
0.6
Metric
0.6
0.4
0.4
0.2
0.2
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-1.0
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-1.0
PCBM Confidence
PCBM Confidence
Figure 6: Effect of the residual predictor the confidence. Here we show the PCBM confidence and mean absolute deviation between the PCBM confidence and the PCBM-h confidence. We observe that PCBM-h has less effect on model confidence as the model gets more confident.
give the best results in the CUB dataset. To make the comparison equal, we use the same set of concepts in both models, and use a linear predictor layer. Further, in both cases, we use the same frozen ResNet18 backbone. We searched over grid of learning rates from {0.01, 0.01, 0.1, 1.0) and À E {0.001, 0.01, 0.1, 1.0} where is the coefficient of the concept predictors in the joint training objective (see the CBM paper, Section 3), and the ElasticNet regularization strength in (0.001, 0.01, 0.1, 1.03. In Table 5 we report the model performance. Overall, we observe some benefit from using dense concept annotations over the entire training dataset, where CBMs achieve slightly better performance than PCBMs, and the original backbone. We note that this was at the cost of 1 12 concept annotations for each of the training samples.
We further analyze the behavior of CBM under different number of annotations. In Figure we train the CBM with varying number of annotations. CBMs require dense annotations, e.g. 11200 annotations would mean 1 11200/11 1122 100 images. On the x-axis, we give the number of annotations used to train the CBM. On the y-axis, we give the accuracy on the test set. We observe that CBMs require much larger amount of annotations achieve the accuracy PCBMs.
15

---

Published as a conference paper at ICLR 2023
Table Comparison CBM. We compare PCBM CBM in the CUB dataset. The accuracy over the test set is reported.
ResNet18(Backbollg ((Bakbooe))
PCBM
PCBM-h
CBM(Backbone Fixed)
CUB Accuracy
0.612
0.588
0.610
0.629
0.6
Accuracy
0.5
0.3
CBM
PCBM
0.2
11200
50176
188160
313600
# of Annotations
Figure 7: Effect of the number of annotations in CBM. Here train the CBM with varying number of annotations. CBMs require dense annotations, e.g. 11200 annotations would mean 11200 112 100 images. On the x-axis, we give the number of annotations used train the CBM. On the y-axis, we give the accuracy on the test set. We observe that CBMs require a much larger amount of annotations achieve the same accuracy PCBMs.
D CONTROLLED METASHIFT EXPERIMENTS FOR MODEL EDITING
For Metashift, we have 2 tasks. Both tasks are 5-class object recognition tasks, where in the first one classes are airplane, bed, car, cow, keyboard, and for the second one have beach, computer, motorcycle, stove, table. For each of these, we use ResNet18 pretrained on ImageNet as the backbone of the P-CBM, and then use 100 images per class to train the concept bottleneck. For all experiments, we use the Adam Optimizer with learning rate of 0.05, the regularization parameters 0.05, a 0.99. Similar to CIFAR experiments, we use the Broden Concept dataset. Below we give the entire set of results.
16

---

Published as a conference paper at ICLR 2023
Train
Test
Model
Original
Prune
Prune+NNrmmalize
Fine-Tune
bed(dog)
bed(cat)
P-CBM
0.760
0.760
0.760
0.920
bed(cat)
bed(dog)
P-CBM
0.680
0.700
0.720
0.940
table(dog)
table(cat)
P-CBM
0.520
0.540
0.620
0.760
table(cat)
table(dog)
P-CBM
0.660
0.700
0.740
0.760
table(books)
table(dog)
P-CBM
0.600
0.580
0.780
0.720
table(books)
table(cat)
P-CBM
0.620
0.680
0,800
0.820
car(dog)
car(cat)
P-CBM
0.718
0.718
0.744
0.949
car(cat)
car(dog)
P-CBM
0.620
0.760
0.840
0.840
cow(dog)
cow(cat)
P-CBM
0.778
0.750
0.778
0.944
(eyboard(dog)
keyboard(cat)
P-CBM
0.620
0.580
0.720
0.940
bed(dog)
bed(cat)
HP-CBM
0.760
0.760
0.780
0.900
bed(cat)
bed(dog)
HP-CBM
0.760
0.740
0.760
0.940
table(dog)
table(cat)
HP-CBM
0.600
0.620
0.640
0.780
table(cat)
table(dog)
HP-CBM
0.540
0.580
0.640
0.820
lable(books)
(able(dog)
HP-CBM
0.660
0.700
0,760
0.680
table(books)
table(cat)
HP-CBM
0.760
0.800
0.820
0.780
car(dog)
car(cat)
HP-CBM
0.795
0.769
0.795
0.974
car(cat)
car(dog)
HP-CBM
0.640
0.660
0.740
0.720
cow(dog)
cow(cat)
HP-CBM
0.639
0.639
0.639
0.917
keyboard(dog)
keyboard(cat)
HP-CBM
0.400
0.460
0.560
0.940
Table 6: Results of the Metashift editing experiments.
USER STUDY
E.
USER INTERFACE
Below, we provide screenshots of the initial launch page and the concluding summary page from the user study. For concept-selection interface, refer main paper.
17

---

Published as a conference paper at ICLR 2023
Background
Typical end-to-end deep learning models for image classification difficultto-interpree black boxes that map input pixels directly prediction. A Post-hoc Concept Bottleneck Model (PCBM) enhances interpretability by making output predictions using human-understandabll concepts. It first maps the image input "bottleneck" of concepts, that maps the concepts to predictions. A PCBM offers insight into what concepts the model "sees" in the input and what concepts it deems as "important" and "relevant" for making the prediction.
Instructions
We are giving you the power to trained PCBM! You will asked to perform edits 9 scenarios. In each scenario, model been trained for S-class classification task. 1 classes will designated class interest. You will shown 10 concepts our model relies on most predict this class of interest, i.e, 10 concepts with most positive weights.
Your job is remove concepts from the model that might hinder generalization. In other words, the goal of your edits to improve the prediction accuracy the class of interest in the real world For instance, might make sense zero out concepts that seem completely unrelated the class, You may also use other rationales to choose which concepts to remove.
Name
Start
Figure 8: Launch page for user study. Before starting the editing tasks, participants are shown
launch page containing brief background section on PCBMs and set of instructions.
Scenario 1/9
The model was trained to distinguish Beach, Bus, Airplane, Keyboard, and Bird.
Before removing concepts
After removing concepts
Largest Concept Weights Keyboard
Largest Concopt Weights Keyboard
spccebor
spocetar
15.74
12,0n
clans dark moin
kry
Tut
1.5
cockult 0.00
devioe 6:5
felime 3.30
6.00
for
Weight
Weight
Spurious
Unedited Accuracy
Unedited keyboard
Edited Accuracy
Edited keybooard
Class
You removed
Accuracy
Accuracy
('feline', 'sharp claws,, whisker,
keyboard cat
'claws; fur, 'giblet"]
11.38
0.75
0.52
0.81
0.84
Figure 9: Summary page for user study. After completing all scenarios, participants shown summary page that includes the user's choice of concepts, the accuracy of the unedited model, and the accuracy achieved by the edited model.
E.2 HUMAN-CUIDED PCBM EDITING: DETAILED PERFORMANCE RESULTS
In each of the 9 scenarios in the user study, the underlying classification task 5-class object recognition task, and one of the classes has spurious correlation in the training set. For instance, all
18

---

Published as a conference paper at ICLR 2023
training images for the class keyboard also contain a in the image. In Table 7 we list the classes in the underlying classification task and the spurious correlation for each scenario.
Table Classification tasks and spurious correlations in user study scenarios.
Classification Task
Spurious Correlation
airplane, bed, car, cow, keyboard beach, bus, airplane, keyboard, bird beach, car, airplane, bed, bird beach, motorcycle, airplane, couch, bird bus, painting, cat, computer, snowboard bus, pillow, cat, computer, snowboard bus, television, cat, computer, snowboard car, fork, table, bed, computer dog, car, airplane, couch, bird
bed(dog)
keyboard(cat)
bed(cat)
couch(cat)
painting(lamp)
pillow(clock)
television( fireplace)
(trk(tomato)
car(snow)
Below, we report the test accuracies for each the 9 scenarios our user study, both for the shifted class and for the overall classification task.
Table 8: Model accuracy of PCBM-h with CLIP concepts after editing (N=30) We report the test accuracy in the shifted class and the overall test accuracy for each individual scenario in the user study. Accuracy for the pruning strategies averaged error.
and is shown + standard
Scenario
Unedited
Random Prune
User Prune
Greedy Prune
Fine-tune
Shifted Class Test Accuracy
bed(dog)
0.720
0.651 + 0.023
0.832 0.008
0.838 0.008
0.940
keyboard(cat)
0.520
0.425 # 0.040
0.788 + 0.021
0.803 0.015
0.940
bed(cat)
0.700
0.663
0.789 0.006
4.008
0.860
couch(cat)
0.700
0.013
0.880
0.960
(aining(lampp)
0.640
0.617
0.006
0.059
0.840
pillow(clock)
0.800
0.776
0.015
0.005
0.820
television fireplace)
0.480
0.485 0.004
0.008
0.009
0.760
fork(tomato)
0.580
0.583 4.012
0.655 0.010
0.707 0.009
0.840
car(snow)
0.640
0.676 2 0.014
0.807 + 0.013
0.817 + 0.011
0.840
Overall Test Accuracy
bed(dog)
0.864
0.850 + 0.005
0.882 + 0.001
0.882 + 0.001
0.900
keyboard(cat)
0.764
0.741 0.008
0.811 0.004
0.814
0.860
bed(cat)
0.820
0.808 # 0.004
0.834 0.001
0.841
0.832
couch(cat)
0.840
0.839 + 0.005
0.851
0.002
0.904
mainting(lamp)
0.852
0.841 08410000 0.001
0.845
0.054
0.884
(cloow(clock)
0.888
0.882 + 0.002
0.001
0.001
0.888
television (fevisiio((freplacee)
0.816
0.817 0.001
0.827
0.832
0.856
(tommtoo
0.700
0.695 0.002
0.706 0.001
0.711 + 0.001
0.732
car(snow)
0.784
0.791 0.003
0.816 0.002
0.817 0.002
0.848
F ANALYSIS ON THE COCO-STURE BIASES
Singh et al (Singh et al. 2020 identifies co-occurence biases in the COCO-Stuff dataset, where 20 categories frequently co-occur with other identified categories. In Table 10 reader can find the concepts. taken from Singh al. In Table TO report the Top-5 concepts for the PCBM trained with CLIP concepts. In the Biased Context column, we give the category that co-occurs frequently with the given context. In PCBM Top-5 Concepts column, we give the concepts that have the highest
19

---

Published as a conference paper at ICLR 2023
Table Model aceuracy of PCBM with CLIP concepts after editing (N=30) We report the test accuracy in the shifted class and the overall test accuracy for each individual scenario in the user study. Accuracy for the pruning strategies averaged over users and is shown as mean + standard error.
Scenario
Unedited
Random Prune
User Prune
Greedy Prune
Fine-tune
Shifted Class Test Accuracy
bed(dog)
0.720
0.647 0.024
0,830 + 0.008
0.832 0.008
0.900
keyboard(cat)
0.520
0.412 0.040
0.787 0.021
0.799
0.016
0.980
bed(cat)
0.700
0.677 # 0.017
0.813 0.008
0.804 0.007
0.820
couch(cat)
0.680
0.701 = 0.034
0.751
0.882 0.012
0.920
Sainting(lapp)
0.580
0.579
0.607
0.635
0.520
pillow(clock)
0.740
0.811 0.008
0.009
0.720
television(fireplle
0.420
0.428 0.004
0,459 + 0.006
0,483 0.007
0.880
fork(tomato)
0.600
0.581 0.012
0.642 0.007
0.685 + 0.006
0.960
car(snow)
0.620
0.651 # 0.014
0.777 0.011
0.737 0.008 0.008
0.720
Overall Test Accuracy
bed(dog)
0.860
0.878 0.001
0.877 0.001
0.880
keyboard(cat)
0.748
0.727 0.008
0,802 + 0.004
0.800
bed(cat)
0.804
0.801 # 0.004
0.830 0.002
0.829
0.668
couch(cat)
0,828
0.826 0.006
0,841 0.002
0.000
0.892
painting(lamp)
0.836
0.835
0.041
0.001
0.840
(illow(clock)
0.876
#.002
0.890
0.888 1.001
0.848
(fevissoo((irppaae))
0.800
0.802 1.000
0.808
0.813
0.812
fork(tomato)
0.704
0.699 + 0.002
0.710
0.705 + 0.002
0.720
car(snow)
0.768
0.776 0.003
0.799 1 0.002
0.792 0.002
0.732
Biased
Biased Context
PCBM Top-5 Concepts
Category
cup
dining table
table, coaster brake, crockery, dining table. column
handbag
person
bag, platform, woman, person, toiletry
apple
fruit
apple tree, fruit, edible fruit, citrus fruit, produce
car
road
traffic circle. intersection, car window, car mirror, trunk
bus road
public transport, transportation, tube, carriages, traffic circle
potted vase
tall plant, patio, vase, plant, plant organ
plant
spoon
bowl
eating utensil, utensil, cutlery, crockery, flatware
microwave
oven
kitchen, washing machine, kitchen appliance, oven, countertop
keyboard
mouse
computer, computer system, portable computer, computer brand, mouse clock face, timepiece, the dial, plank, chimney
clock
building-other
hair drier
towel
shower, bathroom, toiletry, plumbing fixture, toilet
skatchoard
person
paved surface, person, outsole, instep, footwear
Table 10: COCO-Stuff Concepts
weight for the given category. Looking PCBM categories, we can see that we can identify the biased contexts. For instance, cup category has table, dining table, apple has fruit, fresh vegetables, car has truffic circle, intersection in the concepts identified as important, which are parallel to biased context. In other cases, our model surfaced different potential co-occurence biases, such as chimney for clock, and toilet for hair drier. One important limitation for this analysis is that the desired concepts should exist in the concept bank. To further improve the pipeline, automatic concept discovery approaches shall be fruitful research direction (Ghorbami al. 2019 Ych et al. 2020).
20


Reviews:
[{"review_id": "jz87Hap2yS6", "reviewer": "Reviewer_cGSM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for creating a concept bottleneck model from any given model. The authors claim that the proposed approach does not sacrifice the performance of the model, and retains the interpretability benefits of concept bottleneck models along with easy model editing.", "summary_of_the_review": "It is an interesting paper on an important and timely problem, generating post-hoc concept-based explanations for trained neural networks. It proposes a novel approach that extends upon several well-known and widely used methods. The experimentation seems thorough and conclusive, and the methods presented seem flexible and useful in a wide array of problems. However, the lack of discussion around the causal validity of the generated concepts is missing.\n", "strength_and_weaknesses": "Strengths:\n\nIt is an extremely well-written paper on an important and timely problem, generating post-hoc concept-based explanations for trained neural networks. It clearly describes the limitations of existing methods and builds upon them in an elegant way, leading to a superior solution. The experiments are thorough and seem convincing, clearly demonstrating the usefulness and flexibility of the proposed method.\n\nWeaknesses:\n\nWhile I am generally supportive of the paper and its claims, I do find the lack of discussion around the causal validity of the generated concepts missing. There have been many studies in the literature highlighting the causal nature of post-hoc model explanations, which the authors do not address. Please see missing references at the next section (Clarity, Quality, Novelty And Reproducibility).\n\n\n\nI understand that this is not the goal of this paper, but addressing the fact that the generated concepts and their estimated effect might not faithfully reflect their true causal effect on the trained model sounds important to me and should be a part of this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RdaMevj0sc", "reviewer": "Reviewer_uQCo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new class of interpretable neural network classifiers, denoted Post-hoc Concept Based Models (PCBMs).  PCBMs assume a concept vocabulary is given, fit linear representations of the various concepts in the embedding space of the target model using auxiliary positive and negative examples of each concept, then acquire an interpretable classifier on top of these embeddings (or rather the projection of the input on them).  The key advantage of PCBMs over regular CBMs is that the target data set needs not be densely annotated.  The authors also introduce a partially interpretable version of PCBMs (PCBM-h's) that upgrade performance by also making use of possibly uninterpretable features.  Both classes of models are compared on several data sets in terms of accuracy (against a black-box baseline) and amount of supervision required.", "summary_of_the_review": "Simple but useful new class of models that overcome major issue with prior work, with a couple of key issues that can be easily fixed.", "strength_and_weaknesses": "PROS:\n\n- PCBMs mix existing techniques into a novel recipe in a sensible manner.\n\n- PCBMs overcome one of the main limitations of CBMs, namely the need for dense annotations.\n\n- The manipulation experiment adds some flavour to the main message.\n\n- The text is very well written and easy to follow.\n\nCONS:\n\n- Empirical results do not entirely match the claims in the text.\n\n- Missing comparison with non-CBM baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JGxD0YzJqG", "reviewer": "Reviewer_n4tz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present Post-hoc Concept Bottleneck Models (PCBMs) as a method for converting any model into a CBM with its interpretability benefits while retaining the original model performance ( by introducing a residual layer that circumvents the bottleneck and helps with the concept set isn’t rich enough ).   When there isn’t concept supervision for a task, they use CLIP and ConceptNet as a method to learn concepts based on task classes.  They also show that PCBMs can be leveraged to perform global model interventions, specifically to reduce spurious correlations in a contrived dataset.  Through a model-editing user study, they show that editing PCBMs via concept-level feedback can provide significant performance gains without using any data from a target domain or model retraining for tasks in new data distributions.  \n", "summary_of_the_review": "The authors demonstrate the utility of their method for constructing CBMs in a post-hoc fashion using Concept net & CLIP if need be along with their model variant which uses a residual layer PCBM-h to get near same accuracy as the original model  ( at the expense of intepretability, but still a novel and simple addition to the architecture that is quite useful.\n\nThe paper mostly clearly written, though it took me a bit to understand certain parts such as when ConceptNet and Clip were used ( ie, when labeled concept examples don’t exist ).  The experiments well done and model editing was explained\nwell for the use case of datashift ( which is relatively simplified ), but still has utility.  The human user study was well done and very convincing.  It’d be interesting to explain the differences between the fine-tune oracle and user pruned concept weights in that section to see if fine tuning is actually picking up spurious correlations which are useful for task accuracy or as a way to suggest concepts for a human to consider?", "strength_and_weaknesses": "**Strength:**  \nThe authors demonstrate the utility of their method for constructing CBMs in a post-hoc fashion using Concept net & CLIP if need be along with their model variant which uses a residual layer PCBM-h to get near same accuracy as the original model  ( at the expense of interpretability, but still a novel and simple addition to the architecture that is quite useful.\n\n**Weaknesses:**  \nThe paper had a few areas that could have been clearer (1) how to select negatives when given concept supervision and (2) it took me until after reading section 3 to understand when Visual or Multimodal concepts are used ( maybe that could be added to Figure 1?) You all mention it in your limitations section, but the use of Concept Net will not translate well to domain specific tasks or tasks with classes not in ConceptNet.\n\nFinally on Page 6 before Figure 2, you mention PCBM-h and PCBM provide the same concept interpretations, but this is not exactly the same interpretation since the final prediction in PCBM-h is no longer a linear combination of concept weights, but rather concept weights and residual weights so  although the concept bottleneck learns the  same weights, the overall model behavior is different and the concepts influence in each model differ.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%