[Re] On the Reproducibility of Post-Hoc Concept Bottleneck Models
Authors: Nesta Midavaine
Gregory Hok Tjoan Go
Diego Canez
Ioana Simion
Satchit Chatterji
Venue: Accepted by TMLR ()

Abstract: To obtain state-of-the-art performance, many deeper artificial intelligence models sacrifice human explainability in their decision-making. One solution proposed for achieving top performance and retaining explainability is the Post-Hoc Concept Bottleneck Model (PCBM) (Yuksekgonul et al., 2023), which can convert the embeddings of any deep neural network into a set of human-interpretable concept weights. In this work, we reproduce and expand upon the findings of Yuksekgonul et al. (2023). Our results show that while most of the authors’ claims and results hold, some of the results they obtained could not be sufficiently replicated. Specifically, the claims relating to PCBM performance preservation and its non-requirement of labeled concept datasets were generally reproduced, whereas the one claiming its model editing capabilities was not. Beyond these results, our contributions to their work include evidence that PCBMs may work for audio classification problems, verification of the interpretability of their methods, and updates to their code for missing implementations. The code for our implementations can be found at https://github.com/dgcnz/FACT.

Published in Transactions on Machine Learning Research (05/2024)
[Re] On the Reproducibility of Post-Hoc Concept Bottleneck Models
Nesta Midavaine', Gregory Hok Tjoan Go', Diego Cánez IIdefonso,, Ioana Simion", Satchit. Chatterji
midareine, gregorg.go, diego. cwnez. ioana, simion, satchit. chatterj Estudent. wea.nl Graduate School of Informatics University of Amsteriam
Reviewed on OpenReview: htips: // openreview. net/ normf? id- 8UfhCZJOV7
Abstract
To obtain state-of-the-art performance, many deeper artificial intelligence models sacrifice human explainability in their decision-making. One solution proposed for achieving top per- formance and retaining explainability the Post-Hoc Concept Bottleneck Model (PCBM) (Yuksekgomul et al. 2023 which convert the embeddings of any deep neural network into set of cumann-ineerprees
concept weights. In this work, we reproduce and expand upon the findings of Yuksekgonul et al. (202)), showing that while their claims and results do generally hold, some of them could not be sufficiently replicated. Specifically, the claims labeled concept
relating to PCBM performance preservation and its non-requirement datasels were generally reproduced, whereas the one claiming its model editing capabili- ties was not. Beyond these results, our contributions their work include evidence that PCBMs may work for audio classification problems, verification of the interpretability of their methods, and updates to their code for missing implementations The code for our implementations can be found in https: http://ggttubb.oom /dgen//gn///// digen//FACT /ACT
1 Introduction
There an increasing demand within society to make artificially intelligent systems more explainable due to concerns of algorithmic bias (Pessach & Shimueli, 2023 Suresh & Guttag, 2021). One method of doing so involves Concept Bottleneck Models (CBMs), which train model in end-to-end fashion via concept prediction (Koh et al. 2020). These concepts are then used to predict the label attached to the data point, thus improving the model explainability and allowing for the detection of concept prediction-related mistakes through model-based interprelability. However, despite their benefits, CBMs have been widely criticized for being unable to deliver on its goals of interpretability, predictability, and intervenability (Raman et al. 2024 Havasi al.] 2022 Furby et al.] [2044] Margeloiu et al. 2021). For example, Margeloiu et al. (2021) have found that when the concept predictor and classilier of a CBM are trained jointly, extra information about the targets is learned in the concept predictor. As such, interpretability and intervenability become lost
other factors beyond the concepts influence the decisions made.
By combining model-based and post-hoc paradigms, Yuksekgonul et al. (0233 proposed Post-hoc Concept Bottleneck Models (PCBMs). Through these, they address the CBM's limitations of data usage, decreased performance, and lack of model editing capabilities. The limitation of data usage is tackled by the imple- mentation of multimodal models obtain concepts automatically and decreased performance is addressed through the introduction of residual modeling step.
This paper attempts to reproduce and extend upon their main findings through the following:
Equaa contribution.
SSpperisor..

---

Published in Transactions on Machine Learning Research (05/2024)
Reproducing their results using their provided codebase. This was done to identify which of the results supporting their claims can be reproduced, and the resource costs involved (i.e., computational cost and development effort).
Reproducing their user study. This relates to their claim stating that human-guided editing is fast and improves classification accuracy, which was supported using survey with the participants being machine learning practitioners and researchers. We investigate this by replicating said survey with different group of participants who come from a similar demographic to that of the original.
Verifying the interpretability of their approach. The original paper assesses the interpretabil- ity of PCBMs by highlighting crucial concepts in specific classes for some of the used datasets. It also argues that the performance increase from model editing stems from the PCBM's interpretability, To inspect this, we examine two directions: First, we analyze what happens when the original au- thors' concept collection process is replaced with using meaningless concepts, which we then evaluate based on interprctability and performance. This serves as a baseline for our other interpretability experiments and also for the performance of PCBMs as answers the question: How much of the performance retained by the bottleneck can be attributed to the usefulness of concepts?
Furthermore, we evaluate the interpretability of PCBMs by examining if the concepts used by the model correctly correspond to objects in the input space. Similar to the work of Margeloiu et al. (2021) crilicizing CBMs for not being truly interpretable, we utilize salieney maps (Simonyan et al., 2014 Smilkov et al., (017) to visualize the correspondence between the concepts used and parts of the image. Additionally, we conduct another experiment to show the correspondence between concepts and the input space more generally for entire datasets instead of single images.
Extending their work through an additional experiment. This experiment examines how the original implementation can extend to another modality, namely, audio. This was performed using AudioCLIP (Guzhov et al. 2022), which is multinodal model we can utilize to automatically obtain concepts for audio. It was utilized in the same way (Yuksekgonul et al. 2023) did for images.
Improving the original code by implementing additional scripts. This includes, for exam- ple, bash and python scripts which allow for easier reproduction, and master Jupyter Notebooks containing essentially all the setup and code needed to run the experiments.
After performing the above, we successfully verified their claims on model performance and the usability of CLIP concepts, though encountered issues replicating the claim on PCBMs enabling global model editing, In addition, we discovered limitations in the meaningfulness of concepts and their correspondence to objects in imput images, and that PCBMs could be applied to imputs that are not in the form of image data.
As such, our replication efforts yielded mixed results due to potential implementation mistakes and incom- plete experiment setup details. While some performance similarities were observed, inconsistencies emerged, particularly with the COCO-Stuff and SIIM-ISIC datasets. While PCBM-h's exhibited similar performance to the original model, the PCBMs showed mixed results. Also, our interpretability analysis revealed that despite performing well, random concepts lack interprotability. There were also limitations interpretability due to concept feature values not being solely derived from the concept's 's presence in the image.
2 Scope of Reproducibility
PCBMs were introduced to imbue interpretability into any neural network while preserving both its perfor- mance and flexibility. They build upon the idea of concept analysis (Kim et al. 2018), which aims to com- prehend how such networks generate and leverage high-level, numan-understandablle features. This inspired the development of Concept Activation Vectors (CAVs), forming the foundation of PCBMs. Yuksekgonul et al. 2023) proposed three claims in their paper which we aim to investigate:
Claim 1: PCBMs achieve comparable performance to the original model. The authors claim in their paper that after applying the Post-Hoc Concept Bottlenecks to various models, clas- sification performance was comparable and in some cases even identical to the original baselines in
2

---

Published in Transactions on Machine Learning Research (05/2024)
all scenarios. It must be noted, however, that this claim only holds for their hybrid implementation which they call "PCBM-h." An explanation for it can be found in subsection 3.1
2. Claim 2: PCBMs do not require labeled concept datascts. Another claim made is that multimodal representations can be used in the absence of labeled data for concept generation. The original authors demonstrated this using CLIP for concept generation and showed that it improved the performance of the PCBM due to the concepts gencrated being more expressive.
3. Claim 3: PCBMs allow for global model editing. The claim states that by simply adjusting the concept weights, the PCBMs can be adapted to new distributions. In addition, it is shown through a user study that pruning the least human-logical concepts always results in improved classification performance.
3 Methodology
3.1 Model descriptions
The original paper introduces a post-hoc method aimed at enhancing the interpretability of neural networks. It demonstrates this approach using three different pre-trained backbone models: ResNet18 (He et al. 2016) CLIP-ResNet50 (Radford et al., 2021), and Inception (Szegedy et al. 2015). These models are freely available online, with the weights specifically for the Inception model trained on HAM10000 accessible from Daneshjou et al. )022a). All our experiments involve classification tasks, focusing on the performance PCBM and PCBM-h. As a baseline, we employ a linear probe at the output layer for model-dataset combinations lacking a classification head. This involves using logistic regression with l7 regularization.
To convert the backbone model into PCBM, the original authors projected the final embeddings of the backbone model onto concept subspace. This projection is then used to train an interpretable linear clas- sifier. The concept subspace is defined using concept library which can be denoted as I = {i1, $2....,4y, iN. } where N₂ denotes the number of concepts. Each concept can be learned directly from the data or selected
by a domain expert (Ghorbani et al. 2019 Yeh et al. 2020
The original authors used two different methods learn concept representations, with the first being through CAVs (Kim et al. 2018). To obtain a CAV e; for each concept i, we need two image embedding sets Pa and N. The former comprises the embeddings of N, 50 images containing the concept, which we call the positive image examples Tp- Meanwhile, the latter comprises the embeddings of N, = 50 random images not containing said concept which we refer to as the negative image examples In- This gives us P; = (f(mm), f returns the CAV via its vector normal to the linear classification boundary. Note that obtaining these CAVs would require a densely annotated dataset with positive examples for each concept.
and N; = (f(Em)))...,
)}. which we use to train a linear SVM that
Meanwhile, the second approach involves utilizing the image and text encoders of multimodal model (i.e., CLIP (Radford et al. 2021)) to generate embeddings for each modality. These encoders map to shared embedding space, meaning that we can use the vector representations of the concept's natural language descriptions as the concept vectors. As such, we have for the concept "stripes" that clent = freze (stripps) The vector representations of concepts are collected in projection matrix C E RR Nexd such that each row represents concept vector of dimensionality d.
After we obtain C, the final embeddings of the backbone are projected onto the concept subspace. This matrix is then used to compute fc(x) f Ne in way such that p(l (x) |e..1]
f(z) A way
to interpret (00) (x) is as the concept feature value of concept i for image I, which we further examine in subsection 3.4. This means the concept feature value says something about the correspondence wetween concept and image I. Thus, the vector fc(x) can be used as the feature matrix of some interpretable model. Using this function, the original authors minimized the following loss for the computation:
min E. (
N.K 2(g).
(1)
3

---

Published in Transactions on Machine Learning Research (05/2024)
where & represents the cross-entropy loss and the function g represents sparse linear model as determined by the authors, denoted as g(f((z)) = Sc(r)W + b. Here W is the vector of concept weights, these concept weights give the importance of a concept for a given class. This means the concept weights are determined on the class lovel, while concept feature values are different for each image. Additionally D(g) is the (multiclass) elastic-net penalty. Defined as Q(g) = ail Wil++ + a with a as a regularization factor. Furthermore, if g is a classification problem, we apply softmax to its output. Moreover, K denotes the number of classes in the classification problem and À gives the regularization strength.
There are cases where the concept set is not expressive enough for the PCBMs alone to recover the original model's performance. To solve this, the original authors reintroduce the embeddings to "fit the residuals." They thus introduce Hybrid Post-Hoc CBMs (PCBM-h) to solve the following optimization problem:
mén
+ +r( ((f())),))
(2)
where r : R + Y represents the residual predictor mapping embeddings to the classes Y in our classification task. Also, minimizing the loss in Equation 2 is a sequential process where both the concept bank and the interpretable predictor g remain fixed, while a linear layer is employed to fit the residuals.
For the claim regarding model editing, Yuksekgonul et al. (2023) again use both PCBM and PCBM-h, with the model editing performed on the sparse linear model g. Both the user study and the controlled experiment use the Net50 backbone model from Radford el al. 2021
3.2 Datasets
In total, the original authors used seven different datasets for experimentation, either to evaluate the per- formance of PCBMs across different. domains, the quality of generated CLIP concepts, or the results of global model editing. All datasets used for binary classification were evaluated using the Area-Under-Cuu (AUC), the multi-class binary classification-based COCO-Stuff using the mAP, and the rest using accuracy. An overview of each dataset and its purpose can be found in Table 8
For COCO-Stuff and SH1M-ISIC, we followed the original paper to create subsets for each to reduce the required disk space for experimentation! []] The specifications for how they were created can be found in our repository. Meanwhile, for the model editing experiments and the survey, multiple datasets were generated using Metashift with the Visual Genome dataset
The concepts used for evaluating the performance across different domains were taken from 3 different datasets. For CIFAR-10, CIFAR-100, and COCO-Stuff, the concept set was taken from BRODEN (Fong & Vedaldi 2018), which contains visual concepts including for objects, settings, and image qualitics, among others. Meanwhile, the version of the CUB dataset used from Koh al. (2020) is annotated with pre- processed bird-related concepts, and the medical concepts for the HAM10000 and SITM-ISIC originated from the Derm7pt dataset (Kawahara et al. 2019).
For the experiments that evaluated the use of -ele-generated concepts, ConceptNet was employed to gen- erate the utilized concepts (Speer et al., 2017). As an open knowledge graph, it can be used to find concepts with particular relations for any given query concept, such as "wheel" and "engine" for "car." Similar to the original authors, only five relations sets were used build the concept space for each class, which are the "hasA," " "isA," "partOr," "HasProperty," " and "MadeOr" relations.
3.3 Hyperparameters
For a comparable replication, we used the same hyperparameters specified in the original paper whenever they were. This was the case for everything apart from the regularization parameters Crom and Xagisticc regreessonn Crem is used by the SVM for CAV computation. The open-source reposilory supplies the majority of the necessary
code, including an example grid for fine-tuning C' values, which is the following: [0.001, 0.01, 0.1, 1.0, 10.0). Meanwhile, X logistic
regression is employed when investigating the original models for CIFAR10, CIFAR100,
The trimmed-down datasets can he found here: (OCOO-Stfff STIM-ISIC
2he generated datasets can be found here: Model editing Survey
4

---

Published in Transactions on Machine Learning Research (05/2024)
and COCO-Stuff. The original model is RetP- for these three datasets, thus we determine the hyperparameter in the same way utilized by Radford et al. (2021). As such, we conduct a hyperaarameter sweep on validation sets over a range from 10 to 10,, with 96 logarithmically spaced steps.
A hyperparameter warranting some attention is the regularization strength A. It determines the weight decay for the parameters of our sparse linear model g as defined in Subsection 3.1 influencing the interpretability of the model and thus the PCBM. Appendix A of the original paper specifies the values used per dataset, tuned on a validation set, but omits the metric used for tuning. Additionally, the original code notes trade-off between interpretability and accuracy for this parameter, which is discussed in subsection 4.1
3.4 Experimental setup and code
To test the first two main claims outlined in section 2 we perform reproductions of the original paper's results for CAVs and CLIP concepts using the authors' code repository as well as the same datasets, parameters, backbone models, and number of training epochs used for the PCBMs outlined in their paper. An overview of the backbones and datasets used can be found in Appendix A.
In addition, we test the third claim by evaluating their techniques for doing the modcl-cditing experiments, using an unedited PCBM as a baseline. As such, we replicate their methodology by utilizing 6 scenarios where we artificially introduce a spurious correlation between class and a concept. For example, in one scenario we use dataset where all images of bed contain dogs for training and test the resulting model on a dataset where they contain cats instead. A description of these scenarios can be found in Appendix D
The first technique we evaluate is called "pruning," " where we set the corresponding weights for "bed" and "dog" to zero in the PCBM layer following the above example, ideally resulting in the model assigning less importance to "dog" when predicting "bod." Moreover, we evaluate "pruning with normalization," " which is pruning combined with a re-scaling of the weight vector corresponding to the affected class, thus making the new LI norm match the previous one. We evaluate these methods alongside "fine-tuning," which involves first training PCBM on the training domain, then fine-tuning in the test domain, and then testing on a held-out set of the test domain.
Another way we evaluate their third claim is through a survey. Here, we use a datasct similar to the one mentioned previously, with the primary distinction being the absence of spurious correlations in the test class. For example, we use images of beds containing dogs for training and then only use bed images without dogs during testing. In addition, we reproduce the survey in comparable setting to the one presented in the original paper. Using a variant of the base PCBM, we identify the top 10 ranked concept weights for each class of interest and guide the users through the model editing process. Most participants (94.11%) were machine learning students or practitioners and had a median age of 24, alongside a male-to-femall ratio of 76.2% to 23.8%. Additionally, informed consent was obtained from all participants.
We follow the original baselines to obtain an accurate benchmark: random pruning (randomly select a subset of the top ten concepts to prune, matching the number of concepts pruned by the user for fair comparison) and greedy pruning (greedily select from the top 10 concepts that improve model accuracy when pruned, again matching the number of concepts pruned by users). Further details of the setup and survey questions can be found in Appendix F
3.4.1 Assumptions made
Unfortunately, due to missing implementations and details, we had to update the repository to include additional components such as dataset installation scripts, code for the missing implementations, and an environment file. Moreover, we had to make several assumptions in our experimentation:
1. For the COCO-Stuff experiments, binary cross-entropy loss minimization was approached as 20 binary classifications. Performance was averaged across runs, treating the target class positive and the remaining classes as negative.
2. For the SIMM-ISIC experiments, we implemented our data selection method based on the limited details provided by the authors. These details state that they utilized 2000 images (400 malignant,
5

---

Published in Transactions on Machine Learning Research (05/2024)
1600 benign) for training and 500 images (100 malignant, 400 benign) for model evaluation. held-out set of 500 images (100 malignant, 400 benign).
3. For the global model editing experiments, design flexibility existed for the dataset, training set size, class names, optimizers, and hyperparamoterss due conflicting information. Examples of this are: A 170x5 0.05 vs 170x3 0.002, CLIP ResNet.50 vs ResNet and Adam SGD Optimizer. After testing many configurations, decided to proceed using À = 1.7, an LI ratio of a 0.99, BRODEN concepts, a CLIP-ResNet50 encoder, and the SGD optimizer.
A
4. For the user study experiments, insights from the Metashift study guided us in selecting hyperpa- rameters and matching the reported weight magnitudes. We experimented with the regularization and settled on X 0.002, which made our results consistent with the original. Also, the architecture, which was built existing models, now included adjustments and additional implementation of the three pruning methodologies.
3.5 Additional experiments
To further assess the efficacy of PCBMs, we explore three research directions. The first investigates PCBMs' interpretability and performance using randomly generated concept vectors. Meanwhile, the second involves the interpretability of the concepts used PCBMs. Lastly, the third direction aims to verify the original authors' claim that any neural network can be converted into a PCBM by testing for audio classification.
3.5.1 Random projection
Random projections, known for their desirable dimensionality reduction properties, preserve input data distance and similarity (Dasgupta) 2000 Bingham & Mannila 2001 Cannings 2021). For this experiment, we substitute the embedding matrix C with a random one, where each i'th row is a normally distributed and normalized vector. In other words, we replace the original concept matrix, obtained by the original author's methods, with a random and meaningless concept matrix. The exact details of this random projection experiment are as follows. We train PCBM and PCBM-h on CIFARIO, CIFAR100, COCO-StufT, and HAM10000, For the first three datasets, our new random concept matrix has the same dimensionality as the CLIP concept matrix. Mcanwhile, for HAM10000, we have eight random concepts, similar to the number of concepts used in the original paper. The results of this experiment will determine whether meaningful concepts are important for only performance, interpretability, or both. Additionally, this experiment gives a baseline on the performance PCBMs.
3.5.2 Object- correspondence
We examine whether the concepts used in PCBMs correctly correspond to objects in the input space. To see what this mcans in practice, we give the following general interpretation of the interpretability of PCBMs. When we refer to the PCBM as interpretable, we implicitly rely on the following two assumptions:
1. The concept weight of concept i for class k is high when i is important for correctly classifying k. The concept feature value of concept i for image j, reflects ONLY the visibility of i in j.
The first assumption immediately holds for the used class of interpretable models. However, the second one is less trivial. Assumption two related to specifically the interpretability of the concepts used. In practice, it mcans that when we have an image of a bicycle the concept fcature value for the concept "green" " should only be high when the bicycle is green. Thus issues arise when representations of different concepts become entangled since they will be projected onto the image in similar way.
Kim et al. 20188) experimentally show that higher-ranked images represent the concept better when ranking images based on the concept. feature value. However, they do use a different projection method called the TCAV-score, meaning their findings do not directly carry over to our scenario. Also, Kim et al. (2028) show that CAVs are sensitive to the chosen negative and positive samples, leading to unintended entanglement of concept representations. They find that this happens when the positive images for a concept almost always include different concept, such how images of a bicycle frame will almost always include handlebar.
6

---

Published in Transactions on Machine Learning Research (05/2024)
For CLIP concepts, some potential issues exist relating to assumption two. Firstly, CLIP is trained for single- label classification, whereas we are trying to identify multiple parts of an object in an image. Secondly, a type of entanglement has also been found in CLIP concept representations which involves it treating images as bags of concepts, where any one concopt from the bag can be used to explain an entire part of the image (Tang et al. 2023 Lewis et al. 2024p. Also, Taan et al. (0023) specifically discovered that there exists a Concept Association Bias (CAB) within CLIP between objects and their attributes, showing that this holds for part-whole relationships and colour. As such, the textual representation of only one of these two can be used to explain the occurrence of both in the image.
Our first method to test object-concept correspondence is by testing assumption two for PCBMs. We do so by evaluating the concept feature values fc(#) for the positive and negative concept images in BRODEN and CUB. We do this for CAVs, CLIP concepts, and random concept vectors, with the latter having the same shape as the CLIP concepts. For each method, we have a set of concepts I = {i1, Ne is the number of concepts we obtained using this method. We have two image sets for each concept P. (Fpp....., pppp and N; (#in1...... Ingo each containing 50 positive and 50 negative images for concept i respectively. We report the following statistic for each method:
).
gap
1
Σ VET
1
Σ Sc(pos) PoBEP.
1
Σ sc(neg)
(3)
7
50
50
negEN.
This mcans that if this assumption holds, we should see generally higher concept fcature values for positive concept images than for negative ones, alongside a positive average gap when using the interpretable concept vectors (CAVs and CLIP concepts). Meanwhile, for random concepts, we should not see difference.
Meanwhile, our second method involves using saliency maps (Simonyan et al. 2014 Smilkov et al., 2017] to see exactly where in the image our model "sees" the concept. We construct saliency maps from the concept projection to the input. Due to this, our saliency maps visualize the gradient of the concept feature value concerning the input. Below we define a saliency map for concept e concerning the input æ as Mc(x):
(4) Additionally, we employ the SmoothGrad implementation of saliency maps (Smikkov et al. 2017 which enhances these maps by averaging gradients over multiple forward passes while adding noise to the input æ. Consequently, our saliency maps depict the sensitivity of concept feature values to input changes, with brighter regions indicating higher importance.
Mc(#) = /
We investigate if the most crucial parts of an image for given concept visually align with it. For instance, the saliency map for the concept "green" should highlight the green parts of bicycle in an image. To conduct this experiment, we utilize CIFAR100 images and the CLIP backbone, meaning the model from which we compute gradients comprises the CLIP ResNet50 backbone followed by the concept projection described in subsection 3.1 where the concept matrix C' contains the concept vector of a single concept. We generate saliency maps for ten different concepts, with five using CAV concept vectors and the remaining using CLIP ones. For both methods, we select the four concepts with the highest concept weight for the class, along with one hand-chosen concept corresponding to co-occurring object in the image. This hand-chosen concept aids in examining the less fine-grained behavior of concept vectors.
3.5.3 Audio classification
The third experiment aims to verify the original authors' claim that any neural network can be converted into PCBM. We therefore evaluate its performance in audio classification, a domain that has yet been untested. AudioCLIP was utilized for this purpose, which is implementation of CLIP that has been integrated with compatible and well-performing audio encoder (Guzhov et al.] 2022). This encoder is an ESResNeXt instance that has been jointly trained alongside the other CLIP encoders (Guzhov et al., 2021
We assess the audio model performance on ESC-50 (Slamon et al.) 2014) and UrbanSound8K (Piczak 2015). Similar to the models used in the main reproduction study, we create a bascline for AndioCLIIPs performance via linear probing. To obtain the CLIP-based concept matrix, we collect textual concepts and feed them through the AudioCL.IP text encoder, similar to how was done with standard CL.I..
7

---

Published in Transactions on Machine Learning Research (05/2024)
We derive textual concepts as follows: From ESC-50, ConceptNet generates 146 concepts from ground-truth labels, following the original authors' method. UrbanSound8K provides 31 concepts from non-terminal nodes with depths 1 to 4 in its taxonomy, excluding terminal nodes with data point labels. The final set of concepts is derived from the AudioSet ontology, totalling 610 concepts after excluding coostaanel-containinn nodes. Thus, the concept. matrix has a total of 787 concept. vectors.
To obtain the CAV-based concept matrix, we perform the following weakly supervised approach: For each label in a dataset, we generated its associated concepts using ConceptNect Then, we gathered 50 audio clips for each concept whose labels were associated with the given concept and treated them as positive examples, using a similar process for negative examples. Afterwards, we procoeded to train a SVM to separate positive and negative examples and took the vector normal to the linear boundary as the concept's vector. This approach lead to 31 concepts for UrbanSoundSK and 171 for ESC-50.
We evaluate PCBM and PCBM-h on various datasots using CAV and CLIP-basod concept matrices. The AudioCLIP audio head's final embedding dimension has size of 2048, making the residual classifier of PCBM-h function 2015 + N Nelases. For UrbanSoundKK,
set ACLIP = 2 10 and λслv = 30 Similarly, for ESC-50, ACLP = 2 10-19 and dcAv = 171x50 Early stopping was applied to the ESC-50 models due to observed overfitting.
3.6 Computational requirements
All of our experiments were conducted using Google Colab in region "curope-west4," " which has carbon efficiency of 0.57 kgCO2eq/kWh, However, most experiments were CPU-based as almost all training and evaluation was done only to evaluate the PCBM performances given the usage of pre-trained models for the backbones. As such, only the PCBM-h instances required GPU computation as they are neural networks. We utilized T4 GPU and Intel(R) Xeon(R) CPU for these experiments, resulting in total computational cost. of roughly 30 CPU and 30 GPU hours for all experiments. This would amount to 2.39 kgCOzeq in emissions from GPU usage which was entirely offset by the cloud provider. These estimates were conducted using the Machine Learning Impact calculator presented in Lacoste et al. (019).
4 Results
4.1 Results reproducing original paper
Claim 1 One of the main claims originally made is that PCBMs achieve comparable sometimes even identical performance to the original basclines across various models. Our results (shown in Table 1) mostly support this claim, as observed results similar to the original for all datasets besides CUB, which returned performance that was 10 percentage points higher, aligning more with the CUB ResNet18 pre-trained model's performance This suggests a potential mistake in the author's evaluation code. In any case, our results show that the performances achieved and trends found match those of the original study quite closely.
Original
CIFAR10
CIFAR100
COCO-Stuff
CUB
HAM10000
SIIM-ISIC
Original Model
0.888
0.701
0.770
0.612
0.963
0.821
PCBM
0.777 + 0.003
0.520 + 0.005
0.741 + 0.002
0.588 + 0.008
0.947 + 0.001
0.736 + 0.012
PCBM-h
0.871 + 0.001
0.680 + 0.001
0.768 + 0.01
0.610 + 0.01
0.962 : 0.002
0.801 + 0.056
Reproduction
CIFAR10
CIFAR100
COCO-Stuff
CUB
HAM10000
SHIM-IIC
Original Model
0.885
0.699
0.838
0.744
0.963
0.761
PCBM
0.773 + 0.001
0.511 + 0.002
0.796 + 0.005
0.577 + 0.007
0.926 + 0.004
0.511 + 0.002
PCBM-h
0.883 + 0.002
0.688 + 0.002
0.797 + 0.001
0.595 + 0.005
0.957 + 0.004
0.751 + 0.021
Table 1: Original Yukkskkonul et al. 2023) and reproduction results of the claim that PCBMs achieve
comparable performance to the original model. The reported metrics are AUROC for HAM10000 and SITM- ISIC, mAP for COCO-Stuff, and accuracy for CIFAR and CUB.
The pre-trained model and performance can be found here: hutpe://stthaa..cmm/mmm////gggg
8

---

Published in Transactions on Machine Learning Research (05/2024)
Claim 2- We evaluate if PCBMs could achieve at least the same, if not higher, performance using generated concepts. However, our findings (as shown in Tabll 2) contradict this claim, evidenced by how for both CIFARIO and CIFARI00, the PCBM results are approximately 10 percentage points lower than originally reported. Notably, the PCBM for CIFAR10 performed worse when using CLIP concepts compared to BRODEN concepts, suggesting potential limitations the expressiveness of CLIP concepts. Moreover, our results indicate that the performance reported the original paper could be replicated using a lower À value than mentioned. This is further explained in Appendix B
It is noteworthy that the PCBM-h with CLIP concepts achieves comparable performance to CAVs, which is expected due to the sequential procedure in addition to the reintroduction of the embeddings when training PCBM-h. Because of this, the performance of PCBM-h depends only minimally on the performance of PCBM, and by extension, the concepts used. Furthermore, Figure 3 (in Appendix Cj displays example concept weights for the same classes and datasets as in the original paper. While the HAM10000 concept weights are quite similar, the weight values for CIFAR100 are one order of magnitude higher (though the important weights are similar). This aligns with the potential use of a lower X, as lower weight decay lypically results in larger concept weights.
Original
CIFAR10
CIFAR100
COCO-Stuff
Original Model
0.888
0.701
0.770
PCBM & labeled concepts
0.777 + 0.003
0.520 = 0.005
0.741 + 0.002
PCBM-h & labeled concepts
0.871 + 0.001
0.680 = 0.001
0.768 0.01
PCBM & CLIP concepts
0.833 + 0.003
0.600 = 0.003
0.755 + 0.001
PCBM-h & CLIP concepts
0.874 + 0.001
0.691 + 0.006
0.769 + 0.001
Reproduction
CIFAR10
CIFAR100
COCO-Stuff
Original Model
0.885
0.699
0.838
PCBM & labeled concepts
0.773 + 0.001
0.511 + 0.002
0.796 + 0.005
PCBM-h & labeled concepts
0.883 + 0.002
0.688 + 0.002
0.797 + 0.001
PCBM & CLIP concepts
0.736 + 0.005
0.535 + 0.003
0.801 + 0.002
PCBM-h & CLIP concepts
0.881 + 0.002
0.688 = 0.004
0.797 + 0.001
Original Yuuksekgonul et al. 2023 and reproduction results of the claim that PCBMs do not require labeled concept datasets, alongside comparisons to the original performances.
Table 2:
Original PCBM Accuracy
Unedited
Prune
Prune + Normalize
Fine-tune (Oracle) 0.859 + 0.028 0.202
0.656 + 0.025
0.686 + 0.026
0.750 + 0.019
PCBM Edit Gain
0.029
0.093
Reproduction PCBM Accuracy
Unedited
Prunc
Prunc + Normalize
Finc-tune (Oracle)
0.864 + 0.033
0.874 + 0.025
0.873 + 0.025
0.939 = 0.004 0.075 + 0.031
PCBM Edit Gain
0.010 0.009
0.009 + 0.008
Table 3: Original (Yuksckgonul et al.
2023 and reproduction results of the claim that PCBMs allow for
global model editing.
Claim 3 While we see that global model editing (both pruning and pruning with normalization) results in improvements relative to the bascline, the performance increase is half that of the original results, seen in Table 3 Note that the results are averaged over 10 seeds, such that the PCBM edit gain is the average gain across 10 seeds. For a more detailed description of the per-class edit gain, refer to Appendix E, With regards to the user study, we find that users demonstrated enhancements in 4 of the 9 scenarios, as illustrated in our user study (see Table 15 replicating the original paper's Table 9). However, the extent of these improvements does not align with the initial claim (comparing with the original paper's Table 4), as the performance increases are generally modest. In fact, we observed declines in performance for most scenarios, especially when the number of prumed concepts increased (see Table 4/.
9

---

Published in Transactions on Machine Learning Research (05/2024)
Greedy Prune
Original
Layer
Unedited
Random Prune
User Prune
(Oracle)
PCBM Accuracy
0.620 0.035
0.604 0.039
0.719 + 0.042
0.740 + 0.041
Spurious
PCBM Edit Gain
-0.016
0.099
0.120 0.766 + 0.034
Class
PCBM-h Accuracy
0.642 + 0.034
0.622 = 0.037
0.736 + 0.034
PCBM-h Edit Gain
0.020
0.094
0.124
Greedy Prune (Oracle)
Reprod.
Layer
Unedited
Random Prune
User Prune
PCBM Accuracy
0.731 + 0.172
0.217 + 0.270
0.199 + 0.249
0.276 + 0.291
Spurious
PCBM Edit Gain
-0.514
-0.532
-0.455
Class
PCBM-h Accuracy
0.740 0.145
0.736 + 0.144
0.735 + 0.144
0.733 + 0.145
PCBM-h Edit Gain
-0.004
-0.005
-0.007
PCBM Accuracy
0.822 0.038
0.737 = 0.059
0.733 + 0.577
0.747 + 0.066
All Classes
PCBM Edit Gain
-0.085
-0.089
-0.075
PCBM-h Accuracy
0.854 0.056
0.857 0.052
0.856 + 0.052
0.855 +'t 0.053
PCBM-h Edit Gain
0.003
0.002
0.001
Table 4: Original
Yuksekgonul al. 2023 and reproduction results of the claim that luman-guided editing
improves accuracy.
In study, users pruned 3.11 + 2.12 concepts average, showing larger variation compared to the original average of 3.65 + 0.39. This variance affected both the greedy and random methodologies, they rely on the number of pruned concepts. Further details on how these techniques were employed and design decisions regarding the initial selection of classes in the multimodal approach can be found in Appendix F and [Appendix H respectively. Unfortunately, we could not provide a detailed breakdown of the number of sconarios improved per user, as the observed improvements were mostly minor. Morcover, some scenarios that showed enhancements in one instance also demonstrated declines in performance across many runs.
Nonetheless, the initial claim that users could complete the pruning task faster than the Greedy approach with only a relatively minor decrease in performance indeed holds. Users achieved comparable outcomes (a slight decrease in PCBM score and an increase in PCBM-h) while maintaining mean time of 47.48 + 31 seconds per scenario, in contrast to the Greedy approach's average time of 52.49 3.58.
4.2 Results beyond original paper
4.2.1 Random projection
The experiment results of using random projections to assess the necessity of meaningful concepts (e.g., CLIP and CAV concepts) for performance and interpretability are displayed in Table 2 PCBM retains sub- stantial performance when using random concepts for CIFAR10, CIFAR100, and COCO-Stule indicating the preservation of backbone embeddings information post-projection. However, the HAM10000 performance is not retained, possibly because random projection with eight concepts cannot maintain enough information about the original dataset.
CIFAR10
CIFAR100
COCO-stuff
HAM10000
Original Model PCBM
0.885
0.699
0.838
0.963
0.746 + 0.002
0.490 + 0.003
0.810 + 0.002
0.500 + 0.000
PCBM-h
0.882 + 0.002
0.685 + 0.004
0.797 + 0.001
0.961 + 0.004
Table 5: Extension results of the random projection experiment.
The concopt weights for two classes in CIFAR100 and HAM10000 are shown Figure 4 and Figure 5 [Appendix C). For CIFAR100, the concept weights for random concepts lack intuitiveness. Meanwhile, all concept weights are for HAM10000. This is consistent with the earlier finding that PCBMs fail to
10

---

Published in Transactions on Machine Learning Research (05/2024)
maintain the original model's performance on this dataset, suggesting the necessity of collecting meaningful concepts for straightforward interpretability. Comparing the performance results with those obtained using meaningful concepts, we find that the latter does outperform this baseline in all cases.
4.2.2 Object-concent correspondence
This section contains our findings from testing the assumption that concept feature value reflects the visibility of concept in the image. The results for this can be found in Table 2. which presents the average concept feature values for BRODEN and CUB. Looking at the results, CAVs seem to have the greatest separation between positive and negative images, with gaps of 0.274 and 1.694 respectively. Meanwhile, the CLIP concopt vectors return a notably weaker signal for a concept's appearance, with gaps of 0.053 and 0.022 for BRODEN and CUB respectively. It especially weak for the latter since we see that there are incorrect and negative signals within one standard deviation.
We also see that random concept vectors have an average of around zero in both cases expected, given that no relevant information about the images is encoded. Thus, we find that CAV and CLIP concept feature values do indeed represent signal for whether a concept is in the image.
BRODEN
CUB
Concept Activation Vectors
0.274 + 0.124
1.694 0.037
CLIP Concept Vectors
0.053 0.046
0.022
0.037
Random Concept Vectors
0.002 + 0.018
-0.006 + 0.088
Table 6: Extension results of the concept feature value experiment. Reported here are the mean and standard deviations of the concept feature value gaps between 50 positive and negative images, averaged across concepts.
Both F Figure and Appendix T depict. the outcomes of our experiment, revealing the most relevant parts of the image for each concept via saliency maps for two CIFAR100 images. The former illustrates saliency maps for the four most significant concepts of the class "bicycle," " along with maps for the concept "grass," " chosen due to its co-occurrence with bicycles in the image corners. Notably, most saliency maps mainly emphasize the primary object, regardless of the concept's referenced part of the image. For example, both "chain wheel" and "handlebar" saliency maps for CAVs are bicyclo-shaped, consistent with the findings of Kim et al. (2023), ; which highlight the unintended ent entanglement in CAVs. In addition, the bicycle-shaped saliency map for the concept "book" likely stems from entanglement, as it is a top 4 concept for the bicycle class despite its apparent irrelevance. Notably, CAV saliency maps exhibit greater diversity compared to CLIP, highlighting different, incorrect parts the bicycle, possibly due to CAV's sensitivity to sampled positive and negative images, as discussed Kim et al. (2023).
A similar entanglement is present for the CLIP concepts, the saliency maps for "bicycle wheel," " "coaster brake," "two wheels," " and "bicycle seat" are all bicycle-shaped. This matches previous findings showing that CLIP suffers from CAB and thus cannot distinguish well between an object and its attributes (Tang) et al. 2023). A related possible reason why our saliency maps do not correctly highlight the regions of the image corresponding to the concepts is that the clip was not trained for such fine-grained classification. A related finding by [Zoong et al. (2021) is that CLIP does not transfer well to region detection tasks. They hypothesize that this is bocause CLIP was trained to only match whole images to the corresponding text descriptions without capturing the fine-grained alignment between image regions and text spans.
Further evidence that our saliency map findings are due to entanglement rather than other factors (i.e., our Nes550 backbone encoder only extracting fcatures related to the main object in scene) is provided by Chen et al. (2023), who discovered that CLIP-ResNeL50 channels exhibit more noise compared to a ResNet50 trained solely on ImageNet. Using various saliency map types, they demonstrated that this backbone examines more than just the main object within scene. Despite this, a positive observation for PCBMs is that concepts such as "green" and "greenness" successfully highlight the top right green patch of the image, indicating that concept feature values sometimes reflect broader relevant parts of the image.
11

---

Published in Transactions on Machine Learning Research (05/2024)
inpur image
birycle brabe
groen
CLi crecol
bicycle
bar
gracnness
city arroato
Figure 1:
Extension results of the saliency map experiment. Shown here ten saliency maps for an image from the class "bicycle" of CIFAR100. Five CAV and five CLIP concepts were used.
4.2.3 Audio classification
An overview of the audio classification extension results is provided in Table 7 PCBM struggles to match the original model's performance when using UrbanSoundSK with CLIP concepts, though PCBM-h narrows this gap. This pattern does not hold, however, for other dataset-method combinations. For example, PCBM-h fails to reach the original model's performance when using UrbanSound8K with CAVs. Meanwhile, CLIP performs poorly for ESC-50, and the CAV-based approach falls short of the original model despite improving over CLIP. The discrepancy in CLIP-based performances may stem from the number of CLIP concepts not scaling with the number of classes in ESC-50, which is something discussed by Oikarinen et al. 2023 This means that PCBMs are harder to scale to datasets with many labels.
The inadequacy of CLIP's concept space is evident in Table 12 (Appendix Dj, where top concepts like heevvy metal," " "molorcycle," and "whirr" are mismatched with their class label "cow." Conversely, CAVs' consistency across datasets may arise from the customized concepts tailored to each dataset. The results in Table 11 and Table 12 (in Appennixx D) support the interprctability of CAVs over CLIP. Each class has multiple fitting concepts within the lop three most important concepts. Thus, based on this, extending PCBMs to different domains can yield mixed results, heavily influenced by the chosen concept subspace.
UrbanSoundsK
ESC-50
Original Model PCBM CLIP concepts
0.613
0.670
0.558 + 0.002
0.280 + 0.035
PCBM-h & CLIP concepts
0.603 = 0.006
0.280 = 0.035
PCBM & labeled concepts
0.411 + 0.001
0.400 + 0.006
PCBM-h & labeled concepts
0.462 + 0.009
0.410 + 0.006
Table 7: Extension results of the audio classification experiment.
5
Discussion
Our study aimed to replicate the findings from Yuksekgonul et al. (2023) and somewhat successfully repro- duced their first two claims, though had challenges with doing so for the third. The first claim, asserting that Post-Hoc Concept Bottlenccks do not reduce model performance, aligns with our results. Our PCBM experiments show performance similar to that in the original paper and PCBM-h retrieves a similar perfor- mance to the baseline, with negligible differences observed in certain scenarios. As outlined before, some of these differences are caused by potential implementation mistakes by the original authors, relating to the conflicting results obtained for CUB. Furthermore, other differences stem from missing details regarding the experiment setup, which is the case for the experiments related to COCO-Stuff and, to extent, SIIM-ISIC.
12

---

Published in Transactions on Machine Learning Research (05/2024)
The second claim, suggesting the usability and potential advisability of CLIP concepts, is partially sup- ported. While the PCBM-h here performs similarly to the original model, the PCBM results are mixed and not consistently superior to the CAV-based approach. Meanwhile, the third claim regarding PCBMs enabling global model editing is not truly supported by our results. From the Mctashift experiment, the observed increase was significantly less than the original results, and the user study indicated only marginal performance improvements and, many cases, declines. Similar to some of the other experiments performed, this discrepancy is due to incomplete implementation details provided by the authors, especially relating to how the Metashift PCBMs were trained.
Beyond these three claims, we evaluated the interpretability of PCBMs by first testing whether meaningful concepts are necessary for both performance and interpretability, discovering that random, meaningless, concepts perform well but are not easily interpretable. We then examined if the concepts used by the model correctly correspond to objects in the input image using saliency maps and observed that the concept projection generally signals an object's present within an image correctly, though this signal is not basod on only the image part containing said object. Based on this, our conclusion is that the interpretability of PCBMs is limited due to how the concept feature values used by the interpretable model are not obtained only by looking at the concept in the image. Instead, many concepts return high feature values only because they are correlated with the main object we are classifying.
Furthermore, another of our extensions partially supports the claim that PCBMs apply to any neural network. We find that results when applying PCBMs to audio are mixed. The CAV-based approach is consistent and gives interpretable concepts, but does not perform as well as the original model. The CLIP-based approach does as well as the original model for one dataset but does poorly on the other. Additionally, the CLIP concepts are harder to interpret in combination with the classes.
Given these results, we recommend further research focusing on the interpretability of both concept vectors discussed. Additionally, exploring more suitable projection methods for CLIP concepts is crucial to enhance their responsiveness in the presence of specific concepts in images. Another recommendation is to evaluate the performance of additional audio datasets and archilectures to investigate if other, non-CT. -Tn-CT-P--ased audio classifiers still function with the PCBM architecture. Lastly, given how frequently PCBMs are utilized Oikarinen et al. (2023); ; Yang et al.) (2023); Panousis et al. (2023); Daneshjou et al. (202bb)), ), we suggest that future researchers be more careful when using them, as our lindings show that proper interpretability is not fully guaranteed. It would be especially beneficial to examine further research utilizing PCBMs for object attribute entanglement in concepts, to see if they suffer from the same issues covered here.
5.1 What was easy, and what was difficult?
An appendix with all used hyperparameters and implementation details was provided by the original paper. This made it simple to start with the experiments, especially when coupled with the publicly available repository and the complete model implementations.
That aside, the repository provided was stiill incomplete, meaning that several code parts, such the COCO- Stuff implementations, model editing files, and Metashift concept sources, were unavailable. The explanation for some of the experiment setups is also insufficient for producing accurate reproductions, resulting in the assumptions outlined in subsection 3.4
5.2 Communication with original authors
We have attempted to contact the original authors for clarification on how some of the experiments wore set up due to missing details/imphl settall/iimleenentationns such as how the COCO-Stulf binary classification and Metashift experiments were performed. However, as of writing, they have not responded any of our inquiries.
References
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery
13

---

Published in Transactions on Machine Learning Research (05/2024)
and data mining, pp. 243-250 San Francisco California, August 2001. ACM ISBN 978-1-58113-391-22 doi: 10. 00.1145/502512.502255 URL https://dl.acee. www.aa...w........../.. http//////111111/5501000 110.:001111///5222522222222
Timothy T. Cannings. Random projections:
Data perturbation for classification problems.
WIREs Computational Statistics, 13(1)99 2021.
ISSN 1939-0068.
doi: 10.1002/
wics. URL
wiley...w....................... com/ http://daaa//...00/020// 1499
eprint:
Peijie Chen, Qi Li, Saad Biaz, Trung Bui, and Anh Nguyen. gScoreCAM: What Objects Is CLIP Looking At? In Lei Wang, Juergen Gall, Tat-Jun Chin, Imari Sato, and Rama Chellappa (ecss), Computer Vision -ACCV 2022, volume 13844, pp. 588-604. Springer Nature Switzerland, Cham, 2023. ISBN 978-3-031- 26315-6 978-3-031--2316--3 doi: 10.1007 000077977---00126631
-
35. URL aattps://lint http::::/iinnsspiiggeerr com/10. Series Title: Lecture Notes in Computer Science.
1007/978-3-031 26316- 3_35
Roxana Dancshjou, Kailas Vodrahalli, Weixin Liang, Roberto A. Novoa, Mclissa Jenkins, Veronica Rotem- berg, Justin Ko, Susan M. Swetter, Elizabeth E. Bailey, Olivier Gevaert, Pritam Mukherjee, Michelle Phung, Kiana Yekrang, Bradley Fong, Rachna Sahasrabudhe, James Zou, and Albert Chiou. Disparities in Dermatology AI: Assessments Using Diverse Clinical Images. Science Advances, (332)14b,61147, Au- gust 2022a. ISSN 2375-2548. doi: 10.1126/sciadiee abbbadd..a.a61147 URL http://arxiv, http:///axxxv..rrabb//////s arXiv:2111.08000 [cs, eess].
08006.
Roxana. Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skin- Con: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 18137-18167. Curran Associates, Inc., 2022b. URL https::/pproceeddngg neurips. cc cc/paper_ files//paper/222/Eiie/ and Be__BBnchmarks... pdf
Sanjoy Dasgupta. Experiments with Random Projection. Proceedings of the Sixteenth Conference on Un- cerlainly in Arlificial Intelligence, pp. 143 151, 2000.
Ruth Fong and Andrea Vedaldi. Vec: Quantifying and Explaining How Concepts are Encoded by Filters in Deep Neural Networks. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8730 8738, Salt Lake City, UT, June 2018 IEEE. ISBN 978- 978-1-5386-6420-4 doi: 10.1109CCVPR.2018 00910. URL https://ieeexplorre ieee. wwtp//rgg//eeeeoooooooo /tt//mmmt/t//7900/////////
Jack Furby, Daniel Cunnington, Dave Braines, and Alun Preece. Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?, February 2024. URL http://arxivv.rg. abs/2402. 00912 arXiv:2402.00912 (cs).
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and 2017 IEEE International Conference on Acoustics, Speech and Sigmal Processing (ICASSP), pp. 776-780, New Orleans, LA, March 2017. IEEE. ISBN 978-1-5090-4117-6 doi: 10.110//CCASSPP2217...
dataset for audio events. In
URL
http://ieeexploree ieee. wwweeee..rr//oooeeeeoeeeeeeooeee /td////mmmmt/tt2221111111
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards Automatic Concept-based Expla- nations. Advances in Neural Information Processing Systems, 32, 2019.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. ESResNe(X))-fbbp: Learning Ro- bust TTme-FFrqqennn Transformation of Audio, April 2021. URL http:://arxiv.orr/bbb//200000 11587 arXiv:2104.11887 eess].
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending Clip to Image, Text and Audio. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980, May 2022. doi: 1LL11///CAASSS4492220002222 //ieeexplore. ieee. htt.epppooe..eee.oooo/cccaaa/tte ottpe///mecmmeet/97771/111111111 TSSN: 2379-190X.
URL https:
14

---

Published in Transactions on Machine Learning Research (05/2024)
Marton Havasi, Sonali Parbhoo, and Finale Desei-Velez. Addressing Leakage in Concept Bottle- neck Models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 23386 23397. Curran Associates, Inc., 2022. URL https::/proceeddingg neurips www.rips.ccppapppeeeeeeeoo cc/paper fwwes/ sille//paper2222/filel betpeccaaaeeeereaaaaaradaararare
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851111 ieeexplore ieee leee. org/ document/7780455//
doi: 10.1109/CVPR.2016.990 URL http://
Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, and Ghassan Hamarneh. Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets. IEEE Journal of Biomedical and Health Informatics, 23(2):538-546 546, March 2019. ISSN 2168-2208. doi: JJ00/JJH..011 2824327. 2018.22277. URL ieee. org wtwp///ccmee/////3ea330 Conference Name: IEEE Journal of Biomedical and Health Informatics.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vec- tors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, pp. 2668 26777. PMLR, July 2018. URL htttp:://proceddii
mlr ppess/v80/kim1e/ html
ISSN: 2640-3498.
Siwon Kim, Jinoh Oh, Sungjin Lee, Seunghak Yu, Jaeyoung Do, and Tara Taghavi. Grounding Counter- factual Explanation of Image Classifiers to Textual Concept Space. In 2023 IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10942 10942-10950, Vancouver, BC, Canada, June 2023. URL httts:// ieeee
IEEE. ISBN 9798350301298 298. doi: 10.100//CVPR32722200233 200.7229..22.01055.. org/ //cc/mmmtt///203300////////
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept Bottleneck Models. Proceedings of the 37th International Conference on Machine Learning, 119, 2020.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Car- bon Emissions of Machine Learning, November 2019. URL http:///axxiv.org/ab//11000 arXiv:1910.09700 [cs].
Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo, Qinan Yu, Stephen Bach, and Ellie Pavlick. Does CLIP Bind Concepts? Probing Compositionality in Large Image Models. In Yvette Graham and Matthew Purver (ods.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 1487-1500, St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL https://aclaathhooogy org/2024. www.iinnnnnnnc..nnn 101
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do Concept Bottleneck Models Learn as Intended?, May 2021. URL http://aaxivv.oogg/a/a///// 04289.
arXiv:2105.04289 [cs].
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui- Wei Weng. Label-Free Concept Bottleneck Models, June 2023. URL httt:://arxiv.org/eeee www.////////..o.//////// 06129 arXiv:2304.06129 [es].
Konstantinos P. Panousis, Dino lenco, and Diego Marcos. Sparse Linear Concept Discovery Models. In 2023 IEEE/CVF International Conference on Compuler Vision Workshops (ICCVW), pp. 2759 2763, Paris, France, October 2023. IEEE. ISBN 9798350307443,
doi: 1011109/1CCCWW6T88822222..0000 URL
https::/1eeexplore.e ieee. org/ htt.aeeeer/.//eeeee/////00000 /ocumeet/103500600/
Dana Pessach and Erez Shmueli. Review on Fairness in Machine Learning. ACM Computing Surveys, 55 (3):1 14, March 2023. ISSN 0360-0300, 1557-7341, doi: 10.1145/3494672.
URL https://dl acm. org/
doi 201/10.110 1141/444499
15

---

Published in Transactions on Machine Learning Research (05/2024)
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd ACM international conference on Mullimediaa, pp. 1015-1018, Brisbane Australia, October 2015. ACM. ISBN 978- 178-1-4503-3459-44 doi: 10.1145/2778333328866333 2806390
URL https://dl... http//lllacmm..ommmmmmmmm. www../org////////oo/////// 110...1311/2/333...
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv. srg/abb/21033.000 arXiv:2103.00020 [cs].
Naveen Raman, Mateo Espinosa Zarlenga, Juyeon Heo, and Mateja Jamnik. Do Concept. Bottleneck Models Obey Locality?, January 2024. URL httt:://axxiyy htng/abs/2401... 01259 arXiv:2401.11259 [cs].
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A Dataset and Taxonomy for Urban Sound Research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, Orlando Florida USA, November 2014. ACM. ISBN 978-1-4503-3063-1
doi: 10.1145/2647868. 265544 URL
https://dl acm. www.aaa......................a.. cecm.oog//oo///// wet...11/////11..111 2655045
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Networks Visualising Image Classification Models and Saliency Maps, April 2014. URL http://aariv..or//bbb//12..... arXiv:1312.6034 [es].
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. SmoothGrad: removing noise by adding noise, June 2017. URL http::tttp::rtt wrg/ 038651325565553333 MxXiv:1706.03822 [cs, stat].
Robyn Speer, Joshua Chin, and Catherine Havasi. Concept 5.5: an open multilingual graph of general knowledge. In Proceedings of the Thirty-Firss First AAAI Conference on Artificial Intelligence, AAAI11, pp. 4444 4451, San Francisco, California, USA, February 2017. AAAI Press.
Harini Suresh and John V. Guttag. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. In Equity and Access in Algorithms,, Mechanisms, and Optimization, pp. 1 9, October 2021. doi: 111144/3465416.34883833 stat].
URL http://arxxvv.rrgaab//1111. 10002 arXiv:1901.10002 [cs,
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1-9. IEEE Computer Society, June 2015. TSBN 978-1-4673--6664-. doi: 20..100//CPPR.20552200005 URL https: //::::/wwwww http:/:/wwwwwwwwwpee/aaaeat (rtpopputerrorgsssll.. ISSN: 1063-6919.
Yingtian Tang, Yutaro Yamada, Yoyo Zhang, and Ilker Yildirim. When are Lemons Purple? The Concopt Association Bias of Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14333-143444 Singapore, 2023. Association for Computational Linguis- tics. doi: 10.18653/v1. v1/2022.2222222 htt//2..emnlp-pm.... emnll-main..eee 886. URL
emnlp-maaaea.eeeee malipp-miiini.mm ieaainn8866.. 886
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Fallison-Burch and Mark Yatskar. Language in a Bottle: Language Model Guided Concept. Bottlenecks for Interpretable Image Classification. In 2023 TEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19187-19197, Vancouver, BC, Canada, June 2023. IEEE. ISBN 9798350301298. doi:
URL httpps://eeexplloe ieee. org/ httg///ccmmon//004225///////////
Chih-Kuan Yeh, Been Kim, Sercan Ó. Arik, Chun-Tiang Li, Tomas Pfister, and Pradeep Ravikumar. On Completeness-awart aware Concept-Based Based Explanations in Deep Neural Networks. Advances in Neural Infor- mation Processing Systems, 33:20554 20565, 2020.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post- Concept Bottleneck Models. The Eleventh International Conference on Learning Representations, February 2023.
16

---

Published in Transactions on Machine Learning Research (05/2024)
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP: Region-based Language-Tmage Pretraining, December 2021. URL wtt.vvvorr//bbb/2...00110
arXiv:2112.09106 [cs].
17

---

Published in Transactions on Machine Learning Research (05/2024)
A
Specifications of datasets used
Table 8 contains an overview of all datascts used by the original authors, which were experimented on for
reproduction.
Number of concepts used
Task
Dataset
Backbone model
Concepts used
Classes
CIFAR-10
10
Evaluating
BRODEN
CIFAR-100
CLIP-ResNet50
170
100
performance
Fong Vedaldi
2018
COCO-Stuff
20
across
CUB
ResNet18
From Koh et al.
2020
112
200
different
HAM10000
Derm/pt
domains
Inception
8
2
SIIM-ISIC
Kawahara et al.
2019
Evaluating
CIFAR-10
ConceptNct generated concepts
10
generated
CIFAR-100
CLIP-ResNet50
100
concepts
COCO-Stuff
20
Controlled
Metashift
BRODEN Fong & Vedaldi 2018
Experiments For Model Editing
Metashift
CLIP-ResNet50
170
5
ConceptNet
User
Metashift.
CT.PPReeNNt50
generated concepts
440
Study
Table 8: Overview of datasets used from the original study.
As part of the audio classification extension, the ESC-50, UrbanSoundSK, and AudioSot datasots were created to tackle the issue of data scarcity in automatic urban sound classification and are used to further fine-tune the AudioCLIP audio encoder
utilized (Salamon et al., 2014; Piczak] 2015). All datasets
Guuzhov et al. 2022 An overview of them can be found in Table 9
Number of concepts used
Dataset
Backbone model
Concepts used
ESC-50
Generated by Concept.Net
146
UrbanSoun8KK
AndioCLIP
From Salamon et al. (0144
31
AudioSet
From Gemmeke et al. 2017
610
Table 9: Overview of the datascts used for audio classification.
18

---

Published in Transactions on Machine Learning Research (05/2024)
B Accuracy/Interpretaallly trade-off
Table 10 shows the different PCBM accuracies when varying the regularization strength. We see there that for both CIFARIO and CIFARI00, the accuracy is highest when the regularization strength is KN after which it decreases. To examine the trade-off between accuracy and interpretability that exists when tuning the regularization strength, we plot the regularization strength against the number of non-zero parameters and the absolute sum of weights. For both metrics lower better. Figure 2a and Figure 2b show these plots for CIFAR10, where we that lower regularization strength leads to the trained PCBM doing worse the interpretability metrics. As such, the regularization strength not be decreased to obtain higher accuracies without harming interpretability. Figure 20 and Figure 2d show similar trends for CIFAR100.
0.1
Regularization strength
10.0
1.0
0.01
0.001
K.V
K..
KN.
K.N.
K.N.
CIFARIO
0.318
0.552
0.559
0.507
0.461
CIFAR100
0.648
0.793
0.839
0.824
0.820
Table 10: PCBM accuracies for different regularization strengths.
84
ALCUrIcy
70
76
74
74
68
500
2000
1500
2000
2500
100
200
300
400
500
600
700
800
Sun of weights
N non-2erro weights
(a) The absolute sum of class weights against accu- racy for PCBM trained on CIFAR10.
(b) The number of non-zero weights against accuracy for PCBM trained on CIFAR10.
00000
5000
70000
Absolutt qum of wnights
non 2213 wogghts
5000
40000
4000
30000
2000
1000
Regulaticaion strengih
Regulerization strength
(c) The absolute sum of all class weights against accuracy for PCBM trained CIFAR100.
(d) The number of non-zero weights against Accuracy for PCBM trained on CIFAR10.
Figure& 2: Tradeoffs between accuracy, class weights, and regularization strength for selected scenarios.
19

---

Published in Transactions on Machine Learning Research (05/2024)
C Example weights obtained
Below are examples of weights obtained for certain classes from the HAM10000, CIFARI00, and audio experiments. As mentioned in the /sccioon 4 that the obtained concept weights are indeed human- logical, which supports the usability of PCBMs for interpretability.
HAM00000
CIFAR100
Imageler
Straak
Whirhh
Fir
Iregy ky
Blws
34700
Soraak
Thems
500
Whrnh Vala
Itsk
2840
Figure 3: Example weights for classes of HAM10000
(CAVs) and CIFAR100 (CLIP concepts).
We also attach examples of weights obtained from some of the random projection experiments and for CIFAR100. Both figures can be found below:
CIFARI00
HAME300
A.19
Ieglla Soolo
whrch 00X
Daglas
M000
1222
004
005
Inoudar
Maligram
Rezrae
Buc
Whith
10.01
1302
10.
121b
Figure 4: Example weights for classes of CIFAR100 (CLIP concepts).
Figure 5: Example weights for classes of HAM10000
(Random Concept Vectors).
D Preprocessing Metashift for model editing experiments
To generate the 10 scenarios used in the model editing experiments, the original authors used Metashift. They first defined two 5-way classification tasks where they specified different independent scenarios for each. Furthermore, every scenario contained one class which is spuriously corrclated with a different concept in its train and test datasets, and each split consists of 100 images per class. For instance, the first scenario of the first task in Table 13 consists of a training set where all images corresponding to the class bed contain dog and test set where all images of beds contain cats along with 100 unconstrained images for each remaining class.
We can see in Table 13 eachh task with its classes and corresponding scenarios. Although there conflicting information on the number of images per class (either 50 or 100), is worth noting that the 10 scenarios present, four of them do not have 100 images. We considered that 250 images per split was too little, so we discarded the four problematic scenarios.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Class
CLIP Concepts (weights)
Labeled Concepts
air conditioner
Doorbell (17.85)
machine (10.698)
Bicycle bell (17.81)
automobile (10.238)
Cat communication (16.87)
air (4.116)
car horn
Shofar (14.49)
honk (4.858)
Sonar (12.40)
bullet (0.334)
Harmonic (11.56)
bang (0.297)
children playing
Babbling (20.03)
loy (6.711)
Hubbub, speech noise, speech babble (19.47)
school (5.524)
Chatter (18.26)
fun (5.484)
dog bark
Growling (29.54)
cry (12.929)
Yip (27.51)
dog (10.003)
Bow-wow (21.07)
woof (8.170)
drilling
Cattle, bovinae (22.48)
construction (4.338)
Honk (17.71)
machine (0.000)
Canidac, wolves (13.47)
air (0.000)
engine idling
Purr (19.06)
automobile (7.059)
Meow (17.67)
machine (6.176)
Claws (16.95)
motorcycle (5.037)
gun gun_shoo shot
Silence (14.86)
bang (4.549)
Arrow (13.78)
bullet (4.545)
Chainsaw (12.55)
honk (1.769)
jackhammer
Snake (23.94)
rock (5.604)
Steelpan (19.34)
drill (4.969)
Tambourine (18.31)
construction (0.166)
siren
Ice cream truck, ice cream van (28.41)
wail (16.096)
Air horn, truck horn (27.81)
alarm (9.479)
Ambulance (27.61)
noise (5.034)
street music
Shofar (16.56)
entertainment (8. (8.744)
Water tap, faucet (16.43)
automobile (4.411)
Clarinct (14.77)
rap (3.860)
Table 11: Comparison of the top three concept weights between CAVs (labeled concepts) and concepts obtained using CLIP for UrbanSoundSK.
Furthermore, it unclear to us how the domain cow(cat) could even be generated, as the Metashift domain lookup Labled does not register any such image and has not been changed since the creation of the Metashift GitHub repository. We speculate that the original authors may have used different base dataset (such COCO) for Metashift, but chose to stick with the Visual Genome dataset as it was the default. option and because custom base dalasets were mentioned in the original paper.
We make the chosen six scenarios available on HuggingFace along with the entirety of our deterministic pre-processing pipeline for full transparency. Although we report all of our results using a cherry-picked dataset (meaning that we manually pick the best images that represent the domain shift), we also provide randomly selected datasct.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Class
CLIP Concepts (weights)
Labeled Concepts
dog
Growling (3260.187)
canine (9.666)
Yip (2679.440)
woof (8.258)
Bow-wow (2495.401)
four legged animal (7.040)
rooster
Power windows, electric windows (2253.037)
crowing (12.931)
male chicken (1976.757)
chicken (10.502)
Crowing, cock-a-dooolee-oo (1950.169)
farm animal (10.376)
pig
Ping (2314.484)
engine (4.299)
Chewing, mastication (2257.323)
amphibian (3.932)
Snort (2160.123)
oink_oine oink (2.580)
COW
Heavy metal (2201, .455)
moo (12.866)
Motorcycle (2009.351)
living_cceettuine creature (10.481)
Whir (1978.380)
nature (7.279)
frog
Pig (2828.164)
amphibian (18.596)
Ringtone (2783.966)
ribbit (17.952)
Frog (2761.661)
windshicld (4.473)
cat
Cat (2598.395)
feline (10.581)
Cat communication (2558.293)
meow (8.726)
Caterwaul (2385.920)
scratch furniture (7.948)
hen
Pigeon, dove (1894.772)
chirps (7.573)
Dishes, pots, and pans (1873.768)
bird (4.638)
Chink, clink (1797.078)
farm animal (3. 7597
insects
Strum (1400.697)
move (7.170)
Whale vocalization (1386.632) Croak (1372.076)
bugs (6.840)
animals (4.380)
Comparison of the top three concept weights between CAVs (labeled concepts) and concepts Table 12:
obtained using CLIP for ESC-50.
Task
Train domain
Test domain
Availability
bed(dog)
bed(cat)
Enough images
Task (airplane, bed, car, cow, keyboard)
(ed(cat)
bed(dog)
Enough images
car(dog)
car(cat)
50 < car(cat) 100
car(cat)
car(dog)
50 < car(cat) < 100
cow (dog)
cow(cat)
cow(cat) =
key board (dog)
key (oybboad(cct))
< keyboard < 50
Task (beach, computer, motorcycle, stove, table)
table(cat)
table(dog)
Enough images
table(dog)
table(cat,
Enough images
table(books)
table(dog)
Enough images
table(books)
table(cat)
Enough images
Table 13: Metashift task splits and their availability.
E
Per scenario performance on model editing experiments
Similar to Table of the original paper, we compute the accuracies and standard error of each scenario and method as described in Appendix D As shown in Table 14 both pruning and pruning with normalization do not show the improvements seen in the original study, even when averaging over 10 seeds and trying all the ambiguous configurations of regularization strengths and backbones mentioned. (more information on Figures 6a and 6b) For Table 14 we use CLIP's pre-trained ResNet50 with a À of 7.7 (which results in an overall regularization strength of 0.002), learning rate of 0.5, a of 0.99 and using the SGDClassifier.
In Figure 6. we see that no matter the regularization strength or the backbone used, the accuracy change does not compare to the original paper. In addition, the ResNet18 accuracy change is negligible at best.
22

---

Published in Transactions on Machine Learning Research (05/2024)
Train
Test
Original
Prune
Prune + Normalize
Finetune
bed(cat)
(dog)
0.928 0.001
0.930
0.001
0.930 + 0.001
0.946 0.001
(eog)
(ea(c
0.926 0.001
0.925
0.001
0.925 + 0.001
0.943 0.001
table( (books)
(cat)
0.761 0.002
0.795
0.004
0.792 + 0.003
0.924 0.001
table( (bookk)k))
(dole
0.765 = 0.002
0.805 = 0.002
0.804 = 0.002
0.948 = 0.001
table(cat)
table(dog)
0.923 + 0.002
0.918 + 0.003
0.918 + 0.003
0.946 + 0.001
table(dog)
tablo(cat)
0.883 + 0.001
0.874 + 0.001
0.872 + 0.002
0.927 + 0.002
Table 14: Accuracy and Standard Error for Metashift model editing tasks over 10 seeds.
To make sure that our impleementation does not differ substantially from the intended, we make two more: First, one that is as close as possible the existing author's code, which we'll name Strict ResNet50, and socond, implementation that uses the Adam Optimizer for the training procodure (as said in the original paper), which we'll name Adam ResNet50. In Figures 7a and 7b perform considerably worse than ours.
can observe that both implementationss
Prune
Prune Prune+NNrmann NNormalizze
Prune
Prune+ Normalize
0.03
0.03
0.03
0.03
Accursc gain
Accuracy gain
0.01
0.01
0.01
0.01
0.00
0.01
-0.1
00.1
-0.01
-0.02
-0.02
-0.0
-0.02
(a) Boxplots of ResNet18's accuracy gain compared to Original for different regularization strengths
(b) Boxplots of ResNet50's accuracy gain compared to Unedited model for different regularization strengths
Figure Boxplots for the accuracy gain of Pruning and Pruning with Normalization over the Unedited mode for a given model and value.
F Human-guided editing experiment setup
To replicate the human-guided editing experiment, we used the following setup and design assumptions:
We based our dataset on the same source as the Metashift experiment The training dataset consisted of 100 samples of class with spurious correlation, while the test dataset comprised 100 samples of the same class with correlations to any concepts except the one used in training.
We utilized the same model and training methods as in the original repository with minor modifi- cations (i.e., to the dataloaders and pruning capabilities) to align with initial results.
Greedy (Oracle) and random pruning used the same number of simultaneous pruned concepts as the users. For this, we assumed that for each task, each unique number of concepts pruned by users results in model (e.g., if users pruned between 1 and concepts for task, we would have greedy horlly//rrar rrandom model for 1 concept pruned, 2 concepts pruned and so on).
We reproduced the user study following the details provided in the original paper by introducing the terminology to the users (Figure 8) and presenting a similar task of choosing concepts to prunc per scenarios Figure 9
23

---

Published in Transactions on Machine Learning Research (05/2024)
Prune+ Normalize
0.04
0.04
Prune
Prune Prune+Norman -Normalize
0.04
0.04
0.03
0.03
0.03
0.03
0.02
Accuracy gain
0.02
0.02
Accuracy
0.01
0.01
0.01
0.00
0.00
0.00
0.00
-0.01
-0.01
-0.01
-0.02
42.5
1.7
42.5
-0.0
-0.02
42.5
1.7
42.5
(b) Boxplots of Striet ResNet50's accuracy gain com- pared to Unedited model for different regularization strengths
(a) Boxplots of Adam ResNet50's accuracy gain com- pared to Original for different regularization strengths
Figure 7: Boxplots for the accuracy gain of Pruming and Pruning with Normalization over the Unedited mode for a given model and À value of other implementations.
19VL
Concegrs Sorted by weigh:
Melilt Geureel
Background and purpose
Coonessa zan human dodeetannio fenuuee iesureentioou They can levetaged impreve the imarpeeacilly by of sene heaming mecels, they can mapped to the predictoono made: This offere an naightt into what modil bu the input iy providor raminng ooreepia bsocs on then relovano maiiing Ihc prodtattos.
Instructions
Through thes survey you actively partoipare the process mode editing Afssc contepts will displeyed oogale their comsoponcingg inng weight (io,. 10 sooplo with sarget weighss Thoce concigns one ocores are rectoved to gart s-aas
20000
dasss iication took, One the thee casses hos beer designaled as cloos sterest
The besk surnyy is help prune the concepts Winder model izadion siming improve prediction accur
glven dasanee
acamplx, mightt rone senoe preeeted oben je cintapt cowe not the model chess tytng penerol imoqpee at's we should hemooee si Yourmay aiko other ontoraiiss chicsss whicch concapts to semove You may remove as many fmmllvant cianses as you quantioon
our capra thee seem camp aoedy unmatated
Scenario
The mode was trained to ditiigussh Alpplane, Bed Cas Cow, and Kayboard Remove concepts that you mins tinder genersization for dasstying bed
Note
Movic rlidi survey one go Also, you recdl veto uncendand any conocgts of tig, do quick ontine soarch making cneaie eeaioo.
Figure 8: First page of the survey introducing the task.
Figure Example scenario with choice of concepts
G User study for model editing
Replicating Table 8 of the original paper, Table 15 contains full overview of the results obtained from the user study, The original authors used only the PCBM-h results for their breakdown. We attempt to replicate those results and while we also observe increases and decreases in performance across scenarios, the magnitude of the increases does not align with the original claim. Moreover, the usage of PCBM-h for reporting the final results does not provide a reliable insight into the method's performance. By visualizing the PCBM results instead (shown in Table 16), we can see that pruning (in the settings defined, where we highlight larger number of simultancous concepts pruned than the original paper) reduces the per-class accuracy considerably.
24

---

Published in Transactions on Machine Learning Research (05/2024)
Scenario
Unedited
Random Prune
User Prune
Greedy Prune
Shifted Class
Test Accuracy
bed(dog)
0.732 1 0.018
0.727 1 0.019
0.738 1 0.027
0.729 1 0.021
(cyb
0.918 1 0.003
0.906 + 0.009
0.903 + 0.010
0.911 +: 0.010
bed(cat)
0.864 +-0.006
0.863
0.008
0.854 + 0.009
0.911
0.010
couch(cat)
0.965 = 0.005
0.962 0.005
0.961 = 0.007
0.964 0.004
painting(lamp)
0.534 + 0.021
0,509 = 0.012
0.509 = 0.012
0.511 = 0.012
pillow(clock)
0.764 + 0.010
0.762 + 0.005
0.759 + 0.007
0.763 + 0.002
television( (elepision(freelacce)
0.701 + 0.012
0.686 + 0.019
0.678 1 0.015
0.685 1 0.020
fork (trk(tomato)
0.565 0.015
0.580
0.016
0.579 1 0.014
0.575 0.019
car(snow)
0.618 1 0.008
0.625 1 0.015
0.620 1 0.016
0.617 1 0.012
Overall Test Accuracy
(edg)
0.903 + 0.002
0.903 = 0.003
0.905 + 0.006
0.903 0.003
(eyboard(cat))
0.901 + 0.002
0.898 + 0.002
0.898 = 0.002
0.899 = 0.002
bed(cat)
0.853 + 0.002
0.852 = 0.002
0.851 + 0.002
0.853 + 0.002
couch(cat)
0.898 0.001
0.898 0.002
0.897 + 0.002
0.898 0.002
painting(lamp)
0,840 + 0.002
0.840 + 0.002
0.841 + 0.003
0.841 + 0.002
pillow(clock)
0.877 1 0.002
0.879 1 0.002
0.878 1 0.002
0.879 1 0.002
television (elepision(freepace)
0.888 + 0.003
0.886 + 0.004
0.884 + 0.003
0.886 + 0.005
fork fork(tomato)
0.719 + 0.003
0.726 = 0.004
0.726 + 0.003
0.725 + 0.004
car(snow)
0.808 0.002
0.809 0.003
0.807 + 0.003
0.807 0.003
Table 15: Reproduction results of the user study praning with PCBM-h.
Multimodal performance based on initial choice of classes
In designing our user study experiment, we explored and considered different approaches with the dataset and multimodal representations. Notably, we had to consider the choice initial elements (classes) used as basis for utilizing CLIP and ConceptNet as can significantly influence the results obtained.
When determining the classes for concept retrieval in our user study scenarios, compared several ap- proaches:
Using the 5 main classes on which we are conducting classification.
Incorporating these 5 classes along with the class exhibiting spurious correlations.
Including all classes and spurious correlation classes used across the scenarios.
Employing the CIFAR100 classes, which were part of the original code.
These different configurations allowed us to assess how class selection impacts the study's outcomes. Even with a 5-class simple classification lask, the difference in accuracy (shown in Table 17 suggests that more classes/inine /initaal objects lead to better accuracy. Adding on to this, better interpretability of the data and predictions are showcased by the results with more suitable high-weighted concepts (for class exhibiting "bed" with spurious context "cat" we see terms such as "bedroom furniture" and "feline" for CIFAR100 Figure 10d but in limited initial categories we see terminology from other classes spilling over such as "car seat" in Figure 10a).
25

---

Published in Transactions on Machine Learning Research (05/2024)
Scenario
Unedited
Random Prune
User Prune
Greedy Prune
Shifted Class
Test Accuracy
bod(dog)
0.614 0.166
0.099 0.163
0.051 0.107
0.208 0.228
(eyboarddcat)
0.843 + 0.049
0.199 + 0.274
0.051 0.107
0.236 + 0.293
bed(cat)
0,886 + 0.029
0.415 + 0.318
0.239 + 0.279
0.389 = 0.341
couch(cat)
0.979 + 0.010
0.390 + 0.370
0.310 + 0.339
0.461 + 0.385
painting(lamp)
0.467 + 0.061
0.132 = 0.143
0.132 + 0.128
0,188 + 0.165
pillow(clock)
0.707 0.035
0.264 0.234
0.279 + 0.252
0.364 0.271
srevisioon(freplaee
0.539 + 0.059
0.117 +-t 0.167
0.153 + 0.201
0.219 +-2 0.225
fork (trm(tomat)
0.790 + 0.035
0.174 + 0.229
0.156 1 0.236
0.216 1 0.269
car(snow)
0.762 + 0.061
0.175 + 0.024
0.256 + 0.279
0.224 + 0.269
Overall Test Accuracy
(dog)
0.874 + 0.025
0.774 + 0.030
0.764 + 0.021
0.795 + 0.044
(cyboarddcat)
0.839 0.012
0.714 #0.055
0.718 0.043
0.722 0.058
bed(cat)
0,820 + 0.014
0.735 = 0.064
0.698 = 0.057
0.739 = 0.068
couch(cat)
0,852 + 0.010
0.750 + 0.070
0.736 + 0.065
0.764 + 0.073
painting(lamp)
0.806 + 0.013
0.756 + 0.024
0.758 + 0.021
0.766 + 0.028
pillow(clock)
0.854 + 0.005
0.781 0.042
0.784 + 0.045
0.798 0.049
iilevisionffreplaee
0.825 + 0.009
0.755 + 0.029
0.762 0.034
0.772 + 0.039
fork(tomato)
0.738 + 0.012
0.661 + 0.036
0.661 + 0.034
0.668 + 0.040
car(snow)
0.800 + 0.007
0.699 + 0.050
0.714 + 0.054
0.707 + 0.055
Table 16: Performance PCBMs in the context of user study pruning.
26

---

Published in Transactions on Machine Learning Research (05/2024)
Scenario and Spurious classes
Scenario classes
All Scenarios classes
CIFAR100 classes
Overall
0.797 0.007
0.798 + 0.015
0.802 + 0.008
0.819 + 0.014
Test Accuracy
Shifted Class
0.864 + 0.036
0.82 + 0.051
0.852 + 0.037
0.875 = 0.024
Test Accuracy
Table 17: Top concepts retrieved based on the initial choice of classes.
bedrcom tumiture
1.04
bed foom fumntrr
8.50
bodstood
4.03
feline
4.77
headboord
4.32
heeditoard
4.34
Coppe
Concept
mattress
roclator
2.03
whisker
sylin
2.30
sharp cloms
1.05
Scorboard
2.14
1.04
#istioone
boadd
10
Weight
Weight
(a) Concepls extracted using the scenario classes ex- plained above.
(b) Coucepts extracted using the 5 scenario and spurious classes.
bodrcom fumiture
7:30
bod room fambure
22
pows
Mhisters
feine
4.29
hw
3.40
Concept
Iwedboard
beditwed
mntttross
3.46
loft
2.07
uuiiion
2.83
mullase
2.57
2:15
Weight
Weight
(c) Concepts extracted using all scenario classes.
(d) Concepts extracted using C1FAR100 classes.
Figure 10: Concept extracted from the various scenarios established.
27

---

Published in Transactions on Machine Learning Research (05/2024)
Additional examples of obtained saliency maps
Below, we attach an example of the saliency maps for a television image. The concepts used are the four most important concepts for the class "television" " and the concept floor which co-occurs with the television at the bottom of the image. The results obtained here align with the "bike" saliency map results.
input image
television eq equipment
montor
cathode-rnt tube
CLn cencole
computer
hud:peaker
micrwave
cilire
flco
Cit concppo
Figure 11: Extension results of the saliency map experiment. Shown here are ten saliency maps for an image from the class "television" of CIFAR100. Five CAV concepts and five CLIP concepts are used.
28


Reviews:
[{"review_id": "VX0Hw1bkwC", "reviewer": "Reviewer_XbjT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\n-it identifies the missing information required for reproduction and makes a plausible effort either by looking in the authors’ codebase or prior work to fill in the gap, implement, and experiment accordingly. \n- the list of required assumptions corresponding to the missing information is further provided.\n- addition of a new benchmark of a different modality (audio) is interesting\n- the conduction of a new user study helps with understanding the reliability of the original work.\n- there are informative findings regarding the reproducibility of [1]\n    - the main claim, regarding the matching performance, is generally reproducible, including on the new audio modality\n    - underperformance of the reported CUB baseline in [1]\n   - smaller improvement using manual editing of concepts.\n   - inexisting improvement and sometimes degradation of results with the userstudy, especially when it comes to non-hybrid PCBM.\n\n**Weaknesses.**\n\n- presentation can be significantly improved, some examples in the requested changes.\n- it is not clear how the use of “random projection” for the concepts is relevant for the PCBM study. \n- it is not clear how the object-concept correspondence values in Table 6 are produced.\n- it is not clear how the concepts are formed for the audio classification experiments from section 4.2.3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work is a reproduction effort of [1]. As such, it \n\n-  checks the reproducability of experimental results in support of [1]'s claims using [1]'s released codes as well as own investigation and implementation for the missing parts.\n-  conducts an independent user study mostly using the conditions of [1]'s study.  \n-  investigates the importance and validity of concepts in concept-bottleneck interpretability of [1] through an ablation study of generated concepts as well as a semantic evaluation of the discovered concepts.\n-  evaluates the method on audio data that is beyond [1]'s evaluation setup\n-  improves codebase to be more user-friendly\n\n[1] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" The Eleventh International Conference on Learning Representations. 2022.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Are the reported improvements in Table 3 (“Edit Gain”) statistically significant? Can this be repeated to make sure the differences, especially in case of pruning, is reliable?\n\n- p3. <.,.> should be defined as the dot product.\n- eq1. closing parenthesis for the loss should come after label y.\n- eq1. sampling (~) or membership ($\\in$) are usually not represented by the minus sign (-)\n- p2. if $\\mathbf{\\omega}$ is a vector, then g() can only represent a binary classifier, is that the intention? \n- eq2. parentheses need to be corrected\n- check when the differences are meant to be percentage points and use percentage points in the text instead of percent (%).\n- claim 2 discussions in p.8 do not refer to the relevant table. Related to that, it is better to have the results in Table 1 repeated in Table 2 and named differently to compare with and without provided concepts.\n- similarly, the results for the userstudy in Claim 3 do not refer to the corresponding tables.\n- for Table 5. good to have the results from “meaningful” concepts repeated here for comparison.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qKEzP9e5UO", "reviewer": "Reviewer_VnXc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Authors have successfully identified and distilled the key claims of the original paper, and systematically validated the claims through experiments. In particular, the categorization of the main points into three claims have been very effective and added clarity to the overall manuscript.\n- The paper has tediously reported small experimental details, even including the hardware details, for future references.\n- The paper accurately points out the details that have been missing in the original paper, and clearly explains how the authors approached to fill the gap.\n\n**Weaknesses**\n- Writing could have been a bit more clearer. I could not locate the main messages of the manuscript very easily. In many cases, the key sentences appeared in the middle of a lengthy section or a paragraph, instead the beginning.\n- The motivation behind the additional \"random projection\" experiment was not very clear to me. Is this merely a sanity check that the concept matrix actually plays a significant role in PCBMs? Also, it is not super clear to me why the authors brought up the J-L lemma; it adds some mathematical fanciness, but I do not think the lemma adds any concreteness to the discussion.\n- The choice to reduce the COCO-Stuff and SIIM-ISIC datasets to a smaller scale is totally understandable, but nevertheless undermines the credibility of the reproducibility study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "_Note: To my understanding, this is a submission for the reproducibility challenge (MLRC) 2023. Please correct me if this is not._\n\nThis report aims to reproduce the results of Yuksekgonul et al. (2023), which proposed post-hoc concept bottleneck models (PCBM). The authors mainly focus on validating three core claims in the original paper. (1) PCBMs closely achieves the original model's performance. (2) PCBMs can be trained without labeled concept datasets. (3) PCBMs enable global model editing. This report successfully validates the first claim, but finds that the second claim can only be partially supported. The third claim, unfortunately, has not been reproduced well. Stepping further, the paper also conducts several additional experiments on PCBMs. One of the experiments, interestingly, suggest that the interpretabiliy of PCBMs may be more limited than it looks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main issues that I have is about the clarity of the manuscript. In particular, I recommend:\n- adding several sentences to the introduction that summarizes the findings---whether the results were satisfactory, what were the key differences, etc.\n- slightly more explanations on the intentions behind the random projection experiment.\nOther than these, I am generally happy with the manuscript.", "broader_impact_concerns": "I believe that the current version is already good in this respect.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dlW3hjnduJ", "reviewer": "Reviewer_JLhh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides clear explanations of what is clear and unclear in order to reproduce the results.\n- The validation of three scenarios to verify the operation of post-hoc CBM, beyond the results shown in the paper, is highly valid. Particularly, the validation of object-concept correspondence appears crucial.\n\nWeaknesses:\n\n- The analysis of experimental results is overly simplistic. Additional discussion seems necessary.\n- Verification in other domains has been limited to performance aspects only, which is regrettable. More diverse examples and experimental validations are needed in this regard.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors reproduced Post-hoc Concept Bottleneck Models (PCBMs) to validate the experimental results presented in the original paper. This validation includes confirming whether PCBMs can 1) achieve performance similar to the original model, 2) operate without the need for labeled concept datasets, and 3) enable global model editing. Additionally, the interpretability and scalability of PCBMs were validated through a sanity check of concept vectors, assessment of alignment with actual concepts, and extension experiments into audio classification. Through these validations, this paper demonstrates that to some extent, the experimental results of previous literature can be reproduced, with the exception of global model editing. Additionally, the paper shows that PCBMs possess interpretability and scalability.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) In this paper, the authors compared the performances of their trained model with the reported performances in the original paper to initially verify if the performance is reproducible. To facilitate a proper comparison, it would be beneficial to include the performance reported in the original paper in the tables of the paper, allowing readers to compare effectively.\n \n  \n2) The interpretability aspect is crucial, and Section 3.5.2 touches on important points. However, there are considerations to be made regarding the following:\n\n[a] mentions that CAVs are sensitive to negative samples, suggesting that issues may arise not only due to the method of projection but also inherent to CAV itself.\n[b] demonstrates issues with concept association bias in vision-language models like CLIP, which could potentially enhance the discussion in Section 3.5.2.\n\n[a] Kim, Siwon, et al. \"Grounding Counterfactual Explanation of Image Classifiers to Textual Concept Space.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[b] Tang, Yingtian, et al. \"When are Lemons Purple? The Concept Association Bias of Vision-Language Models.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.\n\n3) In Figure 1, the discrepancy between CLIP concepts and CAV concepts in textual representation begs further explanation. Why are they different? The observation that saliency maps highlight primary objects rather than fine-grained parts relevant to concepts is closely related to findings in various papers analyzing CLIP. Citing these papers could enrich the discussion.\n\n4) Expanding the validation of audio classification would be beneficial. Currently, the paper only covers the use of multimodal models like AudioCLIP. Similar to CAVs, exploring the application of techniques like CAVs to audio data could provide a broader validation of Post-hoc CBM across different data domains. More examples are needed besides Figure 4, and the results presented in Section 4.2.3 only cover UrbanSound8K despite the use of various audio datasets mentioned in Section 3.5.3. Additionally, detailed explanations of the settings for PCBM-h (e.g., dimensions for the residual path) are lacking. Overall, there is a shortage of explanation and validation for these experiments.\n\n5) Further minor comments include:\n\n- Ensuring consistent terminology for the SIIM-ISIC dataset.\n- Properly indicating in the main text when referring to figures and tables included in the appendix for readability.", "broader_impact_concerns": "There is no section for Broader Impact Statement. The significance of this research lies in verifying reproducibility of Post-hoc CBM and conducting its validation in various scenarios, under the assumption that Post-hoc CBM holds significance. It would be beneficial to include a Broader Impact Statement section based on these considerations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is reviewed by three expert reviewers. During the rebuttal period, the authors addressed the concerns raised by the reviewers well.
After the rebuttal, all the reviewers were willing to accept the paper.

Following the TMLR official acceptance criteria (https://jmlr.org/tmlr/acceptance-criteria.html), I mainly focused on whether this paper has sufficient generalized insights and actionable lessons for the TMLR audience.

Note that all the experiments of this reproducibility report are based on the authors' code repository. I don't think showing the re-run results of the original code repository itself is sufficient generalized insights. Thus, my decision is based on the additional contributions of the authors. Namely, random projection experiments, object-concept correspondence, and audio classifier results. After carefully reading the reviewers' comments and the revised paper, I think the contribution of this paper is slightly above the bar of the TMLR criterion.

Specifically, I think the conclusion of this paper would be somewhat interesting for the audiences interested in applying PCBM (or any other CBM variants) to their own task rather than the well-known benchmarks (such as CUB). We still have a large gap for improving CBMs in terms of its interpretability and generalizability.
Decision: Accept (poster)