CLIP-QDA: An Explainable Concept Bottleneck Model
Authors: Rémi Kazmierczak
Eloïse Berthier
Goran Frehse
Gianni Franchi
Venue: Accepted by TMLR ()

Abstract: In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLIP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron is linked to a specific word. Observing that this latent space can be modeled with simple distributions, we use a Mixture of Gaussians (MoG) formalism to enhance the interpretability of this latent space. Then, we introduce CLIP-QDA, a classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both local and global explanations. 
These explanations come from the inner design of our architecture, our work is part of a new family of greybox models, combining performances of opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP-QDA achieves similar accuracy with state-of-the-art methods CBMs. Our explanations compete with existing XAI methods while being faster to compute.

Published in Transactions on Machine Learning Research (MM/YYY)
CLIP-QDA: An Explainable Concept Bottleneck Model
Rémi Kazmierczak
Unité d d'Informatique et d' d'Ingénierie des Systèmes ENSTA Paris, Institut Polytechnique de Poris
Eloïse Berthier
eloise. Settebbbrrriiie@estttp-prrrr
Unité d d'Informatique d' d'Ingénierie des Systèmes ENSTA Paris, Institut Polytechnique de Paris
Goran Frehse
goran.frehee@enste--poe
Unité d' d'Informatique et d'Ingénierie des Systèmes ENSTA Paris, Institut Polytechnique de Poris
Gianni Franchi
giorni.franchiienstte.eo pa
Unité d' d'Informatqque et d' d'Ingénierie des Systèmes
ENSTA Paris, Institut Polylechnique de Paris
Reviewed on OpenReview: https: // openreview. net/ form?? id= 7
Abstract
In this paper, we introduce an explainable algorithm designed from a multi-modal foundation model, that performs fast and explainable image classification. Drawing inspiration from CLiP-based Concept Bottleneck Models (CBMs), our method creates a latent space where each neuron linked to specific word. Observing that this latent space can be modeled with simple distributions, interpretability of this latent. space. Then, we introduce CLIP-QDA, classifier that only uses statistical values to infer labels from the concepts. In addition, this formalism allows for both sample-wist wise and datasct-wise explanations. These explanations come from the inner design of our architecture, our work is part of new lamily of greybox models, combining performances opaque foundation models and the interpretability of transparent models. Our empirical findings show that in instances where the MoG assumption holds, CLIP- QDA achieves similar accuracy with state-of-the-art CBMs. Our explanations compete with existing XAI methods while being faster to compute.
use Mixture of Gaussians (MoG) formalism to enhance the
Introduction
The field of artificial intelligence is advancing rapidly, driven by sophisticated models like Deep Neural Networks (LeCun et al. (015)) (DNNs). These models find extensive applications various real-world sconarios, including conversational chatbots (Ouyang et al., 2022), neural machine translation (Liu et al., 2020), and image generation (Rombach et al.j (0021). Although these systems demonstrate remarkable accuracy, the process behind their decision-making often remains obscure. Consequently, deep learning encounters certain limitations and drawbacks. The most notable one is the lack of transparency regarding their behavior, which leaves users with limited insight into how specific decisions are reached. This lack of transparency becomes particularly problematic in high-stakes situations, such as medical diagnoses or autonomous vehicles.
The imperative to scrutinize the behavior of DNNs has become increasingly compelling as the field gravi- tates towards methods of larger scale in terms of both data utilization and number of parameters involved,

---

Published in Transactions on Machine Learning Research (MM/YYYY)
culminating in what is commonly referred to as "foundation models" [Brown et al. 2020 Radford et al, 2021: Ramesh et al. 2021). These models have demonstrated remarkable performance, particularly in the domain of generalization, while concurrently growing more intricate and opaque. Additionally, there is a burgeoning trend in the adoption of multimodality (Reed et al. 2022). wherein various modalitics such as sound, image, and lext are employed to depict a single concept. This strategic use of diverse data repre- sentations empowers neural networks to transcend their reliance on a solitary data format. Nonetheless, the underlying phenomena that govern the amalgamation of these disparate inputs into coherent representations remain shrouded in ambiguity and require further investigation.
The exploration of latent representations is crucial for understanding the internal dynamics of a DNN. DNNs possess the capability to transform input data into space, called latent space, where inputs representing the same semantic concept are nearby. For example, in the latent space of a DNN trained to classify images, two different images of cats would be mapped to points that are close to each other (Johnson et al. 2016
This capability is further reinforced through the utilization of multimodality (Akkus et al., 2023). granting access to neurons that represent abstract concepts inherent to multiple types of data signals.
A class of networks that effectively exploits this notion is Concept Bottleneck Models (CBMs) (Koh et al. 2020). CBMs are characterized by their deliberate construction of representations for high-level human- understandable concepts, frequently denoted as words. Remarkably, there is a growing trend in employing CLIP (Radford et al. 2021), a foundation model that establishes a shared embedding space for both text and images, to generate concept bottleneck models in an unsupervised manner.
Unfortunately, while CLIP embeddings represent tangible concepts, the derived values, often termed "CLIP scores" pose challenges in terms of interpretation. Furthermore, to the best of our knowledge, there is table absence of studies that seek to formally characterize CLIP's latent space. The underlying objective here is to gain insights into how the pre-trained CLIP model organizes a given imput distribution. Consequently, there is an opportunity to develop mathematically rigorous methodologies for clucidating the behavior of CLIP.
Then, our contributions are summarized as follows:
We propose to represent the distribution of CLIP scores by a mixture of Gaussians. This represen- tation enables a mathematically interpretable classification of images using human-understandable concepts.
Utilizing the modeling approach presented in this study, we use Quadratic Discriminant Analysis (QDA) to classify the labels from the concepts, we name this method CLIP-QDA. CLIP-QDA demonstrates competitive performance when compared to existing CBMs based on CLIP. Notably, CLIP-QDA achieves this level of performance while utilizing a reduced set of parameters, limited solely to statistical values, including means, covariance matrices, and label probabilities.
We propose two efficient and mathematically grounded XAI methods for model explanation, named CLIP-QDAIoch local and CLIP-QDAalobul These methods encompass both global and local assessments of how the model behaves. The global metric directly emanates from our Gaussian modeling ap- proach, providing a comprehensive evaluation of CLIP-QDA's performance. Additionally, our local metric draws inspiration from counterfactual analysis, furnishing insights into individual data points.
We extend two established post-hoc XAI methods, LIME and SHAP, to formulate a novel XAI ap- proach specifically tailored for CBMs. Departing from the conventional application of these methods, which typically produce explanations on the image level, CLIP-LIME and CLIP-SHAP generate ex- planations on the concept level allowed by CBMs.
We propose a new evaluation protocol, specifically designed for the unique characteristics of CBMs lo assess the effectiveness of explanations. This protocol includes a deletion metric, which examines faithfulness to the model, and detection metric, which evaluates faithfulness to the data.
2

---

Published in Transactions on Machine Learning Research (MM/YYY)
[kg] i=
Nose
ZY=1
Ear
Score (Nose) = 23
Whechs
Score (Ear)
CLIP
Conaepts
N E1)
XY=
Score (Wheels) 15
Class diarrbbttin prior class
CLIP latent space class
Zy
Gaussial modeling
N Ee)
Images class
(Nose) 14
Class distribution prior class C
Score (Lar) =
X Xy=
CLIP
Score (Wheels) - 26
CLIP latent space class
Imagesclass
Figure Overview of our modeling method. By considering the whole dataset CLIP scores as class conditioned distributions [z] z^J, we model the CLIP latent space as mixture of Gaussians, allowing for mathematically grounded explanations.
2
Background and Related Work
2.1 Contrastive Image Language Pre-training (CLIP)
CLIP [Radford et al. 2021 is multi-modal model that jointly process image and text inputs. The model was pre-trained on large datasel of images and texts to learn to associate visual and textual concepts. Then, the capacity of CLIP to create a semantically rich encoding induced the creation of many emergent modols in detection, few-shot lcarning, or image captioning.
The widespread adoption CLIP stems from the remarkable robust ness exhibited by pre-trained model. Through training on an extensive multimodal dataset, such as SSchhmaann et al. (2022), the model achieves impressive performance. Thus, on few- and zcro-shot lcarning, for which it was designed, it obtains impressive results across wide range of datasets. Notably, CLIP provides straightforward and efficient means of obtaining semantically rich representations of images in low-dinensional spaces. This capability enables researchers and practitioners to divert the original use of CLIP to various other applications (Luo et al.
2022
Menon & Vondrick 2022 Gabeff et al.) 2023
2.2
CLiP-based Concept Bottleneck Models (CLIP- CBMs)
The term Concept Bottleneck Model (CBM), as outlined in Koh et al. 2020)) refers to to the implementation of bottleneck reliant on human-specifted concepts to execute a task, predominantly image classification. Consequently, the resultant algorithm inherently facilitates enhanced interpretability. While the term itself is relatively recent, it characterizes lincage of methods that were employed in carlier research (Kumar al., 2009 Lampert al. 2009 Koh et al. 2020 Losch et al., 2019). However, despite the advantages offered by CBMs in terms of better understanding, early implementations encountered challenges stemming from the requirement for dedicated datasets. These datasets needed to encompass not only inputs and labels but also incorporate human-specified concepts for each sample.
In this context, the emergence of multimodal foundational models has opened up novel opportunities. Recent rescarch (Yang et al. 2023 Oikarinen et al.) (023)) has leveraged large language models to directly construct concepts from CLIP text embeddings, opening the door to family of CLIP-basee CBMs. Additionally,
efforts have been made to create sparse Chaee- CBMs (Panousis et al. 2023 Feng et al. 2023). Yan
3

---

Published in Transactions on Machine Learning Research (MM/YYYY)
al.
(023a
explore methods to achieve superior representations with minimal labels. Yuksekgonul et al. capitalize on the CLIP embedding spaces, considering concepts as activation vectors. Finally, Kim
2022
et al. 2023) build upon the idea of activation vectors to discover counterfactuals.
2.3
Explainable AI
According to Arrieta et al. 20200, we can define an explainable model as a computational model, that is designed to provide specific details or reasons to ensure clarity and ease of understanding regarding its functioning. In broader terms, an explanation denotes the information or output that an explainable model delivers to elucidate its operational processes.
The literature shows clear distinction between non-transparent (or blackbox) and transparent (or whitebox) models. Transparent models are characterized by their inherent explainability. These models can be readily explained due to their simplicity and easily interpretable features. Examples of such models include linear regression (Galton 1886). logistic regression (McCullagh 2019), and decision trees (Quinlan 1986p. In
contrast, non-transparent models are inherently non-explainable. This category encompasses models that could have been explainable if they possessed simpler and more interpretable features (Galton 1886 Quinlan 1986), as well as models that inherently lack explainability, including deep neural networks. The distinction between these two types of models highlights the trade-off between model complexity and interpretability (Arriela et al. 2020 with transparent models offering inherent explainability while non-transparent models allow for better performance but require the use of additional techniques for explanations, named post-hoc methods. Commonly used post-hoc methods include visualization techniques, such as saliency maps, which highlight the inftuential features in an image that contribute to the model's decision-making Within this category of methods, notable examples include approaches such as Grad-CAM (Selvaraju et al. 2017), which generates activation maps by computing the gradients of the output labels. Sensitivity analysis (Cortez & Embreehtts 2011 represents another avenue, involving the analysis of model predictions by varying input data. Sample-wiee wise explanation techniques
also used to explain the model from local simplification of the model around point of interest (Ribeiro et al. 2016 Plumb et al.] 2018). Finally, feature relevance techniques aim at estimating the impact of each feature on the decision (Lundberg & Lee 2017
In an endeavor to integrate the strengths of both black and whitebox models, the concept greybox XAI has been introduced by Bennctot et al. (2022). These models divide the overall architccture into two distinct components. Initially, blackbox model is employed to process high-entropy input signals, such as images, and transform them into lower-entropy latent space that is semantically meaningful and understandable by humans. By leveraging the blackbox model's ability to simplify complex problems, whitebox model is then used to deduce the final result based on the output of the blackbox model. This approach yields a partially explainable model that outperforms traditional whitebox models while retaining partial transparency, in a unified framework.
Feature Attribution Methods, a category of techniques employed address the complexity of DNNs' out- put for explainability, strive to identify crucial features in the input. These methods leverage mapping function to reduce input complexity. Notable examples in this family include DeepSHAP and KerneISHAP, as proposed by Lundberg & Lee| (2017). This approach resonates with the concept of greybox models, wherein the input is initially simplified to be explainable by transparent classifier. However, greyboxes differ from Feature Attribution Methods in that the mapping function is independent from the input under consideration.
3
A Greybox Concept Bottleneck Model: CLIP-QDA
3.1
General Framework
For our experimental investigations, we consider a general framework based on prior work on CLIP-basod CBMs (Yang et al. 2023 Oikarinen et al. 2023). This framework consists of two core components. The first component centers the extraction of multi-modal features, enabling the creation of connections between
4

---

Published in Transactions on Machine Learning Research (MM/YYY)
Freze
Tainal
Nose
Far
Z
Wheels
Cat
Concepts
Score (Nose)
Score (Ear) =
Prediction
Score (Wheels) 15
Chussfier
Concepts scores
CLIP mag
Car
Dataset training Sample
Figure 2: Training procedure of the general framework. First, CLIP scores computed for cach of the concepts {k' J.V {k) then classifier (-), with parameters wh is trained to classify the label from the concatenation of the CLIP scores.
images and text. The second component encompasses classifier head. A visual depiction of this process is presented Figure 2
We build upon CLIP DNN [Rdlord et al. 2021 which enables the creation of multi-modal latent space through the fusion of image and text information. Rather than relying on single text prompt, we employ a set of diverse prompts, each representing distinct concepts. These concepts remain consistent across the dataset and are not subject to alterations. The purpose of CLIP's representation is to gauge the similarities between each concept and an image, thereby giving rise to latent space. To prevent ambiguity, we denote the resulting space of CLIP scores as the "CLIP latent space", while the spaces generated by the text and image encoders are respectively referred to as the "CLIP text embedding space" and the "CLIP image embedding space". Here, "CLIP score" denotes the value derived from cosine distance computation between the image and text encodings.
The selection of concepts is guided by expert input and acts as a hyperparameter within our framework. For comprehensive examples of concept sets, please refer to Section A.3 Notably, there is no requirement for individual image annotation with these concepts. This due to CLIP's inherent design, which allows it score concepts in zero-shot manner.
Following the acquisition of the CLIP latent space, it is given as input to classifier head, which is responsible for learning to predict the class. Thanks to the low dimension of the latent space and the clear semantics of each component (concepts), it is possible to design simple and explainable classifiers.
3.2
CLIP Latent Space Analysis
3.2.1
Notations and formalism
Let introduce the following notations used in the rest of the paper. X and Y represent two random variables (RVs) with joint distribution (X,Y) Px,.. A realization of this distribution is a pair (æ, that concretely represents one image and its label. In particular, y takes values in [1, CJ], with C' e N the number of classes. From this distribution, we can deduce the marginal distributions X N Px and Y Py. We can also describe for each class c, an RV Xy=e ~ Pxrza that represents the conditional distribution of images that have the class

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Let (k'))- 13=1 denote set of N N concepts, where each k' is character string representing the concept in natural language. We consider "CLIP's DNN" to refer to the vector of its pre-trained weights, denoted as w and function g that represents the architecture of the deep neural network (DNN). Given an image a and a concept k² the output of CLIP's DNN is represented as 2 = 9wo (x, k²). The projection in the multi-modal latent space of an image a is the vector z = | (ar, as the random variable associated with the observation 2². It should be noted that Z = [z] is the random variable representing the concatenation of the CLIP scores associated with the concepts. Furthermore, we denote the conditional distributions of z having class e as Zy= [
Iwo (ax, k k} We define Z zM]
Finally, we define the classifier as function h (z) with parameters wh that, given vector z, outputs the predicted class.
3.2.2 Gaussian modeling of CLIP's latent space
To analyze the behavior of the CLIP latent space, we conduct a thorough examination of the distribution of CLIP scores. To elucidate our modeling approach, we suggest to visualize a large set of samples from Z by observing the CLIP scores of an entire set of images taken from toy example, which consists of images reprosenting only cats and cars (sce the Cats/Dogs/ /Cars dataset in Section 4b. In this instance, the concept denoted by "j" corresponds to "Pointy-eared..
In Figure 8 which illustrates the histogram of CLIP scores, we observe that the distribution exhibits char- actoristics that can be effectively modcled as a mixture of two Gaussians. The underlying intuition here is that the distribution Z represents two types of images: those without pointy cars, resulting in the left mode (low scores) of CLIP scores, and those with pointy ears, resulting in the right mode (high scores). Since this concept uniquely characterizes the classes cats have ears but not cars specific class. This intuition is corroborated by the visualization of the distribution Zy=1 (Car) in violet and the distribution Zz=2 (Cat) in red. Since the extracted distributions exhibit similarities to normal distributions, are motivated to describe Z as mixture of Gaussians. Yet, we also discuss the validity and limitations of this modeling approach in Section 5.1
can assign each mode to
016
614
Weighted component Cat Weighted component Car Histogram Cat Histogram Car
Gaussian mixture
0.12
0.12
Histogram all classes
0.0
0.30
probabiltty
probabllity
0.08
0.08
0.08
0.06
0.04
0.64
0.82
0.07
0.0
0.0
0
10
15
20
25
30
5
10
15
20
25
30
Score
Score
(a) Modeling of Zy
(b) Modeling of Z.
Figure 3: Normalized histogram of scores Z specifically for the concept "Pointy-eared", On the left, we observe that the different classes can be modeled as weighted Gaussians. On the right, we show the resulting Gaussian mixture modeling.
Mathematically, the Gaussian prior assumption is equivalent to:
p z = Y = c) = - pe, Ec).
(1)
where Ee and Me are the mean vectors and the covariance matrices, different for each class. Morcover, given the multinomial distríbution of Y, with the notation P. P(Y = c), can model Z as mixture of

---

Published in Transactions on Machine Learning Research (MM/YYX)
Gaussians:
C (ZZ = 2) Σ
Po.N(2 0==1
pes Eei)
(2)
3.3 CLIP Quadratic Discriminant Analysis (CLIP-QDA)
Based on the Gaussian distribution assumption described Section 3.2.2 a natural choice for hig (the classifier in Figure 2 is the Quadratic Discriminant Analysis (QDA) as defined in Hastie et al. (2009). To compute it, we need estimate the parameters (Ee: Mee Pe) of the probability distributions Zy=c and Y, which is done by computing the maximum likelihood estimators the training data.
Subsequently, with the knowledge of the functions p(Z 2 Y c) and p(Y =c), we can apply Bayes theorem to make an inference on p(Y = Z = 2):
PoN Mo. 2o) E No. Mon 2o;)
p(Y = c Z 2)
(3)
Then, the output of the QDA classifier can be described as:
(w) arg max (2m) 1/2 e 112--o) E5
Pe
(4)
In practice, we leverage the training data estimate wh (Ec; Me: Pe), which enables us to bypass the standard stochastic gradient descent process, resulting in an immediate "training time". Furthermore, this classifier offers the advantage of transparency, akin to the approach outlined by Arrieta al. 2020 with its parameters comprising identifiable statistical values and its output values representing probabilities.
3.4 Explainable Al for Concept Bottleneck Models
Our CLIP-QDA model is founded upon a transparent probabilistic framework, hence, we have at disposal variety of statistical tools to explain the functioning of our classifier illustrated in Figure In this section, present two distinct lypes of explanations: CLIP-QDA global offering a global perspective that sheds light on the classifier's behavior across the entire dataset (refer to Section 3.4.1p and CLIP-QDAlace local providing a local explanation tailored to clucidate the model's actions on individual samples (refer to Scction 3.4.2
Furthermore, we propose the adaptation two well-established XAT post-hoc methods, LIME (Ribeiro et al. 2016) and SHAP (Lundberg & Lee, 2017). the unique characteristics of CBMs. These methods are denoted as CLIP-SHAP and CLIP-LIME, respectively (sce Section 3.4.3). An overview of the methods incorporated in our investigation is depicted in Figure
CLIP-QDA
Transparent Transparentexplaaattio
Post-hoc explanation
CLIP-QDA local
CLIP-LIME
CLIP-QDA global
CLIP-SHAP
Figure The different source of explanation of CLIP-QDA. Global olaebly vs. Local
(CLIP-QDA local) explanations offer insights into classifier behavior across the dataset and on individual sam- ples, respectively. Post-hoc explanations with CLIP-SHAP and CLIP-LIME use traditional XAI techniques.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
3.4.1 Dataset-wise explanation with CLIP-QDA global
As we have access to priors that describe the distribution of each class, a valuable insight to gain an understanding of which concept our classifier aligns with the measurement of distances between these distributions. Specifically, we focus on the conditional distributions of two classes of interest ei and 02; that we denote by 2yzo, and Zyzzg- The underlying intuition behind measuring the distance between these distributions is that the larger the distance, the more the attribute j can differentiate between the classes and Cz.
To accomplish this, we propose to use the Wasserstein-2 distance (Ramdas et al. 2017 as a metric for quantifying the separation wetween the two conditional distributions. It worth noting that calculating the Wasserstein-2 distance can be a complex task in general. However, for Gaussian distributions, there exists closed-form solution for computing the Wasserstein-2 distance. In addition, we sigu the distance to keep the information of the position of 01 relative to c2:
W2(Zk=o5 =q: Zz=oz) sign( - (maalen) (Qualon- - (Meloo)) + A N22,02)
Wº
where A2 = (i.)) + (Eoliss> 2/12
E (o))) Ecllis)
Note that the resulting value is no longer a distance since we lost the commutativity property. Examples of explanations based on this metric are given in Sections 5.4 and A.7 Also, an alternative way to produce dataset-wise explanations, oriented on example-based explanations, is also available in Section A.4
3.4.2 Sample- wise explanation with CLIP-QDA
local local
One would like to identify the key concepts associated with a particular image that plays a pivotal role in achieving the task's objective. To delve deeper into the importance and relevance of concepts in the decision- decision-making process of the classifier, a widely accepted approach is to generate counterfactuals (Plumb) ett al. 2022 Luo et al., 2023 Kim et al.) 2023). If a small perturbation of a concept score changes the class, the concept is considered important. We now formalize this mathematically.
Consider a pro-trained classifier denoted as h (-). In this context, wh represents the set of weights associated with the CLIP-QDA, specifically Wh = (E., Me,Pe). Given a score vector 2, we define counterfactuals as hypothetical values z + e,, ed, being called the perturbation. This perturbation aims to be of minimal magnitude and is obtained by solving the following optimization problem:
min|le_'l" s.t. ha + et) + hwr(())
(5)
The idea behind this equation is to find the minimal perturbation ei of the input z that makes the classifier produce a different label than h (z). However, in our case, two important restrictions are applied to eg:
1. Sparsity: for interpretability, we only change one attribute at a time, indicated by the index j.
Then eº = [o,.., O, el, ......... 0].
2. Sign:
we take into account the sign s E {-,+} of the perturbation. Then, we separate the positive counterfactuals e = [o, 0, 0, O..,, 0], e E R+ and the negative counterfactuals.
d' = =0 0,,,,,,,,,,000 O..., d E
These two constraints are imposed to generate concise and, consequently, more informative counterfactuals. In this context, if a solution to equation 5 denoted as ex,,, exists, it represents the minimal modification (addition subtraction) to the coordinate of the original vector that results in change from h (z) to hegh (z + c...). Note that this approach allows for an explicit evaluation of the effect of an intervention, denoted as do(Z = z + e using a common notation in causal inference (Peters et al. 2017). Concretely, this emulates answers to questions of the form: "Would the label of my cat's 's image change if T removed a certain amount of its pointy ears?".
Another important point to notice is that to obtain all possible counterfactuals, this equation must be solved for all concepts j and both signs 8. A practical way to compute counterfactuals is given below.
8

---

CLIP-LIME. To adapt LIME to the operation of CBMs, we begin with the image input x. From this input, caleulate the projection in the latent space [9 (x, ... Iws k>)]. Subsequently, following the LIME method, we train a surrogate model to approximate his in the vicinity of by training Published in Transactions on Machine Learning Research (MM/YYY)
Gaussian
Gaussian A
2
Possible solutions 2'
N
N
60
Gaussian
Gaussian
-30
2
-35
Possiole solutions
-15.
(a) In this case, the imput sample (in green) does not have counter- factuals by only changing 21.
(b) In this case, the input sample (in green) has counterfactuals ch,- and e.... Changing 21 leads to two intersections with the equiprobabil- ity line, one by adding score, the other removing score.
(c) In this case, the imput sample (in green) only has counter- factual e'., and no counterfactual e... Changing 21 leads two in- tersections with the equiprobability line, but since both by remov- ing score, only the one of minimal amagnitude conserved.
Figure 5: Visualization of the counterfactuals in the two Gaussians toy example. Samples of the two distributions are plotted in blue and orange. The equiprobability line is plotted in black.
Proposition 1. Let consider pre-trained QDA classifter ha)(() (.) with paramelers wh Assume that the input data is drawn from the corresponding Gaussian Mizture model, as defined in 4 and that e perturbation with the above sparsity and sign restrictions. Then, there is a closed-form solution to problem which is a function of the paromelers (Ee, Me: Po)
The proof and expression given in Appendix A.5 We illustrate the behavior of our classifier and our local metric with a toy example which consists of two Gaussians (C'= = 2) among two concepts Zi and Ze (N = 2). We find the counterfactuals for both signs following the first concept (c'.. and c!..). Results are presented Figure
It is worth noting that these counterfactual values are initially expressed in CLIP score units, which may not inherently provide meaningful interpretability. To mitigate this limitation, introduce scaled counterfac- tuals, denoted E s.,.eceeed: obtained by dividing each counterfactual by the standard deviation associated with its respective distribution:
(6)
VEelGa G.))
Then, the value of each counterfactual can be expressed as "the addition (or subtraction) of standard deviations in accordance to Zy (x= that changes the label". Examples of such explanations are given in Sections 5.4 and
3.4.3 CLIP-LIME and CLIP-SHAP
it on dataset comprised of perturbed inputs around z. Finally, the explanation is derived from the importance weights the resulting surrogate model.
CLIP-SHAP. To adapt SHAP to CBMs, we also consider the projection z [owo (x, k²)
(x, }^) To compute statistical values relevant to CLIP-SHAP, we incorpo- rate the projections of all the images in the training set. Given that the number of concepts can be high, we employ the Kernel version of SHAP (Kernel SHAP) reduce the computational cost. The resulting explanation comprises the Shapley values associated with each of the concepts.
9

---

Published in Transactions on Machine Learning Research (MM/Y YYY)
4 Experimental Setup
4.1 Datasets
We evaluate our methods ImageNet (Deng et al. 2009), PASCAT-PPart DDnadeello & Serafini 2016), MIT Indoor Scenes dataset QQuattoni & Torralba 2009), MonuMA1 Lamas et al. 2021] and Flowers Nilsbbcck & Zisserman 2008). In addition to these well-ostablished datasets, we introduce a custom dataset named Cats/ Cats/Dogs/ Cars dataset (Section A.2). To construct this dataset, we concatenated two widely recognized datasets, namely, the Kaggle Cats and Dogs Dataset (Cukierski] 2013) and the Standford Cars Dataset (Krause et al,, 2013). Subsequently, we filtered the dataset to contain images of white and black animals and cars exclusively, This curation resulted in six distinct subsets: "Black Cars", "Black Dogs", "Black Cats", "White Cars", "White Dogs", "White Cats". The primary objective of this dataset is to facilitate experiments under conditions of substantial data bias, such classifying white cats when the training data has only encountered while dogs and black cats. Specifically, one containing cats and cars of both colors (referred to as the unbiased setup) and the other one with only black cats and white cars (referred to as the biased sctup). When none of theses scenarios are referred to (for example in the Del values of Table and 50, the complete dataset is used. In its final form, the dataset comprises 6,436 images. Additional information is available in Section A.2
refer to two distinet scenarios (Table D
Table 1:
/Coog//rrrrs// Cars compositions used in our study
Complete dataset
Biased setup
Unbiased setup
Composition
Black Cats, White Cats, Black Cars, White Cats, Black Dogs, White Dogs
Black Cats, White Cars
Black Cats, White Cats, Black Cars, White Cars
Num samples
6436
2536
4031
4.2 Baselines
Classifiers. Here, we present the baseline used for comparing performance in terms of inference (i.e., accuracy) with other algorithms performing classification tasks. First, we evaluate various classifiers, which are represented as the trainable component illustrated in Figure 2 We denote the method proposed by Yan et al., which involves training linear layer classifier from CLIP scores, one of the baselines. As used notably in (Yan et al. 2023a b). this approach is referred to as linear probe. Additionally, we adopt LaBo, introduced by Yang et al. (2023). which employs a class-concept matrix as the classifier of the CBMs. For all these methods, the same set of concepts is used, with a comprehensive description of the procedure for acquiring the concept set provided in Appendix AA.3] We also incorporate previous CBMs such as Greybox XAI (Bennetot et al.) )002), X-NeSyL (Diaz-Rodríguez et al. (2022,, and the method proposed by Morales Rodríguez et al. (2024) into our study, which provides explainability but requires additional training data related with annotated concepts. Tt is noteworthy that these methods, requiring training from assess methods widely used classifiers
images, entail significantly longer training times. Additionally, that are not CBMs, including the use of CLIP as a zero-shot classifier. We also include ResNet (He et al. 2016) and ViT (Dosovitskiy et al.) 2021) trained in a supervised manner using images as inputs. Further implementation details can be found in Section A.1
XAI Methods. To compare our proposed XAI methods (CLIP-QDA CLIP-QDA CLIP-LIME, and CLIP-SHAP) with existing works, we evaluated several methods, including those providing explanations at both the image and concept levels. At the concept level, we examined explanations provided by LaBo (Yang et al. 2023 and Yan al. (Yan et al., (002bb]. For image-based explanations, we assessed SHAP Lundberg & Leek 2017) and LIME (Ribeiro et al. 2016). Further implementation details can be found in
Section A.1
10

---

Published in Transactions on Machine Learning Research (MM/YYX)
4.3 CBM Quantitative Evaluation Process
Assessing the quality of explanations has long been challenge, given its subjective nature. While quanti- tative evaluation processes exist for evaluating performance in general setup (Hedström et al. 2023). they may not be well-suited to the specific characteristics of CBMs. Consequently, we present a novel method designed to evaluate XAI solutions, tailored to CBMs. This method comprises two metrics: the Deletion metric, gauging faithfulness to the model, and the Detection metric, quantifying faithfulness to the data.
Deletion metric. The deletion metric is adapted from the methodology introduced by Petsiuk et al. (2018). The procedure involves taking each sample from the test set and nullifying (i.e., setting the averago value of the score across classes) a cortain number, Nuult, of concepts. We mullify the concepts basod their importance order determined by each explanation method. If nullifying the concepts leads to significant decrease in performance, classifier's 's decision. The intuition behind this idea is that if the concept is important, its absence will result in a loss of performance. Given the different values of accuracy obtained by nullifying Naut concepts, denoted by Aco(Nautl) Nau)), we deduce the deletion score Del by computing the area under the curve of Ace(Nauti The interest to probe the ability to correctly order the important concepts in the explanation:
consider it successful selection of concepts that influenced the
N Accli 1) + Acc(i)
Del Aec(O)
2
Here, Nmar represents a hyperparameter that defines the maximum number of delctions considered. The selection of this hyperparameter is critical: it balances between capturing the metric's capacity to fit the model and ensuring that the computed inputs remain plansible. A default value of 9 is assigned to maintain this equilibrium. Note that we normalize the result by the maximum accuracy Acc(0) to allow for better comparisons among different classifiers.
The evaluation framework is structured into two sctups: the first uses the concept set outlined in Table 0 (referred to Set 1), while the second one uses equivalent number of concepts randomly chosen from dictionary of words (referred to as Set 2).
Detection metric. The detection metric evaluates the model's ability to identify relevant concepts. For each sample s, a set of ground truth concepts S, to detect is defined by an oracle. Then, we construct a set of concepts T, consisting of the top [S,] concepts in the explanation to be tested. The resulting detection metric is the average ratio of agreements among all samples the lest datasel:
S
S.O S.
Det
(8)
where S is the total number of samples in the dataset. The selection of ground truth concepts depends on the available options within each dataset, as elaborated in Section 5.3
5
Experiments
5.1 Assessing the Gaussian Prior Hypothesis
In this section, investigate to which extent the Gaussian prior hypothesis (Equation holds. To assess this, we use Chi-Square Q-Q plots (Chambers) 2018 Mahalanobis] 2018), normality assessment method adapted to multidimensional data. We display Chi-Square Q-Q plots on the conditional distribution Y = c), with data sourced from PASCAL-Part and class c = "aeroplane". First, we compute a set of concepts adapted to the output class, following the procedure described in Appendix [A.33 Subsequently, we perform the same experiment with a set of words specifically dedicated to the class of interest (Figure (Ga) In addition, we show visualization with randomly chosen concepts from the PASCAL-Part set of concepts, depicted in Figure Gb
11

---

Published in Transactions on Machine Learning Research
0.20
probabliity
0.15
0.10
0.05
Gaussian fitting
Histogram
15
20
25
30
Score
Figure 7: Histogram and Gaussian fitting of CLIP scores of the attribute "Multi-doored", for images that have the class "acroplanc".
Chi-Square Q-Q Plot
Chi-Square Plot
312
Chi-Square Quantile
Chi-Square Quantile
154
x
Squared Mahalanobis
Squared Mahalanobis Distance
(a) CBM with of concepts related to the label " "Aeroplane"; [Winged, Jet en- gines, Tail fin, Fuselage, Landing gear]
(b) CBM with of concepts unrelated to the label "Aeroplane": [Furry, Equine,
Container or pot, Saddle or seat, Multi-doored]
Figure Multivariate Q-Q plot illustrating the Gaussian fit for conditional modeling of the latent space, given fixed label. The plot is specifically for PASCAL-Part images labeled "Aeroplane'.
Notably, employing a less precise set of concepts can introduce disturbances, as shown the observations. As indicated in Scction 3.2.2 the ambiguity associated with certain concepts, such as "Multi-doorod", can lead to bimodal distributions (an airplane having one, multiple, or no doors). In Figure 7 we show the histogram of the clip scores of the images that have the class "aeroplane". Compared to the histogram of less ambiguous cases (like in Figure 3). we observe that the histogram presents anomalies, especially around the mean.
Additionally, we performed a similar experiment with larger sets of concepts. We selected random subsets containing 10, 15, and 20 concepts from the PASCAI-Part set listed Table The outcomes are displayed in Figure S In this scenario, becomes obvious that the Gaussian assumption increasingly violated the number of concepts grows. Indeed, the number of concepts increases, the likelihood of encountering
12

---

Published in Transactions on Machine Learning Research (MM/YYY)
ambignous concepts in each sample significantly increases, which undermines the feasibility of modeling the data an unimodal Gaussian distribution.
Chi-Square Quantile
Chi- Squure Quantile
Chi-Square Quantile
Squared Mahalanobis Distance
Squared Mahalanobis Distance
Squared Mahalanobis Distance
(a) CBM with 10 concepts
(b) CBM with 15 concepts
(c) CBM with 20 concepts
Figure 8:
Multivariate Q-Q plot illustrating the Gaussian fit for conditional modeling of the latent space, given a fixed label. The plot is specifically for PASCAL-Part images labeled as 'Acroplane'.
5.2 Assessing the Accuracy
Comparison with other classifiers. In this section, we undertake a comparative evaluation of the performance of our CLIP-QDA classifier in contrast to classifiers used in the bascline (sec Section 4.2) Results of these experiments available in Table 2 The image encoder used in all our experiments based CLiP-based CBMs is Vi7 I.114336ppz provided by OpenAT's public repository.
Test set accuracy. the top are methods that require full training on images, and on the bottom, CBMs. Because Greybox XAI, X-NeSyT. and Morales Rodríguez al. (2024) require concept annotations, their results on MIT scenes and ImageNet are available.
Table 2:
Method
PASCAL-Part t MIT scenes t MonuMAI 1
ImageNet
Non XAI
Resnet
0.84
0.86
0.95
0.80
He al. 2016
VIT-L 336px
0.95
0.94
0.94
0.85
Dosovitskiy et al. 2021
Greyboz XAJ
0.88
0.94
Bennetot et al 2022
X-N 2029
0.82
0.90
Diaz- Rodríguez et al.
CBMs
Morales Rodríguez al.
2024
0.86
0.98
CLIP (zcro-shot)
0.81
0.63
0,52
0.76
Rodford et al. 2021
LaBo
0.83
0.75
0.74
0.69
Yang et at 2023
Von et al. 2029h
0.91
0.77
0.77
0.81
Yan et al.
20236
CLIP-QDA (ours)
0.90
0.81
0.89
0.60
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Our findings reveal that fine-tuning using CBM, either a linear QDA probe, significantly improves performance, as evidenced by the increase in accuracy compared to using CLIP as a zero-shot classifier This improvement is particularly pronounced on datasets dedicated to specialized tasks, such as MonuMAI. Additionally, CLIP CBMs tend to achieve performances comparable to networks trained from raw images, making these models appealing for image classification due to their reduced training cost both time and resources, as well as their interpretability. Notably, CLIP-QDA demonstrates competitive performance, when compared to linear probe techniques.
However, CLIP-QDA faces challenges in delivering competitive results for datasets like PASCAL-Part and ImageNet, which feature significantly larger number of labels and concepts. The accuracy decline could be attributed to the use of a broader set of concepts tailored specifically for these datasets. This challenges the Gaussian assumption and potentially impacts the effectiveness of our classifier. Notably, there appears be a correlation between the tests conducted on the different datasets and the obsorvations depicted Figure [ For instance, while MonuMAI uses 20 concepts, MIT scenes use 25, PASCAL-Part employs 80, and ImageNet involves staggering 5000 concepts.
Influence of the number of concepts. To assess the influence of the number of concepts C' on the accuracy, we conduct experiments on the PASCAL-Part datasot. These experiments involve testing accuracy for both QDA and linear probe with concept sets of different lengths, all generaled following the methodology described in Section AA.3 As seen in Figure 9 CLIP-QUI Q.lP--DDerrformss better than linear probe when the number of concepts is relatively low. In contrast, the linear probe outperforms CLIP-QDA the number of concepts increases. This observation aligns with the insights gained from the discussion on Gaussian modeling in Section 5.1. where a higher number of concepts challenges the grounding assumptions of CLIP-QDA.
0.90
0.88
Test accuracy
0.86
0.84
0.82
0.80
QDA
Linear probe
0.78
20
Number of concepts
Figure 9: Accuracy of the classifiers on PASCAL-Part for different concept sizes. QDA refers to the use of CLIP-QDA as classifier. Linear probe refers to the use of a linear layer as a classifier.
5.3 Assessing the Sample-wise Interpretability of XAI Methods
The objective of this subsection is to present our method within the context current XAI methods (see Scction 4.2 applicd CBMs. comparative overview of the distinctive features of these methods is presented Table Since Greybox XAI, X-NeSyL and Morales Rodríguez et al. (2024) require additional annotations, chose to omit, these methods from our study.
14

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 3: Comparison the different features of the XAI methods. Top: existing methods. Bottom: ours. Dataset- wise refers to methods that provide dataset-wise explanations. Sample-wise refers to methods that provide sample-wise explanations. Closed-form solution refers to methods that do not require an optimization process to produce explanations. Computable from weights refers to methods that produce explanations from the model parameters and imput values. Image level refers to methods that produce explanations from the image imput. Concept level refers to methods that produce explanations from the concept input. Compatible with CLIP-QDA refers to methods that can produce explanations with CLIP- QDA as a classifier. Compotible with CLIP linear probe refers to methods that can produce explanations with CLIP linear probe as a classifier.
Compatible with CLIP-QDA
Compatible with CLIP linear probe
Metric
Datasct
Sample wise
Closed-form solution
Computable from weights
Image level
Concept lenel
wisee
GradCAM
LIME
SHAP
LaBo
Yan et al
Greybox XAI
X-NeSyL
Morales et al
CLIP-LIME
CLIP-SHAP QDA-CBM
Qualitative analysis. To facilitate the comparison of various explanations generated for the same sample in the dataset, we computed the results for one image from PASCAL-Part (Donadello & Seralini 2016). The results are categorized into two sections: image-level explanations and concept-level explanations. The subsequent section focuses on explaining the model's prediction process given the image [0a] as the input, which is labeled as "person". The top two predicted labels by the classifier for this image are "person" and "potted plant". Note that additional samples are available in Section A.7
Image-level explanations are presented in Figure 10 For each of these samples, a weight assigned to each pixel. All the results are computed by applying these post-hoc explanation methods on CLIP-QDA. Additional implementation and computation details be found in A.1 While the results easy to comprehend, criticism may arise due to potential misunderstandings regarding the specific focus of the method. While these methods successfully highlight the object of interest, it is challenging to discern the exact pattern that the method emphasizes, Another observation is that, compared to classical classifiers, the results on CLIP-based CBMs are more imprecise, highlighting large areas interest.
(a) Input image
(b) GradCAM Explana- tion
(c) LIME Explanation
(d) SHAP explanation
Figure 10: Sample-wise explanations (image level). Note that the classifier classified the image cor- rectly.
15

---

Published in Transactions on Machine Learning Research (MM/YYN)
Concept-level sample explanations computed in Figure I1 On sample-wise basis, CLIP-LIME and CLIP-SHAP are computed using CLIP-QDA. Given that the method is model-specific, the Yan et al. method is computed on a linear classifier.
Human Clargel cart
Facial features <e 13.79
Counterfactua
Leaves (Target pottedplanti
Transparent translucent < 15.82
Concept
Maned ITbrget bird)
Landing gear <= 14.15
Hormed (in cases) (larget bird)
Woolly 14.66
Tall 13.98
-0.02
Deviation
Lime magnitude
(a) CLIP-QDA locul explanation. The first row can be read follows: removing little of the con- cept "Human" the vector z would change the label to "Car".
(b) CLIP-LIME explanation. The first row can be read as follows: the fact that the concept score of "Facial features" is below 13.79 has a negative impact the predicted label.
Posture
Concept
Facial features
Stand or wall mounted
Landing gear
-1
2
Weight value
(c) Yan et al. explanation (sample). The first row can be read as follows: concept "Posture or stance" has positive impact on the predicted label.
Nigher
base value
1.00
-0.5
6.26
1.3
(d) CLIP-SHAP explanation. The highest value can be read follows: the concept "Fuselage" has positive impact on the predicted label.
Figure 11: Samplo-wisc explanations (concept level) of the image 10a Except for CLIP-SHAP,
only the top 5 concepts are displayed. Note that the classifier correctly labeled the image.
16

---

Published in Transactions on Machine Learning Research (MM/YYYY)
First, notice that there is high diversity of concepts displayed among the methods. Overall, there are concepts associated with the prediction label ("Posture or stance", "Human") and concepts that arise from the specificities of the image ("Leaves"). Some concepts seem irrelevant ("Landing Gear", "Windshield", "Wooly"), which may be explained by the fact that their scores are useful to the classification process despite their absence in the image. In addition, is worth noting that the context provided by the counterfactual analysis helps to unveil insights into the reasons behind the explanation. For example, when taking into account the concept "Leaves" seems inappropriate, CLIP-QDA local explanations suggest that adding that concept inftuences the prediction of the class "potted plant," " indicating that this concept is important because it reveals the decision process behind predicting a person rather than plant.
Quantitative analysis. To conduct more comprehensive analysis beyond qualitative observations, we subject the various methods to quantitative evaluations. We apply the procedure outlined in Section 4.3 to the PASCAL-Part and CBM explanations, we also tested the explanations provided by LaBo and Yan al. For the selection of S, in Equation [8] in the case of the PASCAL-Part dataset, we opted to designate the concepts to detect the concepts that are associated with the two highost probabilities of the model's inference (refer to the /Concepls association in Table G). In the Cats/Dogs/ /Cars dataset, we chose to identify the concepts "Black" and "White" in the biased setup to spot potential biases. The resulting Det score for this dataset is calculated by considering S. = [Black, White for classifiers trained with biased setups as described in [] To reduce uncertainties, we do not only take the Det score for the set [Black Cats, White Cars] but also for all possible similar biased binary classification tasks (Black Cats, White Cars], [Black Dogs, White Cars), etc.). Additionally, we incorporated inference time as parameter in our experiments referring to the time taken to produce explanations inferences the entire validation set. Results PASCAL-Part are displayed in Table and results Cats/Dogs/ Dougs/Car Cars are displayed in Table
datasets. In addition to CLIP-LIME, CLIP-SHAP, and QDA-
Table 4: Quantitative results for different XAI methods (Cats/Dogy /logs/Cars CCars dataset). Top: existing methods. Bottom: ours. Del refers to the deletion score (7) on random set of concepts (Set 1) and the set of concepts defined in 6 (Set 2). Det refers to the detection score (S. The explanations of the three upper
methods being direct, they present no computation time.
Method
Del (Set 1) + Del (Set 2) |
Det f
Inference time (im/
Yan et al. LaBo
0.6253
0.7522
0.3183
0.5971
0.6224
0.2140
Random
0.8278
0.7474
0.1171
CLIP-QDA local (ours)
0.7609
0.5820
0.2724
3.01
CLIP-LIME (ours)
0.5397
0.5646
0.4042
76.69
CLIP-SHAP (ours)
0.4821
0.3831
0.3696
256.76
Regarding the Cats/Doge Carr dataset, our methods exhibit superior performance compared to the state of the art, emphasizing the reliability of CLIP-QDA on datasets with a low number of concepts. Particularly, QDA-CBM demonstrates significantly faster inference times than CLIP-LIME and CLIP-SHAP, albeit the expense of slightly lower deletion and detection scores. On datasets with higher mumber of concepts and classes, such as PASCAT-Part, existing methods maintain higher scores.
5.4 Assessing the Dataset-wise Interpretability of XAI Methods
5.4.1 Evaluation of CLIP LLIP-DAAstee global on the PascalPART dataset
Following the setup expressed in Section 5.3 for dataset-wise explanations (Figure 12 we present the explanations for the CLIP-QDA method, computed on CLIP-QDA, the LaBo explanation, computed on the LaBo classifier, and the Yan al. explanation, computed on linear classifier. Additional samples are available in Section A.7
17

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 5: Quantitative results for different XAI methods (PASCAL-Part dataset). Top: existing methods. Bottom: ours. Del refers to the deletion score ( on random set of concepts (Set 1) and the usual of concepts defined in 6 (Set 2). Det refers to the detection score The explanations of the
three upper methods being direct, they present no computation time.
Method
Del (Set 1) 4
Del (Set 2)
Det 1 Inference time (im/s)
Yan et al. LaBo
0.6968
0.8824
0.4157
0.4446
0.8822
0.4213
Random
0.8452
0.8999
0.1258
CLIP-QDA local (ours)
0.7983
0.8189
0.1564
2341.12
CLIP-LIME (ours)
0.7313
0.8511
0.0873
2857.28
CLIP-SHAP (ours) 0.5698
0.5510
0.2679
7207.58
We observe that the concepts obtained from all the dataset- wise methods are generally coherent, including concepts commonly associated with persons, such as "Posture or stance" or "Facial features". However, in contrast to the other two methods that impose positive magnitudes, the CLIP-QDAstoba explanation allows for negalive values, providing insights into whether the impact negative or positive. Additionally, our CLIP-QDAstobe method emphasizes the comparison between two classes of interest, enabling detailed exploration of specific disparities, like here between "potted plant" and "person".
5.4.2 Evaluation of CLIP-QDA global on the Cats/Dogs/ Cars dataset
We also present a toy example from our Cats/Doge / Cars dataset by constructing a CBM consisting of the concepts Table plus the concepts "Black" and "White". Next, display the 10 most influential concepts according to our global metric (the top 10 concepts that have the highest Wasserstein distance) in both biased and unbiased selups. Results are presented in Figure 13
We can observe that the concepts "Black" and "White" hold significantly higher importance in the biased sctup, indicating that the classifier is likely to be biased about these concepts. This shows that our global explanation method has the potential for detecting biases in dalasets (Tommasi et al., 2017). Additional explanation samples for various use cases are available in Appendix A.7
Finally, show application of our local metric within the framework of our biased setup, previously described. Subsequently, we feed an image of a white cat into our classifier (Figure 14h. It is noteworthy that the image is misclassified as a car. Our local metric demonstrates sensitivity to the dataset's 's color bias, corroborating the warning issued by the global explanation.
18

---

Published in Transactions on Machine Learning Research
Contal ner or pot
Displaying image
Soil potting mix
Posture or stance
Concept
Concept
Leaves
Manad
Potted
Large
Posture stance
4
3
-2 1
10
Wassertein Distance
Weight value
(a) CLIP-QDA global explanation associated with (b) LaBo explanation associated with the top pre- be read fol-
the top two predictions (*person" and "potted diction (*person"). The first plant"). The first row can be read as follows: the dis- lows: the weight associated with the concept "Displaying tribution of the concept. "Container pot" for the class image" has the highest value among the weights "person" is mostly to the left (smaller values) of the one lated to the class "person". for the class "potted plant".
Posture
Facial features
Concept
Landinn gear
Snout
0.05
0.10
0.25
0.30
Weight value
(c) Yan et al. explanation (dataset) associated with the top prediction ("person").. The first be read as follows: the weight associated with the con- cept "Posture or stance" has the highest value among the weights related to the class "person".
Figure 12: Dataset-wise wise explanations of the image 10a Only the top concepts are displayed. Note that the classifier correctly labeled the image.
19

---

Published in Transactions on Machine Learning Research (MM/YYY)
Whikered
Wiite
Poilty-aaned
Pomyycares
Concept
Happy
Concept
Pourwheoled
Sitted-eyed
Four aehi ed
Furry
Flapsy-sored
bourlegged
Siitiad eyed
White
Burry
Wassertein Distance
Wassertein Distance
(a) Unbiased setup
(b) Biased setup
Figure 13: Global explanation on subsets of Cats/ Calis/Dogs/Crat Here, q==Catt and c2="Car". Positive values indicate concepts that are more prevalent cat images than car images, while negative values indicate concepts that more common in car images compared to cat images. We display here only the top 10 concepts that have the highest Wasserstein distance (the concept "Black" is positioned 15th in the unbiased setup).
White (Target
Counterfactua
(Target Cat)
Slittad ayed (arget - Cat)
Four-Wheeeed (Target = Cat)
Deviation
(b) CLIP-QDA local explanation. Only the top 5 concepts are displayed. Note that classifier mis- classified the image "Car". The first row must read as follows: removing a little the concept "White" to the vector induces change of label to "Cat".
(a) Input image
Figure 14: Local explanation on subsets of Cats/Doge DDoss/Crss.. On the right figure, the blue/ret scale represents the scaled counterfactuals as in equation 6 the text each box corresponds to the label predicted after the perturbation, followed lo the concept changed to obtain its result. (in parentheses).
20

---

Published in Transactions on Machine Learning Research (MM/YYY)
6 Conclusion
In this paper, we introduce modeling approach for the embedding space of CLIP, foundation model trained on vast. dataset. Our observalions reveal that CLIP can organize information in distribution that exhibits similarities with a mixture of Gaussians. Building upon this insight, we develop an adapted concept bottleneck model that demonstrates competitive performance along with transparency. While the model that we have presented offers the advantage of simplicity and a limited number of parameters, it does encounter challenges when dealing with a broader and more ambiguous set of concepts. As a suggestion for future research, we propose extend this modeling approach incorporate other priors, such as Laplacian distributions, and to explore more complex models, including those with multiple components, f.e., using more than one Gaussian to describe class for example. Another avemue to potentially enhance the perfor- mance of our model is to explore guiding the latent space of CLIP's scores towards a Gaussian distribution. Additionally, our research is centered around a specific embedding space (CLIP scores). Exploring similar work on other latent spaces, particularly those associated with multimodal foundation models, could be valuable to determine similar patterns exist in those spaces.
7 Acknoledgements
This work was performed using HPC resources from GENCI-IDRIS (Grant 2023 AD011014675).
References
Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Mauch-Walsed Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schncider, et al. Multimodal deep lcarning. arXiv preprint Xrv::22001.00 04856, 2023.
Alejandro Barredo Arrieta, Natalia Díaz- Rodriguzz, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Gareía, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information. Fusion, 58:82-118 115, 2020.
Adrien Bennetot, Gianni Franchi, Javier Del Ser, Raja Chatila, and Natalia Diaz- Ridz-Rodrig Greybox XAI: A neural-symbolic learning framework to produce interpretable predictions for image classification. Knowledgge-Bel -Based Systems, 258: 258:109947, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877 1901, 2020.
John M. Chambers. Graphical Methods for Data Analysis. CRC Press, 2018.
Paulo Cortez and Mark J. Embrechts. Opening black box data mining models using sensitivity analysis. In 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pp. 341 348. IEEE, 2011.
Will Cukierski. Dogs vs. cats, 2013. URI. http:///aaggggaaoooooooooooto/t/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Natalia Díaz Roaz-Rodrgg Alberto Lamas, Jules Sanchez, Gianni Franchi, Ivan Donadello, Siham Tabik, David Filliat, Policarpo Cruz, Rosana. Montes, and Francisco Herrera. Explainable neural-symbolic learning (x- nesyl) methodology fuse deep learning representations with expert knowledge graphs: The monumai cultural heritage case. Information Fusion, 79:58 83, 2022.
21

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Ivan Donadello and Luciano Serafini. Integration of numeric and symbolic information for semantic image interpretation. Intelligenza Artificiale, 10(1):33-47, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Rep- resentations, 2021.
Zhili Feng, Anna Bair, and J. Zico Kolter. Leveraging multiple descriptive features for robust few-shot image learning. arXiv preprint arXiv:2307.04311 2023.
Valentiu Gabeff, Mare Russwurm, Devis Tuia, and Alexander Mathis. Wildelip: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models. bioRriv, pp. 2023 12, 2023.
Francis Galton. Regression lowards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15:246-268 1886.
Jacob Gildenblat and contributors. Pytorch library for cam methods.
pytorch- wrtorch-grad-cat 2021.
Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Dala Mining, Inference, and Prediction, volume 2. Springer, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770 778, 2016.
Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojeiech Samek, Sebastian Lapuschkin, and Marina Marina M.-C. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34): 1-11, 2023. URL. http://jmlr org/ http:/apprrs/vvv/ 2//////22222220002 html
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Port II 14, pp. 694 711. Springer, 2016.
Siwon Kim, Jinoh Oh, Sungjin Lec, Scunghak Yu, Jacyoung Do, and Tara Taghavi. Grounding counterfactual explanation of image classifiers to lextual concept space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 10942 -09500, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338 5348, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- 3d object representations for fine-grained catego- rization. In 4th International IEEE Workshop on 3D Represenlation and Recoguition (3dRR-13) Sydney, Australia, 2013.
Neeraj Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th International Conference on Computer Vision, pp. 365-372, IEEE, 2009.
Alberto Lamas, Siham Tabik, Policarpo Cruz, Rosana Montes, Álvaro Martinez-Sevilla, Teresa Cruz, and Francisco Herrera. Monumai: Datasct, deep learning pipelinc and citizen science based app for monumental heritage laxonomy and classification. Neurocompuling, 420:266 280, 2021.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference Compuler Vision and Pollern Recognition, pp. 951-958. IEEE, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(3333) 444, 2015.
22

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural machine translation. ar XrXiv preprint arXim:2008.0711 2020.
Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic bottleneck networks. Xrxi preprint arXi:: 19x. 10882, 2019.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end vidco clip retricval and captioning. Neurocomputing, 508:293-300 -044, 2022.
Jinqi Luo, Zhaoning Wang, Chen Henry Wu, Dong Huang, and Fernando De la Torre. Zero-shot model diagnosis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
11631-11640, 2023.
Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Sankhyā: The Indian Journal of Slalisties, Series A (2008-), 80:S1 17, 2018.
Peter McCullagh. Generalized Linear Models. Routledge, 2019.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2022.
David Morales Rodríguez, Manuel Pegalajar Cuellar, and Diego P Morales. On the fusion of soft-decision- trees and concept-basse based models. Available at SSRN 4402768, 2024.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, 12 2008.
Tuomas Oikarinen, Subhro Das, Lam Nguyen, and Lily Weng. Label-free concept. bottleneck models. Tn International Conference on Learning Representations, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Konstantinos P. Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2767-2771, 2023.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized input sampling for explanation of black-box models. Xi preprint arxi 180::8800.7421 2018.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. Model agnostic supervised local explanations.
Advances in neural information processing systems, 31, 2018.
Gregory Plumb, Marco Tulio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:
Expert Certification.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pallern Recognition, pp. 1134-00..22 IEEE, 2009.
J. Ross Quinlan. Induction of decision trees. Machine Learning, 1:81-106, 1986.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknnn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 18638,77, 2021.
23

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Aaditya Ramdas, Nicolás García Trillos, and Marco Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott. Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever, Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831, 2021.
Scott Reed, Konrad Zolna, Emilio Parisolto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research, 2022.
Marco Tulio Ribciro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVF Conference on Computer Vision and Pattern Recog- nilion (CVPR), pp.
2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Newral Information Processing Systems, 35:25278-25222 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. Domain Adaptation in Computer Vision Applications, pp. 37 37-55, 2017.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 3090 3100, 2023a.
An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingbo Shang, al. Robust and interpretable medical image classifiers via concept bottleneck models. arXiv preprint arXi::2310.0118 2023b.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2022.
A Appendix
A.1 Implementation details
GradCAM. To compute GradCAM explanations, we applied the method to the 2h block of the image encoder CLIP-QDA, using the PyTorch package pytoreh-grad-cam) (Gildenblat & contributors) 2021) Specifically, we generated the heatmap by backpropagating the gradient from the classifier's inference class, considering the entire network (CI.I+tcaassfiier).
24

---

Published in Transactions on Machine Learning Research (MM/YYYY)
LIME. The LIME image-level explanations are generated using the superpixel approach defined in (Ribeiro et al. 2016) using the official author's repository All of our samples are generated using the default parameters for image-level explanations, i.e. an exponential kernel of width of 0.25 on an image segmented using quickshift clustering. Concerning the visualization, cach explanation underlines the top 10 superpixels that exert the most influence on the prediction. Superpixels that positively contribute to the prediction of the label are in red and superpixels that positively contribute to the prediction of the label are in blue.
The SHAP image-level explanations are generated using the partitioning approach defined in the official author's repository All of our samples are generated using the default parameters for image-level explanations, i.., running blur image masker to partition the imput image with rectangular masks. The estimation of Shapley values is obtained by running 200 evaluations. Partitions that positively contribute to the prediction of the label are in red and superpixels that negatively contribute to the prediction of the label are in blue.
SHAP.
CLIP-LIME. The CLIP-LIME explanations are generated using the approach used for tabular data as defined in (Ribeiro et al., 2016 using the jofficaal author's repository All of our samples are generated using the default parameters for tabular-level explanations, i.e., an exponential kernel of width 0.75xN, (N being the number of concepts).
The CLIP-SHAP explanations are generated using the approach used for tabular data in the official author's repository) All of our samples generated using the default parameters for explanations, i..., DcepLIFTT algorithm using the training data as a background dataset.
CLIP-SHAP.
LaBo. For LaBo, we train a LaBo classifier by using Adam optimizer with a learning rate of 0.5, a weight decay of O, and a batch size of 8192. The resulting explanations follow the resulting weight matrix.
Yan et al. For the Yan et al. method, we train a lincar classifier by using Adam optimizer with a learning rate of 10-3, a weight decay of 10 10-4, and a batch size of 512. The sample-wise (concept level) explanation results from the product between the concept score and associated weight. The dataset-wise explanation results from the weights alone.
Resnet. To train the Resnet classifier on PASCAL-Part, MIT scenes and MonuMAT, initialized the network with ImageNet pertaining. Then, trained the network using Adam optimizer with learning rale of 10-3 for the probe, 10-1 for the backbone, batch size of 64, momentum of 0.9 and weight decay of 10-4, For ImageNet results, we use results provided by the authors.
VIT. To train the ViT classifier on PASCAL-Part, MIT scenes and MonuMAI, we initialized the network with ImageNet pertaining. Then, we trained the network using Adam optimizer with a learning rate of 10-2 for the probe, a frozen backbone, a batch size of 128, a momentum of 0.9, and no weight decay. For ImageNet results, we use results provided by the authors.
A.2 Cats/Dogs/Cars
The Cats/Dogs/ Dogs/Cars Cars dataset is available on hugging face To create this dataset, we combined two well- known datasets: the Kaggle Cats and Dogs Dataset (Cukierski 2013) and the Standford Cars Dataset [Krause et al., 2013). We then filtered the dataset include only images featuring black and white animals and cars, leading to six different categories. The primary objective behind assembling this dataset is facilitate research in scenarios characterized by significant data bias, such as the classification of white cats when the training data predominantly consists of images of white dogs and black cats.
A.3 Set of concepts
Inspired by Yang et al. (2023), we use large language models to provide concept sets. Concretely, we use GPT-3 (Brown et al. 2020) with the following preprompt: "In this task, you have give visual descriptions
25

---

Published in Transactions on Machine Learning Research (MM/YYYY)
that describe an image. Respond list. Each item being word." Then, we generate the sets of words by the following prompt: "What are [N] useful visual descriptors to distinguish a [class] in photo?". By doing so, we generated concepts per class, presented in Table 6 An additional ordering of concepts by subcatcgorics is available in Table 7
26

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 6: List of concepts used, ordered by classes of interest.
Dataset
Set of concepts
PASCAL-Part
'acroplanc': Jet engines, Tail fin, Fusclage, Landing 'bicycle's (Two- Pedals, Handlebars, Frame, Chain-driven]: bird':[Feathery Beaked, Wingspread, Perched, Avian], "bottle'[Gllass plastic, Cylindrical, Necked, Cap or cork, Transparent translucent), "bus" (Large, Rectangular, Windows, Wheels, Multi-doored], (attoorrred"""tt((FFF"Farr ccatr:[Furry, Whiskered, Pointy-cared, Slitted-yye, Four-leggedJ, carr:[[eeallic Four- wheeled, Head- lights, Windshield, License "dod'el:oog:(SSoout Wagging-tailed, Snout-nosed, Floppy- eared, srraggingl.ccow::BBBa colored, Grazing (if in field),, 'horse': (Equinee, Hooved, Maned, Tailed, Gallop- ing (if in motion)], 'motoorike' (twoorikke"ewwwwweeeooor (Two-wheeled Engine, Handlebars Exhaust, Sad- dle seat], 'person', [Human, Facial features, Limbs (arms and legs), Clothing, Pos- ture or stance), ppotted plant':[Potted, Green, Leaves, Soil potting mix, Container or pot], 'shee'': (Wooly, Hooved, Grazing, Herded (if in group), White or colored fleece), 'train'n 'taattaiaaoooooniiiii [Locomotive, Railcars, Tracks, Wheels : Carriages], tvmoontoor:((crren Rectangular Frame or bezel, Stand or wall-mounted, Displaying an image]
Hooved, Horned (in some cases), Spotted or solid-
MI
"Baroque'(Ornate, Elaborate sculptures, Intricale details, Curved or asymmetrical de- sign, Historical or aged appearance], 'Gothic': (Ponteed arches, Ribbed vaults, Flying but- tresses, Stained glass windows, Thll spires towers], 'Hispanic muslim's :: [Mudejar style, Intricate geometric patterns, Horseshoe arches, Decorative tilework (azulejos), Islamic- inspired motifs), "Rennaissanc"""(Cless pilasters, Human statues and sculptures, Dome or dome-like structures]
proportions, Symmetrical design, Columns and
MIT scenes
'Store': (Buidding or structure, Signage or banners, Glass windows or doors, Displayed products or merchandise, People entering or exiting (if applicable)], 'Hoome "Homee[[Resddentte Roofed, Windows, Landscaping or yard, Front entrance or door), PPublic space': (Open area, Crowds (if people are present), Benches or seating, Pathways or walkways, Architec- tural features (e.g., buildings, statues)], 'Leisure:: |Recreational, Play equipment (if appli- cable), Greenery or landscaping, Picnic tables or seating, Relaxing atmosphero). 'Working space': :OOfiic equipment (e.g., desks, computers), Task-oriented, Office chairs, Organized or struct ured, People working (if applicable]]
'Public
Cats/ Doos/Cars
'Cat': (Furyy Whiskered, Pointy-eared, Slitted-eyed, Four-legged], 'Car': CCrrr[[MMaallis Four- wheeled, Hcadlights, Windshicld, License plate], 'Do''''o'''on Wagging- Snout- nosed, Floppy-eared, Tail- wagging
27

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table List of concepts used, ordered by subcategories.
Dataset
Set of concepts
PASCAL-Part
'Objocts"(Sccren, Rectangular, Frame or bezel, Stand or wall-mountod, Displaying an image, Glass or plastic, Cylindrical, Necked, Cap cork, Transparent or translu- cent, Potted, Green, Teaves, Soil potting mix, Container or pot), "Transportation- (llated:([Twowwhheeee Engine, Handlebars, Exhaust, Saddle or seat, Four-wheeled, Head- lights, Windshield, License plate, Wheels, Multi-doored, Locomotive, Railcars, Tracks, Carriages, Chain-driven]] AhaineddrveellAAiiicciereeeaeddd Mirrrrafttrelaaeeee:WWggge
Jet engines, Tail fin, Fuselage, Tand- Rectangular, Windows] 'Human Character- istics"; ":HHss(H,,, Facial features, Limbs (arms and legs), Clothing, Posture or stance), 'Avian Characteristics (Feathery Beaked, Wingspread, Perched, Avian], 'Animal Characteris- tics' tics': (Ferry Whiskered, Pointy-cared, Slitted-eyed, Four-legged, Equine, Hooved, Maned, Tailed, Galloping (if in motion), Snout, Wagging-tailed, Snout-nosed, Floppy-eared, Tail- wagging, Woolly, Grazing, Herded (if in group), White or colored fleece, Bovine, Horned (in some cases), Spotted or solid-colored]
ing gearl,"BuildinggSStuuiee /blllding/Structuurrrrraaddd
MonuMAI
'Architectural Styles and Elements': [Muddeja (Mudejar style, Intricate geometric patterns, Horso- shoe arches, Decoralive tilework (azulejos), Islamic-inspired motils, Stained glass win- dows, Pointed arches, Ribbed vaults],'Artisti 'Artistic Details and Features':(Onnan rate sculptures, Intricate details, Curved or asymmetrical design, Human statues and sculptures, Historical or aged appearance] 'Architectural Components' "rroonnnsss:(Flynnnnn Classical proportions, Symmetrical design, Columns and pilasters, Dome or dome-like structures, Tall spires or towers]
Elabo-
:[Flying buttresses,
MIT scenes
'Work Environment's :[Ofiice equipment (e.g., desks, computers), Task-oriented, Office chairs, Organized or structured, People working (if applicable)], 'Leisure Environ- mont':[Play equipment (if applicable), Greenery or landscaping, Picnic tables or seat- ing, Relaxing atmosphere, Recreational], 'Community Landscaping yard, Front entrance door, Building structure, Signage banners, Glass windows doors, Displayed products or merchandise, People entering or exiting (if applicable), Open area, Crowds (if people are present), Benches scating, Pathways or walkways, Architect.ural features (e-g., buildings, stat Residential]
(attronmeet:[RRoolee Windows,
Cats/ Cias/Dogs/ee /Cass
'Vehicle Features') [License plate, Headlights, Windshield, Four-wheeled], 'Animal Baaturees:WWaging-g-tee
walging,, Snout, Whiskered, Pointy-carod, Slittod-oyed, Floppy-eared, Furry, Snout-nosed, Four-legged], Material/Texxuue [Metallic]
A.4 Study of images according to their position in the latent space
In addition to the CLIP-QDA method, we introduce another approach leveraging multivariate Gaussian modeling to generate dataset-wise visual explanations for our model. Specifically, for a given class c, we compute the Mahalanobis distance with respect to A(2 Meg Eg) for each sample's latent space projection z labeled as within a dataset of interest:
d Me: E.) = ((z (z- uc) T Ex'(z -Ma))
(9)
Then, we plot in Figures 15 this distance for PascalPART and MAI.
16 and 18 the closest and farthest images with the dataset, according
28

---

Published in Transactions on Machine Learning Research (MM/YYY)
"eroplane"
(b) "Bccycle
(c) "Bird"
(d) "Bottle"
(e) "Bus"
(f) "Car"
g "Cat" "
(h) "Cow"
"Dog"
ii) "Horse" "
kk) "Motorbike"
(I) "Person"
(m) "Potted plant"
(n) "Sheep"
"Train"
"TV monitor"
Figure 15: Images of minimal distance for each class of PascalPART. Fach subfigure in the plot corresponds the image of class from the test set that exhibits the minimal Mahalanobis distance, defined in Equation with respect to the Gaussian representation of the given class c.
29

---

Published in Transactions on Machine Learning Research (YYYYYY)
BEWARE SHEEP
sar30
Cucs Sameti
Say R2.000
"Aeroplane"
(b) Bicycle
cc) "Bird"
(d) "Bottle"
ving
BIR
(c) "Bus"
(f) "Car"
g "Cat" "
(h) "Cow"
(i) "Dog"
ii) "Horse" "
(k) "Motorbike"
"Person"
MOUNT HOOD
(m) "Potted plant"
(n) "Sheep"
"Train"
(p) "TV monitor"
Figure 16: Images of maximal distance for each class of PascalPART. Fach subfigure in the plot corresponds to the image of class from the test that exhibits the maximal Mahalanobis distance, defined in Equation g with respect to the Gaussian representation of the given class c.
(a) "Baroque" "
(b) "Gothic"
(c) "Hispanic-Musee Muslim
(d) "Renaissance"
Figure 17: Images of minimal distance for each class of MonuMAI. Each subfigure in the plot corresponds to the image of class e from the test set that exhibits the minimal Mahalanobis distance, as defined in Equation with respect to the Gaussian representation of the given class c.
30

---

Published in Transactions on Machine Learning Research (MM/YYY)
(a) "Baroque"
(b) 'Gothic"
(c) "Hispanic-Muslim"
(d) "Renaissance"
Figure 18: Images of maximal distance for each class of MonuMAI. Each subfigure the plot
corresponds to the image of class from the test set that exhibits the minimal Mahalanobis distance, defined in Equation with respect to the Gaussian reprosentation of the given class c.
For PascalPART, the closest images tend to be prodominantly closo-up shots of the object in question. How- ever, this occasionally presents challenges due the inherent pixelation of the images, leading lo somewhat counterintuitive results. Conversely, the farthest images are typically those where the object is distant. Another scenario of failure ariscs when there two classes prcsent within the image, which complicates the retrieval process. On the other hand, for MonumAI, the closest images appear to be the simplest ones, without sign of disturbance. In contrast, the farthest images may include extraneous objects, such as signs.
A.5 Closed-form solution of counterfactuals for CLIP QDA
First, will derive the resolution of equation 5 for the binary case before extending it to the multiclass case. Let us begin with binary classifier, where Y E (Awa (z), Aaw (z)} (for convenience, we note h (2) and haw (z) =z))
Proposition 2. Let hwn a binary classifier and Z following a minture of Gaussians such as Zy=ca N( De,) and Zy=e, ~ N((,, -zd) and eg a perturbation with the above sparsity and sign restrictions. Then, there is closed form solution to problem 3 given by:
1 if 12 <P
or (s + sign(bi) and + sign(b2))
bj if 8 4Pe 0 and sign(b1) =
(10)
and (sign(b2) (b2) # or |b 2 [bll)
by if 1 4Pe > 0 and sign(b2)
and (sign(b1) + s or |b9| > b2l),
with:
P
(6.)) (in))
([2](M) (Moglan)) (1 --=>))) - ([2](0)
[Ma k() 1E kk1k))
k=1
log
Σ
+ log
Pes
+ 7(2 -mag) E mam) 2 (Moa))
1
T - Mca)
2
En ph
Pop
b b Vb2
4Pc
(1I)
2P
b+ Vb² 4Pc
b2
2P
31

---

Published in Transactions on Machine Learning Research (MM/YYY)
Proof. For the binary case, equation 5 can be written as
min |el?
s.l.
Poz
[(+e]] (18te--Ma Meyp" 12 E 2 (z+e) - maa)
<
Po (2T) N/2 E
[(=t)--oa)) "Dn (2te) -M,,)
(2x) 2/2 E
(12)
Let us focus on the incquality constraint:
Per
e - -((xte/-HHg Hip 12 ( E
Per Ean
-(x+e/-Muup) E'y (s+e)--uu)
e
2
1
h
1
= log(pe,))
log E
2 5/2 + cd me, )a Σ 1
+ E m
T
1
1
log(Pcn)
- log Σ
(z
+ c² - -Me)) TT (z+ + e₂ - MMcs)
2
2
+ log(p
- log E
1
7(2 -Mo
-1 T プラ %
(z - mma)
E3
T
E
">1 (z - M
2
c
%
< log(pez) )
1
log Dmal
[/2 (2 Mey
1
"'=1
1 (z p(a))
1
E
T
"n'ed [eee Eg. T -ー'ー'
Σー (2 p )
2
2
N
1
- log(Poz) 2
log |gae|=
(z - Hcme
2g (2 - mem)
E (1)))))())
c
Σ ([21(m) - -IM (())) |1 [1100 160.0
2
2
1
1
log(Pez)
log Eesl
(z Mcr )ah E2 (z Mc )
E (e))
Σ
[[=](k) - [Mcalca)) 16,)
2
k=
P(e)) + beg +c 0.
Then we can rewrite the problem as:
min (E))² s.t. P(e))² + bet + 0.
(13)
To solve this problem, we introduce slack variables X and I, and define Lagrangian as: L(e(, = (e)) + x(P(e)) + bet + +c+ +c+
(14)
Then, we can find the possible solutions by solving the system: :
ØL 0
de?
aA
0L
O,
which corresponds to:
2(AP + 1)e} + Ab =0 0
P(e))² + bel + 12
(15)
2A =0
The third equation of 15 indicates whether the incquality condition is saturated or not. If it is not saturated AA=0), 0), then the label cg is already achieved for 2, resulting in a solution of e 0. This being impossible by construction,
only focus on the case where X + O.
If X + 0, the condition is saturated, the second equation leads to:
P(e!)² + beg + c c==00 0,
32

---

Published in Transactions on Machine Learning Research (MM/YYY)
whose solutions are:
b Vb2
4Pc
-b+ vb² 4Pc
e E
12 4Pe> 0,
2P
2P
which we rewrite
d. E (b)
b² 4Pc 0.
(16)
Considering 16 the validity of the results is conditioned by the sign and the condition of the magnitude of c.... Then, the final result of the problem 5 is either 0, b or by depending on the conditions:
0 if 12 <Pc<0
or (s + sign(bı) and + sign(b2))
bi if ba - 4Pc> 0 and sign(b1) = and (sign(b2) b2) # or 162l 2 1bıl)
(17)
b2 if 17 - <Pe> 0 and sign(b2) = and (sign(b1) + s or |bu lb2l)-
To expand problem 5 to multiclass classification C we consider as a succession of C binary classifications between each i' * C₁ and Cb. Given concept index and sign s, if we denote the solutions of theses problems as the set (e'_..1)--. e 5,,,0 e S, c.,ch the final solution e.e is if it exists, the
value of minimal magnitude among this set.
Examples of explanations based on this metric are given in Sections 5.4 and A.7
A.6 Sample-wise explanation time
In Table S we display the amount of time taken to produce a sample-wise explanation for each image of the test set PASCAL-Part and Paats/Doss/Cars
Table 8 Time (in scconds) to produce explanations on the entire test sct.
Method
CLIP-QDA Local
CLIP-LIME CLIP-SHAP
Cats/ Cats/Dogs/Ci DDoss/Crrs
3.01
76.69
256.76
PASCAL-Parll 2341.12
2857.28
7207.58
MonuMAl
2.19
19.94
56.25
MITscenes 33.41
509.33
1398.64
In this table, we can observe that using the local explanation is the fastest, especially for the cases where the number of concepts and classes are low. This is justified by the fact. that our method consists of using Proposition 2 for each concept, sign, and class other than the inference one. Hence the complexity of this computation for each sample O(2(C 1)N), where LIME's complexity does not depend on C.
A.7 Additional samples
We display here additional samples of both sample-wise and datasct-win explanations from the PASCAL- Part and Cats/Dogs/ Cars dataset.
33

---

Published in Transactions on Machine Learning Research (MM/YYY)
(a) Input image
(b) GradCAM Explanation
(c) LIME Explanation
(d) SHAP explanation
Figure 19: Sample-wise explanations (image level). The first two examples come from the PascalPART Dogs/Cars
dataset, and the last two samples come from the Catt/DDos// /Cars dataset in the biased setup. Note that the classifier mislabeled the 2nd example as "car" and the 4th example as "car".
34

---

Published in Transactions on Machine Learning Research (YYY)
(a) Input image
(b) CLIP-QDA local
Explanation
(c) Yan et al. (sample) explanation
(d) CLIP-LIME explanation
Avin thapet
Counterfactun
Concept
Concept
Sparted colared
22.80 Podala
Petas (bvert pertar
eeanvel
Netiod 24.2
Sceatroced (Dgger partar)
000 bott 107 sam area pors
Deviation
Weight value
Lime magnitude
Heroon thwert sersang
Wasdeet 11.31
Counterfactual
ong nes Ilaget perveo
Windini eit
Land 26-35
Concept
Concept
Headightt (laget perart
Sucidl
Parap anont curalacart
Sadde
Haadiigtt
Facal aeallsakannn 12.79
Weight value
Lime magnitude
Counterfactual
Concept
Concept
Blckk Mlrgee Cat
Biack
Whisheed
Phopps care
Parrycand 36A2
Wor oboeced EDrppt car
Stleeeyed
Pery
-6015-6.33-- 025 686 3.05 100 605
Deviation
Weight value
Lime magnitude
Snoucesaedd Ulrger Car)
Blcck
Black > 26.30
Counterfactual
Darper Cao
Whitleed > 27.66
Concept
Concept
Mibioree
Peitticarad io 3602
Deviation
Weight value
Lime magnitude
Figure 20: Sample-wise explanations (concept level). The first two examples come from the Pascal- PART dataset, and the last two samples come from the Cats/Dogs/ Cars dataset the biased setup. Note that the classifier mislaboled the 2nd example "car" and the 4th example "car".
35

---

Published in Transactions on Machine Learning Research
(a)
Input image
(b) CLIP-SHAP Explanation
higher
base value
10O
1.00
0.6
0.6
Feathery
23.140625
higher
[X]]
base value
0.00
-0.
0,0
0.
Floppy-eared
Grazing 9.4921875
Woolly 11.8359375
higher
1.00
11.6951125
19.3125
Whiskered 18. 703125
higher
lower
1.00
Slitted eyed 17.03125
Black 19.71875
Whiskered 18.546875
Figure 21: Sample-wise explanations (concopt level). The first two examples come from the Pascal- PART dataset, and the last two samples come from the Cats/Doge Malss/oos//Caas Cars dataset in the biased setup. Note that the classifier mislabeled the 2nd example as "car" and the 4th example as "car".
36

---

Published in Transactions on Machine Learning Research (YYY
(a) Input image
(b) CLIP-QDAolobal Explanation
(c) LaBo explanation
(d) Yan et al. (dataset) explanation
Concept
Concept
Concopt
Feather
msaked
uevinegger
Soothed or oen collrdd
Parhhe
Wiogpood
6.1
nM
na
Wassertein Distance
Weight value
Weight value
Jorss pide
Limeor pat
Narsderii
Vindined
Wircsfeda
Seddie or sext
Concept
Concept
Concept
Faur whferied
Banohodded
Winderiedd
Engin
Heedigght
Wassertein Distance
34
40
Weight
Weight
Contept
Concept
Concept
eardd
hertt-oured
Liken
ropaycoree
Metyoard
Stracieyed
0.00 3.605 6:200
0.153 3 6.200 Weight value
0.65 3.1
6.25
Wassertein Distance
Weight value
Wi.tt
Whlcordd
Poittyearod
Concept
Wil dered
Concept
Back
sarddarrd
Whllordd
White
Wassertein Distance
Weight value
Welght value
Figure 22: Dataset-wise explanations. The first two examples come from the PascalPART dataset, and the last two samples come from the Mats/Dogs/CCe CCrss dataset the biased setup. Note that the classifier mislabcled the 2nd example as "car" and the 4t example as "car".
37


Reviews:
[{"review_id": "l2706WyFH9", "reviewer": "Reviewer_xNTY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\nOverall, the writing is clear, and the paper is well-structured.\n\nThe method is sound i) using mixture of Gaussian to model the latent and using QDA to measure the concept probability, ii) decomposing the explanation into local and global.\n\nWell-tackled the limitations of prior CLIP-based concept-based methods (i.e., requires dedicated datasets). \n\n------------\n\n**Weakness**\n\nI think it is somewhat hard to understand the definition of 'concept' in this setup. I believe the concept of interpretable machine learning indicates two things: (i) human interpretable things that are used (ii) to decompose the reasoning process (of the prediction). \nFor instance, one example image of a bird contains a \"beak\" and \"feather.\"\\ \nHowever, since this paper does not use dedicated datasets, the proposed concept is only used for decomposability. I think explicitly adding some clusters of concepts will be helpful and labeling them with human experts will be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an interpretability tool that uses CLIP for image classification. Specifically, this paper uses a Mixture of Gaussians to model the latent distribution where each distribution can be interpreted as a concept. Based on this concept, they interpret the model behavior in terms of local and global behavior. Furthermore, they extend well-founded interpretable methods, including LIME and SHAP, to multi-modal setup by combining them with the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Visualizing the examples that are the centroid of each Gaussain distribution or giving some visualization of images inside the cluster (i.e., each Gaussain distribution)", "broader_impact_concerns": "No ethical concern found", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ilkQUMhLcz", "reviewer": "Reviewer_czSS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not an expert on CLIP-based modeling and Concept Bottleneck Models. \n\nI like that the approach is straightforward, does not require training, and includes interpretability features. The mixture of gaussians seems like a nice assumption, however, it probably does not generalize to real-world cases, as noted in the paper. The paper is very readable and easy to follow. The analysis in Fig. 5 & 7 was nice.\n\nI believe the approach could be strengthened by learning CLIP distributions, and not assuming a particular distribution. Alternatively, you could also try to guide CLIP towards GMM, so that your assumption holds. The evaluation section is somewhat unconvincing. The test cases and baselines seem too basic. Moreover, CLIP-QDA does not improve even in these basic examples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new interpretable classifier based on CLIP. In particular, the authors suggest to model latent distributions with a mixture of gaussians, allowing to classify with a classic ML method such as quadratic discriminant analysis. Moreover, the authors propose global and local metrics to assess the learned representations based on measuring the distance between classes, and finding the minimal code that changes classification. Finally, the proposed approach is evaluated on basic examples, compared to basic baselines and analyzed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2igRa4fy0", "reviewer": "Reviewer_SRtB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* This work appears to be technically sounds and would certainly be of interest to certain individuals in TMLR's audience.\n\n**Weaknesses:**\n* Contextualization of this work in the literature and comparisons with related methods.\n* Details about experimental protocol.\n* Limited numerical results.\n\nSee more details below.\n\nStrengths:\n* Relevance for TMLR audience: CLIP backbones are widely utilized in the machine learning community, and thus a simple method for improving the explainability of CLIP image classifiers, post-hoc, is of interest to the TMLR community in my opinion. Moreover, this work builds on a growing body of work in machine learning revisiting CBMs in the context of multi-modal foundation models, which is likely of independent interest to the TMLR community.\n* Simplicity: The proposed CLIP-QDA model is conceptually simple compared to other model-agnostic approaches for by-design or post-hoc explainability, such as classical shapely value estimation methods, which require re-training several classifiers, each time holding out a different subset of features. Moreover, the proposed explainability metrics are conceptually simple and yield closed-form solutions.\n* Limitations: This work clearly discusses limitations of their proposed method by discussing where assumptions break down and their impact on performance.\n\nWeaknesses:\n* The biggest weakness is the situation of this work in the broader literature, both in discussion and numerical experiments.\n  * For instance, no mention is made of post-hoc explainability methods based on explanation generation, such as GradCAM and related approaches.\n  * Notably missing is also a discussion of Additive Feature Attribution methods, which are closely related to the proposed CLIP-QDA in that they seek to define the importance of specific features in latent space for predictions made by light-weight classifier heads.\n  * Very related to this work is the LaBo work of Yang et al., 2023, which proposes to use an LLM to generate concept words for each image class, and then trains a linear classifier on top of the CLIP concept-word similarity scores, rather than a QDA classifier as proposed in this work.\n  * What about a comparison with CounTEX, which also leverages a CLIP text encoder along with predefined concepts to produce concept activation vectors instead of concept scores.\n  * What about model-agnostic approaches, such as Post-hoc Concept Bottleneck Models (PCBMs), which rely on a multi-modal models to obtain concept vectors, and then project the visual encoder’s embeddings onto this space, and thus can also be applied to image backbones which are not text aligned.\n* Details about experimental protocol:\n  * The paper lists ImageNet classification results, but I could not find these in the main paper, nor could I find them in the appendix.\n  * Missing major details; what is the architecture of the CLIP backbone used in the considered experiments?\n* Numerical results are currently limited to Table 1, showing image classification accuracy on PASCAL-Part, MIT scenes, Monoumi with limited baselines, and Figure 11, which shows deletion metric plots compared to other XAI methods.\n* Results:\n  * In terms of classification accuracy: CLIP linear probing is better than CLIP-QDA as number of concepts increases, which seems to correlated with the observation that the Gaussian assumption starts to break down with a larger number of concept words. But best performance for all models is obtained with increasing number of concept classes.\n  * In terms of explainability, numerical results based on the deletion metric (how much does nullifying a certain number of concepts decrease classifier accuracy; thereby indicating the important concepts for classification) show that the proposed metric is significantly worse than the current state-of-the-art, SHAP, although the proposed method is much lighter to run; e.g., 45 minutes instead of 120 minutes on the largest dataset considered (PASCAL-Part).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### **Algorithm:**\nThis paper proposes a method for constructing explainable image classifiers from pre-trained CLIP backbones. The proposed method is based on the Concept Bottleneck Model (CBM), where the model's predictions are bottlenecked through a set of pre-defined concepts. This work in particular generates concept annotations for each image class by prompting an LLM. Next these concept words are encoded with a CLIP text encoder. Now to generate a class prediction for an image, the output of the CLIP image encoder is used to compute a cosine similarity score with each of the CLIP concept text embeddings. These CLIP scores are then fed to a QDA classifier, the parameters of which are estimated directly by computing statistics on the training data.\n\n### **Assumptions:**\nThe QDA classifier makes the assumption that the conditional distribution of concept scores in the CLIP latent space (conditioned on the class label) are normally distributed, and therefore the CLIP latent space can be factored as a Mixture of Gaussians (MoG).\n\n### **XAI Metrics:**\nThis work then proposes a Global XAI metric for explaining the classifier's predictions based on the Wasserstein-2 distance, which provides an indication into how the distribution of a concept score changes when conditioned on one class (c_1) versus another (c_2). This is used to provide global insights into how discriminative a certain concept word is for discriminating between class c_1 and c_2. Based on the proposed MoG decomposition, this metric yields a closed-form solution.\n\nThis work also proposes a signed Local XAI metric for explaining how the classifier's prediction on a specific instance would have changed if its CLIP score with a certain concept word was manually perturbed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest the following revisions to improve the work:\n* Broaden the discussion of related work to describe the outline the literature in more detail than what is currently provided (see above).\n* Broaden Table 1 to include more baselines, including at least\n  * LaBo, SHAP, LIME\n  * and provide details in the appendix about how these baselines are computed/obtained.\n* Include ImageNet classification results, which are claimed to be included in the experimental protocol in the introduction, but cannot be found in the paper.\n* Provide missing details about experimental protocol (see above), and please include in the main paper the strategy used for generating concept words (LLM prompting).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the method interesting and simple. They raised questions in terms of the performance of the nroposed approach, especially for the ImageNet example. A reviewer also mentioned that the clarity of the paper has decreased after revision.

I would hence recommend that the authors ensure that the text correspond to the results presented, especially when they mention the performance of the method. Proof-reading might help with clarity, as well as including links to code repositories used in the work, adding details about the baselines, and improving on the description of figures 15-19 in appendix.
Decision: Accept (poster)